History

Merge tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton: - "selftests/mm: clean up build output and verbosity" (Li Wang) Remove some noise from the MM selftests build - "mm: Free contiguous order-0 pages efficiently" (Ryan Roberts) Speed up the freeing of a batch of 0-order pages by first scanning them for coalescing opportunities. This is applicable to vfree() and to the releasing of frozen pages - "mm/damon: introduce DAMOS failed region quota charge ratio" (SeongJae Park) Address a DAMOS usability issue: The DAMOS quota often exhausts prematurely because it charges for all memory attempted, causing slow and inconsistent performance when actions fail on unreclaimable memory. To fix this, a new feature lets users set a smaller, flexible quota charge ratio (via a numerator and denominator) for failed regions. Since failed actions cause less overhead, reducing their quota cost ensures more predictable and efficient DAMOS processing - "selftests/cgroup: improve zswap tests robustness and support large page sizes" (Li Wang) Fix various spurious failures and improves the overall robustness of the cgroup zswap selftests - "fix MAP_DROPPABLE not supported errno" (Anthony Yznaga) Fix an issue in the mlock selftests on arm32 - "mm: huge_memory: clean up defrag sysfs with shared" (Breno Leitao) Some maintenance work in the huge_memory code - "treewide: fixup gfp_t printks" (Brendan Jackman) Use the special vprintf() gfp_t conversion in various places - "mm: Fix vmemmap optimization accounting and initialization" (Muchun Song) Fix several bugs in the vmemmap optimization, mainly around incorrect page accounting and memmap initialization in the DAX and memory hotplug paths. It also fixes pageblock migratetype initialization and struct page initialization for ZONE_DEVICE compound pages - "mm/damon: repost non-hotfix reviewed patches in damon/next tree" A sprinkle of unrelated minor bugfixes for DAMON - "mm: remove page_mapped()" (David Hildenbrand) Remove this function from the tree, replacing it with folio_mapped() - "mm/damon: let DAMON be paused and resumed" (SeongJae Park) Allow DAMON to be paused and resumed without losing its current state - "kasan: hw_tags: Disable tagging for stack and page-tables" (Muhammad Usama Anjum) Simplify and speed up kasan by removing its ineffective tagging of stacks and page tables - "mm/damon/reclaim,lru_sort: monitor all system rams by default" (SeongJae Park) Simplify deployment on diverse hardware like NUMA systems by updating DAMON_RECLAIM and DAMON_LRU_SORT to automatically monitor the physical address range covering all System RAM areas by default, replacing the overly restrictive behavior that only targeted the single largest memory block to save on negligible overhead - "mm/damon/sysfs: document filters/ directory as deprecated" (SeongJae Park) Update some DAMON docs - "mm: use spinlock guards for zone lock" (Dmitry Ilvokhin) Switch zone->lock handling over to using the guard() mechanisms - "mm/filemap: tighten mmap_miss hit accounting" (fujunjie) Fix a flaw where the mmap_miss counter over-credited page cache hits during fault-arounds and page-fault retries. This results in significant reduction of redundant synchronous mmap readahead I/O, drastically cutting down execution time and gigabytes read for sparse random or strided memory access workloads - "selftests/cgroup: Fix false positive failures in test_percpu_basic" (Li Wang) Fix a couple of false-positives in the cgroup kmem selftests - "mm/damon/reclaim: support monitoring intervals auto-tuning" (SeongJae Park) Add a new parameter to DAMON permitting DAMON_RECLAIM to automatically tune DAMON's sampling and aggregation intervals - "mm/damon/stat: add kdamond_pid parameter" (SeongJae Park) Change DAMON_STAT to provide the pid of its kdamond - "mm/kmemleak: dedupe verbose scan output" (Breno Leitao) Remove large amounts of duplicated backtraces from the verbose-mode kmemleak output - "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)" (David Hildenbrand) Reduce our use of CONFIG_HAVE_BOOTMEM_INFO_NODE, with a view to removing it entirely in a later series - "mm/damon: validate min_region_size to be power of 2" (Liew Rui Yan) Prevent users from passing a non-power-of-2 value of `addr_unit', as this later results in undesirable behavior - "mm: document read_pages and simplify usage" (Frederick Mayle) - "tools/mm/page-types: Fix misc bugs" (Ye Liu) Fix three issues in tools/mm/page-types.c - "mm: misc cleanups from __GFP_UNMAPPED series" (Brendan Jackman) Implement several cleanups in the page allocator and related code - "mm, swap: swap table phase IV: unify allocation" (Kairui Song) Unify the allocation and charging of anon and shmem swap in folios, provides better synchronization, consolidates the metadata management, hence dropping the static array and map, and improves performance - "mm/damon: introduce data attributes monitoring" (SeongJae Park( Extend DAMON to monitor general data attributes other than accesses - "mm/vmalloc: free unused pages on vrealloc() shrink" (Shivam Kalra) Implement the TODO in vrealloc() to unmap and free unused pages when shrinking across a page boundary - "mm/damon: documentation and comment fixes" (niecheng) - "remove mmap_action success, error hooks" (Lorenzo Stoakes) Eliminate custom hooks from mmap_action by removing the problematic success_hook which allowed drivers to improperly access uninitialized VMAs. It replaces the error_hook with a simple error-code field and updates the memory char driver accordingly - "mm/damon: minor improvements for code readability and tests" (SeongJae Park) - "mm/damon: fix macro arguments and clarify quota goals doc" (Maksym Shcherba) - "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c" (Mike Rapoport) - "mm/mglru: improve reclaim loop and dirty folio" (Kairui Song and others) Clean up and slightly improves MGLRU's reclaim loop and dirty writeback handling. Large performance improvements are measured - "use vma locks for proc/pid/{smaps|numa_maps} reads" (Suren Baghdasaryan) Use per-vma locks when reading /proc/pid/smaps and numa_maps similar to reduce contention on central mmap_lock - "refactors thpsize_shmem_enabled_store() and thpsize_shmem_enabled_show()" (Ran Xiaokai) Some cleanup work in the THP code - "selftests/memfd: fix compilation warnings" (Konstantin Khorenko) Fix a few build glitches in the memfd selftest code. - "memcg: shrink obj_stock_pcp and cache multiple objcgs" (Shakeel Butt) Resolve a 68% performance regression caused by NUMA-node cache thrashing around struct obj_stock_pcp by shrinking its existing fields and expanding it into a multi-slot array that caches up to five obj_cgroup pointers per CPU, allowing per-node variants of the same memcg to coexist within a single 64-byte cache line. - "zram: writeback fixes" (Sergey Senozhatsky) address a couple of unrelated zram writeback issues - "mm: switch THP shrinker to list_lru" (Johannes Weiner) Resolve NUMA-awareness issues and streamlines callsite interaction by refactoring and extending the list_lru API to completely replace the complex, open-coded deferred split queue for Transparent Huge Pages - "mm: improve large folio readahead for exec memory" (Usama Arif) Improve large-folio readahead on systems like 64K-page arm64 by preventing the mmap_miss check from permanently disabling target-oriented VM_EXEC readahead, and by generalizing the force_thp_readahead gate to support mappings with any usefully large maximum folio order under the cache cap. - "userfaultfd/pagemap: pre-existing fixes" (Kiryl Shutsemau) Fix a bunch of minor issues in the userfaultfd/pagemap, all of which were flagged by Sashiko review of proposed new material - "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and vmemmap_check_pmd()" (Muchun Song) Provide generic versions of these two functions so the four arch-specific implementations can be removed. - "mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device" (Youngjun Park) Address a uswsusp-vs-swapoff race and reduces the swap device reference taking/releasing frequency. - "mm/hmm: A fix and a selftest" (Dev Jain) * tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits) selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries fs/proc/task_mmu: do not warn on seeing non-migration pmd entry lib/test_hmm: check alloc_page_vma() return value and handle OOM mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX mm/swap: remove redundant swap device reference in alloc/free mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device mm/filemap: use folio_next_index() for start vmalloc: fix NULL pointer dereference in is_vm_area_hugepages() sparc/mm: drop vmemmap_check_pmd helper and use generic code loongarch/mm: drop vmemmap_check_pmd helper and use generic code riscv/mm: drop vmemmap_pmd helpers and use generic code arm64/mm: drop vmemmap_pmd helpers and use generic code mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd() rust: page: mark Page::nid as inline userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks userfaultfd: gate must_wait writability check on pte_present() mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole() fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry() fs/proc/task_mmu: fix make_uffd_wp_huge_pte() prot-update race ...

Linus Torvalds committed 4 months ago

Commit a552c812 parents c98d767 e3d8707169 files · +9215 −5618Browse files

Unified · Split

169 files
  1. Documentation/ABI/testing/sysfs-kernel-mm-damon +53 −28
  2. Documentation/admin-guide/kernel-parameters.txt +4 −0
  3. Documentation/admin-guide/mm/damon/lru_sort.rst +5 −3
  4. Documentation/admin-guide/mm/damon/reclaim.rst +16 −3
  5. Documentation/admin-guide/mm/damon/stat.rst +7 −0
  6. Documentation/admin-guide/mm/damon/usage.rst +82 −26
  7. Documentation/admin-guide/mm/transhuge.rst +2 −2
  8. Documentation/admin-guide/sysctl/vm.rst +2 −0
  9. Documentation/mm/damon/design.rst +75 −3
  10. Documentation/mm/damon/maintainer-profile.rst +21 −0
  11. Documentation/mm/process_addrs.rst +1 −1
  12. MAINTAINERS +14 −2
  13. arch/arm64/mm/mmu.c +3 −16
  14. arch/loongarch/mm/init.c +3 −13
  15. arch/powerpc/mm/init_64.c +0 −8
  16. arch/powerpc/mm/mem.c +3 −2
  17. arch/powerpc/platforms/pseries/hotplug-memory.c +4 −10
  18. arch/riscv/mm/init.c +3 −15
  19. arch/s390/mm/init.c +3 −2
  20. arch/s390/mm/vmem.c +1 −2
  21. arch/sh/mm/cache-sh4.c +1 −1
  22. arch/sparc/mm/init_64.c +0 −31
  23. arch/x86/mm/init_64.c +3 −2
  24. drivers/base/memory.c +16 −25
  25. drivers/base/node.c +4 −2
  26. drivers/block/zram/zram_drv.c +9 −4
  27. drivers/char/mem.c +6 −19
  28. drivers/dax/kmem.c +6 −0
  29. drivers/gpu/drm/drm_managed.c +2 −2
  30. drivers/s390/char/sclp_mem.c +8 −9
  31. fs/Makefile +0 −1
  32. fs/proc/task_mmu.c +223 −46
  33. fs/userfaultfd.c +0 −2231
  34. include/asm-generic/pgalloc.h +1 −1
  35. include/linux/bootmem_info.h +0 −1
  36. include/linux/compaction.h +5 −5
  37. include/linux/damon.h +111 −25
  38. include/linux/gfp.h +4 −0
  39. include/linux/gfp_types.h +3 −3
  40. include/linux/highmem-internal.h +1 −1
  41. include/linux/huge_mm.h +42 −5
  42. include/linux/list_lru.h +70 −0
  43. include/linux/memcontrol.h +17 −14
  44. include/linux/memory.h +5 −2
  45. include/linux/memory_hotplug.h +5 −3
  46. include/linux/mm.h +44 −18
  47. include/linux/mm_inline.h +1 −1
  48. include/linux/mm_types.h +3 −16
  49. include/linux/mmu_notifier.h +2 −2
  50. include/linux/mmzone.h +6 −17
  51. include/linux/nodemask.h +9 −9
  52. include/linux/page_ref.h +18 −0
  53. include/linux/pageblock-flags.h +3 −3
  54. include/linux/pagemap.h +2 −0
  55. include/linux/swap.h +12 −12
  56. include/linux/swap_cgroup.h +0 −47
  57. include/linux/thread_info.h +1 −1
  58. include/linux/userfaultfd_k.h +2 −38
  59. include/linux/vmalloc.h +3 −1
  60. include/linux/vmpressure.h +5 −4
  61. include/trace/events/damon.h +38 −0
  62. include/trace/events/vmscan.h +52 −0
  63. kernel/bpf/arena.c +1 −1
  64. kernel/fork.c +3 −2
  65. kernel/power/snapshot.c +4 −4
  66. kernel/power/swap.c +1 −1
  67. kernel/power/user.c +12 −3
  68. lib/maple_tree.c +5 −2
  69. lib/test_hmm.c +40 −9
  70. lib/test_meminit.c +1 −1
  71. lib/test_vmalloc.c +62 −0
  72. mm/Kconfig +2 −5
  73. mm/Makefile +0 −3
  74. mm/bootmem_info.c +2 −23
  75. mm/compaction.c +8 −3
  76. mm/damon/core.c +578 −77
  77. mm/damon/lru_sort.c +49 −10
  78. mm/damon/ops-common.c +6 −3
  79. mm/damon/paddr.c +80 −4
  80. mm/damon/reclaim.c +79 −19
  81. mm/damon/stat.c +42 −50
  82. mm/damon/sysfs-common.c +41 −0
  83. mm/damon/sysfs-common.h +2 −0
  84. mm/damon/sysfs-schemes.c +244 −45
  85. mm/damon/sysfs.c +588 −0
  86. mm/damon/tests/core-kunit.h +152 −28
  87. mm/damon/tests/vaddr-kunit.h +20 −7
  88. mm/damon/vaddr.c +50 −29
  89. mm/filemap.c +80 −53
  90. mm/gup.c +4 −4
  91. mm/huge_memory.c +190 −289
  92. mm/hugetlb.c +4 −3
  93. mm/hugetlb_cma.c +16 −19
  94. mm/internal.h +13 −14
  95. mm/kasan/kasan_test_c.c +10 −0
  96. mm/kfence/kfence_test.c +1 −1
  97. mm/khugepaged.c +15 −6
  98. mm/kmemleak.c +140 −8
  99. mm/list_lru.c +162 −76
  100. mm/madvise.c +25 −35
  101. mm/memcontrol-v1.c +44 −24
  102. mm/memcontrol.c +197 −97
  103. mm/memory-failure.c +18 −29
  104. mm/memory.c +79 −127
  105. mm/memory_hotplug.c +18 −16
  106. mm/mempolicy.c +1 −1
  107. mm/memremap.c +2 −2
  108. mm/migrate.c +33 −30
  109. mm/migrate_device.c +3 −6
  110. mm/mm_init.c +36 −37
  111. mm/mmap.c +1 −1
  112. mm/mseal.c +3 −2
  113. mm/page_alloc.c +277 −106
  114. mm/page_io.c +73 −30
  115. mm/page_isolation.c +28 −39
  116. mm/page_owner.c +1 −1
  117. mm/page_vma_mapped.c +7 −2
  118. mm/percpu-internal.h +3 −3
  119. mm/readahead.c +30 −6
  120. mm/rmap.c +4 −4
  121. mm/shmem.c +119 −159
  122. mm/shrinker.c +8 −10
  123. mm/sparse-vmemmap.c +56 −26
  124. mm/sparse.c +5 −63
  125. mm/swap.c +53 −4
  126. mm/swap.h +32 −58
  127. mm/swap_cgroup.c +0 −172
  128. mm/swap_state.c +308 −225
  129. mm/swap_table.h +149 −30
  130. mm/swapfile.c +302 −158
  131. mm/userfaultfd.c +2247 −12
  132. mm/util.c +22 −10
  133. mm/vma.c +3 −0
  134. mm/vmalloc.c +107 −23
  135. mm/vmpressure.c +12 −3
  136. mm/vmscan.c +168 −202
  137. mm/workingset.c +6 −4
  138. mm/zswap.c +10 −15
  139. net/rds/tcp_recv.c +1 −1
  140. rust/kernel/page.rs +1 −0
  141. samples/damon/mtier.c +6 −4
  142. tools/mm/page-types.c +4 −4
  143. tools/testing/selftests/cgroup/lib/cgroup_util.c +9 −9
  144. tools/testing/selftests/cgroup/lib/include/cgroup_util.h +2 −2
  145. tools/testing/selftests/cgroup/test_core.c +1 −1
  146. tools/testing/selftests/cgroup/test_freezer.c +1 −1
  147. tools/testing/selftests/cgroup/test_kmem.c +7 −6
  148. tools/testing/selftests/cgroup/test_memcontrol.c +9 −4
  149. tools/testing/selftests/cgroup/test_zswap.c +127 −54
  150. tools/testing/selftests/damon/_damon_sysfs.py +28 −3
  151. tools/testing/selftests/damon/drgn_dump_damon_status.py +3 −0
  152. tools/testing/selftests/damon/sysfs.py +55 −5
  153. tools/testing/selftests/damon/sysfs.sh +62 −0
  154. tools/testing/selftests/memfd/fuse_test.c +1 −1
  155. tools/testing/selftests/memfd/memfd_test.c +2 −2
  156. tools/testing/selftests/mm/Makefile +7 −3
  157. tools/testing/selftests/mm/check_config.sh +1 −1
  158. tools/testing/selftests/mm/droppable.c +8 −1
  159. tools/testing/selftests/mm/hmm-tests.c +33 −3
  160. tools/testing/selftests/mm/khugepaged.c +15 −3
  161. tools/testing/selftests/mm/ksft_kmemleak_dedup.sh +222 −0
  162. tools/testing/selftests/mm/ksm_functional_tests.c +11 −8
  163. tools/testing/selftests/mm/mlock2-tests.c +75 −9
  164. tools/testing/selftests/mm/mremap_test.c +10 −99
  165. tools/testing/selftests/mm/process_madv.c +28 −0
  166. tools/testing/selftests/mm/run_vmtests.sh +1 −0
  167. tools/testing/selftests/mm/split_huge_page_test.c +8 −4
  168. tools/testing/selftests/proc/proc-maps-race.c +231 −62
  169. tools/testing/vma/include/dup.h +4 −16
Documentation/ABI/testing/sysfs-kernel-mm-damon+53−28
@@ -84,6 +84,13 @@ Description: Writing an integer to this file sets the 'address unit' · 83 unmodified lines
84 parameter of the given operations set of the context. Reading84 parameter of the given operations set of the context. Reading
85 the file returns the last-written 'address unit' value.85 the file returns the last-written 'address unit' value.
86 86
87What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/pause
88Date: Mar 2026
89Contact: SeongJae Park <sj@kernel.org>
90Description: Writing a boolean keyword to this file sets the 'pause' request
91 parameter for the context. Reading the file returns the
92 last-written 'pause' value.
93
87What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/sample_us94What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/sample_us
88Date: Mar 202295Date: Mar 2022
89Contact: SeongJae Park <sj@kernel.org>96Contact: SeongJae Park <sj@kernel.org>
@@ -322,6 +329,18 @@ Contact: SeongJae Park <sj@kernel.org> · 232 unmodified lines
322Description: Writing to and reading from this file sets and gets the329Description: Writing to and reading from this file sets and gets the
323 goal-based effective quota auto-tuning algorithm to use.330 goal-based effective quota auto-tuning algorithm to use.
324 331
332What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_num
333Date: Mar 2026
334Contact: SeongJae Park <sj@kernel.org>
335Description: Writing to and reading from this file sets and gets the
336 action-failed memory quota charging ratio numerator.
337
338What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_denom
339Date: Mar 2026
340Contact: SeongJae Park <sj@kernel.org>
341Description: Writing to and reading from this file sets and gets the
342 action-failed memory quota charging ratio denominator.
343
325What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil344What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil
326Date: Mar 2022345Date: Mar 2022
327Contact: SeongJae Park <sj@kernel.org>346Contact: SeongJae Park <sj@kernel.org>
@@ -377,15 +396,20 @@ Contact: SeongJae Park <sj@kernel.org> · 49 unmodified lines
377Description: Writing to and reading from this file sets and gets the low396Description: Writing to and reading from this file sets and gets the low
378 watermark of the scheme in permil.397 watermark of the scheme in permil.
379 398
380What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/nr_filters399What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
381Date: Dec 2022400Date: Feb 2025
401Contact: SeongJae Park <sj@kernel.org>
402Description: Directory for DAMON core layer-handled DAMOS filters.
403
404What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/nr_filters
405Date: Feb 2025
382Contact: SeongJae Park <sj@kernel.org>406Contact: SeongJae Park <sj@kernel.org>
383Description: Writing a number 'N' to this file creates the number of407Description: Writing a number 'N' to this file creates the number of
384 directories for setting filters of the scheme named '0' to408 directories for setting filters of the scheme named '0' to
385 'N-1' under the filters/ directory.409 'N-1' under the core_filters/ directory.
386 410
387What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/type411What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/type
388Date: Dec 2022412Date: Feb 2025
389Contact: SeongJae Park <sj@kernel.org>413Contact: SeongJae Park <sj@kernel.org>
390Description: Writing to and reading from this file sets and gets the type of414Description: Writing to and reading from this file sets and gets the type of
391 the memory of the interest. 'anon' for anonymous pages,415 the memory of the interest. 'anon' for anonymous pages,
@@ -393,77 +417,78 @@ Description: Writing to and reading from this file sets and gets the type of · 1 unmodified line
393 'addr' for address range (an open-ended interval), or 'target'417 'addr' for address range (an open-ended interval), or 'target'
394 for DAMON monitoring target can be written and read.418 for DAMON monitoring target can be written and read.
395 419
396What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/memcg_path420What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/memcg_path
397Date: Dec 2022421Date: Feb 2025
398Contact: SeongJae Park <sj@kernel.org>422Contact: SeongJae Park <sj@kernel.org>
399Description: If 'memcg' is written to the 'type' file, writing to and423Description: If 'memcg' is written to the 'type' file, writing to and
400 reading from this file sets and gets the path to the memory424 reading from this file sets and gets the path to the memory
401 cgroup of the interest.425 cgroup of the interest.
402 426
403What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_start427What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_start
404Date: Jul 2023428Date: Feb 2025
405Contact: SeongJae Park <sj@kernel.org>429Contact: SeongJae Park <sj@kernel.org>
406Description: If 'addr' is written to the 'type' file, writing to or reading430Description: If 'addr' is written to the 'type' file, writing to or reading
407 from this file sets or gets the start address of the address431 from this file sets or gets the start address of the address
408 range for the filter.432 range for the filter.
409 433
410What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_end434What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_end
411Date: Jul 2023435Date: Feb 2025
412Contact: SeongJae Park <sj@kernel.org>436Contact: SeongJae Park <sj@kernel.org>
413Description: If 'addr' is written to the 'type' file, writing to or reading437Description: If 'addr' is written to the 'type' file, writing to or reading
414 from this file sets or gets the end address of the address438 from this file sets or gets the end address of the address
415 range for the filter.439 range for the filter.
416 440
417What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/min441What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/min
418Date: Feb 2025442Date: Feb 2025
419Contact: SeongJae Park <sj@kernel.org>443Contact: SeongJae Park <sj@kernel.org>
420Description: If 'hugepage_size' is written to the 'type' file, writing to444Description: If 'hugepage_size' is written to the 'type' file, writing to
421 or reading from this file sets or gets the minimum size of the445 or reading from this file sets or gets the minimum size of the
422 hugepage for the filter.446 hugepage for the filter.
423 447
424What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/max448What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/max
425Date: Feb 2025449Date: Feb 2025
426Contact: SeongJae Park <sj@kernel.org>450Contact: SeongJae Park <sj@kernel.org>
427Description: If 'hugepage_size' is written to the 'type' file, writing to451Description: If 'hugepage_size' is written to the 'type' file, writing to
428 or reading from this file sets or gets the maximum size of the452 or reading from this file sets or gets the maximum size of the
429 hugepage for the filter.453 hugepage for the filter.
430 454
431What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/target_idx455What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/damon_target_idx
432Date: Dec 2022456Date: Feb 2025
433Contact: SeongJae Park <sj@kernel.org>457Contact: SeongJae Park <sj@kernel.org>
434Description: If 'target' is written to the 'type' file, writing to or458Description: If 'target' is written to the 'type' file, writing to or
435 reading from this file sets or gets the index of the DAMON459 reading from this file sets or gets the index of the DAMON
436 monitoring target of the interest.460 monitoring target of the interest.
437 461
438What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/matching462What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/matching
439Date: Dec 2022463Date: Feb 2025
440Contact: SeongJae Park <sj@kernel.org>464Contact: SeongJae Park <sj@kernel.org>
441Description: Writing 'Y' or 'N' to this file sets whether the filter is for465Description: Writing 'Y' or 'N' to this file sets whether the filter is for
442 the memory of the 'type', or all except the 'type'.466 the memory of the 'type', or all except the 'type'.
443 467
444What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/allow468What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/allow
445Date: Jan 2025469Date: Feb 2025
446Contact: SeongJae Park <sj@kernel.org>470Contact: SeongJae Park <sj@kernel.org>
447Description: Writing 'Y' or 'N' to this file sets whether to allow or reject471Description: Writing 'Y' or 'N' to this file sets whether to allow or reject
448 applying the scheme's action to the memory that satisfies the472 applying the scheme's action to the memory that satisfies the
449 'type' and the 'matching' of the directory.473 'type' and the 'matching' of the directory.
450 474
451What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
452Date: Feb 2025
453Contact: SeongJae Park <sj@kernel.org>
454Description: Directory for DAMON core layer-handled DAMOS filters. Files
455 under this directory works same to those of
456 /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
457 directory.
458
459What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/ops_filters475What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/ops_filters
460Date: Feb 2025476Date: Feb 2025
461Contact: SeongJae Park <sj@kernel.org>477Contact: SeongJae Park <sj@kernel.org>
462Description: Directory for DAMON operations set layer-handled DAMOS filters.478Description: Directory for DAMON operations set layer-handled DAMOS filters.
463 Files under this directory works same to those of479 Files under this directory works same to those of
464 /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters480 /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
465 directory.481 directory.
466 482
483What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
484Date: Dec 2022
485Contact: SeongJae Park <sj@kernel.org>
486Description: Directory for DAMOS filters. Files under this directory works
487 same to those of
488 /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/{core,ops}_filters
489 directory. This is deprecated. Use the core_filters and
490 ops_filters instead.
491
467What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/nr_dests492What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/nr_dests
468Date: Jul 2025493Date: Jul 2025
469Contact: SeongJae Park <sj@kernel.org>494Contact: SeongJae Park <sj@kernel.org>
Documentation/admin-guide/kernel-parameters.txt+4−0
@@ -2067,6 +2067,10 @@ Kernel parameters · 2066 unmodified lines
2067 Format: nn[KMGTPE] or (node format)2067 Format: nn[KMGTPE] or (node format)
2068 <node>:nn[KMGTPE][,<node>:nn[KMGTPE]]2068 <node>:nn[KMGTPE][,<node>:nn[KMGTPE]]
2069 2069
2070 The size must be a multiple of the gigantic page size.
2071 When using node format, this applies to each per-node size.
2072 Missaligned values are dropped with a warning.
2073
2070 Reserve a CMA area of given size and allocate gigantic2074 Reserve a CMA area of given size and allocate gigantic
2071 hugepages using the CMA allocator. If enabled, the2075 hugepages using the CMA allocator. If enabled, the
2072 boot-time allocation of gigantic hugepages is skipped.2076 boot-time allocation of gigantic hugepages is skipped.
Documentation/admin-guide/mm/damon/lru_sort.rst+5−3
@@ -75,7 +75,7 @@ Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``. · 74 unmodified lines
75 75
76Input parameters that updated while DAMON_LRU_SORT is running are not applied76Input parameters that updated while DAMON_LRU_SORT is running are not applied
77by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT reads values77by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT reads values
78of parametrs except ``enabled`` again. Once the re-reading is done, this78of parameters except ``enabled`` again. Once the re-reading is done, this
79parameter is set as ``N``. If invalid parameters are found while the79parameter is set as ``N``. If invalid parameters are found while the
80re-reading, DAMON_LRU_SORT will be disabled.80re-reading, DAMON_LRU_SORT will be disabled.
81 81
@@ -246,7 +246,8 @@ monitor_region_start · 164 unmodified lines
246Start of target memory region in physical address.246Start of target memory region in physical address.
247 247
248The start physical address of memory region that DAMON_LRU_SORT will do work248The start physical address of memory region that DAMON_LRU_SORT will do work
249against. By default, biggest System RAM is used as the region.249against. By default, the system's entire physical memory is used as the
250region.
250 251
251monitor_region_end252monitor_region_end
252------------------253------------------
@@ -254,7 +255,8 @@ monitor_region_end · 1 unmodified line
254End of target memory region in physical address.255End of target memory region in physical address.
255 256
256The end physical address of memory region that DAMON_LRU_SORT will do work257The end physical address of memory region that DAMON_LRU_SORT will do work
257against. By default, biggest System RAM is used as the region.258against. By default, the system's entire physical memory is used as the
259region.
258 260
259addr_unit261addr_unit
260---------262---------
Documentation/admin-guide/mm/damon/reclaim.rst+16−3
@@ -67,7 +67,7 @@ Make DAMON_RECLAIM reads the input parameters again, except ``enabled``. · 66 unmodified lines
67 67
68Input parameters that updated while DAMON_RECLAIM is running are not applied68Input parameters that updated while DAMON_RECLAIM is running are not applied
69by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values69by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
70of parametrs except ``enabled`` again. Once the re-reading is done, this70of parameters except ``enabled`` again. Once the re-reading is done, this
71parameter is set as ``N``. If invalid parameters are found while the71parameter is set as ``N``. If invalid parameters are found while the
72re-reading, DAMON_RECLAIM will be disabled.72re-reading, DAMON_RECLAIM will be disabled.
73 73
@@ -85,6 +85,17 @@ identifies the region as cold, and reclaims it. · 11 unmodified lines
85 85
86120 seconds by default.86120 seconds by default.
87 87
88autotune_monitoring_intervals
89-----------------------------
90
91If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
92sampling and aggregation intervals. The auto-tuning aims to capture meaningful
93amount of access events in each DAMON-snapshot, while keeping the sampling
94interval 5 milliseconds in minimum, and 10 seconds in maximum. Setting this as
95``N`` disables the auto-tuning.
96
97Disabled by default.
98
88quota_ms99quota_ms
89--------100--------
90 101
@@ -229,7 +240,8 @@ Start of target memory region in physical address. · 138 unmodified lines
229 240
230The start physical address of memory region that DAMON_RECLAIM will do work241The start physical address of memory region that DAMON_RECLAIM will do work
231against. That is, DAMON_RECLAIM will find cold memory regions in this region242against. That is, DAMON_RECLAIM will find cold memory regions in this region
232and reclaims. By default, biggest System RAM is used as the region.243and reclaims. By default, the system's entire physical memory is used as the
244region.
233 245
234monitor_region_end246monitor_region_end
235------------------247------------------
@@ -238,7 +250,8 @@ End of target memory region in physical address. · 2 unmodified lines
238 250
239The end physical address of memory region that DAMON_RECLAIM will do work251The end physical address of memory region that DAMON_RECLAIM will do work
240against. That is, DAMON_RECLAIM will find cold memory regions in this region252against. That is, DAMON_RECLAIM will find cold memory regions in this region
241and reclaims. By default, biggest System RAM is used as the region.253and reclaims. By default, the system's entire physical memory is used as the
254region.
242 255
243addr_unit256addr_unit
244---------257---------
Documentation/admin-guide/mm/damon/stat.rst+7−0
@@ -89,3 +89,10 @@ percentiles of the idle time values via this read-only parameter. Reading the · 88 unmodified lines
89parameter returns 101 idle time values in milliseconds, separated by comma.89parameter returns 101 idle time values in milliseconds, separated by comma.
90Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle90Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle
91times.91times.
92
93kdamond_pid
94-----------
95
96PID of the DAMON thread.
97
98If DAMON_STAT is enabled, this becomes the PID of the worker thread. Else, -1.
Documentation/admin-guide/mm/damon/usage.rst+82−26
@@ -66,11 +66,17 @@ comma (","). · 65 unmodified lines
66 │ :ref:`kdamonds <sysfs_kdamonds>`/nr_kdamonds66 │ :ref:`kdamonds <sysfs_kdamonds>`/nr_kdamonds
67 │ │ :ref:`0 <sysfs_kdamond>`/state,pid,refresh_ms67 │ │ :ref:`0 <sysfs_kdamond>`/state,pid,refresh_ms
68 │ │ │ :ref:`contexts <sysfs_contexts>`/nr_contexts68 │ │ │ :ref:`contexts <sysfs_contexts>`/nr_contexts
69 │ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit69 │ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit,
70 │ │ │ │ pause
70 │ │ │ │ │ :ref:`monitoring_attrs <sysfs_monitoring_attrs>`/71 │ │ │ │ │ :ref:`monitoring_attrs <sysfs_monitoring_attrs>`/
71 │ │ │ │ │ │ intervals/sample_us,aggr_us,update_us72 │ │ │ │ │ │ intervals/sample_us,aggr_us,update_us
72 │ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us73 │ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us
73 │ │ │ │ │ │ nr_regions/min,max74 │ │ │ │ │ │ nr_regions/min,max
75 │ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes
76 │ │ │ │ │ │ │ 0/filters/nr_filters
77 │ │ │ │ │ │ │ │ 0/type,matching,allow,path
78 │ │ │ │ │ │ │ │ ...
79 │ │ │ │ │ │ │ ...
74 │ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets80 │ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
75 │ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target81 │ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target
76 │ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions82 │ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions
@@ -83,18 +89,23 @@ comma (","). · 6 unmodified lines
83 │ │ │ │ │ │ │ │ sz/min,max89 │ │ │ │ │ │ │ │ sz/min,max
84 │ │ │ │ │ │ │ │ nr_accesses/min,max90 │ │ │ │ │ │ │ │ nr_accesses/min,max
85 │ │ │ │ │ │ │ │ age/min,max91 │ │ │ │ │ │ │ │ age/min,max
86 │ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,effective_bytes,goal_tuner92 │ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,
93 │ │ │ │ │ │ │ effective_bytes,goal_tuner,
94 │ │ │ │ │ │ │ fail_charge_num,fail_charge_denom
87 │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil95 │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
88 │ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals96 │ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals
89 │ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path97 │ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path
90 │ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low98 │ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low
91 │ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters99 │ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters
92 │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max100 │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,damon_target_idx,min,max
93 │ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests101 │ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests
94 │ │ │ │ │ │ │ │ 0/id,weight102 │ │ │ │ │ │ │ │ 0/id,weight
95 │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots103 │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots
96 │ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes104 │ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes
97 │ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed105 │ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed
106 │ │ │ │ │ │ │ │ │ probes
107 │ │ │ │ │ │ │ │ │ │ 0/hits
108 │ │ │ │ │ │ │ │ │ │ ...
98 │ │ │ │ │ │ │ │ ...109 │ │ │ │ │ │ │ │ ...
99 │ │ │ │ │ │ ...110 │ │ │ │ │ │ ...
100 │ │ │ │ ...111 │ │ │ │ ...
@@ -194,9 +205,9 @@ details). At the moment, only one context per kdamond is supported, so only · 93 unmodified lines
194contexts/<N>/205contexts/<N>/
195-------------206-------------
196 207
197In each context directory, three files (``avail_operations``, ``operations``208In each context directory, four files (``avail_operations``, ``operations``,
198and ``addr_unit``) and three directories (``monitoring_attrs``, ``targets``,209``addr_unit`` and ``pause``) and three directories (``monitoring_attrs``,
199and ``schemes``) exist.210``targets``, and ``schemes``) exist.
200 211
201DAMON supports multiple types of :ref:`monitoring operations212DAMON supports multiple types of :ref:`monitoring operations
202<damon_design_configurable_operations_set>`, including those for virtual address213<damon_design_configurable_operations_set>`, including those for virtual address
@@ -214,6 +225,9 @@ reading from the ``operations`` file. · 11 unmodified lines
214``addr_unit`` file is for setting and getting the :ref:`address unit225``addr_unit`` file is for setting and getting the :ref:`address unit
215<damon_design_addr_unit>` parameter of the operations set.226<damon_design_addr_unit>` parameter of the operations set.
216 227
228``pause`` file is for setting and getting the :ref:`pause request
229<damon_design_execution_model_and_data_structures>` parameter of the context.
230
217.. _sysfs_monitoring_attrs:231.. _sysfs_monitoring_attrs:
218 232
219contexts/<N>/monitoring_attrs/233contexts/<N>/monitoring_attrs/
@@ -221,8 +235,8 @@ contexts/<N>/monitoring_attrs/ · 1 unmodified line
221 235
222Files for specifying attributes of the monitoring including required quality236Files for specifying attributes of the monitoring including required quality
223and efficiency of the monitoring are in ``monitoring_attrs`` directory.237and efficiency of the monitoring are in ``monitoring_attrs`` directory.
224Specifically, two directories, ``intervals`` and ``nr_regions`` exist in this238Specifically, three directories, ``intervals``, ``nr_regions`` and ``probes``
225directory.239exist in this directory.
226 240
227Under ``intervals`` directory, three files for DAMON's sampling interval241Under ``intervals`` directory, three files for DAMON's sampling interval
228(``sample_us``), aggregation interval (``aggr_us``), and update interval242(``sample_us``), aggregation interval (``aggr_us``), and update interval
@@ -256,6 +270,29 @@ tuning-applied current values of the two intervals can be read from the · 27 unmodified lines
256``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to270``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to
257the ``state`` file.271the ``state`` file.
258 272
273.. _damon_usage_sysfs_probes:
274
275contexts/<N>/monitoring_attrs/probes/
276-------------------------------------
277
278A directory for registering :ref:`data attributes monitoring
279<damon_design_data_attrs_monitoring>` probes.
280
281In the beginning, this directory has only one file, ``nr_probes``. Writing a
282number (``N``) to the file creates the number of child directories named ``0``
283to ``N-1``. Each directory represents each monitoring probe.
284
285In each probe directory, one directory, ``filters`` exists. The directory
286contains files for installing filters for the probe, that is used to determine
287the data attribute for the probe.
288
289In the beginning, ``filters`` directory has only one file, ``nr_filters``.
290Writing a number (``N``) to the file creates the number of child directories
291named ``0`` to ``N-1``. Each directory represents each filter and works in a
292way similar to that for :ref:`DAMOS filter <sysfs_filters>`. When the filter
293``type`` is ``memcg``, ``path`` file acts as ``memcg_path`` for :ref:`DAMOS
294filter <sysfs_filters>`.
295
259.. _sysfs_targets:296.. _sysfs_targets:
260 297
261contexts/<N>/targets/298contexts/<N>/targets/
@@ -337,7 +374,7 @@ to ``N-1``. Each directory represents each DAMON-based operation scheme. · 75 unmodified lines
337schemes/<N>/374schemes/<N>/
338------------375------------
339 376
340In each scheme directory, eight directories (``access_pattern``, ``quotas``,377In each scheme directory, nine directories (``access_pattern``, ``quotas``,
341``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``,378``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``,
342``stats``, and ``tried_regions``) and three files (``action``, ``target_nid``379``stats``, and ``tried_regions``) and three files (``action``, ``target_nid``
343and ``apply_interval``) exist.380and ``apply_interval``) exist.
@@ -377,9 +414,10 @@ schemes/<N>/quotas/ · 33 unmodified lines
377The directory for the :ref:`quotas <damon_design_damos_quotas>` of the given414The directory for the :ref:`quotas <damon_design_damos_quotas>` of the given
378DAMON-based operation scheme.415DAMON-based operation scheme.
379 416
380Under ``quotas`` directory, five files (``ms``, ``bytes``,417Under ``quotas`` directory, seven files (``ms``, ``bytes``,
381``reset_interval_ms``, ``effective_bytes`` and ``goal_tuner``) and two418``reset_interval_ms``, ``effective_bytes``, ``goal_tuner``, ``fail_charge_num``
382directories (``weights`` and ``goals``) exist.419and ``fail_charge_denom``) and two directories (``weights`` and ``goals``)
420exist.
383 421
384You can set the ``time quota`` in milliseconds, ``size quota`` in bytes, and422You can set the ``time quota`` in milliseconds, ``size quota`` in bytes, and
385``reset interval`` in milliseconds by writing the values to the three files,423``reset interval`` in milliseconds by writing the values to the three files,
@@ -398,6 +436,13 @@ the background design of the feature and the name of the selectable algorithms. · 12 unmodified lines
398Refer to :ref:`goals directory <sysfs_schemes_quota_goals>` for the goals436Refer to :ref:`goals directory <sysfs_schemes_quota_goals>` for the goals
399setup.437setup.
400 438
439You can set the action-failed memory quota charging ratio by writing the
440numerator and the denominator for the ratio to ``fail_charge_num`` and
441``fail_charge_denom`` files, respectively. Reading those files will return the
442current set values. Refer to :ref:`design
443<damon_design_damos_quotas_failed_memory_charging_ratio>` for more details of
444the ratio feature.
445
401The time quota is internally transformed to a size quota. Between the446The time quota is internally transformed to a size quota. Between the
402transformed size quota and user-specified size quota, smaller one is applied.447transformed size quota and user-specified size quota, smaller one is applied.
403Based on the user-specified :ref:`goal <sysfs_schemes_quota_goals>`, the448Based on the user-specified :ref:`goal <sysfs_schemes_quota_goals>`, the
@@ -429,10 +474,12 @@ to ``N-1``. Each directory represents each goal and current achievement. · 25 unmodified lines
429Among the multiple feedback, the best one is used.474Among the multiple feedback, the best one is used.
430 475
431Each goal directory contains five files, namely ``target_metric``,476Each goal directory contains five files, namely ``target_metric``,
432``target_value``, ``current_value`` ``nid`` and ``path``. Users can set and477``target_value``, ``current_value``, ``nid``, and ``path``. Users can set and
433get the five parameters for the quota auto-tuning goals that specified on the478get the five parameters for the quota auto-tuning goals that specified on the
434:ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and479:ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and
435reading from each of the files. Note that users should further write480reading from each of the files. Because the kernel does not update
481``current_value``, reading it only makes sense when ``target_metric`` is
482``user_input``. Note that users should further write
436``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond483``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond
437directory <sysfs_kdamond>` to pass the feedback to DAMON.484directory <sysfs_kdamond>` to pass the feedback to DAMON.
438 485
@@ -447,7 +494,7 @@ given DAMON-based operation scheme. · 8 unmodified lines
447Under the watermarks directory, five files (``metric``, ``interval_us``,494Under the watermarks directory, five files (``metric``, ``interval_us``,
448``high``, ``mid``, and ``low``) for setting the metric, the time interval495``high``, ``mid``, and ``low``) for setting the metric, the time interval
449between check of the metric, and the three watermarks exist. You can set and496between check of the metric, and the three watermarks exist. You can set and
450get the five values by writing to the files, respectively.497get the five values by writing to and reading from the files, respectively.
451 498
452Keywords and meanings of those that can be written to the ``metric`` file are499Keywords and meanings of those that can be written to the ``metric`` file are
453as below.500as below.
@@ -455,7 +502,7 @@ as below. · 1 unmodified line
455 - none: Ignore the watermarks502 - none: Ignore the watermarks
456 - free_mem_rate: System's free memory rate (per thousand)503 - free_mem_rate: System's free memory rate (per thousand)
457 504
458The ``interval`` should written in microseconds unit.505The ``interval_us`` should be written in microseconds unit.
459 506
460.. _sysfs_filters:507.. _sysfs_filters:
461 508
@@ -471,10 +518,10 @@ directory can be used for installing filters regardless of their handled · 9 unmodified lines
471layers. Filters that requested by ``core_filters`` and ``ops_filters`` will be518layers. Filters that requested by ``core_filters`` and ``ops_filters`` will be
472installed before those of ``filters``. All three directories have same files.519installed before those of ``filters``. All three directories have same files.
473 520
474Use of ``filters`` directory can make expecting evaluation orders of given521Use of ``filters`` directory can make filters evaluation orders confusing to
475filters with the files under directory bit confusing. Users are hence522expect. For this reason, ``filters`` directory is deprecated. It is still
476recommended to use ``core_filters`` and ``ops_filters`` directories. The523functioning, but is scheduled for removal in the near future. Users should use
477``filters`` directory could be deprecated in future.524``core_filters`` and ``ops_filters`` directories instead.
478 525
479In the beginning, the directory has only one file, ``nr_filters``. Writing a526In the beginning, the directory has only one file, ``nr_filters``. Writing a
480number (``N``) to the file creates the number of child directories named ``0``527number (``N``) to the file creates the number of child directories named ``0``
@@ -483,9 +530,9 @@ in the numeric order. · 2 unmodified lines
483 530
484Each filter directory contains nine files, namely ``type``, ``matching``,531Each filter directory contains nine files, namely ``type``, ``matching``,
485``allow``, ``memcg_path``, ``addr_start``, ``addr_end``, ``min``, ``max``532``allow``, ``memcg_path``, ``addr_start``, ``addr_end``, ``min``, ``max``
486and ``target_idx``. To ``type`` file, you can write the type of the filter.533and ``damon_target_idx``. To ``type`` file, you can write the type of the
487Refer to :ref:`the design doc <damon_design_damos_filters>` for available type534filter. Refer to :ref:`the design doc <damon_design_damos_filters>` for
488names, their meaning and on what layer those are handled.535available type names, their meaning and on what layer those are handled.
489 536
490For ``memcg`` type, you can specify the memory cgroup of the interest by537For ``memcg`` type, you can specify the memory cgroup of the interest by
491writing the path of the memory cgroup from the cgroups mount point to538writing the path of the memory cgroup from the cgroups mount point to
@@ -495,7 +542,7 @@ files, respectively. For ``hugepage_size`` type, you can specify the minimum · 3 unmodified lines
495and maximum size of the range (closed interval) to ``min`` and ``max`` files,542and maximum size of the range (closed interval) to ``min`` and ``max`` files,
496respectively. For ``target`` type, you can specify the index of the target543respectively. For ``target`` type, you can specify the index of the target
497between the list of the DAMON context's monitoring targets list to544between the list of the DAMON context's monitoring targets list to
498``target_idx`` file.545``damon_target_idx`` file.
499 546
500You can write ``Y`` or ``N`` to ``matching`` file to specify whether the filter547You can write ``Y`` or ``N`` to ``matching`` file to specify whether the filter
501is for memory that matches the ``type``. You can write ``Y`` or ``N`` to548is for memory that matches the ``type``. You can write ``Y`` or ``N`` to
@@ -601,10 +648,19 @@ tried_regions/<N>/ · 99 unmodified lines
601------------------648------------------
602 649
603In each region directory, you will find five files (``start``, ``end``,650In each region directory, you will find five files (``start``, ``end``,
604``nr_accesses``, ``age``, and ``sz_filter_passed``). Reading the files will651``nr_accesses``, ``age`` and ``sz_filter_passed``). Reading the files will
605show the properties of the region that corresponding DAMON-based operation652show the properties of the region that corresponding DAMON-based operation
606scheme ``action`` has tried to be applied.653scheme ``action`` has tried to be applied.
607 654
655tried_regions/<N>/probes/
656-------------------------
657
658In each region directory, one directory (``probes``) also exists. In the
659directory, subdirectories named ``0`` to ``N-1`` exists. ``N`` is the number
660of installed probes. In each number-named directory, a file (``hits``) exist.
661Reading the file shows the number of data attributes monitoring probe-hit
662positive samples of the region.
663
608Example664Example
609~~~~~~~665~~~~~~~
610 666
@@ -677,7 +733,7 @@ show results using tracepoint supporting tools like ``perf``. For example:: · 66 unmodified lines
677 733
678Each line of the perf script output represents each monitoring region. The734Each line of the perf script output represents each monitoring region. The
679first five fields are as usual other tracepoint outputs. The sixth field735first five fields are as usual other tracepoint outputs. The sixth field
680(``target_id=X``) shows the ide of the monitoring target of the region. The736(``target_id=X``) shows the id of the monitoring target of the region. The
681seventh field (``nr_regions=X``) shows the total number of monitoring regions737seventh field (``nr_regions=X``) shows the total number of monitoring regions
682for the target. The eighth field (``X-Y:``) shows the start (``X``) and end738for the target. The eighth field (``X-Y:``) shows the start (``X``) and end
683(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the739(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the
Documentation/admin-guide/mm/transhuge.rst+2−2
@@ -57,7 +57,7 @@ prominent because the size of each page isn't as huge as the PMD-sized · 56 unmodified lines
57variant and there is less memory to clear in each page fault. Some57variant and there is less memory to clear in each page fault. Some
58architectures also employ TLB compression mechanisms to squeeze more58architectures also employ TLB compression mechanisms to squeeze more
59entries in when a set of PTEs are virtually and physically contiguous59entries in when a set of PTEs are virtually and physically contiguous
60and approporiately aligned. In this case, TLB misses will occur less60and appropriately aligned. In this case, TLB misses will occur less
61often.61often.
62 62
63THP can be enabled system wide or restricted to certain tasks or even63THP can be enabled system wide or restricted to certain tasks or even
@@ -210,7 +210,7 @@ PMD-mappable transparent hugepage:: · 146 unmodified lines
210 cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size210 cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
211 211
212All THPs at fault and collapse time will be added to _deferred_list,212All THPs at fault and collapse time will be added to _deferred_list,
213and will therefore be split under memory presure if they are considered213and will therefore be split under memory pressure if they are considered
214"underused". A THP is underused if the number of zero-filled pages in214"underused". A THP is underused if the number of zero-filled pages in
215the THP is above max_ptes_none (see below). It is possible to disable215the THP is above max_ptes_none (see below). It is possible to disable
216this behaviour by writing 0 to shrink_underused, and enable it by writing216this behaviour by writing 0 to shrink_underused, and enable it by writing
Documentation/admin-guide/sysctl/vm.rst+2−0
@@ -1034,6 +1034,8 @@ min(3% of current process size, user_reserve_kbytes) of free memory. · 1033 unmodified lines
1034This is intended to prevent a user from starting a single memory hogging1034This is intended to prevent a user from starting a single memory hogging
1035process, such that they cannot recover (kill the hog).1035process, such that they cannot recover (kill the hog).
1036 1036
1037This setting has no effect when overcommit_memory is set to 0 or 1.
1038
1037user_reserve_kbytes defaults to min(3% of the current process size, 128MB).1039user_reserve_kbytes defaults to min(3% of the current process size, 128MB).
1038 1040
1039If this is reduced to zero, then the user will be allowed to allocate1041If this is reduced to zero, then the user will be allowed to allocate
Documentation/mm/damon/design.rst+75−3
@@ -19,6 +19,13 @@ types of monitoring. · 18 unmodified lines
19To know how user-space can do the configurations and start/stop DAMON, refer to19To know how user-space can do the configurations and start/stop DAMON, refer to
20:ref:`DAMON sysfs interface <sysfs_interface>` documentation.20:ref:`DAMON sysfs interface <sysfs_interface>` documentation.
21 21
22Users can also request each context execution to be paused and resumed. When
23it is paused, the kdamond does nothing other than applying online parameter
24update.
25
26To know how user-space can pause/resume each context, refer to :ref:`DAMON
27sysfs context <sysfs_context>` usage documentation.
28
22 29
23Overall Architecture30Overall Architecture
24====================31====================
@@ -140,7 +147,7 @@ as Idle page tracking does. · 115 unmodified lines
140Address Unit147Address Unit
141------------148------------
142 149
143DAMON core layer uses ``unsinged long`` type for monitoring target address150DAMON core layer uses ``unsigned long`` type for monitoring target address
144ranges. In some cases, the address space for a given operations set could be151ranges. In some cases, the address space for a given operations set could be
145too large to be handled with the type. ARM (32-bit) with large physical152too large to be handled with the type. ARM (32-bit) with large physical
146address extension is an example. For such cases, a per-operations set153address extension is an example. For such cases, a per-operations set
@@ -269,6 +276,45 @@ interval``, DAMON checks if the region's size and access frequency · 122 unmodified lines
269(``nr_accesses``) has significantly changed. If so, the counter is reset to276(``nr_accesses``) has significantly changed. If so, the counter is reset to
270zero. Otherwise, the counter is increased.277zero. Otherwise, the counter is increased.
271 278
279.. _damon_design_data_attrs_monitoring:
280
281Data Attributes Monitoring
282~~~~~~~~~~~~~~~~~~~~~~~~~~
283
284Data access pattern is only one type of data attributes. In some use cases,
285users need to know more data attributes information. For example, users may
286need to know how much of a given hot or cold memory region is backed by
287anonymous pages, or belong to a specific cgroup. For such use case, data
288attributes monitoring feature is provided.
289
290Using the feature, users can register data attributes of their interest to the
291DAMON :ref:`context <damon_design_execution_model_and_data_structures>`. The
292registration is made by specifying a probe per attribute. Each of the probe
293specifies a rule to determine if a given memory region has the related
294attribute. The rule is constructed with multiple filters. The filters work
295same to :ref:`DAMOS filters <damon_design_damos_filters>` except the supported
296filter types. Currently only ``anon`` and ``memcg`` filter types are supported
297for data attributes monitoring.
298
299If such probes are registered, DAMON executes the probes for each region's
300sampling memory when it does the access :ref:`sampling
301<damon_design_region_based_sampling>`. The number of samples that identified
302as having the data attribute (hitting the probe) per :ref:`aggregation interval
303<damon_design_monitoring>` is accounted in a per-region per-probe counter.
304Users can therefore know how much of a given DAMON region has a specific data
305attribute by reading the per-region per-probe probe hits counter after each
306aggregation interval.
307
308This is a sampling based mechanism. Hence, it is lightweight but the output
309may include some measurement errors. The output should be used with good
310understanding of statistics.
311
312Another way to do this for higher accuracy is using :ref:`DAMOS filter
313<damon_design_damos_filters>` with ``stat`` :ref:`action
314<damon_design_damos_action>` and ``sz_ops_filter_passed`` :ref:`stat
315<damon_design_damos_stat>`. This approach provides the data attributes
316information in page level. But, because it is operated in page level, the
317overhead is proportional to the size of the memory.
272 318
273Dynamic Target Space Updates Handling319Dynamic Target Space Updates Handling
274~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~320~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -371,7 +417,7 @@ with theoretical maximum ``nr_accesses``, which can be calculated as · 96 unmodified lines
371``aggregation interval / sampling interval``.417``aggregation interval / sampling interval``.
372 418
373The mechanism calculates the ratio of access events for ``aggrs`` aggregations,419The mechanism calculates the ratio of access events for ``aggrs`` aggregations,
374and increases or decrease the ``sampleing interval`` and ``aggregation420and increases or decrease the ``sampling interval`` and ``aggregation
375interval`` in same ratio, if the observed access ratio is lower or higher than421interval`` in same ratio, if the observed access ratio is lower or higher than
376the target, respectively. The ratio of the intervals change is decided in422the target, respectively. The ratio of the intervals change is decided in
377proportion to the distance between current samples ratio and the target ratio.423proportion to the distance between current samples ratio and the target ratio.
@@ -387,7 +433,7 @@ The tuning is turned off by default, and need to be set explicitly by the user. · 9 unmodified lines
387As a rule of thumbs and the Parreto principle, 4% access samples ratio target433As a rule of thumbs and the Parreto principle, 4% access samples ratio target
388is recommended. Note that Parreto principle (80/20 rule) has applied twice.434is recommended. Note that Parreto principle (80/20 rule) has applied twice.
389That is, assumes 4% (20% of 20%) DAMON-observed access events ratio (source)435That is, assumes 4% (20% of 20%) DAMON-observed access events ratio (source)
390to capture 64% (80% multipled by 80%) real access events (outcomes).436to capture 64% (80% multiplied by 80%) real access events (outcomes).
391 437
392To know how user-space can use this feature via :ref:`DAMON sysfs interface438To know how user-space can use this feature via :ref:`DAMON sysfs interface
393<sysfs_interface>`, refer to :ref:`intervals_goal439<sysfs_interface>`, refer to :ref:`intervals_goal
@@ -474,6 +520,10 @@ that supports each action are as below. · 80 unmodified lines
474 Supported by ``vaddr`` and ``fvaddr`` operations set. When520 Supported by ``vaddr`` and ``fvaddr`` operations set. When
475 TRANSPARENT_HUGEPAGE is disabled, the application of the action will just521 TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
476 fail.522 fail.
523 - ``collapse``: Call ``madvise()`` for the region with ``MADV_COLLAPSE``.
524 Supported by ``vaddr`` and ``fvaddr`` operations set. When
525 TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
526 fail.
477 - ``lru_prio``: Prioritize the region on its LRU lists.527 - ``lru_prio``: Prioritize the region on its LRU lists.
478 Supported by ``paddr`` operations set.528 Supported by ``paddr`` operations set.
479 - ``lru_deprio``: Deprioritize the region on its LRU lists.529 - ``lru_deprio``: Deprioritize the region on its LRU lists.
@@ -565,6 +615,28 @@ interface <sysfs_interface>`, refer to :ref:`weights <sysfs_quotas>` part of · 85 unmodified lines
565the documentation.615the documentation.
566 616
567 617
618.. _damon_design_damos_quotas_failed_memory_charging_ratio:
619
620Action-failed Memory Charging Ratio
621^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
622
623DAMOS action to a given region can fail for some subsets of the memory of the
624region. For example, if the action is ``pageout`` and the region has some
625unreclaimable pages, applying the action to the pages will fail. The amount of
626system resource that is taken for such failed action applications is usually
627different from that for successful action applications. For such cases, users
628can set different charging ratio for such failed memory. The ratio can be
629specified using ``fail_charge_num`` and ``fail_charge_denom`` parameters. The
630two parameters represent the numerator and denominator of the ratio. The
631feature is enabled only if ``fail_charge_denom`` is not zero.
632
633For example, let's suppose a DAMOS action is applied to a region of 1,000 MiB
634size. The action is successfully applied to only 700 MiB of the region.
635``fail_charge_num`` and ``fail_charge_denom`` are set to ``1`` and ``1024``,
636respectively. Then only 700 MiB and 300 KiB of size (``700 MiB + 300 MiB * 1 /
6371024``) will be charged.
638
639
568.. _damon_design_damos_quotas_auto_tuning:640.. _damon_design_damos_quotas_auto_tuning:
569 641
570Aim-oriented Feedback-driven Auto-tuning642Aim-oriented Feedback-driven Auto-tuning
Documentation/mm/damon/maintainer-profile.rst+21−0
@@ -100,3 +100,24 @@ There is also a public Google `calendar · 99 unmodified lines
100<https://calendar.google.com/calendar/u/0?cid=ZDIwOTA4YTMxNjc2MDQ3NTIyMmUzYTM5ZmQyM2U4NDA0ZGIwZjBiYmJlZGQxNDM0MmY4ZTRjOTE0NjdhZDRiY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t>`_100<https://calendar.google.com/calendar/u/0?cid=ZDIwOTA4YTMxNjc2MDQ3NTIyMmUzYTM5ZmQyM2U4NDA0ZGIwZjBiYmJlZGQxNDM0MmY4ZTRjOTE0NjdhZDRiY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t>`_
101that has the events. Anyone can subscribe to it. DAMON maintainer will also101that has the events. Anyone can subscribe to it. DAMON maintainer will also
102provide periodic reminders to the mailing list (damon@lists.linux.dev).102provide periodic reminders to the mailing list (damon@lists.linux.dev).
103
104AI Review
105---------
106
107For patches that are publicly posted to DAMON mailing list
108(damon@lists.linux.dev), AI reviews of the patches will be available at
109sashiko.dev. The reviews could also be sent as mails to the author of the
110patch.
111
112Patch authors are encouraged to check the AI reviews and share their opinions.
113The sharing could be done as a reply to the mail thread. Consider reducing the
114recipients list for such sharing, since some people are not really interested
115in AI reviews. As a rule of thumb, drop stable@vger.kernel.org and individuals
116except DAMON maintainer.
117
118`hkml` also provides a `feature
119<https://github.com/sjp38/hackermail/blob/master/USAGE.md#forwarding-sashikodev-statuscomments-to-mailing-list>`_
120for such sharing. Please feel free to use the feature.
121
122It is only an optional recommendation. DAMON maintainer could also ask any
123question about the AI reviews, though.
Documentation/mm/process_addrs.rst+1−1
@@ -775,7 +775,7 @@ lock, releasing or downgrading the mmap write lock also releases the VMA write · 774 unmodified lines
775lock so there is no :c:func:`!vma_end_write` function.775lock so there is no :c:func:`!vma_end_write` function.
776 776
777Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily777Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily
778modified so that readers can detect the presense of a writer. The reference counter is778modified so that readers can detect the presence of a writer. The reference counter is
779restored once the vma sequence number used for serialisation is updated.779restored once the vma sequence number used for serialisation is updated.
780 780
781This ensures the semantics we require - VMA write locks provide exclusive write781This ensures the semantics we require - VMA write locks provide exclusive write
MAINTAINERS+14−2
@@ -6626,7 +6626,6 @@ F: mm/memcontrol.c · 6625 unmodified lines
6626F: mm/memcontrol-v1.c6626F: mm/memcontrol-v1.c
6627F: mm/memcontrol-v1.h6627F: mm/memcontrol-v1.h
6628F: mm/page_counter.c6628F: mm/page_counter.c
6629F: mm/swap_cgroup.c
6630F: samples/cgroup/*6629F: samples/cgroup/*
6631F: tools/testing/selftests/cgroup/memcg_protection.m6630F: tools/testing/selftests/cgroup/memcg_protection.m
6632F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c6631F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
@@ -16920,6 +16919,7 @@ L: linux-mm@kvack.org · 10287 unmodified lines
16920S: Maintained16919S: Maintained
16921W: http://www.linux-mm.org16920W: http://www.linux-mm.org
16922T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm16921T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
16922F: Documentation/admin-guide/sysctl/vm.rst
16923F: include/linux/folio_batch.h16923F: include/linux/folio_batch.h
16924F: include/linux/gfp.h16924F: include/linux/gfp.h
16925F: include/linux/gfp_types.h16925F: include/linux/gfp_types.h
@@ -16992,6 +16992,7 @@ L: linux-mm@kvack.org · 66 unmodified lines
16992S: Maintained16992S: Maintained
16993W: http://www.linux-mm.org16993W: http://www.linux-mm.org
16994T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm16994T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
16995F: Documentation/ABI/testing/sysfs-kernel-mm-ksm
16995F: Documentation/admin-guide/mm/ksm.rst16996F: Documentation/admin-guide/mm/ksm.rst
16996F: Documentation/mm/ksm.rst16997F: Documentation/mm/ksm.rst
16997F: include/linux/ksm.h16998F: include/linux/ksm.h
@@ -17014,6 +17015,8 @@ L: linux-mm@kvack.org · 16 unmodified lines
17014S: Maintained17015S: Maintained
17015W: http://www.linux-mm.org17016W: http://www.linux-mm.org
17016T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm17017T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17018F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
17019F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
17017F: include/linux/mempolicy.h17020F: include/linux/mempolicy.h
17018F: include/uapi/linux/mempolicy.h17021F: include/uapi/linux/mempolicy.h
17019F: include/linux/migrate.h17022F: include/linux/migrate.h
@@ -17056,6 +17059,10 @@ L: linux-mm@kvack.org · 36 unmodified lines
17056S: Maintained17059S: Maintained
17057W: http://www.linux-mm.org17060W: http://www.linux-mm.org
17058T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm17061T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17062F: Documentation/ABI/testing/sysfs-kernel-mm
17063F: Documentation/ABI/testing/sysfs-kernel-mm-cma
17064F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
17065F: Documentation/ABI/testing/sysfs-kernel-mm-numa
17059F: Documentation/admin-guide/mm/17066F: Documentation/admin-guide/mm/
17060F: Documentation/mm/17067F: Documentation/mm/
17061F: include/linux/cma.h17068F: include/linux/cma.h
@@ -17179,6 +17186,7 @@ R: Barry Song <baohua@kernel.org> · 117 unmodified lines
17179R: Youngjun Park <youngjun.park@lge.com>17186R: Youngjun Park <youngjun.park@lge.com>
17180L: linux-mm@kvack.org17187L: linux-mm@kvack.org
17181S: Maintained17188S: Maintained
17189F: Documentation/ABI/testing/sysfs-kernel-mm-swap
17182F: Documentation/mm/swap-table.rst17190F: Documentation/mm/swap-table.rst
17183F: include/linux/swap.h17191F: include/linux/swap.h
17184F: include/linux/swapfile.h17192F: include/linux/swapfile.h
@@ -17206,6 +17214,7 @@ L: linux-mm@kvack.org · 21 unmodified lines
17206S: Maintained17214S: Maintained
17207W: http://www.linux-mm.org17215W: http://www.linux-mm.org
17208T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm17216T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17217F: Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage
17209F: Documentation/admin-guide/mm/transhuge.rst17218F: Documentation/admin-guide/mm/transhuge.rst
17210F: include/linux/huge_mm.h17219F: include/linux/huge_mm.h
17211F: include/linux/khugepaged.h17220F: include/linux/khugepaged.h
@@ -17224,7 +17233,6 @@ R: Peter Xu <peterx@redhat.com> · 12 unmodified lines
17224L: linux-mm@kvack.org17233L: linux-mm@kvack.org
17225S: Maintained17234S: Maintained
17226F: Documentation/admin-guide/mm/userfaultfd.rst17235F: Documentation/admin-guide/mm/userfaultfd.rst
17227F: fs/userfaultfd.c
17228F: include/asm-generic/pgtable_uffd.h17236F: include/asm-generic/pgtable_uffd.h
17229F: include/linux/userfaultfd_k.h17237F: include/linux/userfaultfd_k.h
17230F: include/uapi/linux/userfaultfd.h17238F: include/uapi/linux/userfaultfd.h
@@ -20352,6 +20360,10 @@ T: git git://git.infradead.org/users/willy/pagecache.git · 3121 unmodified lines
20352F: Documentation/filesystems/locking.rst20360F: Documentation/filesystems/locking.rst
20353F: Documentation/filesystems/vfs.rst20361F: Documentation/filesystems/vfs.rst
20354F: include/linux/pagemap.h20362F: include/linux/pagemap.h
20363F: include/linux/writeback.h
20364F: include/trace/events/filemap.h
20365F: include/trace/events/readahead.h
20366F: include/trace/events/writeback.h
20355F: mm/filemap.c20367F: mm/filemap.c
20356F: mm/page-writeback.c20368F: mm/page-writeback.c
20357F: mm/readahead.c20369F: mm/readahead.c
arch/arm64/mm/mmu.c+3−16
@@ -1781,20 +1781,6 @@ static void free_empty_tables(unsigned long addr, unsigned long end, · 1780 unmodified lines
1781}1781}
1782#endif1782#endif
1783 1783
1784void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node,
1785 unsigned long addr, unsigned long next)
1786{
1787 pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL));
1788}
1789
1790int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
1791 unsigned long addr, unsigned long next)
1792{
1793 vmemmap_verify((pte_t *)pmdp, node, addr, next);
1794
1795 return pmd_leaf(READ_ONCE(*pmdp));
1796}
1797
1798int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,1784int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
1799 struct vmem_altmap *altmap)1785 struct vmem_altmap *altmap)
1800{1786{
@@ -2030,12 +2016,13 @@ err: · 229 unmodified lines
2030 return ret;2016 return ret;
2031}2017}
2032 2018
2033void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)2019void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
2020 struct dev_pagemap *pgmap)
2034{2021{
2035 unsigned long start_pfn = start >> PAGE_SHIFT;2022 unsigned long start_pfn = start >> PAGE_SHIFT;
2036 unsigned long nr_pages = size >> PAGE_SHIFT;2023 unsigned long nr_pages = size >> PAGE_SHIFT;
2037 2024
2038 __remove_pages(start_pfn, nr_pages, altmap);2025 __remove_pages(start_pfn, nr_pages, altmap, pgmap);
2039 __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size);2026 __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size);
2040}2027}
2041 2028
arch/loongarch/mm/init.c+3−13
@@ -119,12 +119,13 @@ int arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) · 118 unmodified lines
119 return ret;119 return ret;
120}120}
121 121
122void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)122void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
123 struct dev_pagemap *pgmap)
123{124{
124 unsigned long start_pfn = start >> PAGE_SHIFT;125 unsigned long start_pfn = start >> PAGE_SHIFT;
125 unsigned long nr_pages = size >> PAGE_SHIFT;126 unsigned long nr_pages = size >> PAGE_SHIFT;
126 127
127 __remove_pages(start_pfn, nr_pages, altmap);128 __remove_pages(start_pfn, nr_pages, altmap, pgmap);
128}129}
129#endif130#endif
130 131
@@ -139,17 +140,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, · 8 unmodified lines
139 set_pmd_at(&init_mm, addr, pmd, entry);140 set_pmd_at(&init_mm, addr, pmd, entry);
140}141}
141 142
142int __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
143 unsigned long addr, unsigned long next)
144{
145 int huge = pmd_val(pmdp_get(pmd)) & _PAGE_HUGE;
146
147 if (huge)
148 vmemmap_verify((pte_t *)pmd, node, addr, next);
149
150 return huge;
151}
152
153int __meminit vmemmap_populate(unsigned long start, unsigned long end,143int __meminit vmemmap_populate(unsigned long start, unsigned long end,
154 int node, struct vmem_altmap *altmap)144 int node, struct vmem_altmap *altmap)
155{145{
arch/powerpc/mm/init_64.c+0−8
@@ -41,7 +41,6 @@ · 40 unmodified lines
41#include <linux/libfdt.h>41#include <linux/libfdt.h>
42#include <linux/memremap.h>42#include <linux/memremap.h>
43#include <linux/memory.h>43#include <linux/memory.h>
44#include <linux/bootmem_info.h>
45 44
46#include <asm/pgalloc.h>45#include <asm/pgalloc.h>
47#include <asm/page.h>46#include <asm/page.h>
@@ -388,13 +387,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, · 340 unmodified lines
388 387
389#endif388#endif
390 389
391#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE
392void register_page_bootmem_memmap(unsigned long section_nr,
393 struct page *start_page, unsigned long size)
394{
395}
396#endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
397
398#endif /* CONFIG_SPARSEMEM_VMEMMAP */390#endif /* CONFIG_SPARSEMEM_VMEMMAP */
399 391
400#ifdef CONFIG_PPC_BOOK3S_64392#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/mm/mem.c+3−2
@@ -158,12 +158,13 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, · 157 unmodified lines
158 return rc;158 return rc;
159}159}
160 160
161void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)161void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
162 struct dev_pagemap *pgmap)
162{163{
163 unsigned long start_pfn = start >> PAGE_SHIFT;164 unsigned long start_pfn = start >> PAGE_SHIFT;
164 unsigned long nr_pages = size >> PAGE_SHIFT;165 unsigned long nr_pages = size >> PAGE_SHIFT;
165 166
166 __remove_pages(start_pfn, nr_pages, altmap);167 __remove_pages(start_pfn, nr_pages, altmap, pgmap);
167 arch_remove_linear_mapping(start, size);168 arch_remove_linear_mapping(start, size);
168}169}
169#endif170#endif
arch/powerpc/platforms/pseries/hotplug-memory.c+4−10
@@ -164,13 +164,7 @@ static int update_lmb_associativity_index(struct drmem_lmb *lmb) · 163 unmodified lines
164 164
165static struct memory_block *lmb_to_memblock(struct drmem_lmb *lmb)165static struct memory_block *lmb_to_memblock(struct drmem_lmb *lmb)
166{166{
167 unsigned long section_nr;167 return memory_block_get(phys_to_block_id(lmb->base_addr));
168 struct memory_block *mem_block;
169
170 section_nr = pfn_to_section_nr(PFN_DOWN(lmb->base_addr));
171
172 mem_block = find_memory_block(section_nr);
173 return mem_block;
174}168}
175 169
176static int get_lmb_range(u32 drc_index, int n_lmbs,170static int get_lmb_range(u32 drc_index, int n_lmbs,
@@ -220,7 +214,7 @@ static int dlpar_change_lmb_state(struct drmem_lmb *lmb, bool online) · 43 unmodified lines
220 else214 else
221 rc = 0;215 rc = 0;
222 216
223 put_device(&mem_block->dev);217 memory_block_put(mem_block);
224 218
225 return rc;219 return rc;
226}220}
@@ -319,12 +313,12 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb) · 92 unmodified lines
319 313
320 rc = dlpar_offline_lmb(lmb);314 rc = dlpar_offline_lmb(lmb);
321 if (rc) {315 if (rc) {
322 put_device(&mem_block->dev);316 memory_block_put(mem_block);
323 return rc;317 return rc;
324 }318 }
325 319
326 __remove_memory(lmb->base_addr, memory_block_size);320 __remove_memory(lmb->base_addr, memory_block_size);
327 put_device(&mem_block->dev);321 memory_block_put(mem_block);
328 322
329 /* Update memory regions for memory remove */323 /* Update memory regions for memory remove */
330 memblock_remove(lmb->base_addr, memory_block_size);324 memblock_remove(lmb->base_addr, memory_block_size);
arch/riscv/mm/init.c+3−15
@@ -1359,19 +1359,6 @@ void __init misc_mem_init(void) · 1358 unmodified lines
1359}1359}
1360 1360
1361#ifdef CONFIG_SPARSEMEM_VMEMMAP1361#ifdef CONFIG_SPARSEMEM_VMEMMAP
1362void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
1363 unsigned long addr, unsigned long next)
1364{
1365 pmd_set_huge(pmd, virt_to_phys(p), PAGE_KERNEL);
1366}
1367
1368int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
1369 unsigned long addr, unsigned long next)
1370{
1371 vmemmap_verify((pte_t *)pmdp, node, addr, next);
1372 return 1;
1373}
1374
1375int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,1362int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
1376 struct vmem_altmap *altmap)1363 struct vmem_altmap *altmap)
1377{1364{
@@ -1742,9 +1729,10 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *param · 364 unmodified lines
1742 return ret;1729 return ret;
1743}1730}
1744 1731
1745void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)1732void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
1733 struct dev_pagemap *pgmap)
1746{1734{
1747 __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);1735 __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, pgmap);
1748 remove_linear_mapping(start, size);1736 remove_linear_mapping(start, size);
1749 flush_tlb_all();1737 flush_tlb_all();
1750}1738}
arch/s390/mm/init.c+3−2
@@ -278,12 +278,13 @@ int arch_add_memory(int nid, u64 start, u64 size, · 277 unmodified lines
278 return rc;278 return rc;
279}279}
280 280
281void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)281void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
282 struct dev_pagemap *pgmap)
282{283{
283 unsigned long start_pfn = start >> PAGE_SHIFT;284 unsigned long start_pfn = start >> PAGE_SHIFT;
284 unsigned long nr_pages = size >> PAGE_SHIFT;285 unsigned long nr_pages = size >> PAGE_SHIFT;
285 286
286 __remove_pages(start_pfn, nr_pages, altmap);287 __remove_pages(start_pfn, nr_pages, altmap, pgmap);
287 vmem_remove_mapping(start, size);288 vmem_remove_mapping(start, size);
288}289}
289#endif /* CONFIG_MEMORY_HOTPLUG */290#endif /* CONFIG_MEMORY_HOTPLUG */
arch/s390/mm/vmem.c+1−2
@@ -4,7 +4,6 @@ · 3 unmodified lines
4 */4 */
5 5
6#include <linux/memory_hotplug.h>6#include <linux/memory_hotplug.h>
7#include <linux/bootmem_info.h>
8#include <linux/cpufeature.h>7#include <linux/cpufeature.h>
9#include <linux/memblock.h>8#include <linux/memblock.h>
10#include <linux/pfn.h>9#include <linux/pfn.h>
@@ -51,7 +50,7 @@ static void vmem_free_pages(unsigned long addr, int order, struct vmem_altmap *a · 40 unmodified lines
51 if (PageReserved(page)) {50 if (PageReserved(page)) {
52 /* allocated from memblock */51 /* allocated from memblock */
53 while (nr_pages--)52 while (nr_pages--)
54 free_bootmem_page(page++);53 free_reserved_page(page++);
55 } else {54 } else {
56 free_pages(addr, order);55 free_pages(addr, order);
57 }56 }
arch/sh/mm/cache-sh4.c+1−1
@@ -248,7 +248,7 @@ static void sh4_flush_cache_page(void *args) · 247 unmodified lines
248 */248 */
249 map_coherent = (current_cpu_data.dcache.n_aliases &&249 map_coherent = (current_cpu_data.dcache.n_aliases &&
250 test_bit(PG_dcache_clean, folio_flags(folio, 0)) &&250 test_bit(PG_dcache_clean, folio_flags(folio, 0)) &&
251 page_mapped(page));251 folio_mapped(folio));
252 if (map_coherent)252 if (map_coherent)
253 vaddr = kmap_coherent(page, address);253 vaddr = kmap_coherent(page, address);
254 else254 else
arch/sparc/mm/init_64.c+0−31
@@ -27,7 +27,6 @@ · 26 unmodified lines
27#include <linux/percpu.h>27#include <linux/percpu.h>
28#include <linux/mmzone.h>28#include <linux/mmzone.h>
29#include <linux/gfp.h>29#include <linux/gfp.h>
30#include <linux/bootmem_info.h>
31 30
32#include <asm/head.h>31#include <asm/head.h>
33#include <asm/page.h>32#include <asm/page.h>
@@ -2477,17 +2476,6 @@ int page_in_phys_avail(unsigned long paddr) · 2443 unmodified lines
2477 return 0;2476 return 0;
2478}2477}
2479 2478
2480static void __init register_page_bootmem_info(void)
2481{
2482#ifdef CONFIG_NUMA
2483 int i;
2484
2485 for_each_online_node(i)
2486 if (NODE_DATA(i)->node_spanned_pages)
2487 register_page_bootmem_info_node(NODE_DATA(i));
2488#endif
2489}
2490
2491void __init arch_setup_zero_pages(void)2479void __init arch_setup_zero_pages(void)
2492{2480{
2493 phys_addr_t zero_page_pa = kern_base +2481 phys_addr_t zero_page_pa = kern_base +
@@ -2498,14 +2486,6 @@ void __init arch_setup_zero_pages(void) · 4 unmodified lines
2498 2486
2499void __init mem_init(void)2487void __init mem_init(void)
2500{2488{
2501 /*
2502 * Must be done after boot memory is put on freelist, because here we
2503 * might set fields in deferred struct pages that have not yet been
2504 * initialized, and memblock_free_all() initializes all the reserved
2505 * deferred pages for us.
2506 */
2507 register_page_bootmem_info();
2508
2509 if (tlb_type == cheetah || tlb_type == cheetah_plus)2489 if (tlb_type == cheetah || tlb_type == cheetah_plus)
2510 cheetah_ecache_flush_init();2490 cheetah_ecache_flush_init();
2511}2491}
@@ -2579,17 +2559,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, · 67 unmodified lines
2579 pmd_val(*pmd) = pte_base | __pa(p);2559 pmd_val(*pmd) = pte_base | __pa(p);
2580}2560}
2581 2561
2582int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
2583 unsigned long addr, unsigned long next)
2584{
2585 int large = pmd_leaf(*pmdp);
2586
2587 if (large)
2588 vmemmap_verify((pte_t *)pmdp, node, addr, next);
2589
2590 return large;
2591}
2592
2593int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,2562int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
2594 int node, struct vmem_altmap *altmap)2563 int node, struct vmem_altmap *altmap)
2595{2564{
arch/x86/mm/init_64.c+3−2
@@ -1300,12 +1300,13 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end) · 1299 unmodified lines
1300 remove_pagetable(start, end, true, NULL);1300 remove_pagetable(start, end, true, NULL);
1301}1301}
1302 1302
1303void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)1303void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
1304 struct dev_pagemap *pgmap)
1304{1305{
1305 unsigned long start_pfn = start >> PAGE_SHIFT;1306 unsigned long start_pfn = start >> PAGE_SHIFT;
1306 unsigned long nr_pages = size >> PAGE_SHIFT;1307 unsigned long nr_pages = size >> PAGE_SHIFT;
1307 1308
1308 __remove_pages(start_pfn, nr_pages, altmap);1309 __remove_pages(start_pfn, nr_pages, altmap, pgmap);
1309 kernel_physical_mapping_remove(start, start + size);1310 kernel_physical_mapping_remove(start, start + size);
1310}1311}
1311#endif /* CONFIG_MEMORY_HOTPLUG */1312#endif /* CONFIG_MEMORY_HOTPLUG */
drivers/base/memory.c+16−25
@@ -649,7 +649,7 @@ int __weak arch_get_memory_phys_device(unsigned long start_pfn) · 648 unmodified lines
649 *649 *
650 * Called under device_hotplug_lock.650 * Called under device_hotplug_lock.
651 */651 */
652struct memory_block *find_memory_block_by_id(unsigned long block_id)652struct memory_block *memory_block_get(unsigned long block_id)
653{653{
654 struct memory_block *mem;654 struct memory_block *mem;
655 655
@@ -659,16 +659,6 @@ struct memory_block *find_memory_block_by_id(unsigned long block_id) · 3 unmodified lines
659 return mem;659 return mem;
660}660}
661 661
662/*
663 * Called under device_hotplug_lock.
664 */
665struct memory_block *find_memory_block(unsigned long section_nr)
666{
667 unsigned long block_id = memory_block_id(section_nr);
668
669 return find_memory_block_by_id(block_id);
670}
671
672static struct attribute *memory_memblk_attrs[] = {662static struct attribute *memory_memblk_attrs[] = {
673 &dev_attr_phys_index.attr,663 &dev_attr_phys_index.attr,
674 &dev_attr_state.attr,664 &dev_attr_state.attr,
@@ -701,7 +691,7 @@ static int __add_memory_block(struct memory_block *memory) · 26 unmodified lines
701 691
702 ret = device_register(&memory->dev);692 ret = device_register(&memory->dev);
703 if (ret) {693 if (ret) {
704 put_device(&memory->dev);694 memory_block_put(memory);
705 return ret;695 return ret;
706 }696 }
707 ret = xa_err(xa_store(&memory_blocks, memory->dev.id, memory,697 ret = xa_err(xa_store(&memory_blocks, memory->dev.id, memory,
@@ -795,9 +785,9 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 87 unmodified lines
795 struct memory_block *mem;785 struct memory_block *mem;
796 int ret = 0;786 int ret = 0;
797 787
798 mem = find_memory_block_by_id(block_id);788 mem = memory_block_get(block_id);
799 if (mem) {789 if (mem) {
800 put_device(&mem->dev);790 memory_block_put(mem);
801 return -EEXIST;791 return -EEXIST;
802 }792 }
803 mem = kzalloc_obj(*mem);793 mem = kzalloc_obj(*mem);
@@ -807,7 +797,6 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 3 unmodified lines
807 mem->start_section_nr = block_id * sections_per_block;797 mem->start_section_nr = block_id * sections_per_block;
808 mem->state = state;798 mem->state = state;
809 mem->nid = nid;799 mem->nid = nid;
810 mem->altmap = altmap;
811 INIT_LIST_HEAD(&mem->group_next);800 INIT_LIST_HEAD(&mem->group_next);
812 801
813#ifndef CONFIG_NUMA802#ifndef CONFIG_NUMA
@@ -825,6 +814,8 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 11 unmodified lines
825 if (ret)814 if (ret)
826 return ret;815 return ret;
827 816
817 mem->altmap = altmap;
818
828 if (group) {819 if (group) {
829 mem->group = group;820 mem->group = group;
830 list_add(&mem->group_next, &group->memory_blocks);821 list_add(&mem->group_next, &group->memory_blocks);
@@ -845,8 +836,8 @@ static void remove_memory_block(struct memory_block *memory) · 14 unmodified lines
845 memory->group = NULL;836 memory->group = NULL;
846 }837 }
847 838
848 /* drop the ref. we got via find_memory_block() */839 /* drop the ref. we got via memory_block_get() */
849 put_device(&memory->dev);840 memory_block_put(memory);
850 device_unregister(&memory->dev);841 device_unregister(&memory->dev);
851}842}
852 843
@@ -880,7 +871,7 @@ int create_memory_block_devices(unsigned long start, unsigned long size, · 27 unmodified lines
880 end_block_id = block_id;871 end_block_id = block_id;
881 for (block_id = start_block_id; block_id != end_block_id;872 for (block_id = start_block_id; block_id != end_block_id;
882 block_id++) {873 block_id++) {
883 mem = find_memory_block_by_id(block_id);874 mem = memory_block_get(block_id);
884 if (WARN_ON_ONCE(!mem))875 if (WARN_ON_ONCE(!mem))
885 continue;876 continue;
886 remove_memory_block(mem);877 remove_memory_block(mem);
@@ -908,7 +899,7 @@ void remove_memory_block_devices(unsigned long start, unsigned long size) · 21 unmodified lines
908 return;899 return;
909 900
910 for (block_id = start_block_id; block_id != end_block_id; block_id++) {901 for (block_id = start_block_id; block_id != end_block_id; block_id++) {
911 mem = find_memory_block_by_id(block_id);902 mem = memory_block_get(block_id);
912 if (WARN_ON_ONCE(!mem))903 if (WARN_ON_ONCE(!mem))
913 continue;904 continue;
914 num_poisoned_pages_sub(-1UL, memblk_nr_poison(mem));905 num_poisoned_pages_sub(-1UL, memblk_nr_poison(mem));
@@ -1015,12 +1006,12 @@ int walk_memory_blocks(unsigned long start, unsigned long size, · 100 unmodified lines
1015 return 0;1006 return 0;
1016 1007
1017 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {1008 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
1018 mem = find_memory_block_by_id(block_id);1009 mem = memory_block_get(block_id);
1019 if (!mem)1010 if (!mem)
1020 continue;1011 continue;
1021 1012
1022 ret = func(mem, arg);1013 ret = func(mem, arg);
1023 put_device(&mem->dev);1014 memory_block_put(mem);
1024 if (ret)1015 if (ret)
1025 break;1016 break;
1026 }1017 }
@@ -1228,22 +1219,22 @@ int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func, · 201 unmodified lines
1228void memblk_nr_poison_inc(unsigned long pfn)1219void memblk_nr_poison_inc(unsigned long pfn)
1229{1220{
1230 const unsigned long block_id = pfn_to_block_id(pfn);1221 const unsigned long block_id = pfn_to_block_id(pfn);
1231 struct memory_block *mem = find_memory_block_by_id(block_id);1222 struct memory_block *mem = memory_block_get(block_id);
1232 1223
1233 if (mem) {1224 if (mem) {
1234 atomic_long_inc(&mem->nr_hwpoison);1225 atomic_long_inc(&mem->nr_hwpoison);
1235 put_device(&mem->dev);1226 memory_block_put(mem);
1236 }1227 }
1237}1228}
1238 1229
1239void memblk_nr_poison_sub(unsigned long pfn, long i)1230void memblk_nr_poison_sub(unsigned long pfn, long i)
1240{1231{
1241 const unsigned long block_id = pfn_to_block_id(pfn);1232 const unsigned long block_id = pfn_to_block_id(pfn);
1242 struct memory_block *mem = find_memory_block_by_id(block_id);1233 struct memory_block *mem = memory_block_get(block_id);
1243 1234
1244 if (mem) {1235 if (mem) {
1245 atomic_long_sub(i, &mem->nr_hwpoison);1236 atomic_long_sub(i, &mem->nr_hwpoison);
1246 put_device(&mem->dev);1237 memory_block_put(mem);
1247 }1238 }
1248}1239}
1249 1240
drivers/base/node.c+4−2
@@ -523,6 +523,7 @@ static ssize_t node_read_meminfo(struct device *dev, · 522 unmodified lines
523#ifdef CONFIG_UNACCEPTED_MEMORY523#ifdef CONFIG_UNACCEPTED_MEMORY
524 "Node %d Unaccepted: %8lu kB\n"524 "Node %d Unaccepted: %8lu kB\n"
525#endif525#endif
526 "Node %d Balloon: %8lu kB\n"
526 "Node %d GPUActive: %8lu kB\n"527 "Node %d GPUActive: %8lu kB\n"
527 "Node %d GPUReclaim: %8lu kB\n"528 "Node %d GPUReclaim: %8lu kB\n"
528 ,529 ,
@@ -559,6 +560,7 @@ static ssize_t node_read_meminfo(struct device *dev, · 30 unmodified lines
559 nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED))560 nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED))
560#endif561#endif
561 ,562 ,
563 nid, K(node_page_state(pgdat, NR_BALLOON_PAGES)),
562 nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)),564 nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)),
563 nid, K(node_page_state(pgdat, NR_GPU_RECLAIM))565 nid, K(node_page_state(pgdat, NR_GPU_RECLAIM))
564 );566 );
@@ -847,13 +849,13 @@ static void register_memory_blocks_under_nodes(void) · 282 unmodified lines
847 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {849 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
848 struct memory_block *mem;850 struct memory_block *mem;
849 851
850 mem = find_memory_block_by_id(block_id);852 mem = memory_block_get(block_id);
851 if (!mem)853 if (!mem)
852 continue;854 continue;
853 855
854 memory_block_add_nid_early(mem, nid);856 memory_block_add_nid_early(mem, nid);
855 do_register_memory_block_under_node(nid, mem);857 do_register_memory_block_under_node(nid, mem);
856 put_device(&mem->dev);858 memory_block_put(mem);
857 }859 }
858 860
859 }861 }
drivers/block/zram/zram_drv.c+9−4
@@ -1127,6 +1127,9 @@ next: · 1126 unmodified lines
1127 if (req)1127 if (req)
1128 release_wb_req(req);1128 release_wb_req(req);
1129 1129
1130 if (blk_idx != INVALID_BDEV_BLOCK)
1131 zram_release_bdev_block(zram, blk_idx);
1132
1130 while (atomic_read(&wb_ctl->num_inflight) > 0) {1133 while (atomic_read(&wb_ctl->num_inflight) > 0) {
1131 wait_event(wb_ctl->done_wait, !list_empty(&wb_ctl->done_reqs));1134 wait_event(wb_ctl->done_wait, !list_empty(&wb_ctl->done_reqs));
1132 err = zram_complete_done_reqs(zram, wb_ctl);1135 err = zram_complete_done_reqs(zram, wb_ctl);
@@ -2131,6 +2134,8 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) · 998 unmodified lines
2131 zs_obj_read_end(zram->mem_pool, handle, size, src);2134 zs_obj_read_end(zram->mem_pool, handle, size, src);
2132 zcomp_stream_put(zstrm);2135 zcomp_stream_put(zstrm);
2133 2136
2137 memzero_page(page, size, PAGE_SIZE - size);
2138
2134 return 0;2139 return 0;
2135}2140}
2136#endif2141#endif
@@ -2329,7 +2334,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) · 192 unmodified lines
2329 * This is a partial IO. Read the full page before writing the changes.2334 * This is a partial IO. Read the full page before writing the changes.
2330 */2335 */
2331static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,2336static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,
2332 u32 index, int offset, struct bio *bio)2337 u32 index, int offset)
2333{2338{
2334 struct page *page = alloc_page(GFP_NOIO);2339 struct page *page = alloc_page(GFP_NOIO);
2335 int ret;2340 int ret;
@@ -2347,10 +2352,10 @@ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, · 11 unmodified lines
2347}2352}
2348 2353
2349static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,2354static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
2350 u32 index, int offset, struct bio *bio)2355 u32 index, int offset)
2351{2356{
2352 if (is_partial_io(bvec))2357 if (is_partial_io(bvec))
2353 return zram_bvec_write_partial(zram, bvec, index, offset, bio);2358 return zram_bvec_write_partial(zram, bvec, index, offset);
2354 return zram_write_page(zram, bvec->bv_page, index);2359 return zram_write_page(zram, bvec->bv_page, index);
2355}2360}
2356 2361
@@ -2747,7 +2752,7 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) · 390 unmodified lines
2747 2752
2748 bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset);2753 bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset);
2749 2754
2750 if (zram_bvec_write(zram, &bv, index, offset, bio) < 0) {2755 if (zram_bvec_write(zram, &bv, index, offset) < 0) {
2751 atomic64_inc(&zram->stats.failed_writes);2756 atomic64_inc(&zram->stats.failed_writes);
2752 bio->bi_status = BLK_STS_IOERR;2757 bio->bi_status = BLK_STS_IOERR;
2753 break;2758 break;
drivers/char/mem.c+6−19
@@ -322,11 +322,6 @@ static const struct vm_operations_struct mmap_mem_ops = { · 321 unmodified lines
322#endif322#endif
323};323};
324 324
325static int mmap_filter_error(int err)
326{
327 return -EAGAIN;
328}
329
330static int mmap_mem_prepare(struct vm_area_desc *desc)325static int mmap_mem_prepare(struct vm_area_desc *desc)
331{326{
332 struct file *file = desc->file;327 struct file *file = desc->file;
@@ -362,8 +357,7 @@ static int mmap_mem_prepare(struct vm_area_desc *desc) · 29 unmodified lines
362 357
363 /* Remap-pfn-range will mark the range with the I/O flag. */358 /* Remap-pfn-range will mark the range with the I/O flag. */
364 mmap_action_remap_full(desc, desc->pgoff);359 mmap_action_remap_full(desc, desc->pgoff);
365 /* We filter remap errors to -EAGAIN. */360 desc->action.error_override = -EAGAIN;
366 desc->action.error_hook = mmap_filter_error;
367 361
368 return 0;362 return 0;
369}363}
@@ -504,17 +498,6 @@ static ssize_t read_zero(struct file *file, char __user *buf, · 134 unmodified lines
504 return cleared;498 return cleared;
505}499}
506 500
507static int mmap_zero_private_success(const struct vm_area_struct *vma)
508{
509 /*
510 * This is a highly unique situation where we mark a MAP_PRIVATE mapping
511 * of /dev/zero anonymous, despite it not being.
512 */
513 vma_set_anonymous((struct vm_area_struct *)vma);
514
515 return 0;
516}
517
518static int mmap_zero_prepare(struct vm_area_desc *desc)501static int mmap_zero_prepare(struct vm_area_desc *desc)
519{502{
520#ifndef CONFIG_MMU503#ifndef CONFIG_MMU
@@ -523,7 +506,11 @@ static int mmap_zero_prepare(struct vm_area_desc *desc) · 2 unmodified lines
523 if (vma_desc_test(desc, VMA_SHARED_BIT))506 if (vma_desc_test(desc, VMA_SHARED_BIT))
524 return shmem_zero_setup_desc(desc);507 return shmem_zero_setup_desc(desc);
525 508
526 desc->action.success_hook = mmap_zero_private_success;509 /*
510 * This is a highly unique situation where we mark a MAP_PRIVATE mapping
511 * of /dev/zero anonymous, despite it not being.
512 */
513 vma_desc_set_anonymous(desc);
527 return 0;514 return 0;
528}515}
529 516
drivers/dax/kmem.c+6−0
@@ -227,6 +227,12 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 226 unmodified lines
227 if (rc)227 if (rc)
228 continue;228 continue;
229 229
230 /* range was never added during probe */
231 if (!data->res[i]) {
232 success++;
233 continue;
234 }
235
230 rc = remove_memory(range.start, range_len(&range));236 rc = remove_memory(range.start, range_len(&range));
231 if (rc == 0) {237 if (rc == 0) {
232 remove_resource(data->res[i]);238 remove_resource(data->res[i]);
drivers/gpu/drm/drm_managed.c+2−2
@@ -232,8 +232,8 @@ void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp) · 231 unmodified lines
232 232
233 dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));233 dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
234 if (!dr) {234 if (!dr) {
235 drm_dbg_drmres(dev, "failed to allocate %zu bytes, %u flags\n",235 drm_dbg_drmres(dev, "failed to allocate %zu bytes, %pGg\n",
236 size, gfp);236 size, &gfp);
237 return NULL;237 return NULL;
238 }238 }
239 dr->node.name = kstrdup_const("kmalloc", gfp);239 dr->node.name = kstrdup_const("kmalloc", gfp);
drivers/s390/char/sclp_mem.c+8−9
@@ -204,7 +204,7 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute · 203 unmodified lines
204 addr = sclp_mem->id * block_size;204 addr = sclp_mem->id * block_size;
205 /*205 /*
206 * Hold device_hotplug_lock when adding/removing memory blocks.206 * Hold device_hotplug_lock when adding/removing memory blocks.
207 * Additionally, also protect calls to find_memory_block() and207 * Additionally, also protect calls to memory_block_get() and
208 * sclp_attach_storage().208 * sclp_attach_storage().
209 */209 */
210 rc = lock_device_hotplug_sysfs();210 rc = lock_device_hotplug_sysfs();
@@ -231,20 +231,19 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute · 20 unmodified lines
231 sclp_mem_change_state(addr, block_size, 0);231 sclp_mem_change_state(addr, block_size, 0);
232 goto out_unlock;232 goto out_unlock;
233 }233 }
234 mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));234 mem = memory_block_get(phys_to_block_id(addr));
235 put_device(&mem->dev);235 memory_block_put(mem);
236 WRITE_ONCE(sclp_mem->config, 1);236 WRITE_ONCE(sclp_mem->config, 1);
237 } else {237 } else {
238 if (!sclp_mem->config)238 if (!sclp_mem->config)
239 goto out_unlock;239 goto out_unlock;
240 mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));240 mem = memory_block_get(phys_to_block_id(addr));
241 if (mem->state != MEM_OFFLINE) {241 if (mem->state != MEM_OFFLINE) {
242 put_device(&mem->dev);242 memory_block_put(mem);
243 rc = -EBUSY;243 rc = -EBUSY;
244 goto out_unlock;244 goto out_unlock;
245 }245 }
246 /* drop the ref just got via find_memory_block() */246 memory_block_put(mem);
247 put_device(&mem->dev);
248 sclp_mem_change_state(addr, block_size, 0);247 sclp_mem_change_state(addr, block_size, 0);
249 __remove_memory(addr, block_size);248 __remove_memory(addr, block_size);
250#ifdef CONFIG_KASAN249#ifdef CONFIG_KASAN
@@ -294,11 +293,11 @@ static ssize_t sclp_memmap_on_memory_store(struct kobject *kobj, struct kobj_att · 43 unmodified lines
294 return rc;293 return rc;
295 block_size = memory_block_size_bytes();294 block_size = memory_block_size_bytes();
296 sclp_mem = container_of(kobj, struct sclp_mem, kobj);295 sclp_mem = container_of(kobj, struct sclp_mem, kobj);
297 mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(sclp_mem->id * block_size)));296 mem = memory_block_get(phys_to_block_id(sclp_mem->id * block_size));
298 if (!mem) {297 if (!mem) {
299 WRITE_ONCE(sclp_mem->memmap_on_memory, value);298 WRITE_ONCE(sclp_mem->memmap_on_memory, value);
300 } else {299 } else {
301 put_device(&mem->dev);300 memory_block_put(mem);
302 rc = -EBUSY;301 rc = -EBUSY;
303 }302 }
304 unlock_device_hotplug();303 unlock_device_hotplug();
fs/Makefile+0−1
@@ -27,7 +27,6 @@ obj-y += anon_inodes.o · 26 unmodified lines
27obj-$(CONFIG_SIGNALFD) += signalfd.o27obj-$(CONFIG_SIGNALFD) += signalfd.o
28obj-$(CONFIG_TIMERFD) += timerfd.o28obj-$(CONFIG_TIMERFD) += timerfd.o
29obj-$(CONFIG_EVENTFD) += eventfd.o29obj-$(CONFIG_EVENTFD) += eventfd.o
30obj-$(CONFIG_USERFAULTFD) += userfaultfd.o
31obj-$(CONFIG_AIO) += aio.o30obj-$(CONFIG_AIO) += aio.o
32obj-$(CONFIG_FS_DAX) += dax.o31obj-$(CONFIG_FS_DAX) += dax.o
33obj-$(CONFIG_FS_ENCRYPTION) += crypto/32obj-$(CONFIG_FS_ENCRYPTION) += crypto/
fs/proc/task_mmu.c+223−46
@@ -132,6 +132,22 @@ static void release_task_mempolicy(struct proc_maps_private *priv) · 131 unmodified lines
132 132
133#ifdef CONFIG_PER_VMA_LOCK133#ifdef CONFIG_PER_VMA_LOCK
134 134
135static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
136{
137 int ret = mmap_read_lock_killable(lock_ctx->mm);
138
139 if (!ret)
140 lock_ctx->mmap_locked = true;
141
142 return ret;
143}
144
145static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
146{
147 mmap_read_unlock(lock_ctx->mm);
148 lock_ctx->mmap_locked = false;
149}
150
135static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx)151static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx)
136{152{
137 lock_ctx->locked_vma = NULL;153 lock_ctx->locked_vma = NULL;
@@ -146,25 +162,11 @@ static void unlock_ctx_vma(struct proc_maps_locking_ctx *lock_ctx) · 8 unmodified lines
146 }162 }
147}163}
148 164
149static const struct seq_operations proc_pid_maps_op;
150
151static inline bool lock_vma_range(struct seq_file *m,165static inline bool lock_vma_range(struct seq_file *m,
152 struct proc_maps_locking_ctx *lock_ctx)166 struct proc_maps_locking_ctx *lock_ctx)
153{167{
154 /*168 rcu_read_lock();
155 * smaps and numa_maps perform page table walk, therefore require169 reset_lock_ctx(lock_ctx);
156 * mmap_lock but maps can be read with locking just the vma and
157 * walking the vma tree under rcu read protection.
158 */
159 if (m->op != &proc_pid_maps_op) {
160 if (mmap_read_lock_killable(lock_ctx->mm))
161 return false;
162
163 lock_ctx->mmap_locked = true;
164 } else {
165 rcu_read_lock();
166 reset_lock_ctx(lock_ctx);
167 }
168 170
169 return true;171 return true;
170}172}
@@ -172,7 +174,7 @@ static inline bool lock_vma_range(struct seq_file *m, · 1 unmodified line
172static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)174static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
173{175{
174 if (lock_ctx->mmap_locked) {176 if (lock_ctx->mmap_locked) {
175 mmap_read_unlock(lock_ctx->mm);177 unlock_ctx_mm(lock_ctx);
176 } else {178 } else {
177 unlock_ctx_vma(lock_ctx);179 unlock_ctx_vma(lock_ctx);
178 rcu_read_unlock();180 rcu_read_unlock();
@@ -213,17 +215,45 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, · 34 unmodified lines
213 return true;215 return true;
214}216}
215 217
218static inline void drop_rcu(struct proc_maps_private *priv)
219{
220 if (priv->lock_ctx.mmap_locked)
221 return;
222
223 rcu_read_unlock();
224}
225
226static inline void reacquire_rcu(struct proc_maps_private *priv)
227{
228 if (priv->lock_ctx.mmap_locked)
229 return;
230
231 rcu_read_lock();
232 /* Reinitialize the iterator. */
233 vma_iter_set(&priv->iter, priv->lock_ctx.locked_vma->vm_end);
234}
235
216#else /* CONFIG_PER_VMA_LOCK */236#else /* CONFIG_PER_VMA_LOCK */
217 237
238static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
239{
240 return mmap_read_lock_killable(lock_ctx->mm);
241}
242
243static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
244{
245 mmap_read_unlock(lock_ctx->mm);
246}
247
218static inline bool lock_vma_range(struct seq_file *m,248static inline bool lock_vma_range(struct seq_file *m,
219 struct proc_maps_locking_ctx *lock_ctx)249 struct proc_maps_locking_ctx *lock_ctx)
220{250{
221 return mmap_read_lock_killable(lock_ctx->mm) == 0;251 return lock_ctx_mm(lock_ctx) == 0;
222}252}
223 253
224static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)254static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
225{255{
226 mmap_read_unlock(lock_ctx->mm);256 unlock_ctx_mm(lock_ctx);
227}257}
228 258
229static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv,259static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv,
@@ -238,6 +268,9 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, · 8 unmodified lines
238 return false;268 return false;
239}269}
240 270
271static inline void drop_rcu(struct proc_maps_private *priv) {}
272static inline void reacquire_rcu(struct proc_maps_private *priv) {}
273
241#endif /* CONFIG_PER_VMA_LOCK */274#endif /* CONFIG_PER_VMA_LOCK */
242 275
243static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos)276static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos)
@@ -538,12 +571,10 @@ static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx) · 294 unmodified lines
538 571
539static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx)572static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx)
540{573{
541 if (lock_ctx->mmap_locked) {574 if (lock_ctx->mmap_locked)
542 mmap_read_unlock(lock_ctx->mm);575 unlock_ctx_mm(lock_ctx);
543 lock_ctx->mmap_locked = false;576 else
544 } else {
545 unlock_ctx_vma(lock_ctx);577 unlock_ctx_vma(lock_ctx);
546 }
547}578}
548 579
549static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_locking_ctx *lock_ctx,580static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_locking_ctx *lock_ctx,
@@ -1280,21 +1311,75 @@ static const struct mm_walk_ops smaps_shmem_walk_ops = { · 730 unmodified lines
1280 .walk_lock = PGWALK_RDLOCK,1311 .walk_lock = PGWALK_RDLOCK,
1281};1312};
1282 1313
1314#ifdef CONFIG_PER_VMA_LOCK
1315
1316static const struct mm_walk_ops smaps_walk_vma_lock_ops = {
1317 .pmd_entry = smaps_pte_range,
1318 .hugetlb_entry = smaps_hugetlb_range,
1319 .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
1320};
1321
1322static const struct mm_walk_ops smaps_shmem_walk_vma_lock_ops = {
1323 .pmd_entry = smaps_pte_range,
1324 .hugetlb_entry = smaps_hugetlb_range,
1325 .pte_hole = smaps_pte_hole,
1326 .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
1327};
1328
1329static inline const struct mm_walk_ops *
1330get_smaps_walk_ops(struct proc_maps_private *priv)
1331{
1332 if (priv->lock_ctx.mmap_locked)
1333 return &smaps_walk_ops;
1334 return &smaps_walk_vma_lock_ops;
1335}
1336
1337static inline const struct mm_walk_ops *
1338get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
1339{
1340 if (priv->lock_ctx.mmap_locked)
1341 return &smaps_shmem_walk_ops;
1342 return &smaps_shmem_walk_vma_lock_ops;
1343}
1344
1345#else /* CONFIG_PER_VMA_LOCK */
1346
1347static inline const struct mm_walk_ops *
1348get_smaps_walk_ops(struct proc_maps_private *priv)
1349{
1350 return &smaps_walk_ops;
1351}
1352
1353static inline const struct mm_walk_ops *
1354get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
1355{
1356 return &smaps_shmem_walk_ops;
1357}
1358
1359#endif /* CONFIG_PER_VMA_LOCK */
1360
1283/*1361/*
1284 * Gather mem stats from @vma with the indicated beginning1362 * Gather mem stats from @vma with the indicated beginning
1285 * address @start, and keep them in @mss.1363 * address @start, and keep them in @mss.
1286 *1364 *
1287 * Use vm_start of @vma as the beginning address if @start is 0.1365 * Use vm_start of @vma as the beginning address if @start is 0.
1288 */1366 */
1289static void smap_gather_stats(struct vm_area_struct *vma,1367static void smap_gather_stats(struct proc_maps_private *priv,
1290 struct mem_size_stats *mss, unsigned long start)1368 struct vm_area_struct *vma,
1369 struct mem_size_stats *mss, unsigned long start)
1291{1370{
1292 const struct mm_walk_ops *ops = &smaps_walk_ops;1371 const struct mm_walk_ops *ops = get_smaps_walk_ops(priv);
1293 1372
1294 /* Invalid start */1373 /* Invalid start */
1295 if (start >= vma->vm_end)1374 if (start >= vma->vm_end)
1296 return;1375 return;
1297 1376
1377 if (vma == get_gate_vma(priv->lock_ctx.mm))
1378 return;
1379
1380 /* Might sleep. Drop RCU read lock but keep the VMA locked. */
1381 drop_rcu(priv);
1382
1298 if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {1383 if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {
1299 /*1384 /*
1300 * For shared or readonly shmem mappings we know that all1385 * For shared or readonly shmem mappings we know that all
@@ -1312,15 +1397,16 @@ static void smap_gather_stats(struct vm_area_struct *vma, · 11 unmodified lines
1312 !(vma->vm_flags & VM_WRITE))) {1397 !(vma->vm_flags & VM_WRITE))) {
1313 mss->swap += shmem_swapped;1398 mss->swap += shmem_swapped;
1314 } else {1399 } else {
1315 ops = &smaps_shmem_walk_ops;1400 ops = get_smaps_shmem_walk_ops(priv);
1316 }1401 }
1317 }1402 }
1318 1403
1319 /* mmap_lock is held in m_start */
1320 if (!start)1404 if (!start)
1321 walk_page_vma(vma, ops, mss);1405 walk_page_vma(vma, ops, mss);
1322 else1406 else
1323 walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss);1407 walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss);
1408
1409 reacquire_rcu(priv);
1324}1410}
1325 1411
1326#define SEQ_PUT_DEC(str, val) \1412#define SEQ_PUT_DEC(str, val) \
@@ -1369,10 +1455,11 @@ static void __show_smap(struct seq_file *m, const struct mem_size_stats *mss, · 42 unmodified lines
1369 1455
1370static int show_smap(struct seq_file *m, void *v)1456static int show_smap(struct seq_file *m, void *v)
1371{1457{
1458 struct proc_maps_private *priv = m->private;
1372 struct vm_area_struct *vma = v;1459 struct vm_area_struct *vma = v;
1373 struct mem_size_stats mss = {};1460 struct mem_size_stats mss = {};
1374 1461
1375 smap_gather_stats(vma, &mss, 0);1462 smap_gather_stats(priv, vma, &mss, 0);
1376 1463
1377 show_map_vma(m, vma);1464 show_map_vma(m, vma);
1378 1465
@@ -1413,7 +1500,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 34 unmodified lines
1413 goto out_put_task;1500 goto out_put_task;
1414 }1501 }
1415 1502
1416 ret = mmap_read_lock_killable(mm);1503 ret = lock_ctx_mm(&priv->lock_ctx);
1417 if (ret)1504 if (ret)
1418 goto out_put_mm;1505 goto out_put_mm;
1419 1506
@@ -1425,7 +1512,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 5 unmodified lines
1425 1512
1426 vma_start = vma->vm_start;1513 vma_start = vma->vm_start;
1427 do {1514 do {
1428 smap_gather_stats(vma, &mss, 0);1515 smap_gather_stats(priv, vma, &mss, 0);
1429 last_vma_end = vma->vm_end;1516 last_vma_end = vma->vm_end;
1430 1517
1431 /*1518 /*
@@ -1434,8 +1521,8 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 2 unmodified lines
1434 */1521 */
1435 if (mmap_lock_is_contended(mm)) {1522 if (mmap_lock_is_contended(mm)) {
1436 vma_iter_invalidate(&vmi);1523 vma_iter_invalidate(&vmi);
1437 mmap_read_unlock(mm);1524 unlock_ctx_mm(&priv->lock_ctx);
1438 ret = mmap_read_lock_killable(mm);1525 ret = lock_ctx_mm(&priv->lock_ctx);
1439 if (ret) {1526 if (ret) {
1440 release_task_mempolicy(priv);1527 release_task_mempolicy(priv);
1441 goto out_put_mm;1528 goto out_put_mm;
@@ -1484,14 +1571,14 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 42 unmodified lines
1484 1571
1485 /* Case 1 and 2 above */1572 /* Case 1 and 2 above */
1486 if (vma->vm_start >= last_vma_end) {1573 if (vma->vm_start >= last_vma_end) {
1487 smap_gather_stats(vma, &mss, 0);1574 smap_gather_stats(priv, vma, &mss, 0);
1488 last_vma_end = vma->vm_end;1575 last_vma_end = vma->vm_end;
1489 continue;1576 continue;
1490 }1577 }
1491 1578
1492 /* Case 4 above */1579 /* Case 4 above */
1493 if (vma->vm_end > last_vma_end) {1580 if (vma->vm_end > last_vma_end) {
1494 smap_gather_stats(vma, &mss, last_vma_end);1581 smap_gather_stats(priv, vma, &mss, last_vma_end);
1495 last_vma_end = vma->vm_end;1582 last_vma_end = vma->vm_end;
1496 }1583 }
1497 }1584 }
@@ -1505,7 +1592,7 @@ empty_set: · 7 unmodified lines
1505 __show_smap(m, &mss, true);1592 __show_smap(m, &mss, true);
1506 1593
1507 release_task_mempolicy(priv);1594 release_task_mempolicy(priv);
1508 mmap_read_unlock(mm);1595 unlock_ctx_mm(&priv->lock_ctx);
1509 1596
1510out_put_mm:1597out_put_mm:
1511 mmput(mm);1598 mmput(mm);
@@ -2042,7 +2129,6 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr, · 530 unmodified lines
2042 flags |= PM_SOFT_DIRTY;2129 flags |= PM_SOFT_DIRTY;
2043 if (pmd_swp_uffd_wp(pmd))2130 if (pmd_swp_uffd_wp(pmd))
2044 flags |= PM_UFFD_WP;2131 flags |= PM_UFFD_WP;
2045 VM_WARN_ON_ONCE(!pmd_is_migration_entry(pmd));
2046 page = softleaf_to_page(entry);2132 page = softleaf_to_page(entry);
2047 }2133 }
2048 2134
@@ -2523,12 +2609,16 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma, · 474 unmodified lines
2523 if (softleaf_is_hwpoison(entry) || softleaf_is_marker(entry))2609 if (softleaf_is_hwpoison(entry) || softleaf_is_marker(entry))
2524 return;2610 return;
2525 2611
2526 if (softleaf_is_migration(entry))2612 if (softleaf_is_migration(entry)) {
2527 set_huge_pte_at(vma->vm_mm, addr, ptep,2613 set_huge_pte_at(vma->vm_mm, addr, ptep,
2528 pte_swp_mkuffd_wp(ptent), psize);2614 pte_swp_mkuffd_wp(ptent), psize);
2529 else2615 } else {
2530 huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,2616 pte_t old_pte, new_pte;
2531 huge_pte_mkuffd_wp(ptent));2617
2618 old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
2619 new_pte = huge_pte_mkuffd_wp(old_pte);
2620 huge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);
2621 }
2532}2622}
2533#endif /* CONFIG_HUGETLB_PAGE */2623#endif /* CONFIG_HUGETLB_PAGE */
2534 2624
@@ -2869,7 +2959,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 334 unmodified lines
2869 if (~categories & PAGE_IS_WRITTEN)2959 if (~categories & PAGE_IS_WRITTEN)
2870 goto out_unlock;2960 goto out_unlock;
2871 2961
2872 if (end != start + HPAGE_SIZE) {2962 if (end != start + huge_page_size(hstate_vma(vma))) {
2873 /* Partial HugeTLB page WP isn't possible. */2963 /* Partial HugeTLB page WP isn't possible. */
2874 pagemap_scan_backout_range(p, start, end);2964 pagemap_scan_backout_range(p, start, end);
2875 p->arg.walk_end = start;2965 p->arg.walk_end = start;
@@ -2886,8 +2976,62 @@ out_unlock: · 10 unmodified lines
2886 2976
2887 return ret;2977 return ret;
2888}2978}
2979
2980/*
2981 * Write-protect the unpopulated hugetlb entries covering [addr, end) by
2982 * installing uffd-wp markers inline, exactly as pagemap_scan_hugetlb_entry()
2983 * does for populated entries.
2984 *
2985 * walk_hugetlb_range() currently calls ->pte_hole() once per huge page, so the
2986 * loop normally runs a single iteration; it is written to cover the full range
2987 * in case the walker ever coalesces adjacent holes.
2988 *
2989 * The obvious route -- uffd_wp_range() -> hugetlb_change_protection() --
2990 * cannot be used here: it takes hugetlb_vma_lock_write(), but the page-table
2991 * walker (walk_hugetlb_range()) already holds hugetlb_vma_lock_read() on the
2992 * same VMA, so the scanning thread would deadlock against itself. PMD sharing
2993 * is disabled on uffd-wp VMAs (hugetlb_unshare_all_pmds() at registration), so
2994 * the vma lock guards nothing that matters for these entries anyway.
2995 */
2996static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
2997 unsigned long addr, unsigned long end)
2998{
2999 struct hstate *h = hstate_vma(vma);
3000 unsigned long psize = huge_page_size(h);
3001 struct mm_struct *mm = vma->vm_mm;
3002 spinlock_t *ptl;
3003 pte_t *ptep;
3004 pte_t pte;
3005
3006 for (addr = ALIGN_DOWN(addr, psize); addr < end; addr += psize) {
3007 ptep = huge_pte_alloc(mm, vma, addr, psize);
3008 if (!ptep)
3009 return -ENOMEM;
3010
3011 i_mmap_lock_write(vma->vm_file->f_mapping);
3012 ptl = huge_pte_lock(h, mm, ptep);
3013 pte = huge_ptep_get(mm, addr, ptep);
3014 make_uffd_wp_huge_pte(vma, addr, ptep, pte);
3015 /*
3016 * A none entry has no cached translation, so installing the
3017 * marker needs no TLB flush. Flush only if a fault populated
3018 * the entry between huge_pte_alloc() and the page table lock.
3019 */
3020 if (!huge_pte_none(pte))
3021 flush_hugetlb_tlb_range(vma, addr, addr + psize);
3022 spin_unlock(ptl);
3023 i_mmap_unlock_write(vma->vm_file->f_mapping);
3024 }
3025
3026 return 0;
3027}
2889#else3028#else
2890#define pagemap_scan_hugetlb_entry NULL3029#define pagemap_scan_hugetlb_entry NULL
3030static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
3031 unsigned long addr, unsigned long end)
3032{
3033 return 0;
3034}
2891#endif3035#endif
2892 3036
2893static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,3037static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
@@ -2907,7 +3051,10 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, · 13 unmodified lines
2907 if (~p->arg.flags & PM_SCAN_WP_MATCHING)3051 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2908 return ret;3052 return ret;
2909 3053
2910 err = uffd_wp_range(vma, addr, end - addr, true);3054 if (is_vm_hugetlb_page(vma))
3055 err = pagemap_scan_hugetlb_hole_wp(vma, addr, end);
3056 else
3057 err = uffd_wp_range(vma, addr, end - addr, true);
2911 if (err < 0)3058 if (err < 0)
2912 ret = err;3059 ret = err;
2913 3060
@@ -3291,6 +3438,31 @@ static const struct mm_walk_ops show_numa_ops = { · 377 unmodified lines
3291 .walk_lock = PGWALK_RDLOCK,3438 .walk_lock = PGWALK_RDLOCK,
3292};3439};
3293 3440
3441#ifdef CONFIG_PER_VMA_LOCK
3442static const struct mm_walk_ops show_numa_vma_lock_ops = {
3443 .hugetlb_entry = gather_hugetlb_stats,
3444 .pmd_entry = gather_pte_stats,
3445 .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
3446};
3447
3448static inline const struct mm_walk_ops *
3449get_show_numa_ops(struct proc_maps_private *priv)
3450{
3451 if (priv->lock_ctx.mmap_locked)
3452 return &show_numa_ops;
3453 return &show_numa_vma_lock_ops;
3454}
3455
3456#else /* CONFIG_PER_VMA_LOCK */
3457
3458static inline const struct mm_walk_ops *
3459get_show_numa_ops(struct proc_maps_private *priv)
3460{
3461 return &show_numa_ops;
3462}
3463
3464#endif /* CONFIG_PER_VMA_LOCK */
3465
3294/*3466/*
3295 * Display pages allocated per node and memory policy via /proc.3467 * Display pages allocated per node and memory policy via /proc.
3296 */3468 */
@@ -3335,8 +3507,13 @@ static int show_numa_map(struct seq_file *m, void *v) · 38 unmodified lines
3335 if (is_vm_hugetlb_page(vma))3507 if (is_vm_hugetlb_page(vma))
3336 seq_puts(m, " huge");3508 seq_puts(m, " huge");
3337 3509
3338 /* mmap_lock is held by m_start */3510 /* Skip walking pages if gate VMA */
3339 walk_page_vma(vma, &show_numa_ops, md);3511 if (vma != get_gate_vma(proc_priv->lock_ctx.mm)) {
3512 /* Might sleep. Drop RCU read lock but keep the VMA locked. */
3513 drop_rcu(proc_priv);
3514 walk_page_vma(vma, get_show_numa_ops(proc_priv), md);
3515 reacquire_rcu(proc_priv);
3516 }
3340 3517
3341 if (!md->pages)3518 if (!md->pages)
3342 goto out;3519 goto out;
fs/userfaultfd.c+0−2231
@@ -1,2231 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0-only
2/*
3 * fs/userfaultfd.c
4 *
5 * Copyright (C) 2007 Davide Libenzi <davidel@xmailserver.org>
6 * Copyright (C) 2008-2009 Red Hat, Inc.
7 * Copyright (C) 2015 Red Hat, Inc.
8 *
9 * Some part derived from fs/eventfd.c (anon inode setup) and
10 * mm/ksm.c (mm hashing).
11 */
12
13#include <linux/list.h>
14#include <linux/hashtable.h>
15#include <linux/sched/signal.h>
16#include <linux/sched/mm.h>
17#include <linux/mm.h>
18#include <linux/mm_inline.h>
19#include <linux/mmu_notifier.h>
20#include <linux/poll.h>
21#include <linux/slab.h>
22#include <linux/seq_file.h>
23#include <linux/file.h>
24#include <linux/bug.h>
25#include <linux/anon_inodes.h>
26#include <linux/syscalls.h>
27#include <linux/userfaultfd_k.h>
28#include <linux/mempolicy.h>
29#include <linux/ioctl.h>
30#include <linux/security.h>
31#include <linux/hugetlb.h>
32#include <linux/leafops.h>
33#include <linux/miscdevice.h>
34#include <linux/uio.h>
35
36static int sysctl_unprivileged_userfaultfd __read_mostly;
37
38#ifdef CONFIG_SYSCTL
39static const struct ctl_table vm_userfaultfd_table[] = {
40 {
41 .procname = "unprivileged_userfaultfd",
42 .data = &sysctl_unprivileged_userfaultfd,
43 .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
44 .mode = 0644,
45 .proc_handler = proc_dointvec_minmax,
46 .extra1 = SYSCTL_ZERO,
47 .extra2 = SYSCTL_ONE,
48 },
49};
50#endif
51
52static struct kmem_cache *userfaultfd_ctx_cachep __ro_after_init;
53
54struct userfaultfd_fork_ctx {
55 struct userfaultfd_ctx *orig;
56 struct userfaultfd_ctx *new;
57 struct list_head list;
58};
59
60struct userfaultfd_unmap_ctx {
61 struct userfaultfd_ctx *ctx;
62 unsigned long start;
63 unsigned long end;
64 struct list_head list;
65};
66
67struct userfaultfd_wait_queue {
68 struct uffd_msg msg;
69 wait_queue_entry_t wq;
70 struct userfaultfd_ctx *ctx;
71 bool waken;
72};
73
74struct userfaultfd_wake_range {
75 unsigned long start;
76 unsigned long len;
77};
78
79/* internal indication that UFFD_API ioctl was successfully executed */
80#define UFFD_FEATURE_INITIALIZED (1u << 31)
81
82static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
83{
84 return ctx->features & UFFD_FEATURE_INITIALIZED;
85}
86
87static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
88{
89 return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
90}
91
92/*
93 * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
94 * meaningful when userfaultfd_wp()==true on the vma and when it's
95 * anonymous.
96 */
97bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
98{
99 struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
100
101 if (!ctx)
102 return false;
103
104 return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
105}
106
107static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
108 int wake_flags, void *key)
109{
110 struct userfaultfd_wake_range *range = key;
111 int ret;
112 struct userfaultfd_wait_queue *uwq;
113 unsigned long start, len;
114
115 uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
116 ret = 0;
117 /* len == 0 means wake all */
118 start = range->start;
119 len = range->len;
120 if (len && (start > uwq->msg.arg.pagefault.address ||
121 start + len <= uwq->msg.arg.pagefault.address))
122 goto out;
123 WRITE_ONCE(uwq->waken, true);
124 /*
125 * The Program-Order guarantees provided by the scheduler
126 * ensure uwq->waken is visible before the task is woken.
127 */
128 ret = wake_up_state(wq->private, mode);
129 if (ret) {
130 /*
131 * Wake only once, autoremove behavior.
132 *
133 * After the effect of list_del_init is visible to the other
134 * CPUs, the waitqueue may disappear from under us, see the
135 * !list_empty_careful() in handle_userfault().
136 *
137 * try_to_wake_up() has an implicit smp_mb(), and the
138 * wq->private is read before calling the extern function
139 * "wake_up_state" (which in turns calls try_to_wake_up).
140 */
141 list_del_init(&wq->entry);
142 }
143out:
144 return ret;
145}
146
147/**
148 * userfaultfd_ctx_get - Acquires a reference to the internal userfaultfd
149 * context.
150 * @ctx: [in] Pointer to the userfaultfd context.
151 */
152static void userfaultfd_ctx_get(struct userfaultfd_ctx *ctx)
153{
154 refcount_inc(&ctx->refcount);
155}
156
157/**
158 * userfaultfd_ctx_put - Releases a reference to the internal userfaultfd
159 * context.
160 * @ctx: [in] Pointer to userfaultfd context.
161 *
162 * The userfaultfd context reference must have been previously acquired either
163 * with userfaultfd_ctx_get() or userfaultfd_ctx_fdget().
164 */
165static void userfaultfd_ctx_put(struct userfaultfd_ctx *ctx)
166{
167 if (refcount_dec_and_test(&ctx->refcount)) {
168 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_pending_wqh.lock));
169 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_pending_wqh));
170 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_wqh.lock));
171 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_wqh));
172 VM_WARN_ON_ONCE(spin_is_locked(&ctx->event_wqh.lock));
173 VM_WARN_ON_ONCE(waitqueue_active(&ctx->event_wqh));
174 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fd_wqh.lock));
175 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fd_wqh));
176 mmdrop(ctx->mm);
177 kmem_cache_free(userfaultfd_ctx_cachep, ctx);
178 }
179}
180
181static inline void msg_init(struct uffd_msg *msg)
182{
183 BUILD_BUG_ON(sizeof(struct uffd_msg) != 32);
184 /*
185 * Must use memset to zero out the paddings or kernel data is
186 * leaked to userland.
187 */
188 memset(msg, 0, sizeof(struct uffd_msg));
189}
190
191static inline struct uffd_msg userfault_msg(unsigned long address,
192 unsigned long real_address,
193 unsigned int flags,
194 unsigned long reason,
195 unsigned int features)
196{
197 struct uffd_msg msg;
198
199 msg_init(&msg);
200 msg.event = UFFD_EVENT_PAGEFAULT;
201
202 msg.arg.pagefault.address = (features & UFFD_FEATURE_EXACT_ADDRESS) ?
203 real_address : address;
204
205 /*
206 * These flags indicate why the userfault occurred:
207 * - UFFD_PAGEFAULT_FLAG_WP indicates a write protect fault.
208 * - UFFD_PAGEFAULT_FLAG_MINOR indicates a minor fault.
209 * - Neither of these flags being set indicates a MISSING fault.
210 *
211 * Separately, UFFD_PAGEFAULT_FLAG_WRITE indicates it was a write
212 * fault. Otherwise, it was a read fault.
213 */
214 if (flags & FAULT_FLAG_WRITE)
215 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;
216 if (reason & VM_UFFD_WP)
217 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;
218 if (reason & VM_UFFD_MINOR)
219 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;
220 if (features & UFFD_FEATURE_THREAD_ID)
221 msg.arg.pagefault.feat.ptid = task_pid_vnr(current);
222 return msg;
223}
224
225#ifdef CONFIG_HUGETLB_PAGE
226/*
227 * Same functionality as userfaultfd_must_wait below with modifications for
228 * hugepmd ranges.
229 */
230static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
231 struct vm_fault *vmf,
232 unsigned long reason)
233{
234 struct vm_area_struct *vma = vmf->vma;
235 pte_t *ptep, pte;
236
237 assert_fault_locked(vmf);
238
239 ptep = hugetlb_walk(vma, vmf->address, vma_mmu_pagesize(vma));
240 if (!ptep)
241 return true;
242
243 pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
244
245 /*
246 * Lockless access: we're in a wait_event so it's ok if it
247 * changes under us.
248 */
249
250 /* Entry is still missing, wait for userspace to resolve the fault. */
251 if (huge_pte_none(pte))
252 return true;
253 /* UFFD PTE markers require userspace to resolve the fault. */
254 if (pte_is_uffd_marker(pte))
255 return true;
256 /*
257 * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
258 * resolve the fault.
259 */
260 if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
261 return true;
262
263 return false;
264}
265#else
266static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
267 struct vm_fault *vmf,
268 unsigned long reason)
269{
270 /* Should never get here. */
271 VM_WARN_ON_ONCE(1);
272 return false;
273}
274#endif /* CONFIG_HUGETLB_PAGE */
275
276/*
277 * Verify the pagetables are still not ok after having registered into
278 * the fault_pending_wqh to avoid userland having to UFFDIO_WAKE any
279 * userfault that has already been resolved, if userfaultfd_read_iter and
280 * UFFDIO_COPY|ZEROPAGE are being run simultaneously on two different
281 * threads.
282 */
283static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
284 struct vm_fault *vmf,
285 unsigned long reason)
286{
287 struct mm_struct *mm = ctx->mm;
288 unsigned long address = vmf->address;
289 pgd_t *pgd;
290 p4d_t *p4d;
291 pud_t *pud;
292 pmd_t *pmd, _pmd;
293 pte_t *pte;
294 pte_t ptent;
295 bool ret;
296
297 assert_fault_locked(vmf);
298
299 pgd = pgd_offset(mm, address);
300 if (!pgd_present(*pgd))
301 return true;
302 p4d = p4d_offset(pgd, address);
303 if (!p4d_present(*p4d))
304 return true;
305 pud = pud_offset(p4d, address);
306 if (!pud_present(*pud))
307 return true;
308 pmd = pmd_offset(pud, address);
309again:
310 _pmd = pmdp_get_lockless(pmd);
311 if (pmd_none(_pmd))
312 return true;
313
314 /*
315 * A race could arise which would result in a softleaf entry such as
316 * migration entry unexpectedly being present in the PMD, so explicitly
317 * check for this and bail out if so.
318 */
319 if (!pmd_present(_pmd))
320 return false;
321
322 if (pmd_trans_huge(_pmd))
323 return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
324
325 pte = pte_offset_map(pmd, address);
326 if (!pte)
327 goto again;
328
329 /*
330 * Lockless access: we're in a wait_event so it's ok if it
331 * changes under us.
332 */
333 ptent = ptep_get(pte);
334
335 ret = true;
336 /* Entry is still missing, wait for userspace to resolve the fault. */
337 if (pte_none(ptent))
338 goto out;
339 /* UFFD PTE markers require userspace to resolve the fault. */
340 if (pte_is_uffd_marker(ptent))
341 goto out;
342 /*
343 * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
344 * resolve the fault.
345 */
346 if (!pte_write(ptent) && (reason & VM_UFFD_WP))
347 goto out;
348
349 ret = false;
350out:
351 pte_unmap(pte);
352 return ret;
353}
354
355static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags)
356{
357 if (flags & FAULT_FLAG_INTERRUPTIBLE)
358 return TASK_INTERRUPTIBLE;
359
360 if (flags & FAULT_FLAG_KILLABLE)
361 return TASK_KILLABLE;
362
363 return TASK_UNINTERRUPTIBLE;
364}
365
366/*
367 * The locking rules involved in returning VM_FAULT_RETRY depending on
368 * FAULT_FLAG_ALLOW_RETRY, FAULT_FLAG_RETRY_NOWAIT and
369 * FAULT_FLAG_KILLABLE are not straightforward. The "Caution"
370 * recommendation in __lock_page_or_retry is not an understatement.
371 *
372 * If FAULT_FLAG_ALLOW_RETRY is set, the mmap_lock must be released
373 * before returning VM_FAULT_RETRY only if FAULT_FLAG_RETRY_NOWAIT is
374 * not set.
375 *
376 * If FAULT_FLAG_ALLOW_RETRY is set but FAULT_FLAG_KILLABLE is not
377 * set, VM_FAULT_RETRY can still be returned if and only if there are
378 * fatal_signal_pending()s, and the mmap_lock must be released before
379 * returning it.
380 */
381vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
382{
383 struct vm_area_struct *vma = vmf->vma;
384 struct mm_struct *mm = vma->vm_mm;
385 struct userfaultfd_ctx *ctx;
386 struct userfaultfd_wait_queue uwq;
387 vm_fault_t ret = VM_FAULT_SIGBUS;
388 bool must_wait;
389 unsigned int blocking_state;
390
391 /*
392 * We don't do userfault handling for the final child pid update
393 * and when coredumping (faults triggered by get_dump_page()).
394 */
395 if (current->flags & (PF_EXITING|PF_DUMPCORE))
396 goto out;
397
398 assert_fault_locked(vmf);
399
400 ctx = vma->vm_userfaultfd_ctx.ctx;
401 if (!ctx)
402 goto out;
403
404 VM_WARN_ON_ONCE(ctx->mm != mm);
405
406 /* Any unrecognized flag is a bug. */
407 VM_WARN_ON_ONCE(reason & ~__VM_UFFD_FLAGS);
408 /* 0 or > 1 flags set is a bug; we expect exactly 1. */
409 VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
410
411 if (ctx->features & UFFD_FEATURE_SIGBUS)
412 goto out;
413 if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
414 goto out;
415
416 /*
417 * Check that we can return VM_FAULT_RETRY.
418 *
419 * NOTE: it should become possible to return VM_FAULT_RETRY
420 * even if FAULT_FLAG_TRIED is set without leading to gup()
421 * -EBUSY failures, if the userfaultfd is to be extended for
422 * VM_UFFD_WP tracking and we intend to arm the userfault
423 * without first stopping userland access to the memory. For
424 * VM_UFFD_MISSING userfaults this is enough for now.
425 */
426 if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
427 /*
428 * Validate the invariant that nowait must allow retry
429 * to be sure not to return SIGBUS erroneously on
430 * nowait invocations.
431 */
432 VM_WARN_ON_ONCE(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
433#ifdef CONFIG_DEBUG_VM
434 if (printk_ratelimit()) {
435 pr_warn("FAULT_FLAG_ALLOW_RETRY missing %x\n",
436 vmf->flags);
437 dump_stack();
438 }
439#endif
440 goto out;
441 }
442
443 /*
444 * Handle nowait, not much to do other than tell it to retry
445 * and wait.
446 */
447 ret = VM_FAULT_RETRY;
448 if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
449 goto out;
450
451 if (unlikely(READ_ONCE(ctx->released))) {
452 /*
453 * If a concurrent release is detected, do not return
454 * VM_FAULT_SIGBUS or VM_FAULT_NOPAGE, but instead always
455 * return VM_FAULT_RETRY with lock released proactively.
456 *
457 * If we were to return VM_FAULT_SIGBUS here, the non
458 * cooperative manager would be instead forced to
459 * always call UFFDIO_UNREGISTER before it can safely
460 * close the uffd, to avoid involuntary SIGBUS triggered.
461 *
462 * If we were to return VM_FAULT_NOPAGE, it would work for
463 * the fault path, in which the lock will be released
464 * later. However for GUP, faultin_page() does nothing
465 * special on NOPAGE, so GUP would spin retrying without
466 * releasing the mmap read lock, causing possible livelock.
467 *
468 * Here only VM_FAULT_RETRY would make sure the mmap lock
469 * be released immediately, so that the thread concurrently
470 * releasing the userfault would always make progress.
471 */
472 release_fault_lock(vmf);
473 goto out;
474 }
475
476 /* take the reference before dropping the mmap_lock */
477 userfaultfd_ctx_get(ctx);
478
479 init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
480 uwq.wq.private = current;
481 uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
482 reason, ctx->features);
483 uwq.ctx = ctx;
484 uwq.waken = false;
485
486 blocking_state = userfaultfd_get_blocking_state(vmf->flags);
487
488 /*
489 * Take the vma lock now, in order to safely call
490 * userfaultfd_huge_must_wait() later. Since acquiring the
491 * (sleepable) vma lock can modify the current task state, that
492 * must be before explicitly calling set_current_state().
493 */
494 if (is_vm_hugetlb_page(vma))
495 hugetlb_vma_lock_read(vma);
496
497 spin_lock_irq(&ctx->fault_pending_wqh.lock);
498 /*
499 * After the __add_wait_queue the uwq is visible to userland
500 * through poll/read().
501 */
502 __add_wait_queue(&ctx->fault_pending_wqh, &uwq.wq);
503 /*
504 * The smp_mb() after __set_current_state prevents the reads
505 * following the spin_unlock to happen before the list_add in
506 * __add_wait_queue.
507 */
508 set_current_state(blocking_state);
509 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
510
511 if (is_vm_hugetlb_page(vma)) {
512 must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason);
513 hugetlb_vma_unlock_read(vma);
514 } else {
515 must_wait = userfaultfd_must_wait(ctx, vmf, reason);
516 }
517
518 release_fault_lock(vmf);
519
520 if (likely(must_wait && !READ_ONCE(ctx->released))) {
521 wake_up_poll(&ctx->fd_wqh, EPOLLIN);
522 schedule();
523 }
524
525 __set_current_state(TASK_RUNNING);
526
527 /*
528 * Here we race with the list_del; list_add in
529 * userfaultfd_ctx_read(), however because we don't ever run
530 * list_del_init() to refile across the two lists, the prev
531 * and next pointers will never point to self. list_add also
532 * would never let any of the two pointers to point to
533 * self. So list_empty_careful won't risk to see both pointers
534 * pointing to self at any time during the list refile. The
535 * only case where list_del_init() is called is the full
536 * removal in the wake function and there we don't re-list_add
537 * and it's fine not to block on the spinlock. The uwq on this
538 * kernel stack can be released after the list_del_init.
539 */
540 if (!list_empty_careful(&uwq.wq.entry)) {
541 spin_lock_irq(&ctx->fault_pending_wqh.lock);
542 /*
543 * No need of list_del_init(), the uwq on the stack
544 * will be freed shortly anyway.
545 */
546 list_del(&uwq.wq.entry);
547 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
548 }
549
550 /*
551 * ctx may go away after this if the userfault pseudo fd is
552 * already released.
553 */
554 userfaultfd_ctx_put(ctx);
555
556out:
557 return ret;
558}
559
560static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
561 struct userfaultfd_wait_queue *ewq)
562{
563 struct userfaultfd_ctx *release_new_ctx;
564
565 if (WARN_ON_ONCE(current->flags & PF_EXITING))
566 goto out;
567
568 ewq->ctx = ctx;
569 init_waitqueue_entry(&ewq->wq, current);
570 release_new_ctx = NULL;
571
572 spin_lock_irq(&ctx->event_wqh.lock);
573 /*
574 * After the __add_wait_queue the uwq is visible to userland
575 * through poll/read().
576 */
577 __add_wait_queue(&ctx->event_wqh, &ewq->wq);
578 for (;;) {
579 set_current_state(TASK_KILLABLE);
580 if (ewq->msg.event == 0)
581 break;
582 if (READ_ONCE(ctx->released) ||
583 fatal_signal_pending(current)) {
584 /*
585 * &ewq->wq may be queued in fork_event, but
586 * __remove_wait_queue ignores the head
587 * parameter. It would be a problem if it
588 * didn't.
589 */
590 __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
591 if (ewq->msg.event == UFFD_EVENT_FORK) {
592 struct userfaultfd_ctx *new;
593
594 new = (struct userfaultfd_ctx *)
595 (unsigned long)
596 ewq->msg.arg.reserved.reserved1;
597 release_new_ctx = new;
598 }
599 break;
600 }
601
602 spin_unlock_irq(&ctx->event_wqh.lock);
603
604 wake_up_poll(&ctx->fd_wqh, EPOLLIN);
605 schedule();
606
607 spin_lock_irq(&ctx->event_wqh.lock);
608 }
609 __set_current_state(TASK_RUNNING);
610 spin_unlock_irq(&ctx->event_wqh.lock);
611
612 if (release_new_ctx) {
613 userfaultfd_release_new(release_new_ctx);
614 userfaultfd_ctx_put(release_new_ctx);
615 }
616
617 /*
618 * ctx may go away after this if the userfault pseudo fd is
619 * already released.
620 */
621out:
622 atomic_dec(&ctx->mmap_changing);
623 VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
624 userfaultfd_ctx_put(ctx);
625}
626
627static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
628 struct userfaultfd_wait_queue *ewq)
629{
630 ewq->msg.event = 0;
631 wake_up_locked(&ctx->event_wqh);
632 __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
633}
634
635int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
636{
637 struct userfaultfd_ctx *ctx = NULL, *octx;
638 struct userfaultfd_fork_ctx *fctx;
639
640 octx = vma->vm_userfaultfd_ctx.ctx;
641 if (!octx)
642 return 0;
643
644 if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
645 userfaultfd_reset_ctx(vma);
646 return 0;
647 }
648
649 list_for_each_entry(fctx, fcs, list)
650 if (fctx->orig == octx) {
651 ctx = fctx->new;
652 break;
653 }
654
655 if (!ctx) {
656 fctx = kmalloc_obj(*fctx);
657 if (!fctx)
658 return -ENOMEM;
659
660 ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
661 if (!ctx) {
662 kfree(fctx);
663 return -ENOMEM;
664 }
665
666 refcount_set(&ctx->refcount, 1);
667 ctx->flags = octx->flags;
668 ctx->features = octx->features;
669 ctx->released = false;
670 init_rwsem(&ctx->map_changing_lock);
671 atomic_set(&ctx->mmap_changing, 0);
672 ctx->mm = vma->vm_mm;
673 mmgrab(ctx->mm);
674
675 userfaultfd_ctx_get(octx);
676 down_write(&octx->map_changing_lock);
677 atomic_inc(&octx->mmap_changing);
678 up_write(&octx->map_changing_lock);
679 fctx->orig = octx;
680 fctx->new = ctx;
681 list_add_tail(&fctx->list, fcs);
682 }
683
684 vma->vm_userfaultfd_ctx.ctx = ctx;
685 return 0;
686}
687
688static void dup_fctx(struct userfaultfd_fork_ctx *fctx)
689{
690 struct userfaultfd_ctx *ctx = fctx->orig;
691 struct userfaultfd_wait_queue ewq;
692
693 msg_init(&ewq.msg);
694
695 ewq.msg.event = UFFD_EVENT_FORK;
696 ewq.msg.arg.reserved.reserved1 = (unsigned long)fctx->new;
697
698 userfaultfd_event_wait_completion(ctx, &ewq);
699}
700
701void dup_userfaultfd_complete(struct list_head *fcs)
702{
703 struct userfaultfd_fork_ctx *fctx, *n;
704
705 list_for_each_entry_safe(fctx, n, fcs, list) {
706 dup_fctx(fctx);
707 list_del(&fctx->list);
708 kfree(fctx);
709 }
710}
711
712void dup_userfaultfd_fail(struct list_head *fcs)
713{
714 struct userfaultfd_fork_ctx *fctx, *n;
715
716 /*
717 * An error has occurred on fork, we will tear memory down, but have
718 * allocated memory for fctx's and raised reference counts for both the
719 * original and child contexts (and on the mm for each as a result).
720 *
721 * These would ordinarily be taken care of by a user handling the event,
722 * but we are no longer doing so, so manually clean up here.
723 *
724 * mm tear down will take care of cleaning up VMA contexts.
725 */
726 list_for_each_entry_safe(fctx, n, fcs, list) {
727 struct userfaultfd_ctx *octx = fctx->orig;
728 struct userfaultfd_ctx *ctx = fctx->new;
729
730 atomic_dec(&octx->mmap_changing);
731 VM_WARN_ON_ONCE(atomic_read(&octx->mmap_changing) < 0);
732 userfaultfd_ctx_put(octx);
733 userfaultfd_ctx_put(ctx);
734
735 list_del(&fctx->list);
736 kfree(fctx);
737 }
738}
739
740void mremap_userfaultfd_prep(struct vm_area_struct *vma,
741 struct vm_userfaultfd_ctx *vm_ctx)
742{
743 struct userfaultfd_ctx *ctx;
744
745 ctx = vma->vm_userfaultfd_ctx.ctx;
746
747 if (!ctx)
748 return;
749
750 if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
751 vm_ctx->ctx = ctx;
752 userfaultfd_ctx_get(ctx);
753 down_write(&ctx->map_changing_lock);
754 atomic_inc(&ctx->mmap_changing);
755 up_write(&ctx->map_changing_lock);
756 } else {
757 /* Drop uffd context if remap feature not enabled */
758 userfaultfd_reset_ctx(vma);
759 }
760}
761
762void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *vm_ctx,
763 unsigned long from, unsigned long to,
764 unsigned long len)
765{
766 struct userfaultfd_ctx *ctx = vm_ctx->ctx;
767 struct userfaultfd_wait_queue ewq;
768
769 if (!ctx)
770 return;
771
772 msg_init(&ewq.msg);
773
774 ewq.msg.event = UFFD_EVENT_REMAP;
775 ewq.msg.arg.remap.from = from;
776 ewq.msg.arg.remap.to = to;
777 ewq.msg.arg.remap.len = len;
778
779 userfaultfd_event_wait_completion(ctx, &ewq);
780}
781
782void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
783{
784 struct userfaultfd_ctx *ctx = vm_ctx->ctx;
785
786 if (!ctx)
787 return;
788
789 userfaultfd_ctx_put(ctx);
790}
791
792bool userfaultfd_remove(struct vm_area_struct *vma,
793 unsigned long start, unsigned long end)
794{
795 struct mm_struct *mm = vma->vm_mm;
796 struct userfaultfd_ctx *ctx;
797 struct userfaultfd_wait_queue ewq;
798
799 ctx = vma->vm_userfaultfd_ctx.ctx;
800 if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
801 return true;
802
803 userfaultfd_ctx_get(ctx);
804 down_write(&ctx->map_changing_lock);
805 atomic_inc(&ctx->mmap_changing);
806 up_write(&ctx->map_changing_lock);
807 mmap_read_unlock(mm);
808
809 msg_init(&ewq.msg);
810
811 ewq.msg.event = UFFD_EVENT_REMOVE;
812 ewq.msg.arg.remove.start = start;
813 ewq.msg.arg.remove.end = end;
814
815 userfaultfd_event_wait_completion(ctx, &ewq);
816
817 return false;
818}
819
820static bool has_unmap_ctx(struct userfaultfd_ctx *ctx, struct list_head *unmaps,
821 unsigned long start, unsigned long end)
822{
823 struct userfaultfd_unmap_ctx *unmap_ctx;
824
825 list_for_each_entry(unmap_ctx, unmaps, list)
826 if (unmap_ctx->ctx == ctx && unmap_ctx->start == start &&
827 unmap_ctx->end == end)
828 return true;
829
830 return false;
831}
832
833int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
834 unsigned long end, struct list_head *unmaps)
835{
836 struct userfaultfd_unmap_ctx *unmap_ctx;
837 struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
838
839 if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
840 has_unmap_ctx(ctx, unmaps, start, end))
841 return 0;
842
843 unmap_ctx = kzalloc_obj(*unmap_ctx);
844 if (!unmap_ctx)
845 return -ENOMEM;
846
847 userfaultfd_ctx_get(ctx);
848 down_write(&ctx->map_changing_lock);
849 atomic_inc(&ctx->mmap_changing);
850 up_write(&ctx->map_changing_lock);
851 unmap_ctx->ctx = ctx;
852 unmap_ctx->start = start;
853 unmap_ctx->end = end;
854 list_add_tail(&unmap_ctx->list, unmaps);
855
856 return 0;
857}
858
859void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf)
860{
861 struct userfaultfd_unmap_ctx *ctx, *n;
862 struct userfaultfd_wait_queue ewq;
863
864 list_for_each_entry_safe(ctx, n, uf, list) {
865 msg_init(&ewq.msg);
866
867 ewq.msg.event = UFFD_EVENT_UNMAP;
868 ewq.msg.arg.remove.start = ctx->start;
869 ewq.msg.arg.remove.end = ctx->end;
870
871 userfaultfd_event_wait_completion(ctx->ctx, &ewq);
872
873 list_del(&ctx->list);
874 kfree(ctx);
875 }
876}
877
878static int userfaultfd_release(struct inode *inode, struct file *file)
879{
880 struct userfaultfd_ctx *ctx = file->private_data;
881 struct mm_struct *mm = ctx->mm;
882 /* len == 0 means wake all */
883 struct userfaultfd_wake_range range = { .len = 0, };
884
885 WRITE_ONCE(ctx->released, true);
886
887 userfaultfd_release_all(mm, ctx);
888
889 /*
890 * After no new page faults can wait on this fault_*wqh, flush
891 * the last page faults that may have been already waiting on
892 * the fault_*wqh.
893 */
894 spin_lock_irq(&ctx->fault_pending_wqh.lock);
895 __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &range);
896 __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
897 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
898
899 /* Flush pending events that may still wait on event_wqh */
900 wake_up_all(&ctx->event_wqh);
901
902 wake_up_poll(&ctx->fd_wqh, EPOLLHUP);
903 userfaultfd_ctx_put(ctx);
904 return 0;
905}
906
907/* fault_pending_wqh.lock must be hold by the caller */
908static inline struct userfaultfd_wait_queue *find_userfault_in(
909 wait_queue_head_t *wqh)
910{
911 wait_queue_entry_t *wq;
912 struct userfaultfd_wait_queue *uwq;
913
914 lockdep_assert_held(&wqh->lock);
915
916 uwq = NULL;
917 if (!waitqueue_active(wqh))
918 goto out;
919 /* walk in reverse to provide FIFO behavior to read userfaults */
920 wq = list_last_entry(&wqh->head, typeof(*wq), entry);
921 uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
922out:
923 return uwq;
924}
925
926static inline struct userfaultfd_wait_queue *find_userfault(
927 struct userfaultfd_ctx *ctx)
928{
929 return find_userfault_in(&ctx->fault_pending_wqh);
930}
931
932static inline struct userfaultfd_wait_queue *find_userfault_evt(
933 struct userfaultfd_ctx *ctx)
934{
935 return find_userfault_in(&ctx->event_wqh);
936}
937
938static __poll_t userfaultfd_poll(struct file *file, poll_table *wait)
939{
940 struct userfaultfd_ctx *ctx = file->private_data;
941 __poll_t ret;
942
943 poll_wait(file, &ctx->fd_wqh, wait);
944
945 if (!userfaultfd_is_initialized(ctx))
946 return EPOLLERR;
947
948 /*
949 * poll() never guarantees that read won't block.
950 * userfaults can be waken before they're read().
951 */
952 if (unlikely(!(file->f_flags & O_NONBLOCK)))
953 return EPOLLERR;
954 /*
955 * lockless access to see if there are pending faults
956 * __pollwait last action is the add_wait_queue but
957 * the spin_unlock would allow the waitqueue_active to
958 * pass above the actual list_add inside
959 * add_wait_queue critical section. So use a full
960 * memory barrier to serialize the list_add write of
961 * add_wait_queue() with the waitqueue_active read
962 * below.
963 */
964 ret = 0;
965 smp_mb();
966 if (waitqueue_active(&ctx->fault_pending_wqh))
967 ret = EPOLLIN;
968 else if (waitqueue_active(&ctx->event_wqh))
969 ret = EPOLLIN;
970
971 return ret;
972}
973
974static const struct file_operations userfaultfd_fops;
975
976static int resolve_userfault_fork(struct userfaultfd_ctx *new,
977 struct inode *inode,
978 struct uffd_msg *msg)
979{
980 int fd;
981
982 fd = anon_inode_create_getfd("[userfaultfd]", &userfaultfd_fops, new,
983 O_RDONLY | (new->flags & UFFD_SHARED_FCNTL_FLAGS), inode);
984 if (fd < 0)
985 return fd;
986
987 msg->arg.reserved.reserved1 = 0;
988 msg->arg.fork.ufd = fd;
989 return 0;
990}
991
992static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
993 struct uffd_msg *msg, struct inode *inode)
994{
995 ssize_t ret;
996 DECLARE_WAITQUEUE(wait, current);
997 struct userfaultfd_wait_queue *uwq;
998 /*
999 * Handling fork event requires sleeping operations, so
1000 * we drop the event_wqh lock, then do these ops, then
1001 * lock it back and wake up the waiter. While the lock is
1002 * dropped the ewq may go away so we keep track of it
1003 * carefully.
1004 */
1005 LIST_HEAD(fork_event);
1006 struct userfaultfd_ctx *fork_nctx = NULL;
1007
1008 /* always take the fd_wqh lock before the fault_pending_wqh lock */
1009 spin_lock_irq(&ctx->fd_wqh.lock);
1010 __add_wait_queue(&ctx->fd_wqh, &wait);
1011 for (;;) {
1012 set_current_state(TASK_INTERRUPTIBLE);
1013 spin_lock(&ctx->fault_pending_wqh.lock);
1014 uwq = find_userfault(ctx);
1015 if (uwq) {
1016 /*
1017 * Use a seqcount to repeat the lockless check
1018 * in wake_userfault() to avoid missing
1019 * wakeups because during the refile both
1020 * waitqueue could become empty if this is the
1021 * only userfault.
1022 */
1023 write_seqcount_begin(&ctx->refile_seq);
1024
1025 /*
1026 * The fault_pending_wqh.lock prevents the uwq
1027 * to disappear from under us.
1028 *
1029 * Refile this userfault from
1030 * fault_pending_wqh to fault_wqh, it's not
1031 * pending anymore after we read it.
1032 *
1033 * Use list_del() by hand (as
1034 * userfaultfd_wake_function also uses
1035 * list_del_init() by hand) to be sure nobody
1036 * changes __remove_wait_queue() to use
1037 * list_del_init() in turn breaking the
1038 * !list_empty_careful() check in
1039 * handle_userfault(). The uwq->wq.head list
1040 * must never be empty at any time during the
1041 * refile, or the waitqueue could disappear
1042 * from under us. The "wait_queue_head_t"
1043 * parameter of __remove_wait_queue() is unused
1044 * anyway.
1045 */
1046 list_del(&uwq->wq.entry);
1047 add_wait_queue(&ctx->fault_wqh, &uwq->wq);
1048
1049 write_seqcount_end(&ctx->refile_seq);
1050
1051 /* careful to always initialize msg if ret == 0 */
1052 *msg = uwq->msg;
1053 spin_unlock(&ctx->fault_pending_wqh.lock);
1054 ret = 0;
1055 break;
1056 }
1057 spin_unlock(&ctx->fault_pending_wqh.lock);
1058
1059 spin_lock(&ctx->event_wqh.lock);
1060 uwq = find_userfault_evt(ctx);
1061 if (uwq) {
1062 *msg = uwq->msg;
1063
1064 if (uwq->msg.event == UFFD_EVENT_FORK) {
1065 fork_nctx = (struct userfaultfd_ctx *)
1066 (unsigned long)
1067 uwq->msg.arg.reserved.reserved1;
1068 list_move(&uwq->wq.entry, &fork_event);
1069 /*
1070 * fork_nctx can be freed as soon as
1071 * we drop the lock, unless we take a
1072 * reference on it.
1073 */
1074 userfaultfd_ctx_get(fork_nctx);
1075 spin_unlock(&ctx->event_wqh.lock);
1076 ret = 0;
1077 break;
1078 }
1079
1080 userfaultfd_event_complete(ctx, uwq);
1081 spin_unlock(&ctx->event_wqh.lock);
1082 ret = 0;
1083 break;
1084 }
1085 spin_unlock(&ctx->event_wqh.lock);
1086
1087 if (signal_pending(current)) {
1088 ret = -ERESTARTSYS;
1089 break;
1090 }
1091 if (no_wait) {
1092 ret = -EAGAIN;
1093 break;
1094 }
1095 spin_unlock_irq(&ctx->fd_wqh.lock);
1096 schedule();
1097 spin_lock_irq(&ctx->fd_wqh.lock);
1098 }
1099 __remove_wait_queue(&ctx->fd_wqh, &wait);
1100 __set_current_state(TASK_RUNNING);
1101 spin_unlock_irq(&ctx->fd_wqh.lock);
1102
1103 if (!ret && msg->event == UFFD_EVENT_FORK) {
1104 ret = resolve_userfault_fork(fork_nctx, inode, msg);
1105 spin_lock_irq(&ctx->event_wqh.lock);
1106 if (!list_empty(&fork_event)) {
1107 /*
1108 * The fork thread didn't abort, so we can
1109 * drop the temporary refcount.
1110 */
1111 userfaultfd_ctx_put(fork_nctx);
1112
1113 uwq = list_first_entry(&fork_event,
1114 typeof(*uwq),
1115 wq.entry);
1116 /*
1117 * If fork_event list wasn't empty and in turn
1118 * the event wasn't already released by fork
1119 * (the event is allocated on fork kernel
1120 * stack), put the event back to its place in
1121 * the event_wq. fork_event head will be freed
1122 * as soon as we return so the event cannot
1123 * stay queued there no matter the current
1124 * "ret" value.
1125 */
1126 list_del(&uwq->wq.entry);
1127 __add_wait_queue(&ctx->event_wqh, &uwq->wq);
1128
1129 /*
1130 * Leave the event in the waitqueue and report
1131 * error to userland if we failed to resolve
1132 * the userfault fork.
1133 */
1134 if (likely(!ret))
1135 userfaultfd_event_complete(ctx, uwq);
1136 } else {
1137 /*
1138 * Here the fork thread aborted and the
1139 * refcount from the fork thread on fork_nctx
1140 * has already been released. We still hold
1141 * the reference we took before releasing the
1142 * lock above. If resolve_userfault_fork
1143 * failed we've to drop it because the
1144 * fork_nctx has to be freed in such case. If
1145 * it succeeded we'll hold it because the new
1146 * uffd references it.
1147 */
1148 if (ret)
1149 userfaultfd_ctx_put(fork_nctx);
1150 }
1151 spin_unlock_irq(&ctx->event_wqh.lock);
1152 }
1153
1154 return ret;
1155}
1156
1157static ssize_t userfaultfd_read_iter(struct kiocb *iocb, struct iov_iter *to)
1158{
1159 struct file *file = iocb->ki_filp;
1160 struct userfaultfd_ctx *ctx = file->private_data;
1161 ssize_t _ret, ret = 0;
1162 struct uffd_msg msg;
1163 struct inode *inode = file_inode(file);
1164 bool no_wait;
1165
1166 if (!userfaultfd_is_initialized(ctx))
1167 return -EINVAL;
1168
1169 no_wait = file->f_flags & O_NONBLOCK || iocb->ki_flags & IOCB_NOWAIT;
1170 for (;;) {
1171 if (iov_iter_count(to) < sizeof(msg))
1172 return ret ? ret : -EINVAL;
1173 _ret = userfaultfd_ctx_read(ctx, no_wait, &msg, inode);
1174 if (_ret < 0)
1175 return ret ? ret : _ret;
1176 _ret = !copy_to_iter_full(&msg, sizeof(msg), to);
1177 if (_ret)
1178 return ret ? ret : -EFAULT;
1179 ret += sizeof(msg);
1180 /*
1181 * Allow to read more than one fault at time but only
1182 * block if waiting for the very first one.
1183 */
1184 no_wait = true;
1185 }
1186}
1187
1188static void __wake_userfault(struct userfaultfd_ctx *ctx,
1189 struct userfaultfd_wake_range *range)
1190{
1191 spin_lock_irq(&ctx->fault_pending_wqh.lock);
1192 /* wake all in the range and autoremove */
1193 if (waitqueue_active(&ctx->fault_pending_wqh))
1194 __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
1195 range);
1196 if (waitqueue_active(&ctx->fault_wqh))
1197 __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, range);
1198 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
1199}
1200
1201static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
1202 struct userfaultfd_wake_range *range)
1203{
1204 unsigned seq;
1205 bool need_wakeup;
1206
1207 /*
1208 * To be sure waitqueue_active() is not reordered by the CPU
1209 * before the pagetable update, use an explicit SMP memory
1210 * barrier here. PT lock release or mmap_read_unlock(mm) still
1211 * have release semantics that can allow the
1212 * waitqueue_active() to be reordered before the pte update.
1213 */
1214 smp_mb();
1215
1216 /*
1217 * Use waitqueue_active because it's very frequent to
1218 * change the address space atomically even if there are no
1219 * userfaults yet. So we take the spinlock only when we're
1220 * sure we've userfaults to wake.
1221 */
1222 do {
1223 seq = read_seqcount_begin(&ctx->refile_seq);
1224 need_wakeup = waitqueue_active(&ctx->fault_pending_wqh) ||
1225 waitqueue_active(&ctx->fault_wqh);
1226 cond_resched();
1227 } while (read_seqcount_retry(&ctx->refile_seq, seq));
1228 if (need_wakeup)
1229 __wake_userfault(ctx, range);
1230}
1231
1232static __always_inline int validate_unaligned_range(
1233 struct mm_struct *mm, __u64 start, __u64 len)
1234{
1235 __u64 task_size = mm->task_size;
1236
1237 if (len & ~PAGE_MASK)
1238 return -EINVAL;
1239 if (!len)
1240 return -EINVAL;
1241 if (start >= task_size)
1242 return -EINVAL;
1243 if (len > task_size - start)
1244 return -EINVAL;
1245 if (start + len <= start)
1246 return -EINVAL;
1247 return 0;
1248}
1249
1250static __always_inline int validate_range(struct mm_struct *mm,
1251 __u64 start, __u64 len)
1252{
1253 if (start & ~PAGE_MASK)
1254 return -EINVAL;
1255
1256 return validate_unaligned_range(mm, start, len);
1257}
1258
1259static int userfaultfd_register(struct userfaultfd_ctx *ctx,
1260 unsigned long arg)
1261{
1262 struct mm_struct *mm = ctx->mm;
1263 struct vm_area_struct *vma, *cur;
1264 int ret;
1265 struct uffdio_register uffdio_register;
1266 struct uffdio_register __user *user_uffdio_register;
1267 vm_flags_t vm_flags;
1268 bool found;
1269 bool basic_ioctls;
1270 unsigned long start, end;
1271 struct vma_iterator vmi;
1272 bool wp_async = userfaultfd_wp_async_ctx(ctx);
1273
1274 user_uffdio_register = (struct uffdio_register __user *) arg;
1275
1276 ret = -EFAULT;
1277 if (copy_from_user(&uffdio_register, user_uffdio_register,
1278 sizeof(uffdio_register)-sizeof(__u64)))
1279 goto out;
1280
1281 ret = -EINVAL;
1282 if (!uffdio_register.mode)
1283 goto out;
1284 if (uffdio_register.mode & ~UFFD_API_REGISTER_MODES)
1285 goto out;
1286 vm_flags = 0;
1287 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
1288 vm_flags |= VM_UFFD_MISSING;
1289 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
1290 if (!pgtable_supports_uffd_wp())
1291 goto out;
1292
1293 vm_flags |= VM_UFFD_WP;
1294 }
1295 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
1296#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
1297 goto out;
1298#endif
1299 vm_flags |= VM_UFFD_MINOR;
1300 }
1301
1302 ret = validate_range(mm, uffdio_register.range.start,
1303 uffdio_register.range.len);
1304 if (ret)
1305 goto out;
1306
1307 start = uffdio_register.range.start;
1308 end = start + uffdio_register.range.len;
1309
1310 ret = -ENOMEM;
1311 if (!mmget_not_zero(mm))
1312 goto out;
1313
1314 ret = -EINVAL;
1315 mmap_write_lock(mm);
1316 vma_iter_init(&vmi, mm, start);
1317 vma = vma_find(&vmi, end);
1318 if (!vma)
1319 goto out_unlock;
1320
1321 /*
1322 * If the first vma contains huge pages, make sure start address
1323 * is aligned to huge page size.
1324 */
1325 if (is_vm_hugetlb_page(vma)) {
1326 unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
1327
1328 if (start & (vma_hpagesize - 1))
1329 goto out_unlock;
1330 }
1331
1332 /*
1333 * Search for not compatible vmas.
1334 */
1335 found = false;
1336 basic_ioctls = false;
1337 cur = vma;
1338 do {
1339 cond_resched();
1340
1341 VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
1342 !!(cur->vm_flags & __VM_UFFD_FLAGS));
1343
1344 /* check not compatible vmas */
1345 ret = -EINVAL;
1346 if (!vma_can_userfault(cur, vm_flags, wp_async))
1347 goto out_unlock;
1348
1349 /*
1350 * UFFDIO_COPY will fill file holes even without
1351 * PROT_WRITE. This check enforces that if this is a
1352 * MAP_SHARED, the process has write permission to the backing
1353 * file. If VM_MAYWRITE is set it also enforces that on a
1354 * MAP_SHARED vma: there is no F_WRITE_SEAL and no further
1355 * F_WRITE_SEAL can be taken until the vma is destroyed.
1356 */
1357 ret = -EPERM;
1358 if (unlikely(!(cur->vm_flags & VM_MAYWRITE)))
1359 goto out_unlock;
1360
1361 /*
1362 * If this vma contains ending address, and huge pages
1363 * check alignment.
1364 */
1365 if (is_vm_hugetlb_page(cur) && end <= cur->vm_end &&
1366 end > cur->vm_start) {
1367 unsigned long vma_hpagesize = vma_kernel_pagesize(cur);
1368
1369 ret = -EINVAL;
1370
1371 if (end & (vma_hpagesize - 1))
1372 goto out_unlock;
1373 }
1374 if ((vm_flags & VM_UFFD_WP) && !(cur->vm_flags & VM_MAYWRITE))
1375 goto out_unlock;
1376
1377 /*
1378 * Check that this vma isn't already owned by a
1379 * different userfaultfd. We can't allow more than one
1380 * userfaultfd to own a single vma simultaneously or we
1381 * wouldn't know which one to deliver the userfaults to.
1382 */
1383 ret = -EBUSY;
1384 if (cur->vm_userfaultfd_ctx.ctx &&
1385 cur->vm_userfaultfd_ctx.ctx != ctx)
1386 goto out_unlock;
1387
1388 /*
1389 * Note vmas containing huge pages
1390 */
1391 if (is_vm_hugetlb_page(cur))
1392 basic_ioctls = true;
1393
1394 found = true;
1395 } for_each_vma_range(vmi, cur, end);
1396 VM_WARN_ON_ONCE(!found);
1397
1398 ret = userfaultfd_register_range(ctx, vma, vm_flags, start, end,
1399 wp_async);
1400
1401out_unlock:
1402 mmap_write_unlock(mm);
1403 mmput(mm);
1404 if (!ret) {
1405 __u64 ioctls_out;
1406
1407 ioctls_out = basic_ioctls ? UFFD_API_RANGE_IOCTLS_BASIC :
1408 UFFD_API_RANGE_IOCTLS;
1409
1410 /*
1411 * Declare the WP ioctl only if the WP mode is
1412 * specified and all checks passed with the range
1413 */
1414 if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_WP))
1415 ioctls_out &= ~((__u64)1 << _UFFDIO_WRITEPROTECT);
1416
1417 /* CONTINUE ioctl is only supported for MINOR ranges. */
1418 if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
1419 ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
1420
1421 /*
1422 * Now that we scanned all vmas we can already tell
1423 * userland which ioctls methods are guaranteed to
1424 * succeed on this range.
1425 */
1426 if (put_user(ioctls_out, &user_uffdio_register->ioctls))
1427 ret = -EFAULT;
1428 }
1429out:
1430 return ret;
1431}
1432
1433static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
1434 unsigned long arg)
1435{
1436 struct mm_struct *mm = ctx->mm;
1437 struct vm_area_struct *vma, *prev, *cur;
1438 int ret;
1439 struct uffdio_range uffdio_unregister;
1440 bool found;
1441 unsigned long start, end, vma_end;
1442 const void __user *buf = (void __user *)arg;
1443 struct vma_iterator vmi;
1444 bool wp_async = userfaultfd_wp_async_ctx(ctx);
1445
1446 ret = -EFAULT;
1447 if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
1448 goto out;
1449
1450 ret = validate_range(mm, uffdio_unregister.start,
1451 uffdio_unregister.len);
1452 if (ret)
1453 goto out;
1454
1455 start = uffdio_unregister.start;
1456 end = start + uffdio_unregister.len;
1457
1458 ret = -ENOMEM;
1459 if (!mmget_not_zero(mm))
1460 goto out;
1461
1462 mmap_write_lock(mm);
1463 ret = -EINVAL;
1464 vma_iter_init(&vmi, mm, start);
1465 vma = vma_find(&vmi, end);
1466 if (!vma)
1467 goto out_unlock;
1468
1469 /*
1470 * If the first vma contains huge pages, make sure start address
1471 * is aligned to huge page size.
1472 */
1473 if (is_vm_hugetlb_page(vma)) {
1474 unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
1475
1476 if (start & (vma_hpagesize - 1))
1477 goto out_unlock;
1478 }
1479
1480 /*
1481 * Search for not compatible vmas.
1482 */
1483 found = false;
1484 cur = vma;
1485 do {
1486 cond_resched();
1487
1488 VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
1489 !!(cur->vm_flags & __VM_UFFD_FLAGS));
1490
1491 /*
1492 * Prevent unregistering through a different userfaultfd than
1493 * the one used for registration.
1494 */
1495 if (cur->vm_userfaultfd_ctx.ctx &&
1496 cur->vm_userfaultfd_ctx.ctx != ctx)
1497 goto out_unlock;
1498
1499 /*
1500 * Check not compatible vmas, not strictly required
1501 * here as not compatible vmas cannot have an
1502 * userfaultfd_ctx registered on them, but this
1503 * provides for more strict behavior to notice
1504 * unregistration errors.
1505 */
1506 if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
1507 goto out_unlock;
1508
1509 found = true;
1510 } for_each_vma_range(vmi, cur, end);
1511 VM_WARN_ON_ONCE(!found);
1512
1513 vma_iter_set(&vmi, start);
1514 prev = vma_prev(&vmi);
1515 if (vma->vm_start < start)
1516 prev = vma;
1517
1518 ret = 0;
1519 for_each_vma_range(vmi, vma, end) {
1520 cond_resched();
1521
1522 /* VMA not registered with userfaultfd. */
1523 if (!vma->vm_userfaultfd_ctx.ctx)
1524 goto skip;
1525
1526 VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx != ctx);
1527 VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, wp_async));
1528 VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
1529
1530 if (vma->vm_start > start)
1531 start = vma->vm_start;
1532 vma_end = min(end, vma->vm_end);
1533
1534 if (userfaultfd_missing(vma)) {
1535 /*
1536 * Wake any concurrent pending userfault while
1537 * we unregister, so they will not hang
1538 * permanently and it avoids userland to call
1539 * UFFDIO_WAKE explicitly.
1540 */
1541 struct userfaultfd_wake_range range;
1542 range.start = start;
1543 range.len = vma_end - start;
1544 wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range);
1545 }
1546
1547 vma = userfaultfd_clear_vma(&vmi, prev, vma,
1548 start, vma_end);
1549 if (IS_ERR(vma)) {
1550 ret = PTR_ERR(vma);
1551 break;
1552 }
1553
1554 skip:
1555 prev = vma;
1556 start = vma->vm_end;
1557 }
1558
1559out_unlock:
1560 mmap_write_unlock(mm);
1561 mmput(mm);
1562out:
1563 return ret;
1564}
1565
1566/*
1567 * userfaultfd_wake may be used in combination with the
1568 * UFFDIO_*_MODE_DONTWAKE to wakeup userfaults in batches.
1569 */
1570static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
1571 unsigned long arg)
1572{
1573 int ret;
1574 struct uffdio_range uffdio_wake;
1575 struct userfaultfd_wake_range range;
1576 const void __user *buf = (void __user *)arg;
1577
1578 ret = -EFAULT;
1579 if (copy_from_user(&uffdio_wake, buf, sizeof(uffdio_wake)))
1580 goto out;
1581
1582 ret = validate_range(ctx->mm, uffdio_wake.start, uffdio_wake.len);
1583 if (ret)
1584 goto out;
1585
1586 range.start = uffdio_wake.start;
1587 range.len = uffdio_wake.len;
1588
1589 /*
1590 * len == 0 means wake all and we don't want to wake all here,
1591 * so check it again to be sure.
1592 */
1593 VM_WARN_ON_ONCE(!range.len);
1594
1595 wake_userfault(ctx, &range);
1596 ret = 0;
1597
1598out:
1599 return ret;
1600}
1601
1602static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
1603 unsigned long arg)
1604{
1605 __s64 ret;
1606 struct uffdio_copy uffdio_copy;
1607 struct uffdio_copy __user *user_uffdio_copy;
1608 struct userfaultfd_wake_range range;
1609 uffd_flags_t flags = 0;
1610
1611 user_uffdio_copy = (struct uffdio_copy __user *) arg;
1612
1613 ret = -EAGAIN;
1614 if (unlikely(atomic_read(&ctx->mmap_changing))) {
1615 if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
1616 return -EFAULT;
1617 goto out;
1618 }
1619
1620 ret = -EFAULT;
1621 if (copy_from_user(&uffdio_copy, user_uffdio_copy,
1622 /* don't copy "copy" last field */
1623 sizeof(uffdio_copy)-sizeof(__s64)))
1624 goto out;
1625
1626 ret = validate_unaligned_range(ctx->mm, uffdio_copy.src,
1627 uffdio_copy.len);
1628 if (ret)
1629 goto out;
1630 ret = validate_range(ctx->mm, uffdio_copy.dst, uffdio_copy.len);
1631 if (ret)
1632 goto out;
1633
1634 ret = -EINVAL;
1635 if (uffdio_copy.mode & ~(UFFDIO_COPY_MODE_DONTWAKE|UFFDIO_COPY_MODE_WP))
1636 goto out;
1637 if (uffdio_copy.mode & UFFDIO_COPY_MODE_WP)
1638 flags |= MFILL_ATOMIC_WP;
1639 if (mmget_not_zero(ctx->mm)) {
1640 ret = mfill_atomic_copy(ctx, uffdio_copy.dst, uffdio_copy.src,
1641 uffdio_copy.len, flags);
1642 mmput(ctx->mm);
1643 } else {
1644 return -ESRCH;
1645 }
1646 if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
1647 return -EFAULT;
1648 if (ret < 0)
1649 goto out;
1650 VM_WARN_ON_ONCE(!ret);
1651 /* len == 0 would wake all */
1652 range.len = ret;
1653 if (!(uffdio_copy.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
1654 range.start = uffdio_copy.dst;
1655 wake_userfault(ctx, &range);
1656 }
1657 ret = range.len == uffdio_copy.len ? 0 : -EAGAIN;
1658out:
1659 return ret;
1660}
1661
1662static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
1663 unsigned long arg)
1664{
1665 __s64 ret;
1666 struct uffdio_zeropage uffdio_zeropage;
1667 struct uffdio_zeropage __user *user_uffdio_zeropage;
1668 struct userfaultfd_wake_range range;
1669
1670 user_uffdio_zeropage = (struct uffdio_zeropage __user *) arg;
1671
1672 ret = -EAGAIN;
1673 if (unlikely(atomic_read(&ctx->mmap_changing))) {
1674 if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
1675 return -EFAULT;
1676 goto out;
1677 }
1678
1679 ret = -EFAULT;
1680 if (copy_from_user(&uffdio_zeropage, user_uffdio_zeropage,
1681 /* don't copy "zeropage" last field */
1682 sizeof(uffdio_zeropage)-sizeof(__s64)))
1683 goto out;
1684
1685 ret = validate_range(ctx->mm, uffdio_zeropage.range.start,
1686 uffdio_zeropage.range.len);
1687 if (ret)
1688 goto out;
1689 ret = -EINVAL;
1690 if (uffdio_zeropage.mode & ~UFFDIO_ZEROPAGE_MODE_DONTWAKE)
1691 goto out;
1692
1693 if (mmget_not_zero(ctx->mm)) {
1694 ret = mfill_atomic_zeropage(ctx, uffdio_zeropage.range.start,
1695 uffdio_zeropage.range.len);
1696 mmput(ctx->mm);
1697 } else {
1698 return -ESRCH;
1699 }
1700 if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
1701 return -EFAULT;
1702 if (ret < 0)
1703 goto out;
1704 /* len == 0 would wake all */
1705 VM_WARN_ON_ONCE(!ret);
1706 range.len = ret;
1707 if (!(uffdio_zeropage.mode & UFFDIO_ZEROPAGE_MODE_DONTWAKE)) {
1708 range.start = uffdio_zeropage.range.start;
1709 wake_userfault(ctx, &range);
1710 }
1711 ret = range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
1712out:
1713 return ret;
1714}
1715
1716static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
1717 unsigned long arg)
1718{
1719 int ret;
1720 struct uffdio_writeprotect uffdio_wp;
1721 struct uffdio_writeprotect __user *user_uffdio_wp;
1722 struct userfaultfd_wake_range range;
1723 bool mode_wp, mode_dontwake;
1724
1725 if (atomic_read(&ctx->mmap_changing))
1726 return -EAGAIN;
1727
1728 user_uffdio_wp = (struct uffdio_writeprotect __user *) arg;
1729
1730 if (copy_from_user(&uffdio_wp, user_uffdio_wp,
1731 sizeof(struct uffdio_writeprotect)))
1732 return -EFAULT;
1733
1734 ret = validate_range(ctx->mm, uffdio_wp.range.start,
1735 uffdio_wp.range.len);
1736 if (ret)
1737 return ret;
1738
1739 if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE |
1740 UFFDIO_WRITEPROTECT_MODE_WP))
1741 return -EINVAL;
1742
1743 mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
1744 mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
1745
1746 if (mode_wp && mode_dontwake)
1747 return -EINVAL;
1748
1749 if (mmget_not_zero(ctx->mm)) {
1750 ret = mwriteprotect_range(ctx, uffdio_wp.range.start,
1751 uffdio_wp.range.len, mode_wp);
1752 mmput(ctx->mm);
1753 } else {
1754 return -ESRCH;
1755 }
1756
1757 if (ret)
1758 return ret;
1759
1760 if (!mode_wp && !mode_dontwake) {
1761 range.start = uffdio_wp.range.start;
1762 range.len = uffdio_wp.range.len;
1763 wake_userfault(ctx, &range);
1764 }
1765 return ret;
1766}
1767
1768static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
1769{
1770 __s64 ret;
1771 struct uffdio_continue uffdio_continue;
1772 struct uffdio_continue __user *user_uffdio_continue;
1773 struct userfaultfd_wake_range range;
1774 uffd_flags_t flags = 0;
1775
1776 user_uffdio_continue = (struct uffdio_continue __user *)arg;
1777
1778 ret = -EAGAIN;
1779 if (unlikely(atomic_read(&ctx->mmap_changing))) {
1780 if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
1781 return -EFAULT;
1782 goto out;
1783 }
1784
1785 ret = -EFAULT;
1786 if (copy_from_user(&uffdio_continue, user_uffdio_continue,
1787 /* don't copy the output fields */
1788 sizeof(uffdio_continue) - (sizeof(__s64))))
1789 goto out;
1790
1791 ret = validate_range(ctx->mm, uffdio_continue.range.start,
1792 uffdio_continue.range.len);
1793 if (ret)
1794 goto out;
1795
1796 ret = -EINVAL;
1797 if (uffdio_continue.mode & ~(UFFDIO_CONTINUE_MODE_DONTWAKE |
1798 UFFDIO_CONTINUE_MODE_WP))
1799 goto out;
1800 if (uffdio_continue.mode & UFFDIO_CONTINUE_MODE_WP)
1801 flags |= MFILL_ATOMIC_WP;
1802
1803 if (mmget_not_zero(ctx->mm)) {
1804 ret = mfill_atomic_continue(ctx, uffdio_continue.range.start,
1805 uffdio_continue.range.len, flags);
1806 mmput(ctx->mm);
1807 } else {
1808 return -ESRCH;
1809 }
1810
1811 if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
1812 return -EFAULT;
1813 if (ret < 0)
1814 goto out;
1815
1816 /* len == 0 would wake all */
1817 VM_WARN_ON_ONCE(!ret);
1818 range.len = ret;
1819 if (!(uffdio_continue.mode & UFFDIO_CONTINUE_MODE_DONTWAKE)) {
1820 range.start = uffdio_continue.range.start;
1821 wake_userfault(ctx, &range);
1822 }
1823 ret = range.len == uffdio_continue.range.len ? 0 : -EAGAIN;
1824
1825out:
1826 return ret;
1827}
1828
1829static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long arg)
1830{
1831 __s64 ret;
1832 struct uffdio_poison uffdio_poison;
1833 struct uffdio_poison __user *user_uffdio_poison;
1834 struct userfaultfd_wake_range range;
1835
1836 user_uffdio_poison = (struct uffdio_poison __user *)arg;
1837
1838 ret = -EAGAIN;
1839 if (unlikely(atomic_read(&ctx->mmap_changing))) {
1840 if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
1841 return -EFAULT;
1842 goto out;
1843 }
1844
1845 ret = -EFAULT;
1846 if (copy_from_user(&uffdio_poison, user_uffdio_poison,
1847 /* don't copy the output fields */
1848 sizeof(uffdio_poison) - (sizeof(__s64))))
1849 goto out;
1850
1851 ret = validate_range(ctx->mm, uffdio_poison.range.start,
1852 uffdio_poison.range.len);
1853 if (ret)
1854 goto out;
1855
1856 ret = -EINVAL;
1857 if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE)
1858 goto out;
1859
1860 if (mmget_not_zero(ctx->mm)) {
1861 ret = mfill_atomic_poison(ctx, uffdio_poison.range.start,
1862 uffdio_poison.range.len, 0);
1863 mmput(ctx->mm);
1864 } else {
1865 return -ESRCH;
1866 }
1867
1868 if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
1869 return -EFAULT;
1870 if (ret < 0)
1871 goto out;
1872
1873 /* len == 0 would wake all */
1874 VM_WARN_ON_ONCE(!ret);
1875 range.len = ret;
1876 if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) {
1877 range.start = uffdio_poison.range.start;
1878 wake_userfault(ctx, &range);
1879 }
1880 ret = range.len == uffdio_poison.range.len ? 0 : -EAGAIN;
1881
1882out:
1883 return ret;
1884}
1885
1886bool userfaultfd_wp_async(struct vm_area_struct *vma)
1887{
1888 return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
1889}
1890
1891static inline unsigned int uffd_ctx_features(__u64 user_features)
1892{
1893 /*
1894 * For the current set of features the bits just coincide. Set
1895 * UFFD_FEATURE_INITIALIZED to mark the features as enabled.
1896 */
1897 return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
1898}
1899
1900static int userfaultfd_move(struct userfaultfd_ctx *ctx,
1901 unsigned long arg)
1902{
1903 __s64 ret;
1904 struct uffdio_move uffdio_move;
1905 struct uffdio_move __user *user_uffdio_move;
1906 struct userfaultfd_wake_range range;
1907 struct mm_struct *mm = ctx->mm;
1908
1909 user_uffdio_move = (struct uffdio_move __user *) arg;
1910
1911 ret = -EAGAIN;
1912 if (unlikely(atomic_read(&ctx->mmap_changing))) {
1913 if (unlikely(put_user(ret, &user_uffdio_move->move)))
1914 return -EFAULT;
1915 goto out;
1916 }
1917
1918 if (copy_from_user(&uffdio_move, user_uffdio_move,
1919 /* don't copy "move" last field */
1920 sizeof(uffdio_move)-sizeof(__s64)))
1921 return -EFAULT;
1922
1923 /* Do not allow cross-mm moves. */
1924 if (mm != current->mm)
1925 return -EINVAL;
1926
1927 ret = validate_range(mm, uffdio_move.dst, uffdio_move.len);
1928 if (ret)
1929 return ret;
1930
1931 ret = validate_range(mm, uffdio_move.src, uffdio_move.len);
1932 if (ret)
1933 return ret;
1934
1935 if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES|
1936 UFFDIO_MOVE_MODE_DONTWAKE))
1937 return -EINVAL;
1938
1939 if (mmget_not_zero(mm)) {
1940 ret = move_pages(ctx, uffdio_move.dst, uffdio_move.src,
1941 uffdio_move.len, uffdio_move.mode);
1942 mmput(mm);
1943 } else {
1944 return -ESRCH;
1945 }
1946
1947 if (unlikely(put_user(ret, &user_uffdio_move->move)))
1948 return -EFAULT;
1949 if (ret < 0)
1950 goto out;
1951
1952 /* len == 0 would wake all */
1953 VM_WARN_ON(!ret);
1954 range.len = ret;
1955 if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) {
1956 range.start = uffdio_move.dst;
1957 wake_userfault(ctx, &range);
1958 }
1959 ret = range.len == uffdio_move.len ? 0 : -EAGAIN;
1960
1961out:
1962 return ret;
1963}
1964
1965/*
1966 * userland asks for a certain API version and we return which bits
1967 * and ioctl commands are implemented in this kernel for such API
1968 * version or -EINVAL if unknown.
1969 */
1970static int userfaultfd_api(struct userfaultfd_ctx *ctx,
1971 unsigned long arg)
1972{
1973 struct uffdio_api uffdio_api;
1974 void __user *buf = (void __user *)arg;
1975 unsigned int ctx_features;
1976 int ret;
1977 __u64 features;
1978
1979 ret = -EFAULT;
1980 if (copy_from_user(&uffdio_api, buf, sizeof(uffdio_api)))
1981 goto out;
1982 features = uffdio_api.features;
1983 ret = -EINVAL;
1984 if (uffdio_api.api != UFFD_API)
1985 goto err_out;
1986 ret = -EPERM;
1987 if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
1988 goto err_out;
1989
1990 /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
1991 if (features & UFFD_FEATURE_WP_ASYNC)
1992 features |= UFFD_FEATURE_WP_UNPOPULATED;
1993
1994 /* report all available features and ioctls to userland */
1995 uffdio_api.features = UFFD_API_FEATURES;
1996#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
1997 uffdio_api.features &=
1998 ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
1999#endif
2000 if (!pgtable_supports_uffd_wp())
2001 uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
2002
2003 if (!uffd_supports_wp_marker()) {
2004 uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
2005 uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
2006 uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
2007 }
2008
2009 ret = -EINVAL;
2010 if (features & ~uffdio_api.features)
2011 goto err_out;
2012
2013 uffdio_api.ioctls = UFFD_API_IOCTLS;
2014 ret = -EFAULT;
2015 if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
2016 goto out;
2017
2018 /* only enable the requested features for this uffd context */
2019 ctx_features = uffd_ctx_features(features);
2020 ret = -EINVAL;
2021 if (cmpxchg(&ctx->features, 0, ctx_features) != 0)
2022 goto err_out;
2023
2024 ret = 0;
2025out:
2026 return ret;
2027err_out:
2028 memset(&uffdio_api, 0, sizeof(uffdio_api));
2029 if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
2030 ret = -EFAULT;
2031 goto out;
2032}
2033
2034static long userfaultfd_ioctl(struct file *file, unsigned cmd,
2035 unsigned long arg)
2036{
2037 int ret = -EINVAL;
2038 struct userfaultfd_ctx *ctx = file->private_data;
2039
2040 if (cmd != UFFDIO_API && !userfaultfd_is_initialized(ctx))
2041 return -EINVAL;
2042
2043 switch(cmd) {
2044 case UFFDIO_API:
2045 ret = userfaultfd_api(ctx, arg);
2046 break;
2047 case UFFDIO_REGISTER:
2048 ret = userfaultfd_register(ctx, arg);
2049 break;
2050 case UFFDIO_UNREGISTER:
2051 ret = userfaultfd_unregister(ctx, arg);
2052 break;
2053 case UFFDIO_WAKE:
2054 ret = userfaultfd_wake(ctx, arg);
2055 break;
2056 case UFFDIO_COPY:
2057 ret = userfaultfd_copy(ctx, arg);
2058 break;
2059 case UFFDIO_ZEROPAGE:
2060 ret = userfaultfd_zeropage(ctx, arg);
2061 break;
2062 case UFFDIO_MOVE:
2063 ret = userfaultfd_move(ctx, arg);
2064 break;
2065 case UFFDIO_WRITEPROTECT:
2066 ret = userfaultfd_writeprotect(ctx, arg);
2067 break;
2068 case UFFDIO_CONTINUE:
2069 ret = userfaultfd_continue(ctx, arg);
2070 break;
2071 case UFFDIO_POISON:
2072 ret = userfaultfd_poison(ctx, arg);
2073 break;
2074 }
2075 return ret;
2076}
2077
2078#ifdef CONFIG_PROC_FS
2079static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
2080{
2081 struct userfaultfd_ctx *ctx = f->private_data;
2082 wait_queue_entry_t *wq;
2083 unsigned long pending = 0, total = 0;
2084
2085 spin_lock_irq(&ctx->fault_pending_wqh.lock);
2086 list_for_each_entry(wq, &ctx->fault_pending_wqh.head, entry) {
2087 pending++;
2088 total++;
2089 }
2090 list_for_each_entry(wq, &ctx->fault_wqh.head, entry) {
2091 total++;
2092 }
2093 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2094
2095 /*
2096 * If more protocols will be added, there will be all shown
2097 * separated by a space. Like this:
2098 * protocols: aa:... bb:...
2099 */
2100 seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
2101 pending, total, UFFD_API, ctx->features,
2102 UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
2103}
2104#endif
2105
2106static const struct file_operations userfaultfd_fops = {
2107#ifdef CONFIG_PROC_FS
2108 .show_fdinfo = userfaultfd_show_fdinfo,
2109#endif
2110 .release = userfaultfd_release,
2111 .poll = userfaultfd_poll,
2112 .read_iter = userfaultfd_read_iter,
2113 .unlocked_ioctl = userfaultfd_ioctl,
2114 .compat_ioctl = compat_ptr_ioctl,
2115 .llseek = noop_llseek,
2116};
2117
2118static void init_once_userfaultfd_ctx(void *mem)
2119{
2120 struct userfaultfd_ctx *ctx = (struct userfaultfd_ctx *) mem;
2121
2122 init_waitqueue_head(&ctx->fault_pending_wqh);
2123 init_waitqueue_head(&ctx->fault_wqh);
2124 init_waitqueue_head(&ctx->event_wqh);
2125 init_waitqueue_head(&ctx->fd_wqh);
2126 seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock);
2127}
2128
2129static int new_userfaultfd(int flags)
2130{
2131 struct userfaultfd_ctx *ctx __free(kfree) = NULL;
2132
2133 VM_WARN_ON_ONCE(!current->mm);
2134
2135 /* Check the UFFD_* constants for consistency. */
2136 BUILD_BUG_ON(UFFD_USER_MODE_ONLY & UFFD_SHARED_FCNTL_FLAGS);
2137
2138 if (flags & ~(UFFD_SHARED_FCNTL_FLAGS | UFFD_USER_MODE_ONLY))
2139 return -EINVAL;
2140
2141 ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
2142 if (!ctx)
2143 return -ENOMEM;
2144
2145 refcount_set(&ctx->refcount, 1);
2146 ctx->flags = flags;
2147 ctx->features = 0;
2148 ctx->released = false;
2149 init_rwsem(&ctx->map_changing_lock);
2150 atomic_set(&ctx->mmap_changing, 0);
2151 ctx->mm = current->mm;
2152
2153 FD_PREPARE(fdf, flags & UFFD_SHARED_FCNTL_FLAGS,
2154 anon_inode_create_getfile("[userfaultfd]", &userfaultfd_fops, ctx,
2155 O_RDONLY | (flags & UFFD_SHARED_FCNTL_FLAGS),
2156 NULL));
2157 if (fdf.err)
2158 return fdf.err;
2159
2160 /* prevent the mm struct to be freed */
2161 mmgrab(ctx->mm);
2162 fd_prepare_file(fdf)->f_mode |= FMODE_NOWAIT;
2163 retain_and_null_ptr(ctx);
2164 return fd_publish(fdf);
2165}
2166
2167static inline bool userfaultfd_syscall_allowed(int flags)
2168{
2169 /* Userspace-only page faults are always allowed */
2170 if (flags & UFFD_USER_MODE_ONLY)
2171 return true;
2172
2173 /*
2174 * The user is requesting a userfaultfd which can handle kernel faults.
2175 * Privileged users are always allowed to do this.
2176 */
2177 if (capable(CAP_SYS_PTRACE))
2178 return true;
2179
2180 /* Otherwise, access to kernel fault handling is sysctl controlled. */
2181 return sysctl_unprivileged_userfaultfd;
2182}
2183
2184SYSCALL_DEFINE1(userfaultfd, int, flags)
2185{
2186 if (!userfaultfd_syscall_allowed(flags))
2187 return -EPERM;
2188
2189 return new_userfaultfd(flags);
2190}
2191
2192static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags)
2193{
2194 if (cmd != USERFAULTFD_IOC_NEW)
2195 return -EINVAL;
2196
2197 return new_userfaultfd(flags);
2198}
2199
2200static const struct file_operations userfaultfd_dev_fops = {
2201 .unlocked_ioctl = userfaultfd_dev_ioctl,
2202 .compat_ioctl = userfaultfd_dev_ioctl,
2203 .owner = THIS_MODULE,
2204 .llseek = noop_llseek,
2205};
2206
2207static struct miscdevice userfaultfd_misc = {
2208 .minor = MISC_DYNAMIC_MINOR,
2209 .name = "userfaultfd",
2210 .fops = &userfaultfd_dev_fops
2211};
2212
2213static int __init userfaultfd_init(void)
2214{
2215 int ret;
2216
2217 ret = misc_register(&userfaultfd_misc);
2218 if (ret)
2219 return ret;
2220
2221 userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache",
2222 sizeof(struct userfaultfd_ctx),
2223 0,
2224 SLAB_HWCACHE_ALIGN|SLAB_PANIC,
2225 init_once_userfaultfd_ctx);
2226#ifdef CONFIG_SYSCTL
2227 register_sysctl_init("vm", vm_userfaultfd_table);
2228#endif
2229 return 0;
2230}
2231__initcall(userfaultfd_init);
include/asm-generic/pgalloc.h+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
4 4
5#ifdef CONFIG_MMU5#ifdef CONFIG_MMU
6 6
7#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO)7#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
8#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)8#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)
9 9
10/**10/**
include/linux/bootmem_info.h+0−1
@@ -82,7 +82,6 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 81 unmodified lines
82 82
83static inline void free_bootmem_page(struct page *page)83static inline void free_bootmem_page(struct page *page)
84{84{
85 kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
86 free_reserved_page(page);85 free_reserved_page(page);
87}86}
88#endif87#endif
include/linux/compaction.h+5−5
@@ -2,6 +2,8 @@ · 1 unmodified line
2#ifndef _LINUX_COMPACTION_H2#ifndef _LINUX_COMPACTION_H
3#define _LINUX_COMPACTION_H3#define _LINUX_COMPACTION_H
4 4
5#include <linux/swap.h>
6
5/*7/*
6 * Determines how hard direct compaction should try to succeed.8 * Determines how hard direct compaction should try to succeed.
7 * Lower value means higher priority, analogically to reclaim priority.9 * Lower value means higher priority, analogically to reclaim priority.
@@ -73,11 +75,9 @@ static inline unsigned long compact_gap(unsigned int order) · 65 unmodified lines
73 * effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum75 * effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum
74 * that the migrate scanner can have isolated on migrate list, and free76 * that the migrate scanner can have isolated on migrate list, and free
75 * scanner is only invoked when the number of isolated free pages is77 * scanner is only invoked when the number of isolated free pages is
76 * lower than that. But it's not worth to complicate the formula here78 * lower than that.
77 * as a bigger gap for higher orders than strictly necessary can also
78 * improve chances of compaction success.
79 */79 */
80 return 2UL << order;80 return min(2UL << order, COMPACT_CLUSTER_MAX);
81}81}
82 82
83static inline int current_is_kcompactd(void)83static inline int current_is_kcompactd(void)
@@ -101,7 +101,7 @@ extern void compaction_defer_reset(struct zone *zone, int order, · 17 unmodified lines
101 bool alloc_success);101 bool alloc_success);
102 102
103bool compaction_zonelist_suitable(struct alloc_context *ac, int order,103bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
104 int alloc_flags);104 int alloc_flags, gfp_t gfp_mask);
105 105
106extern void __meminit kcompactd_run(int nid);106extern void __meminit kcompactd_run(int nid);
107extern void __meminit kcompactd_stop(int nid);107extern void __meminit kcompactd_stop(int nid);
include/linux/damon.h+111−25
@@ -8,23 +8,20 @@ · 7 unmodified lines
8#ifndef _DAMON_H_8#ifndef _DAMON_H_
9#define _DAMON_H_9#define _DAMON_H_
10 10
11#include <linux/math64.h>
11#include <linux/memcontrol.h>12#include <linux/memcontrol.h>
12#include <linux/mutex.h>13#include <linux/mutex.h>
14#include <linux/prandom.h>
13#include <linux/time64.h>15#include <linux/time64.h>
14#include <linux/types.h>16#include <linux/types.h>
15#include <linux/random.h>
16 17
17/* Minimal region size. Every damon_region is aligned by this. */18/* Minimal region size. Every damon_region is aligned by this. */
18#define DAMON_MIN_REGION_SZ PAGE_SIZE19#define DAMON_MIN_REGION_SZ PAGE_SIZE
20/* Maximum number of monitoring probes. */
21#define DAMON_MAX_PROBES (4)
19/* Max priority score for DAMON-based operation schemes */22/* Max priority score for DAMON-based operation schemes */
20#define DAMOS_MAX_SCORE (99)23#define DAMOS_MAX_SCORE (99)
21 24
22/* Get a random number in [l, r) */
23static inline unsigned long damon_rand(unsigned long l, unsigned long r)
24{
25 return l + get_random_u32_below(r - l);
26}
27
28/**25/**
29 * struct damon_addr_range - Represents an address region of [@start, @end).26 * struct damon_addr_range - Represents an address region of [@start, @end).
30 * @start: Start address of the region (inclusive).27 * @start: Start address of the region (inclusive).
@@ -52,6 +49,7 @@ struct damon_size_range { · 21 unmodified lines
52 * @nr_accesses: Access frequency of this region.49 * @nr_accesses: Access frequency of this region.
53 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for50 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
54 * each sampling interval.51 * each sampling interval.
52 * @probe_hits: Number of probe-positive region samples.
55 * @list: List head for siblings.53 * @list: List head for siblings.
56 * @age: Age of this region.54 * @age: Age of this region.
57 *55 *
@@ -80,6 +78,7 @@ struct damon_region { · 22 unmodified lines
80 unsigned long sampling_addr;78 unsigned long sampling_addr;
81 unsigned int nr_accesses;79 unsigned int nr_accesses;
82 unsigned int nr_accesses_bp;80 unsigned int nr_accesses_bp;
81 unsigned char probe_hits[DAMON_MAX_PROBES];
83 struct list_head list;82 struct list_head list;
84 83
85 unsigned int age;84 unsigned int age;
@@ -121,6 +120,7 @@ struct damon_target { · 35 unmodified lines
121 * @DAMOS_PAGEOUT: Reclaim the region.120 * @DAMOS_PAGEOUT: Reclaim the region.
122 * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.121 * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
123 * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.122 * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
123 * @DAMOS_COLLAPSE: Call ``madvise()`` for the region with MADV_COLLAPSE.
124 * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.124 * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
125 * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.125 * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.
126 * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.126 * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.
@@ -140,6 +140,7 @@ enum damos_action { · 13 unmodified lines
140 DAMOS_PAGEOUT,140 DAMOS_PAGEOUT,
141 DAMOS_HUGEPAGE,141 DAMOS_HUGEPAGE,
142 DAMOS_NOHUGEPAGE,142 DAMOS_NOHUGEPAGE,
143 DAMOS_COLLAPSE,
143 DAMOS_LRU_PRIO,144 DAMOS_LRU_PRIO,
144 DAMOS_LRU_DEPRIO,145 DAMOS_LRU_DEPRIO,
145 DAMOS_MIGRATE_HOT,146 DAMOS_MIGRATE_HOT,
@@ -159,6 +160,8 @@ enum damos_action { · 13 unmodified lines
159 * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.160 * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.
160 * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.161 * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.
161 * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.162 * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.
163 * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a
164 * node in basis points (0-10000).
162 * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.165 * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.
163 *166 *
164 * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.167 * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@@ -172,6 +175,7 @@ enum damos_quota_goal_metric { · 7 unmodified lines
172 DAMOS_QUOTA_NODE_MEMCG_FREE_BP,175 DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
173 DAMOS_QUOTA_ACTIVE_MEM_BP,176 DAMOS_QUOTA_ACTIVE_MEM_BP,
174 DAMOS_QUOTA_INACTIVE_MEM_BP,177 DAMOS_QUOTA_INACTIVE_MEM_BP,
178 DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
175 NR_DAMOS_QUOTA_GOAL_METRICS,179 NR_DAMOS_QUOTA_GOAL_METRICS,
176};180};
177 181
@@ -233,6 +237,8 @@ enum damos_quota_goal_tuner { · 55 unmodified lines
233 * @goals: Head of quota tuning goals (&damos_quota_goal) list.237 * @goals: Head of quota tuning goals (&damos_quota_goal) list.
234 * @goal_tuner: Goal-based @esz tuning algorithm to use.238 * @goal_tuner: Goal-based @esz tuning algorithm to use.
235 * @esz: Effective size quota in bytes.239 * @esz: Effective size quota in bytes.
240 * @fail_charge_num: Failed regions charge rate numerator.
241 * @fail_charge_denom: Failed regions charge rate denominator.
236 *242 *
237 * @weight_sz: Weight of the region's size for prioritization.243 * @weight_sz: Weight of the region's size for prioritization.
238 * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.244 * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.
@@ -262,6 +268,10 @@ enum damos_quota_goal_tuner { · 23 unmodified lines
262 *268 *
263 * The resulting effective size quota in bytes is set to @esz.269 * The resulting effective size quota in bytes is set to @esz.
264 *270 *
271 * For DAMOS action applying failed amount of regions, charging those same to
272 * those that the action has successfully applied may be unfair. For the
273 * reason, 'the size * @fail_charge_num / @fail_charge_denom' is charged.
274 *
265 * For selecting regions within the quota, DAMON prioritizes current scheme's275 * For selecting regions within the quota, DAMON prioritizes current scheme's
266 * target memory regions using the &struct damon_operations->get_scheme_score.276 * target memory regions using the &struct damon_operations->get_scheme_score.
267 * You could customize the prioritization logic by setting &weight_sz,277 * You could customize the prioritization logic by setting &weight_sz,
@@ -276,6 +286,9 @@ struct damos_quota { · 8 unmodified lines
276 enum damos_quota_goal_tuner goal_tuner;286 enum damos_quota_goal_tuner goal_tuner;
277 unsigned long esz;287 unsigned long esz;
278 288
289 unsigned int fail_charge_num;
290 unsigned int fail_charge_denom;
291
279 unsigned int weight_sz;292 unsigned int weight_sz;
280 unsigned int weight_nr_accesses;293 unsigned int weight_nr_accesses;
281 unsigned int weight_age;294 unsigned int weight_age;
@@ -617,6 +630,7 @@ enum damon_ops_id { · 335 unmodified lines
617 * @update: Update operations-related data structures.630 * @update: Update operations-related data structures.
618 * @prepare_access_checks: Prepare next access check of target regions.631 * @prepare_access_checks: Prepare next access check of target regions.
619 * @check_accesses: Check the accesses to target regions.632 * @check_accesses: Check the accesses to target regions.
633 * @apply_probes: Apply probes for each region.
620 * @get_scheme_score: Get the score of a region for a scheme.634 * @get_scheme_score: Get the score of a region for a scheme.
621 * @apply_scheme: Apply a DAMON-based operation scheme.635 * @apply_scheme: Apply a DAMON-based operation scheme.
622 * @target_valid: Determine if the target is valid.636 * @target_valid: Determine if the target is valid.
@@ -643,6 +657,8 @@ enum damon_ops_id { · 20 unmodified lines
643 * last preparation and update the number of observed accesses of each region.657 * last preparation and update the number of observed accesses of each region.
644 * It should also return max number of observed accesses that made as a result658 * It should also return max number of observed accesses that made as a result
645 * of its update. The value will be used for regions adjustment threshold.659 * of its update. The value will be used for regions adjustment threshold.
660 * @apply_probes should apply the data attribute probes to each region and
661 * accordingly update the probe hits counter of the region.
646 * @get_scheme_score should return the priority score of a region for a scheme662 * @get_scheme_score should return the priority score of a region for a scheme
647 * as an integer in [0, &DAMOS_MAX_SCORE].663 * as an integer in [0, &DAMOS_MAX_SCORE].
648 * @apply_scheme is called from @kdamond when a region for user provided664 * @apply_scheme is called from @kdamond when a region for user provided
@@ -660,6 +676,7 @@ struct damon_operations { · 11 unmodified lines
660 void (*update)(struct damon_ctx *context);676 void (*update)(struct damon_ctx *context);
661 void (*prepare_access_checks)(struct damon_ctx *context);677 void (*prepare_access_checks)(struct damon_ctx *context);
662 unsigned int (*check_accesses)(struct damon_ctx *context);678 unsigned int (*check_accesses)(struct damon_ctx *context);
679 void (*apply_probes)(struct damon_ctx *context);
663 int (*get_scheme_score)(struct damon_ctx *context,680 int (*get_scheme_score)(struct damon_ctx *context,
664 struct damon_region *r, struct damos *scheme);681 struct damon_region *r, struct damos *scheme);
665 unsigned long (*apply_scheme)(struct damon_ctx *context,682 unsigned long (*apply_scheme)(struct damon_ctx *context,
@@ -721,6 +738,47 @@ struct damon_intervals_goal { · 55 unmodified lines
721 unsigned long max_sample_us;738 unsigned long max_sample_us;
722};739};
723 740
741/**
742 * enum damon_filter_type - Type of &struct damon_filter
743 *
744 * @DAMON_FILTER_TYPE_ANON: Anonymous pages.
745 * @DAMON_FILTER_TYPE_MEMCG: Specific memcg's pages.
746 */
747enum damon_filter_type {
748 DAMON_FILTER_TYPE_ANON,
749 DAMON_FILTER_TYPE_MEMCG,
750};
751
752/**
753 * struct damon_filter - DAMON region filter for &struct damon_probe.
754 *
755 * @type: Type of the region.
756 * @matching: Whether this filter is for the type-matching ones.
757 * @allow: Whether the @type-@matching ones should pass this filter.
758 * @memcg_id: Memcg id of the question if @type is DAMON_FILTER_MEMCG.
759 * @list: Siblings list.
760 */
761struct damon_filter {
762 enum damon_filter_type type;
763 bool matching;
764 bool allow;
765 union {
766 u64 memcg_id;
767 };
768 struct list_head list;
769};
770
771/**
772 * struct damon_probe - Data region attribute probe.
773 *
774 * @filters: Filters for assessing if a given region is for this probe.
775 * @list: Siblings list.
776 */
777struct damon_probe {
778 struct list_head filters;
779 struct list_head list;
780};
781
724/**782/**
725 * struct damon_attrs - Monitoring attributes for accuracy/overhead control.783 * struct damon_attrs - Monitoring attributes for accuracy/overhead control.
726 *784 *
@@ -787,6 +845,7 @@ struct damon_attrs { · 60 unmodified lines
787 * @ops: Set of monitoring operations for given use cases.845 * @ops: Set of monitoring operations for given use cases.
788 * @addr_unit: Scale factor for core to ops address conversion.846 * @addr_unit: Scale factor for core to ops address conversion.
789 * @min_region_sz: Minimum region size.847 * @min_region_sz: Minimum region size.
848 * @pause: Pause kdamond main loop.
790 * @adaptive_targets: Head of monitoring targets (&damon_target) list.849 * @adaptive_targets: Head of monitoring targets (&damon_target) list.
791 * @schemes: Head of schemes (&damos) list.850 * @schemes: Head of schemes (&damos) list.
792 */851 */
@@ -838,13 +897,34 @@ struct damon_ctx { · 45 unmodified lines
838 897
839/* public: */898/* public: */
840 struct damon_operations ops;899 struct damon_operations ops;
900 struct list_head probes;
841 unsigned long addr_unit;901 unsigned long addr_unit;
842 unsigned long min_region_sz;902 unsigned long min_region_sz;
903 bool pause;
843 904
844 struct list_head adaptive_targets;905 struct list_head adaptive_targets;
845 struct list_head schemes;906 struct list_head schemes;
907
908 /* Per-ctx PRNG state for damon_rand(); kdamond is the sole consumer. */
909 struct rnd_state rnd_state;
846};910};
847 911
912/* Get a random number in [@l, @r) using @ctx's lockless PRNG. */
913static inline unsigned long damon_rand(struct damon_ctx *ctx,
914 unsigned long l, unsigned long r)
915{
916 unsigned long span = r - l;
917 u64 rnd;
918
919 if (span <= U32_MAX) {
920 rnd = prandom_u32_state(&ctx->rnd_state);
921 return l + (unsigned long)((rnd * span) >> 32);
922 }
923 rnd = ((u64)prandom_u32_state(&ctx->rnd_state) << 32) |
924 prandom_u32_state(&ctx->rnd_state);
925 return l + mul_u64_u64_shr(rnd, span, 64);
926}
927
848static inline struct damon_region *damon_next_region(struct damon_region *r)928static inline struct damon_region *damon_next_region(struct damon_region *r)
849{929{
850 return container_of(r->list.next, struct damon_region, list);930 return container_of(r->list.next, struct damon_region, list);
@@ -870,15 +950,26 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 19 unmodified lines
870 return r->ar.end - r->ar.start;950 return r->ar.end - r->ar.start;
871}951}
872 952
953#define damon_for_each_filter(f, p) \
954 list_for_each_entry(f, &(p)->filters, list)
955
956#define damon_for_each_filter_safe(f, next, p) \
957 list_for_each_entry_safe(f, next, &(p)->filters, list)
958
959#define damon_for_each_probe(p, ctx) \
960 list_for_each_entry(p, &(ctx)->probes, list)
961
962#define damon_for_each_probe_safe(p, next, ctx) \
963 list_for_each_entry_safe(p, next, &(ctx)->probes, list)
873 964
874#define damon_for_each_region(r, t) \965#define damon_for_each_region(r, t) \
875 list_for_each_entry(r, &t->regions_list, list)966 list_for_each_entry(r, &(t)->regions_list, list)
876 967
877#define damon_for_each_region_from(r, t) \968#define damon_for_each_region_from(r, t) \
878 list_for_each_entry_from(r, &t->regions_list, list)969 list_for_each_entry_from(r, &(t)->regions_list, list)
879 970
880#define damon_for_each_region_safe(r, next, t) \971#define damon_for_each_region_safe(r, next, t) \
881 list_for_each_entry_safe(r, next, &t->regions_list, list)972 list_for_each_entry_safe(r, next, &(t)->regions_list, list)
882 973
883#define damon_for_each_target(t, ctx) \974#define damon_for_each_target(t, ctx) \
884 list_for_each_entry(t, &(ctx)->adaptive_targets, list)975 list_for_each_entry(t, &(ctx)->adaptive_targets, list)
@@ -893,7 +984,7 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 8 unmodified lines
893 list_for_each_entry_safe(s, next, &(ctx)->schemes, list)984 list_for_each_entry_safe(s, next, &(ctx)->schemes, list)
894 985
895#define damos_for_each_quota_goal(goal, quota) \986#define damos_for_each_quota_goal(goal, quota) \
896 list_for_each_entry(goal, &quota->goals, list)987 list_for_each_entry(goal, &(quota)->goals, list)
897 988
898#define damos_for_each_quota_goal_safe(goal, next, quota) \989#define damos_for_each_quota_goal_safe(goal, next, quota) \
899 list_for_each_entry_safe(goal, next, &(quota)->goals, list)990 list_for_each_entry_safe(goal, next, &(quota)->goals, list)
@@ -912,21 +1003,16 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 12 unmodified lines
912 1003
913#ifdef CONFIG_DAMON1004#ifdef CONFIG_DAMON
914 1005
915struct damon_region *damon_new_region(unsigned long start, unsigned long end);1006struct damon_filter *damon_new_filter(enum damon_filter_type type,
1007 bool matching, bool allow);
1008void damon_add_filter(struct damon_probe *probe, struct damon_filter *f);
1009void damon_destroy_filter(struct damon_filter *f);
916 1010
917/*1011struct damon_probe *damon_new_probe(void);
918 * Add a region between two other regions1012void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe);
919 */1013
920static inline void damon_insert_region(struct damon_region *r,1014struct damon_region *damon_new_region(unsigned long start, unsigned long end);
921 struct damon_region *prev, struct damon_region *next,
922 struct damon_target *t)
923{
924 __list_add(&r->list, &prev->list, &next->list);
925 t->nr_regions++;
926}
927 1015
928void damon_add_region(struct damon_region *r, struct damon_target *t);
929void damon_destroy_region(struct damon_region *r, struct damon_target *t);
930int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,1016int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
931 unsigned int nr_ranges, unsigned long min_region_sz);1017 unsigned int nr_ranges, unsigned long min_region_sz);
932void damon_update_region_access_rate(struct damon_region *r, bool accessed,1018void damon_update_region_access_rate(struct damon_region *r, bool accessed,
@@ -994,7 +1080,7 @@ int damon_kdamond_pid(struct damon_ctx *ctx); · 61 unmodified lines
994int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);1080int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
995int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);1081int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
996 1082
997int damon_set_region_biggest_system_ram_default(struct damon_target *t,1083int damon_set_region_system_rams_default(struct damon_target *t,
998 unsigned long *start, unsigned long *end,1084 unsigned long *start, unsigned long *end,
999 unsigned long addr_unit,1085 unsigned long addr_unit,
1000 unsigned long min_region_sz);1086 unsigned long min_region_sz);
include/linux/gfp.h+4−0
@@ -239,6 +239,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 238 unmodified lines
239 struct page **page_array);239 struct page **page_array);
240#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))240#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))
241 241
242void free_pages_bulk(struct page **page_array, unsigned long nr_pages);
243
242unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,244unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
243 unsigned long nr_pages,245 unsigned long nr_pages,
244 struct page **page_array);246 struct page **page_array);
@@ -467,6 +469,8 @@ void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages); · 222 unmodified lines
467void free_contig_range(unsigned long pfn, unsigned long nr_pages);469void free_contig_range(unsigned long pfn, unsigned long nr_pages);
468#endif470#endif
469 471
472void __free_contig_range(unsigned long pfn, unsigned long nr_pages);
473
470DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))474DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))
471 475
472#endif /* __LINUX_GFP_H */476#endif /* __LINUX_GFP_H */
include/linux/gfp_types.h+3−3
@@ -281,9 +281,9 @@ enum { · 280 unmodified lines
281 *281 *
282 * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.282 * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.
283 * Used for userspace and vmalloc pages; the latter are unpoisoned by283 * Used for userspace and vmalloc pages; the latter are unpoisoned by
284 * kasan_unpoison_vmalloc instead. For userspace pages, results in284 * kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc
285 * poisoning being skipped as well, see should_skip_kasan_poison for285 * is skipped too. For userspace pages, results in poisoning being skipped as
286 * details. Only effective in HW_TAGS mode.286 * well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode.
287 */287 */
288#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)288#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
289#define __GFP_COMP ((__force gfp_t)___GFP_COMP)289#define __GFP_COMP ((__force gfp_t)___GFP_COMP)
include/linux/highmem-internal.h+1−1
@@ -262,7 +262,7 @@ static inline bool is_kmap_addr(const void *x) · 261 unmodified lines
262 * @__addr: Virtual address to be unmapped262 * @__addr: Virtual address to be unmapped
263 *263 *
264 * Unmaps an address previously mapped by kmap_atomic() and re-enables264 * Unmaps an address previously mapped by kmap_atomic() and re-enables
265 * pagefaults. Depending on PREEMP_RT configuration, re-enables also265 * pagefaults. Depending on PREEMPT_RT configuration, re-enables also
266 * migration and preemption. Users should not count on these side effects.266 * migration and preemption. Users should not count on these side effects.
267 *267 *
268 * Mappings should be unmapped in the reverse order that they were mapped.268 * Mappings should be unmapped in the reverse order that they were mapped.
include/linux/huge_mm.h+42−5
@@ -237,6 +237,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 236 unmodified lines
237 return true;237 return true;
238}238}
239 239
240/*
241 * Make sure huge_gfp is always more limited than limit_gfp.
242 * Some shmem users want THP allocation to be done less aggressively
243 * and only in certain zone.
244 */
245static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
246{
247 gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
248 gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
249 gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
250 gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
251
252 /* Allow allocations only from the originally specified zones. */
253 result |= zoneflags;
254
255 /*
256 * Minimize the result gfp by taking the union with the deny flags,
257 * and the intersection of the allow flags.
258 */
259 result |= (limit_gfp & denyflags);
260 result |= (huge_gfp & limit_gfp) & allowflags;
261
262 return result;
263}
264
240/*265/*
241 * Filter the bitfield of input orders to the ones suitable for use in the vma.266 * Filter the bitfield of input orders to the ones suitable for use in the vma.
242 * See thp_vma_suitable_order().267 * See thp_vma_suitable_order().
@@ -414,10 +439,10 @@ static inline int split_huge_page(struct page *page) · 171 unmodified lines
414{439{
415 return split_huge_page_to_list_to_order(page, NULL, 0);440 return split_huge_page_to_list_to_order(page, NULL, 0);
416}441}
442
443int folio_memcg_alloc_deferred(struct folio *folio);
444
417void deferred_split_folio(struct folio *folio, bool partially_mapped);445void deferred_split_folio(struct folio *folio, bool partially_mapped);
418#ifdef CONFIG_MEMCG
419void reparent_deferred_split_queue(struct mem_cgroup *memcg);
420#endif
421 446
422void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,447void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
423 unsigned long address, bool freeze);448 unsigned long address, bool freeze);
@@ -581,6 +606,11 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 157 unmodified lines
581 return false;606 return false;
582}607}
583 608
609static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
610{
611 return huge_gfp;
612}
613
584static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,614static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
585 unsigned long addr, unsigned long orders)615 unsigned long addr, unsigned long orders)
586{616{
@@ -649,8 +679,15 @@ static inline int try_folio_split_to_order(struct folio *folio, · 62 unmodified lines
649 return -EINVAL;679 return -EINVAL;
650}680}
651 681
652static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}682static inline int folio_memcg_alloc_deferred(struct folio *folio)
653static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}683{
684 return 0;
685}
686
687static inline void deferred_split_folio(struct folio *folio, bool partially_mapped)
688{
689}
690
654#define split_huge_pmd(__vma, __pmd, __address) \691#define split_huge_pmd(__vma, __pmd, __address) \
655 do { } while (0)692 do { } while (0)
656 693
include/linux/list_lru.h+70−0
@@ -81,8 +81,75 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker · 80 unmodified lines
81 81
82int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,82int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
83 gfp_t gfp);83 gfp_t gfp);
84
85#ifdef CONFIG_MEMCG
86/**
87 * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
88 * @folio: the newly allocated & charged folio
89 * @lru: the list_lru this might be queued on
90 * @gfp: gfp mask
91 *
92 * Allocate list_lru heads (per-memcg, per-node) needed to queue this
93 * particular folio down the line.
94 *
95 * This does memcg_list_lru_alloc(), but on the memcg that @folio is
96 * associated with. Handles folio_memcg() access rules in the fast
97 * path (list_lru heads allocated) and the allocation slowpath.
98 *
99 * Returns 0 on success, a negative error value otherwise.
100 */
101int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
102 gfp_t gfp);
103#else
104static inline int folio_memcg_list_lru_alloc(struct folio *folio,
105 struct list_lru *lru, gfp_t gfp)
106{
107 return 0;
108}
109#endif
110
84void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);111void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
85 112
113/**
114 * list_lru_lock: lock the sublist for the given node and memcg
115 * @lru: the lru pointer
116 * @nid: the node id of the sublist to lock.
117 * @memcg: pointer to the cgroup of the sublist to lock. On return,
118 * updated to the cgroup whose sublist was actually locked,
119 * which may be an ancestor if the original memcg was dying.
120 *
121 * Returns the locked list_lru_one sublist. The caller must call
122 * list_lru_unlock() when done.
123 *
124 * You must ensure that the memcg is not freed during this call (e.g., with
125 * rcu or by taking a css refcnt).
126 *
127 * Return: the locked list_lru_one, or NULL on failure
128 */
129struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
130 struct mem_cgroup **memcg);
131
132/**
133 * list_lru_unlock: unlock a sublist locked by list_lru_lock()
134 * @l: the list_lru_one to unlock
135 */
136void list_lru_unlock(struct list_lru_one *l);
137
138struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
139 struct mem_cgroup **memcg);
140void list_lru_unlock_irq(struct list_lru_one *l);
141
142struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
143 struct mem_cgroup **memcg, unsigned long *irq_flags);
144void list_lru_unlock_irqrestore(struct list_lru_one *l,
145 unsigned long *irq_flags);
146
147/* Caller-locked variants, see list_lru_add() etc for documentation */
148bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
149 struct list_head *item, int nid, struct mem_cgroup *memcg);
150bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
151 struct list_head *item, int nid);
152
86/**153/**
87 * list_lru_add: add an element to the lru list's tail154 * list_lru_add: add an element to the lru list's tail
88 * @lru: the lru pointer155 * @lru: the lru pointer
@@ -115,6 +182,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren · 26 unmodified lines
115bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,182bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
116 struct mem_cgroup *memcg);183 struct mem_cgroup *memcg);
117 184
185bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
186 struct mem_cgroup *memcg);
187
118/**188/**
119 * list_lru_add_obj: add an element to the lru list's tail189 * list_lru_add_obj: add an element to the lru list's tail
120 * @lru: the lru pointer190 * @lru: the lru pointer
include/linux/memcontrol.h+17−14
@@ -29,6 +29,7 @@ struct obj_cgroup; · 28 unmodified lines
29struct page;29struct page;
30struct mm_struct;30struct mm_struct;
31struct kmem_cache;31struct kmem_cache;
32struct swap_cluster_info;
32 33
33/* Cgroup-specific page state, on top of universal node page state */34/* Cgroup-specific page state, on top of universal node page state */
34enum memcg_stat_item {35enum memcg_stat_item {
@@ -277,10 +278,6 @@ struct mem_cgroup { · 242 unmodified lines
277 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];278 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
278#endif279#endif
279 280
280#ifdef CONFIG_TRANSPARENT_HUGEPAGE
281 struct deferred_split deferred_split_queue;
282#endif
283
284#ifdef CONFIG_LRU_GEN_WALKS_MMU281#ifdef CONFIG_LRU_GEN_WALKS_MMU
285 /* per-memcg mm_struct list */282 /* per-memcg mm_struct list */
286 struct lru_gen_mm_list mm_list;283 struct lru_gen_mm_list mm_list;
@@ -646,8 +643,8 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 359 unmodified lines
646 643
647int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);644int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
648 645
649int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,646int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
650 gfp_t gfp, swp_entry_t entry);647 struct mm_struct *mm, gfp_t gfp);
651 648
652void __mem_cgroup_uncharge(struct folio *folio);649void __mem_cgroup_uncharge(struct folio *folio);
653 650
@@ -1137,7 +1134,7 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp) · 483 unmodified lines
1137}1134}
1138 1135
1139static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,1136static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
1140 struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)1137 unsigned short id, struct mm_struct *mm, gfp_t gfp)
1141{1138{
1142 return 0;1139 return 0;
1143}1140}
@@ -1899,9 +1896,6 @@ static inline void mem_cgroup_exit_user_fault(void) · 755 unmodified lines
1899 current->in_user_fault = 0;1896 current->in_user_fault = 0;
1900}1897}
1901 1898
1902void memcg1_swapout(struct folio *folio, swp_entry_t entry);
1903void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages);
1904
1905#else /* CONFIG_MEMCG_V1 */1899#else /* CONFIG_MEMCG_V1 */
1906static inline1900static inline
1907unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,1901unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
@@ -1929,14 +1923,23 @@ static inline void mem_cgroup_exit_user_fault(void) · 21 unmodified lines
1929{1923{
1930}1924}
1931 1925
1932static inline void memcg1_swapout(struct folio *folio, swp_entry_t entry)1926#endif /* CONFIG_MEMCG_V1 */
1927
1928#if defined(CONFIG_MEMCG_V1) && defined(CONFIG_SWAP)
1929
1930void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci);
1931void memcg1_swapin(struct folio *folio);
1932
1933#else
1934
1935static inline void __memcg1_swapout(struct folio *folio,
1936 struct swap_cluster_info *ci)
1933{1937{
1934}1938}
1935 1939
1936static inline void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)1940static inline void memcg1_swapin(struct folio *folio)
1937{1941{
1938}1942}
1939 1943#endif
1940#endif /* CONFIG_MEMCG_V1 */
1941 1944
1942#endif /* _LINUX_MEMCONTROL_H */1945#endif /* _LINUX_MEMCONTROL_H */
include/linux/memory.h+5−2
@@ -158,7 +158,11 @@ int create_memory_block_devices(unsigned long start, unsigned long size, · 157 unmodified lines
158void remove_memory_block_devices(unsigned long start, unsigned long size);158void remove_memory_block_devices(unsigned long start, unsigned long size);
159extern void memory_dev_init(void);159extern void memory_dev_init(void);
160extern int memory_notify(enum memory_block_state state, void *v);160extern int memory_notify(enum memory_block_state state, void *v);
161extern struct memory_block *find_memory_block(unsigned long section_nr);161struct memory_block *memory_block_get(unsigned long block_id);
162static inline void memory_block_put(struct memory_block *mem)
163{
164 put_device(&mem->dev);
165}
162typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);166typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);
163extern int walk_memory_blocks(unsigned long start, unsigned long size,167extern int walk_memory_blocks(unsigned long start, unsigned long size,
164 void *arg, walk_memory_blocks_func_t func);168 void *arg, walk_memory_blocks_func_t func);
@@ -171,7 +175,6 @@ struct memory_group *memory_group_find_by_id(int mgid); · 6 unmodified lines
171typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);175typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);
172int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,176int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
173 struct memory_group *excluded, void *arg);177 struct memory_group *excluded, void *arg);
174struct memory_block *find_memory_block_by_id(unsigned long block_id);
175#define hotplug_memory_notifier(fn, pri) ({ \178#define hotplug_memory_notifier(fn, pri) ({ \
176 static __meminitdata struct notifier_block fn##_mem_nb =\179 static __meminitdata struct notifier_block fn##_mem_nb =\
177 { .notifier_call = fn, .priority = pri };\180 { .notifier_call = fn, .priority = pri };\
include/linux/memory_hotplug.h+5−3
@@ -135,9 +135,10 @@ static inline bool movable_node_is_enabled(void) · 134 unmodified lines
135 return movable_node_enabled;135 return movable_node_enabled;
136}136}
137 137
138extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap);138extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
139 struct dev_pagemap *pgmap);
139extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,140extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
140 struct vmem_altmap *altmap);141 struct vmem_altmap *altmap, struct dev_pagemap *pgmap);
141 142
142/* reasonably generic interface to expand the physical pages */143/* reasonably generic interface to expand the physical pages */
143extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,144extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -307,7 +308,8 @@ extern int sparse_add_section(int nid, unsigned long pfn, · 163 unmodified lines
307 unsigned long nr_pages, struct vmem_altmap *altmap,308 unsigned long nr_pages, struct vmem_altmap *altmap,
308 struct dev_pagemap *pgmap);309 struct dev_pagemap *pgmap);
309extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,310extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
310 struct vmem_altmap *altmap);311 struct vmem_altmap *altmap,
312 struct dev_pagemap *pgmap);
311extern struct zone *zone_for_pfn_range(enum mmop online_type,313extern struct zone *zone_for_pfn_range(enum mmop online_type,
312 int nid, struct memory_group *group, unsigned long start_pfn,314 int nid, struct memory_group *group, unsigned long start_pfn,
313 unsigned long nr_pages);315 unsigned long nr_pages);
include/linux/mm.h+44−18
@@ -496,6 +496,21 @@ enum { · 495 unmodified lines
496#else496#else
497#define VM_UFFD_MINOR VM_NONE497#define VM_UFFD_MINOR VM_NONE
498#endif498#endif
499
500/*
501 * vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on
502 * the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits
503 * -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose
504 * bitmap cannot hold it.
505 */
506#define VMA_UFFD_MISSING mk_vma_flags(VMA_UFFD_MISSING_BIT)
507#define VMA_UFFD_WP mk_vma_flags(VMA_UFFD_WP_BIT)
508#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
509#define VMA_UFFD_MINOR mk_vma_flags(VMA_UFFD_MINOR_BIT)
510#else
511#define VMA_UFFD_MINOR EMPTY_VMA_FLAGS
512#endif
513
499#ifdef CONFIG_64BIT514#ifdef CONFIG_64BIT
500#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)515#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
501#define VM_SEALED INIT_VM_FLAG(SEALED)516#define VM_SEALED INIT_VM_FLAG(SEALED)
@@ -1238,6 +1253,30 @@ static __always_inline void vma_flags_set_mask(vma_flags_t *flags, · 736 unmodified lines
1238#define vma_flags_set(flags, ...) \1253#define vma_flags_set(flags, ...) \
1239 vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))1254 vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))
1240 1255
1256static __always_inline vma_flags_t __mk_vma_flags_from_masks(size_t count,
1257 const vma_flags_t *masks)
1258{
1259 vma_flags_t flags = EMPTY_VMA_FLAGS;
1260 size_t i;
1261
1262 for (i = 0; i < count; i++)
1263 vma_flags_set_mask(&flags, masks[i]);
1264 return flags;
1265}
1266
1267/*
1268 * Combine pre-computed vma_flags_t masks into one value, e.g.:
1269 *
1270 * vma_flags_t flags = mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR);
1271 *
1272 * Unlike mk_vma_flags(), which takes bit numbers, this takes whole masks --
1273 * each of which may be EMPTY_VMA_FLAGS when its feature is unavailable -- so a
1274 * bit that does not exist on the current build is never materialised.
1275 */
1276#define mk_vma_flags_from_masks(...) \
1277 __mk_vma_flags_from_masks(COUNT_ARGS(__VA_ARGS__), \
1278 (const vma_flags_t []){__VA_ARGS__})
1279
1241/* Clear all of the to-clear flags in flags, non-atomically. */1280/* Clear all of the to-clear flags in flags, non-atomically. */
1242static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,1281static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,
1243 vma_flags_t to_clear)1282 vma_flags_t to_clear)
@@ -1489,6 +1528,11 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma) · 245 unmodified lines
1489 vma->vm_ops = NULL;1528 vma->vm_ops = NULL;
1490}1529}
1491 1530
1531static inline void vma_desc_set_anonymous(struct vm_area_desc *desc)
1532{
1533 desc->vm_ops = NULL;
1534}
1535
1492static inline bool vma_is_anonymous(struct vm_area_struct *vma)1536static inline bool vma_is_anonymous(struct vm_area_struct *vma)
1493{1537{
1494 return !vma->vm_ops;1538 return !vma->vm_ops;
@@ -1888,16 +1932,6 @@ static inline bool folio_mapped(const struct folio *folio) · 393 unmodified lines
1888 return folio_mapcount(folio) >= 1;1932 return folio_mapcount(folio) >= 1;
1889}1933}
1890 1934
1891/*
1892 * Return true if this page is mapped into pagetables.
1893 * For compound page it returns true if any sub-page of compound page is mapped,
1894 * even if this particular sub-page is not itself mapped by any PTE or PMD.
1895 */
1896static inline bool page_mapped(const struct page *page)
1897{
1898 return folio_mapped(page_folio(page));
1899}
1900
1901static inline struct page *virt_to_head_page(const void *x)1935static inline struct page *virt_to_head_page(const void *x)
1902{1936{
1903 struct page *page = virt_to_page(x);1937 struct page *page = virt_to_page(x);
@@ -4855,18 +4889,10 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) · 2951 unmodified lines
4855}4889}
4856#endif4890#endif
4857 4891
4858void *sparse_buffer_alloc(unsigned long size);
4859unsigned long section_map_size(void);4892unsigned long section_map_size(void);
4860struct page * __populate_section_memmap(unsigned long pfn,4893struct page * __populate_section_memmap(unsigned long pfn,
4861 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,4894 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
4862 struct dev_pagemap *pgmap);4895 struct dev_pagemap *pgmap);
4863pgd_t *vmemmap_pgd_populate(unsigned long addr, int node);
4864p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node);
4865pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node);
4866pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node);
4867pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
4868 struct vmem_altmap *altmap, unsigned long ptpfn,
4869 unsigned long flags);
4870void *vmemmap_alloc_block(unsigned long size, int node);4896void *vmemmap_alloc_block(unsigned long size, int node);
4871struct vmem_altmap;4897struct vmem_altmap;
4872void *vmemmap_alloc_block_buf(unsigned long size, int node,4898void *vmemmap_alloc_block_buf(unsigned long size, int node,
include/linux/mm_inline.h+1−1
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · 246 unmodified lines
247 (folio_test_dirty(folio) || folio_test_writeback(folio))))247 (folio_test_dirty(folio) || folio_test_writeback(folio))))
248 gen = MIN_NR_GENS;248 gen = MIN_NR_GENS;
249 else249 else
250 gen = MAX_NR_GENS - folio_test_workingset(folio);250 gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
251 251
252 return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));252 return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
253}253}
include/linux/mm_types.h+3−16
@@ -845,23 +845,10 @@ struct mmap_action { · 844 unmodified lines
845 enum mmap_action_type type;845 enum mmap_action_type type;
846 846
847 /*847 /*
848 * If specified, this hook is invoked after the selected action has been848 * If non-zero, replace errors that arise from mmap actions with this
849 * successfully completed. Note that the VMA write lock still held.849 * value instead. Only valid error codes may be specified.
850 *
851 * The absolute minimum ought to be done here.
852 *
853 * Returns 0 on success, or an error code.
854 */
855 int (*success_hook)(const struct vm_area_struct *vma);
856
857 /*
858 * If specified, this hook is invoked when an error occurred when
859 * attempting the selected action.
860 *
861 * The hook can return an error code in order to filter the error, but
862 * it is not valid to clear the error here.
863 */850 */
864 int (*error_hook)(int err);851 int error_override;
865 852
866 /*853 /*
867 * This should be set in rare instances where the operation required854 * This should be set in rare instances where the operation required
include/linux/mmu_notifier.h+2−2
@@ -134,8 +134,8 @@ struct mmu_notifier_ops { · 133 unmodified lines
134 * Invalidation of multiple concurrent ranges may be134 * Invalidation of multiple concurrent ranges may be
135 * optionally permitted by the driver. Either way the135 * optionally permitted by the driver. Either way the
136 * establishment of sptes is forbidden in the range passed to136 * establishment of sptes is forbidden in the range passed to
137 * invalidate_range_begin/end for the whole duration of the137 * invalidate_range_start/end for the whole duration of the
138 * invalidate_range_begin/end critical section.138 * invalidate_range_start/end critical section.
139 *139 *
140 * invalidate_range_start() is called when all pages in the140 * invalidate_range_start() is called when all pages in the
141 * range are still mapped and have at least a refcount of one.141 * range are still mapped and have at least a refcount of one.
include/linux/mmzone.h+6−17
@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt) · 176 unmodified lines
177 return mt < MIGRATE_PCPTYPES;177 return mt < MIGRATE_PCPTYPES;
178}178}
179 179
180#define for_each_migratetype_order(order, type) \180#define for_each_free_list(list, zone, order) \
181 for (order = 0; order < NR_PAGE_ORDERS; order++) \181 for (order = 0; order < NR_PAGE_ORDERS; order++) \
182 for (type = 0; type < MIGRATE_TYPES; type++)182 for (unsigned int __type = 0; \
183 __type < MIGRATE_TYPES && \
184 (list = &(zone)->free_area[order].free_list[__type], 1); \
185 __type++)
183 186
184extern int page_group_by_mobility_disabled;187extern int page_group_by_mobility_disabled;
185 188
@@ -211,7 +214,6 @@ enum numa_stat_item { · 25 unmodified lines
211#endif214#endif
212 215
213enum zone_stat_item {216enum zone_stat_item {
214 /* First 128 byte cacheline (assuming 64 bit words) */
215 NR_FREE_PAGES,217 NR_FREE_PAGES,
216 NR_FREE_PAGES_BLOCKS,218 NR_FREE_PAGES_BLOCKS,
217 NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */219 NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */
@@ -222,7 +224,6 @@ enum zone_stat_item { · 4 unmodified lines
222 NR_ZONE_UNEVICTABLE,224 NR_ZONE_UNEVICTABLE,
223 NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */225 NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */
224 NR_MLOCK, /* mlock()ed pages found and moved off LRU */226 NR_MLOCK, /* mlock()ed pages found and moved off LRU */
225 /* Second 128 byte cacheline */
226#if IS_ENABLED(CONFIG_ZSMALLOC)227#if IS_ENABLED(CONFIG_ZSMALLOC)
227 NR_ZSPAGES, /* allocated in zsmalloc */228 NR_ZSPAGES, /* allocated in zsmalloc */
228#endif229#endif
@@ -1428,14 +1429,6 @@ struct zonelist { · 1199 unmodified lines
1428 */1429 */
1429extern struct page *mem_map;1430extern struct page *mem_map;
1430 1431
1431#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1432struct deferred_split {
1433 spinlock_t split_queue_lock;
1434 struct list_head split_queue;
1435 unsigned long split_queue_len;
1436};
1437#endif
1438
1439#ifdef CONFIG_MEMORY_FAILURE1432#ifdef CONFIG_MEMORY_FAILURE
1440/*1433/*
1441 * Per NUMA node memory failure handling statistics.1434 * Per NUMA node memory failure handling statistics.
@@ -1561,10 +1554,6 @@ typedef struct pglist_data { · 119 unmodified lines
1561 unsigned long first_deferred_pfn;1554 unsigned long first_deferred_pfn;
1562#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */1555#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
1563 1556
1564#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1565 struct deferred_split deferred_split_queue;
1566#endif
1567
1568#ifdef CONFIG_NUMA_BALANCING1557#ifdef CONFIG_NUMA_BALANCING
1569 /* start time in ms of current promote rate limit period */1558 /* start time in ms of current promote rate limit period */
1570 unsigned int nbp_rl_start;1559 unsigned int nbp_rl_start;
include/linux/nodemask.h+9−9
@@ -24,23 +24,23 @@ · 23 unmodified lines
24 * void nodes_setall(mask) set all bits24 * void nodes_setall(mask) set all bits
25 * void nodes_clear(mask) clear all bits25 * void nodes_clear(mask) clear all bits
26 * int node_isset(node, mask) true iff bit 'node' set in mask26 * int node_isset(node, mask) true iff bit 'node' set in mask
27 * int node_test_and_set(node, mask) test and set bit 'node' in mask27 * bool node_test_and_set(node, mask) test and set bit 'node' in mask
28 *28 *
29 * void nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]29 * bool nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
30 * void nodes_or(dst, src1, src2) dst = src1 | src2 [union]30 * void nodes_or(dst, src1, src2) dst = src1 | src2 [union]
31 * void nodes_xor(dst, src1, src2) dst = src1 ^ src231 * void nodes_xor(dst, src1, src2) dst = src1 ^ src2
32 * void nodes_andnot(dst, src1, src2) dst = src1 & ~src232 * bool nodes_andnot(dst, src1, src2) dst = src1 & ~src2
33 * void nodes_complement(dst, src) dst = ~src33 * void nodes_complement(dst, src) dst = ~src
34 *34 *
35 * int nodes_equal(mask1, mask2) Does mask1 == mask2?35 * bool nodes_equal(mask1, mask2) Does mask1 == mask2?
36 * int nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?36 * bool nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
37 * int nodes_subset(mask1, mask2) Is mask1 a subset of mask2?37 * bool nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
38 * int nodes_empty(mask) Is mask empty (no bits sets)?38 * bool nodes_empty(mask) Is mask empty (no bits sets)?
39 * int nodes_full(mask) Is mask full (all bits sets)?39 * bool nodes_full(mask) Is mask full (all bits sets)?
40 * int nodes_weight(mask) Hamming weight - number of set bits40 * int nodes_weight(mask) Hamming weight - number of set bits
41 *41 *
42 * unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES42 * unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES
43 * unsigend int next_node(node, mask) Next node past 'node', or MAX_NUMNODES43 * unsigned int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
44 * unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,44 * unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,
45 * or MAX_NUMNODES45 * or MAX_NUMNODES
46 * unsigned int first_unset_node(mask) First node not set in mask, or46 * unsigned int first_unset_node(mask) First node not set in mask, or
include/linux/page_ref.h+18−0
@@ -71,6 +71,12 @@ static inline int page_ref_count(const struct page *page) · 70 unmodified lines
71 * folio_ref_count - The reference count on this folio.71 * folio_ref_count - The reference count on this folio.
72 * @folio: The folio.72 * @folio: The folio.
73 *73 *
74 * Folios contain a reference count. When that reference count reaches
75 * zero, the folio is referred to as frozen. At this point, it will
76 * usually be returned to the memory allocator, but some parts of the
77 * kernel freeze folios in order to perform unusual operations on them
78 * such as splitting or migration.
79 *
74 * The refcount is usually incremented by calls to folio_get() and80 * The refcount is usually incremented by calls to folio_get() and
75 * decremented by calls to folio_put(). Some typical users of the81 * decremented by calls to folio_put(). Some typical users of the
76 * folio refcount:82 * folio refcount:
@@ -82,6 +88,18 @@ static inline int page_ref_count(const struct page *page) · 5 unmodified lines
82 * - Pipes88 * - Pipes
83 * - Direct IO which references this page in the process address space89 * - Direct IO which references this page in the process address space
84 *90 *
91 * The reference count has three components: expected, temporary and
92 * spurious. The expected reference count of a folio is that which
93 * we would logically expect it to be from just reading the code.
94 * Temporary refcounts are gained by threads which need a temporary
95 * reference to make sure the folio isn't reallocated while they use it.
96 * Spurious refcounts are gained by threads which, thanks to RCU walks
97 * of the page tables or file cache, find a stale pointer to a folio.
98 * These threads will drop the refcount after discoveering the pointer
99 * is stale, but it can surprise other users to see the spurious refcount
100 * on a freshly allocated folio (eg they may see a refcount of 2 instead
101 * of 1).
102 *
85 * Return: The number of references to this folio.103 * Return: The number of references to this folio.
86 */104 */
87static inline int folio_ref_count(const struct folio *folio)105static inline int folio_ref_count(const struct folio *folio)
include/linux/pageblock-flags.h+3−3
@@ -36,12 +36,12 @@ enum pageblock_bits { · 35 unmodified lines
36 36
37#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))37#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))
38 38
39#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))39#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
40 40
41#ifdef CONFIG_MEMORY_ISOLATION41#ifdef CONFIG_MEMORY_ISOLATION
42#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate))42#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
43#else43#else
44#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK44#define PAGEBLOCK_ISO_MASK 0
45#endif45#endif
46 46
47#if defined(CONFIG_HUGETLB_PAGE)47#if defined(CONFIG_HUGETLB_PAGE)
include/linux/pagemap.h+2−0
@@ -1350,6 +1350,7 @@ struct readahead_control { · 1349 unmodified lines
1350 struct file_ra_state *ra;1350 struct file_ra_state *ra;
1351/* private: use the readahead_* accessors instead */1351/* private: use the readahead_* accessors instead */
1352 pgoff_t _index;1352 pgoff_t _index;
1353 pgoff_t _max_index; /* limit readahead to _max_index, inclusive */
1353 unsigned int _nr_pages;1354 unsigned int _nr_pages;
1354 unsigned int _batch_count;1355 unsigned int _batch_count;
1355 bool dropbehind;1356 bool dropbehind;
@@ -1363,6 +1364,7 @@ struct readahead_control { · 7 unmodified lines
1363 .mapping = m, \1364 .mapping = m, \
1364 .ra = r, \1365 .ra = r, \
1365 ._index = i, \1366 ._index = i, \
1367 ._max_index = ULONG_MAX, \
1366 }1368 }
1367 1369
1368#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)1370#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)
include/linux/swap.h+12−12
@@ -213,6 +213,7 @@ enum { · 212 unmodified lines
213 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */213 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
214 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */214 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
215 SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */215 SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
216 SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
216 /* add others here before... */217 /* add others here before... */
217};218};
218 219
@@ -252,7 +253,6 @@ struct swap_info_struct { · 33 unmodified lines
252 struct plist_node list; /* entry in swap_active_head */253 struct plist_node list; /* entry in swap_active_head */
253 signed char type; /* strange name for an index */254 signed char type; /* strange name for an index */
254 unsigned int max; /* size of this swap device */255 unsigned int max; /* size of this swap device */
255 unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
256 struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */256 struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
257 struct list_head free_clusters; /* free clusters list */257 struct list_head free_clusters; /* free clusters list */
258 struct list_head full_clusters; /* full clusters list */258 struct list_head full_clusters; /* full clusters list */
@@ -433,7 +433,9 @@ static inline long get_nr_swap_pages(void) · 174 unmodified lines
433}433}
434 434
435extern void si_swapinfo(struct sysinfo *);435extern void si_swapinfo(struct sysinfo *);
436int swap_type_of(dev_t device, sector_t offset);436extern int pin_hibernation_swap_type(dev_t device, sector_t offset);
437extern void unpin_hibernation_swap_type(int type);
438extern int find_hibernation_swap_type(dev_t device, sector_t offset);
437int find_first_swap(dev_t *device);439int find_first_swap(dev_t *device);
438extern unsigned int count_swap_pages(int, int);440extern unsigned int count_swap_pages(int, int);
439extern sector_t swapdev_block(int, pgoff_t);441extern sector_t swapdev_block(int, pgoff_t);
@@ -571,33 +573,31 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp) · 131 unmodified lines
571#endif573#endif
572 574
573#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)575#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
574int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);576int __mem_cgroup_try_charge_swap(struct folio *folio);
575static inline int mem_cgroup_try_charge_swap(struct folio *folio,577static inline int mem_cgroup_try_charge_swap(struct folio *folio)
576 swp_entry_t entry)
577{578{
578 if (mem_cgroup_disabled())579 if (mem_cgroup_disabled())
579 return 0;580 return 0;
580 return __mem_cgroup_try_charge_swap(folio, entry);581 return __mem_cgroup_try_charge_swap(folio);
581}582}
582 583
583extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);584extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
584static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)585static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
585{586{
586 if (mem_cgroup_disabled())587 if (mem_cgroup_disabled())
587 return;588 return;
588 __mem_cgroup_uncharge_swap(entry, nr_pages);589 __mem_cgroup_uncharge_swap(id, nr_pages);
589}590}
590 591
591extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);592extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
592extern bool mem_cgroup_swap_full(struct folio *folio);593extern bool mem_cgroup_swap_full(struct folio *folio);
593#else594#else
594static inline int mem_cgroup_try_charge_swap(struct folio *folio,595static inline int mem_cgroup_try_charge_swap(struct folio *folio)
595 swp_entry_t entry)
596{596{
597 return 0;597 return 0;
598}598}
599 599
600static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,600static inline void mem_cgroup_uncharge_swap(unsigned short id,
601 unsigned int nr_pages)601 unsigned int nr_pages)
602{602{
603}603}
include/linux/swap_cgroup.h+0−47
@@ -1,47 +0,0 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2#ifndef __LINUX_SWAP_CGROUP_H
3#define __LINUX_SWAP_CGROUP_H
4
5#include <linux/swap.h>
6
7#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
8
9extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
10extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
11extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
12extern int swap_cgroup_swapon(int type, unsigned long max_pages);
13extern void swap_cgroup_swapoff(int type);
14
15#else
16
17static inline
18void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
19{
20}
21
22static inline
23unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
24{
25 return 0;
26}
27
28static inline
29unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
30{
31 return 0;
32}
33
34static inline int
35swap_cgroup_swapon(int type, unsigned long max_pages)
36{
37 return 0;
38}
39
40static inline void swap_cgroup_swapoff(int type)
41{
42 return;
43}
44
45#endif
46
47#endif /* __LINUX_SWAP_CGROUP_H */
include/linux/thread_info.h+1−1
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart, · 91 unmodified lines
92#define THREAD_ALIGN THREAD_SIZE92#define THREAD_ALIGN THREAD_SIZE
93#endif93#endif
94 94
95#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO)95#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)
96 96
97/*97/*
98 * flag set/clear/test wrappers98 * flag set/clear/test wrappers
include/linux/userfaultfd_k.h+2−38
@@ -23,8 +23,8 @@ · 22 unmodified lines
23/* The set of all possible UFFD-related VM flags. */23/* The set of all possible UFFD-related VM flags. */
24#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)24#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
25 25
26#define __VMA_UFFD_FLAGS mk_vma_flags(VMA_UFFD_MISSING_BIT, VMA_UFFD_WP_BIT, \26#define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \
27 VMA_UFFD_MINOR_BIT)27 VMA_UFFD_MINOR)
28 28
29/*29/*
30 * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining30 * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -147,26 +147,12 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at · 116 unmodified lines
147/* Flags controlling behavior. These behavior changes are mode-independent. */147/* Flags controlling behavior. These behavior changes are mode-independent. */
148#define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)148#define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
149 149
150extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
151 unsigned long src_start, unsigned long len,
152 uffd_flags_t flags);
153extern ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
154 unsigned long dst_start,
155 unsigned long len);
156extern ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long dst_start,
157 unsigned long len, uffd_flags_t flags);
158extern ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
159 unsigned long len, uffd_flags_t flags);
160extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
161 unsigned long len, bool enable_wp);
162extern long uffd_wp_range(struct vm_area_struct *vma,150extern long uffd_wp_range(struct vm_area_struct *vma,
163 unsigned long start, unsigned long len, bool enable_wp);151 unsigned long start, unsigned long len, bool enable_wp);
164 152
165/* move_pages */153/* move_pages */
166void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);154void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
167void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);155void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
168ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
169 unsigned long src_start, unsigned long len, __u64 flags);
170int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,156int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,
171 struct vm_area_struct *dst_vma,157 struct vm_area_struct *dst_vma,
172 struct vm_area_struct *src_vma,158 struct vm_area_struct *src_vma,
@@ -239,9 +225,6 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 66 unmodified lines
239 return vma->vm_flags & __VM_UFFD_FLAGS;225 return vma->vm_flags & __VM_UFFD_FLAGS;
240}226}
241 227
242bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
243 bool wp_async);
244
245static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)228static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
246{229{
247 struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;230 struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
@@ -271,25 +254,6 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm, · 23 unmodified lines
271extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);254extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
272extern bool userfaultfd_wp_async(struct vm_area_struct *vma);255extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
273 256
274void userfaultfd_reset_ctx(struct vm_area_struct *vma);
275
276struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
277 struct vm_area_struct *prev,
278 struct vm_area_struct *vma,
279 unsigned long start,
280 unsigned long end);
281
282int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
283 struct vm_area_struct *vma,
284 vm_flags_t vm_flags,
285 unsigned long start, unsigned long end,
286 bool wp_async);
287
288void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
289
290void userfaultfd_release_all(struct mm_struct *mm,
291 struct userfaultfd_ctx *ctx);
292
293static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)257static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
294{258{
295 /* Only wr-protect mode uses pte markers */259 /* Only wr-protect mode uses pte markers */
include/linux/vmalloc.h+3−1
@@ -265,7 +265,9 @@ static inline bool is_vm_area_hugepages(const void *addr) · 264 unmodified lines
265 * allocated in the vmalloc layer.265 * allocated in the vmalloc layer.
266 */266 */
267#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC267#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
268 return find_vm_area(addr)->page_order > 0;268 struct vm_struct *area = find_vm_area(addr);
269
270 return area && area->page_order > 0;
269#else271#else
270 return false;272 return false;
271#endif273#endif
include/linux/vmpressure.h+5−4
@@ -30,8 +30,8 @@ struct vmpressure { · 29 unmodified lines
30struct mem_cgroup;30struct mem_cgroup;
31 31
32#ifdef CONFIG_MEMCG32#ifdef CONFIG_MEMCG
33extern void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,33void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
34 unsigned long scanned, unsigned long reclaimed);34 unsigned long scanned, unsigned long reclaimed);
35extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);35extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);
36 36
37extern void vmpressure_init(struct vmpressure *vmpr);37extern void vmpressure_init(struct vmpressure *vmpr);
@@ -44,8 +44,9 @@ extern int vmpressure_register_event(struct mem_cgroup *memcg, · 6 unmodified lines
44extern void vmpressure_unregister_event(struct mem_cgroup *memcg,44extern void vmpressure_unregister_event(struct mem_cgroup *memcg,
45 struct eventfd_ctx *eventfd);45 struct eventfd_ctx *eventfd);
46#else46#else
47static inline void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,47static inline void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg,
48 unsigned long scanned, unsigned long reclaimed) {}48 bool tree, unsigned long scanned,
49 unsigned long reclaimed) {}
49static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,50static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,
50 int prio) {}51 int prio) {}
51#endif /* CONFIG_MEMCG */52#endif /* CONFIG_MEMCG */
include/trace/events/damon.h+38−0
@@ -130,6 +130,44 @@ TRACE_EVENT(damon_monitor_intervals_tune, · 129 unmodified lines
130 TP_printk("sample_us=%lu", __entry->sample_us)130 TP_printk("sample_us=%lu", __entry->sample_us)
131);131);
132 132
133TRACE_EVENT_CONDITION(damon_region_aggregated,
134
135 TP_PROTO(unsigned int target_id, struct damon_region *r,
136 unsigned int nr_regions, unsigned int nr_probes),
137
138 TP_ARGS(target_id, r, nr_regions, nr_probes),
139
140 TP_CONDITION(nr_probes > 0),
141
142 TP_STRUCT__entry(
143 __field(unsigned long, target_id)
144 __field(unsigned long, start)
145 __field(unsigned long, end)
146 __field(unsigned int, nr_regions)
147 __field(unsigned int, nr_accesses)
148 __field(unsigned int, age)
149 __dynamic_array(unsigned char, probe_hits, nr_probes)
150 ),
151
152 TP_fast_assign(
153 __entry->target_id = target_id;
154 __entry->start = r->ar.start;
155 __entry->end = r->ar.end;
156 __entry->nr_regions = nr_regions;
157 __entry->nr_accesses = r->nr_accesses;
158 __entry->age = r->age;
159 memcpy(__get_dynamic_array(probe_hits), r->probe_hits,
160 sizeof(*r->probe_hits) * nr_probes);
161 ),
162
163 TP_printk("target_id=%lu nr_regions=%u %lu-%lu: %u %u probe_hits=%s",
164 __entry->target_id, __entry->nr_regions,
165 __entry->start, __entry->end,
166 __entry->nr_accesses, __entry->age,
167 __print_hex(__get_dynamic_array(probe_hits),
168 __get_dynamic_array_len(probe_hits)))
169);
170
133TRACE_EVENT(damon_aggregated,171TRACE_EVENT(damon_aggregated,
134 172
135 TP_PROTO(unsigned int target_id, struct damon_region *r,173 TP_PROTO(unsigned int target_id, struct damon_region *r,
include/trace/events/vmscan.h+52−0
@@ -96,6 +96,58 @@ TRACE_EVENT(mm_vmscan_kswapd_wake, · 95 unmodified lines
96 __entry->order)96 __entry->order)
97);97);
98 98
99TRACE_EVENT(mm_vmscan_balance_pgdat_begin,
100
101 TP_PROTO(int nid, int order, int highest_zoneidx),
102
103 TP_ARGS(nid, order, highest_zoneidx),
104
105 TP_STRUCT__entry(
106 __field(int, nid)
107 __field(int, order)
108 __field(int, highest_zoneidx)
109 ),
110
111 TP_fast_assign(
112 __entry->nid = nid;
113 __entry->order = order;
114 __entry->highest_zoneidx = highest_zoneidx;
115 ),
116
117 TP_printk("nid=%d order=%d highest_zoneidx=%-8s",
118 __entry->nid,
119 __entry->order,
120 __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE))
121);
122
123TRACE_EVENT(mm_vmscan_balance_pgdat_end,
124
125 TP_PROTO(int nid, int order, int highest_zoneidx,
126 unsigned long nr_reclaimed),
127
128 TP_ARGS(nid, order, highest_zoneidx, nr_reclaimed),
129
130 TP_STRUCT__entry(
131 __field(int, nid)
132 __field(int, order)
133 __field(int, highest_zoneidx)
134 __field(unsigned long, nr_reclaimed)
135 ),
136
137 TP_fast_assign(
138 __entry->nid = nid;
139 __entry->order = order;
140 __entry->highest_zoneidx = highest_zoneidx;
141 __entry->nr_reclaimed = nr_reclaimed;
142 ),
143
144 TP_printk("nid=%d order=%d highest_zoneidx=%-8s nr_reclaimed=%lu",
145 __entry->nid,
146 __entry->order,
147 __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE),
148 __entry->nr_reclaimed)
149);
150
99TRACE_EVENT(mm_vmscan_wakeup_kswapd,151TRACE_EVENT(mm_vmscan_wakeup_kswapd,
100 152
101 TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),153 TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),
kernel/bpf/arena.c+1−1
@@ -891,7 +891,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 890 unmodified lines
891 891
892 llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {892 llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {
893 page = llist_entry(pos, struct page, pcp_llist);893 page = llist_entry(pos, struct page, pcp_llist);
894 if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */894 if (page_cnt == 1 && page_ref_count(page) > 1) /* maybe mapped by user space */
895 /* Optimization for the common case of page_cnt==1:895 /* Optimization for the common case of page_cnt==1:
896 * If page wasn't mapped into some user vma there896 * If page wasn't mapped into some user vma there
897 * is no need to call zap_pages which is slow. When897 * is no need to call zap_pages which is slow. When
kernel/fork.c+3−2
@@ -205,7 +205,7 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]); · 204 unmodified lines
205 * accounting is performed by the code assigning/releasing stacks to tasks.205 * accounting is performed by the code assigning/releasing stacks to tasks.
206 * We need a zeroed memory without __GFP_ACCOUNT.206 * We need a zeroed memory without __GFP_ACCOUNT.
207 */207 */
208#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO)208#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
209 209
210struct vm_stack {210struct vm_stack {
211 struct rcu_head rcu;211 struct rcu_head rcu;
@@ -343,7 +343,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) · 131 unmodified lines
343 }343 }
344 344
345 /* Reset stack metadata. */345 /* Reset stack metadata. */
346 kasan_unpoison_range(vm_area->addr, THREAD_SIZE);346 if (!kasan_hw_tags_enabled())
347 kasan_unpoison_range(vm_area->addr, THREAD_SIZE);
347 348
348 stack = kasan_reset_tag(vm_area->addr);349 stack = kasan_reset_tag(vm_area->addr);
349 350
kernel/power/snapshot.c+4−4
@@ -1244,8 +1244,9 @@ unsigned int snapshot_additional_pages(struct zone *zone) · 1243 unmodified lines
1244static void mark_free_pages(struct zone *zone)1244static void mark_free_pages(struct zone *zone)
1245{1245{
1246 unsigned long pfn, max_zone_pfn, page_count = WD_PAGE_COUNT;1246 unsigned long pfn, max_zone_pfn, page_count = WD_PAGE_COUNT;
1247 struct list_head *free_list;
1247 unsigned long flags;1248 unsigned long flags;
1248 unsigned int order, t;1249 unsigned int order;
1249 struct page *page;1250 struct page *page;
1250 1251
1251 if (zone_is_empty(zone))1252 if (zone_is_empty(zone))
@@ -1269,9 +1270,8 @@ static void mark_free_pages(struct zone *zone) · 17 unmodified lines
1269 swsusp_unset_page_free(page);1270 swsusp_unset_page_free(page);
1270 }1271 }
1271 1272
1272 for_each_migratetype_order(order, t) {1273 for_each_free_list(free_list, zone, order) {
1273 list_for_each_entry(page,1274 list_for_each_entry(page, free_list, buddy_list) {
1274 &zone->free_area[order].free_list[t], buddy_list) {
1275 unsigned long i;1275 unsigned long i;
1276 1276
1277 pfn = page_to_pfn(page);1277 pfn = page_to_pfn(page);
kernel/power/swap.c+1−1
@@ -341,7 +341,7 @@ static int swsusp_swap_check(void) · 340 unmodified lines
341 * This is called before saving the image.341 * This is called before saving the image.
342 */342 */
343 if (swsusp_resume_device)343 if (swsusp_resume_device)
344 res = swap_type_of(swsusp_resume_device, swsusp_resume_block);344 res = find_hibernation_swap_type(swsusp_resume_device, swsusp_resume_block);
345 else345 else
346 res = find_first_swap(&swsusp_resume_device);346 res = find_first_swap(&swsusp_resume_device);
347 if (res < 0)347 if (res < 0)
kernel/power/user.c+12−3
@@ -71,7 +71,7 @@ static int snapshot_open(struct inode *inode, struct file *filp) · 70 unmodified lines
71 memset(&data->handle, 0, sizeof(struct snapshot_handle));71 memset(&data->handle, 0, sizeof(struct snapshot_handle));
72 if ((filp->f_flags & O_ACCMODE) == O_RDONLY) {72 if ((filp->f_flags & O_ACCMODE) == O_RDONLY) {
73 /* Hibernating. The image device should be accessible. */73 /* Hibernating. The image device should be accessible. */
74 data->swap = swap_type_of(swsusp_resume_device, 0);74 data->swap = pin_hibernation_swap_type(swsusp_resume_device, 0);
75 data->mode = O_RDONLY;75 data->mode = O_RDONLY;
76 data->free_bitmaps = false;76 data->free_bitmaps = false;
77 error = pm_notifier_call_chain_robust(PM_HIBERNATION_PREPARE, PM_POST_HIBERNATION);77 error = pm_notifier_call_chain_robust(PM_HIBERNATION_PREPARE, PM_POST_HIBERNATION);
@@ -90,8 +90,10 @@ static int snapshot_open(struct inode *inode, struct file *filp) · 12 unmodified lines
90 data->free_bitmaps = !error;90 data->free_bitmaps = !error;
91 }91 }
92 }92 }
93 if (error)93 if (error) {
94 unpin_hibernation_swap_type(data->swap);
94 hibernate_release();95 hibernate_release();
96 }
95 97
96 data->frozen = false;98 data->frozen = false;
97 data->ready = false;99 data->ready = false;
@@ -115,6 +117,7 @@ static int snapshot_release(struct inode *inode, struct file *filp) · 17 unmodified lines
115 data = filp->private_data;117 data = filp->private_data;
116 data->dev = 0;118 data->dev = 0;
117 free_all_swap_pages(data->swap);119 free_all_swap_pages(data->swap);
120 unpin_hibernation_swap_type(data->swap);
118 if (data->frozen) {121 if (data->frozen) {
119 pm_restore_gfp_mask();122 pm_restore_gfp_mask();
120 free_basic_memory_bitmaps();123 free_basic_memory_bitmaps();
@@ -235,11 +238,17 @@ static int snapshot_set_swap_area(struct snapshot_data *data, · 114 unmodified lines
235 offset = swap_area.offset;238 offset = swap_area.offset;
236 }239 }
237 240
241 /*
242 * Unpin the swap device if a swap area was already
243 * set by SNAPSHOT_SET_SWAP_AREA.
244 */
245 unpin_hibernation_swap_type(data->swap);
246
238 /*247 /*
239 * User space encodes device types as two-byte values,248 * User space encodes device types as two-byte values,
240 * so we need to recode them249 * so we need to recode them
241 */250 */
242 data->swap = swap_type_of(swdev, offset);251 data->swap = pin_hibernation_swap_type(swdev, offset);
243 if (data->swap < 0)252 if (data->swap < 0)
244 return swdev ? -ENODEV : -EINVAL;253 return swdev ? -ENODEV : -EINVAL;
245 data->dev = swdev;254 data->dev = swdev;
lib/maple_tree.c+5−2
@@ -5727,13 +5727,16 @@ int mtree_store(struct maple_tree *mt, unsigned long index, void *entry, · 5726 unmodified lines
5727EXPORT_SYMBOL(mtree_store);5727EXPORT_SYMBOL(mtree_store);
5728 5728
5729/**5729/**
5730 * mtree_insert_range() - Insert an entry at a given range if there is no value.5730 * mtree_insert_range() - Insert an entry from [first, last] at a given range
5731 * if there is no value.
5731 * @mt: The maple tree5732 * @mt: The maple tree
5732 * @first: The start of the range5733 * @first: The start of the range
5733 * @last: The end of the range5734 * @last: The end of the range (inclusive)
5734 * @entry: The entry to store5735 * @entry: The entry to store
5735 * @gfp: The GFP_FLAGS to use for allocations.5736 * @gfp: The GFP_FLAGS to use for allocations.
5736 *5737 *
5738 * Note that @last is inclusive. That is, @last = @first + length - 1;
5739 *
5737 * Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid5740 * Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid
5738 * request, -ENOMEM if memory could not be allocated.5741 * request, -ENOMEM if memory could not be allocated.
5739 */5742 */
lib/test_hmm.c+40−9
@@ -1063,6 +1063,25 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, · 1062 unmodified lines
1063 /* Try with smaller pages if large allocation fails */1063 /* Try with smaller pages if large allocation fails */
1064 if (!dpage && order) {1064 if (!dpage && order) {
1065 dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr);1065 dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr);
1066 if (!dpage) {
1067 /* Unlock and free pages already allocated. */
1068 while (i > 0) {
1069 struct page *fpage;
1070
1071 fpage = migrate_pfn_to_page(dst[--i]);
1072 unlock_page(fpage);
1073 __free_page(fpage);
1074 }
1075 /* Clear remaining dst entries to avoid
1076 * migrate_vma_pages/finalize() using
1077 * uninitialized values.
1078 */
1079 while (i < (1 << order)) {
1080 dst[i] = 0;
1081 i++;
1082 }
1083 return VM_FAULT_OOM;
1084 }
1066 lock_page(dpage);1085 lock_page(dpage);
1067 dst[i] = migrate_pfn(page_to_pfn(dpage));1086 dst[i] = migrate_pfn(page_to_pfn(dpage));
1068 dst_page = pfn_to_page(page_to_pfn(dpage));1087 dst_page = pfn_to_page(page_to_pfn(dpage));
@@ -1111,9 +1130,6 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 42 unmodified lines
1111 unsigned long *src_pfns;1130 unsigned long *src_pfns;
1112 unsigned long *dst_pfns;1131 unsigned long *dst_pfns;
1113 1132
1114 src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
1115 dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
1116
1117 start = cmd->addr;1133 start = cmd->addr;
1118 end = start + size;1134 end = start + size;
1119 if (end < start)1135 if (end < start)
@@ -1123,6 +1139,9 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 3 unmodified lines
1123 if (!mmget_not_zero(mm))1139 if (!mmget_not_zero(mm))
1124 return -EINVAL;1140 return -EINVAL;
1125 1141
1142 src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
1143 dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
1144
1126 cmd->cpages = 0;1145 cmd->cpages = 0;
1127 mmap_read_lock(mm);1146 mmap_read_lock(mm);
1128 for (addr = start; addr < end; addr = next) {1147 for (addr = start; addr < end; addr = next) {
@@ -1148,7 +1167,11 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 19 unmodified lines
1148 goto out;1167 goto out;
1149 1168
1150 pr_debug("Migrating from device mem to sys mem\n");1169 pr_debug("Migrating from device mem to sys mem\n");
1151 dmirror_devmem_fault_alloc_and_copy(&args, dmirror);1170 if (dmirror_devmem_fault_alloc_and_copy(&args, dmirror)) {
1171 migrate_vma_finalize(&args);
1172 ret = -ENOMEM;
1173 goto out;
1174 }
1152 1175
1153 migrate_vma_pages(&args);1176 migrate_vma_pages(&args);
1154 cmd->cpages += dmirror_successful_migrated_pages(&args);1177 cmd->cpages += dmirror_successful_migrated_pages(&args);
@@ -1253,8 +1276,8 @@ out: · 98 unmodified lines
1253 mmap_read_unlock(mm);1276 mmap_read_unlock(mm);
1254 mmput(mm);1277 mmput(mm);
1255free_mem:1278free_mem:
1256 kfree(src_pfns);1279 kvfree(src_pfns);
1257 kfree(dst_pfns);1280 kvfree(dst_pfns);
1258 return ret;1281 return ret;
1259}1282}
1260 1283
@@ -1679,12 +1702,20 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) · 418 unmodified lines
1679 if (order)1702 if (order)
1680 args.flags |= MIGRATE_VMA_SELECT_COMPOUND;1703 args.flags |= MIGRATE_VMA_SELECT_COMPOUND;
1681 1704
1682 if (migrate_vma_setup(&args))1705 /*
1683 return VM_FAULT_SIGBUS;1706 * In practice migrate_vma_setup() should never fail unless the
1707 * test is wrong as it just tests some static VMA properties.
1708 */
1709 if (migrate_vma_setup(&args)) {
1710 ret = VM_FAULT_SIGBUS;
1711 goto err;
1712 }
1684 1713
1685 ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror);1714 ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror);
1686 if (ret)1715 if (ret) {
1716 migrate_vma_finalize(&args);
1687 goto err;1717 goto err;
1718 }
1688 migrate_vma_pages(&args);1719 migrate_vma_pages(&args);
1689 /*1720 /*
1690 * No device finalize step is needed since1721 * No device finalize step is needed since
lib/test_meminit.c+1−1
@@ -386,7 +386,7 @@ static int __init test_kmemcache(int *total_failures) · 385 unmodified lines
386 ctor = flags & 1;386 ctor = flags & 1;
387 rcu = flags & 2;387 rcu = flags & 2;
388 zero = flags & 4;388 zero = flags & 4;
389 if (ctor & zero)389 if (ctor && zero)
390 continue;390 continue;
391 num_tests += do_kmem_cache_size(size, ctor, rcu, zero,391 num_tests += do_kmem_cache_size(size, ctor, rcu, zero,
392 &failures);392 &failures);
lib/test_vmalloc.c+62−0
@@ -55,6 +55,7 @@ __param(int, run_test_mask, 7, · 54 unmodified lines
55 "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n"55 "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n"
56 "\t\tid: 1024, name: vm_map_ram_test\n"56 "\t\tid: 1024, name: vm_map_ram_test\n"
57 "\t\tid: 2048, name: no_block_alloc_test\n"57 "\t\tid: 2048, name: no_block_alloc_test\n"
58 "\t\tid: 4096, name: vrealloc_test\n"
58 /* Add a new test case description here. */59 /* Add a new test case description here. */
59);60);
60 61
@@ -421,6 +422,66 @@ cleanup: · 360 unmodified lines
421 return nr_allocated != map_nr_pages;422 return nr_allocated != map_nr_pages;
422}423}
423 424
425static int vrealloc_test(void)
426{
427 void *ptr, *tmp;
428 int i;
429
430 for (i = 0; i < test_loop_count; i++) {
431 int err = -1;
432
433 ptr = vrealloc(NULL, PAGE_SIZE, GFP_KERNEL);
434 if (!ptr)
435 return -1;
436
437 *((__u8 *)ptr) = 'a';
438
439 /* Grow: beyond allocated pages, triggers full realloc. */
440 tmp = vrealloc(ptr, 4 * PAGE_SIZE, GFP_KERNEL);
441 if (!tmp)
442 goto error;
443 ptr = tmp;
444
445 if (*((__u8 *)ptr) != 'a')
446 goto error;
447
448 /* Shrink: crosses page boundary, frees tail pages. */
449 tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
450 if (!tmp)
451 goto error;
452 ptr = tmp;
453
454 if (*((__u8 *)ptr) != 'a')
455 goto error;
456
457 /* Shrink: within same page, no page freeing. */
458 tmp = vrealloc(ptr, PAGE_SIZE / 2, GFP_KERNEL);
459 if (!tmp)
460 goto error;
461 ptr = tmp;
462
463 if (*((__u8 *)ptr) != 'a')
464 goto error;
465
466 /* Grow: within allocated page, in-place, no realloc. */
467 tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
468 if (!tmp)
469 goto error;
470 ptr = tmp;
471
472 if (*((__u8 *)ptr) != 'a')
473 goto error;
474
475 err = 0;
476error:
477 vfree(ptr);
478 if (err)
479 return err;
480 }
481
482 return 0;
483}
484
424struct test_case_desc {485struct test_case_desc {
425 const char *test_name;486 const char *test_name;
426 int (*test_func)(void);487 int (*test_func)(void);
@@ -440,6 +501,7 @@ static struct test_case_desc test_case_array[] = { · 13 unmodified lines
440 { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test, },501 { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test, },
441 { "vm_map_ram_test", vm_map_ram_test, },502 { "vm_map_ram_test", vm_map_ram_test, },
442 { "no_block_alloc_test", no_block_alloc_test, true },503 { "no_block_alloc_test", no_block_alloc_test, true },
504 { "vrealloc_test", vrealloc_test, },
443 /* Add a new test case here. */505 /* Add a new test case here. */
444};506};
445 507
mm/Kconfig+2−5
@@ -590,7 +590,7 @@ endchoice · 589 unmodified lines
590 590
591config MEMORY_HOTREMOVE591config MEMORY_HOTREMOVE
592 bool "Allow for memory hot remove"592 bool "Allow for memory hot remove"
593 select HAVE_BOOTMEM_INFO_NODE if (X86_64 || PPC64)593 select HAVE_BOOTMEM_INFO_NODE if X86_64
594 depends on MEMORY_HOTPLUG594 depends on MEMORY_HOTPLUG
595 select MIGRATION595 select MIGRATION
596 596
@@ -863,7 +863,6 @@ if TRANSPARENT_HUGEPAGE · 266 unmodified lines
863 863
864choice864choice
865 prompt "Transparent Hugepage Support sysfs defaults"865 prompt "Transparent Hugepage Support sysfs defaults"
866 depends on TRANSPARENT_HUGEPAGE
867 default TRANSPARENT_HUGEPAGE_ALWAYS866 default TRANSPARENT_HUGEPAGE_ALWAYS
868 help867 help
869 Selects the sysfs defaults for Transparent Hugepage Support.868 Selects the sysfs defaults for Transparent Hugepage Support.
@@ -893,7 +892,6 @@ endchoice · 23 unmodified lines
893 892
894choice893choice
895 prompt "Shmem hugepage allocation defaults"894 prompt "Shmem hugepage allocation defaults"
896 depends on TRANSPARENT_HUGEPAGE
897 default TRANSPARENT_HUGEPAGE_SHMEM_HUGE_NEVER895 default TRANSPARENT_HUGEPAGE_SHMEM_HUGE_NEVER
898 help896 help
899 Selects the hugepage allocation policy defaults for897 Selects the hugepage allocation policy defaults for
@@ -939,7 +937,6 @@ endchoice · 39 unmodified lines
939 937
940choice938choice
941 prompt "Tmpfs hugepage allocation defaults"939 prompt "Tmpfs hugepage allocation defaults"
942 depends on TRANSPARENT_HUGEPAGE
943 default TRANSPARENT_HUGEPAGE_TMPFS_HUGE_NEVER940 default TRANSPARENT_HUGEPAGE_TMPFS_HUGE_NEVER
944 help941 help
945 Selects the hugepage allocation policy defaults for942 Selects the hugepage allocation policy defaults for
@@ -984,7 +981,7 @@ endchoice · 38 unmodified lines
984 981
985config THP_SWAP982config THP_SWAP
986 def_bool y983 def_bool y
987 depends on TRANSPARENT_HUGEPAGE && ARCH_WANTS_THP_SWAP && SWAP && 64BIT984 depends on ARCH_WANTS_THP_SWAP && SWAP && 64BIT
988 help985 help
989 Swap transparent huge pages in one piece, without splitting.986 Swap transparent huge pages in one piece, without splitting.
990 XXX: For now, swap cluster backing transparent huge page987 XXX: For now, swap cluster backing transparent huge page
mm/Makefile+0−3
@@ -103,9 +103,6 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o · 102 unmodified lines
103obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o103obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
104obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o104obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
105obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o105obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
106ifdef CONFIG_SWAP
107obj-$(CONFIG_MEMCG) += swap_cgroup.o
108endif
109ifdef CONFIG_BPF_SYSCALL106ifdef CONFIG_BPF_SYSCALL
110obj-$(CONFIG_MEMCG) += bpf_memcontrol.o107obj-$(CONFIG_MEMCG) += bpf_memcontrol.o
111endif108endif
mm/bootmem_info.c+2−23
@@ -19,7 +19,6 @@ void get_page_bootmem(unsigned long info, struct page *page, · 18 unmodified lines
19{19{
20 BUG_ON(type > 0xf);20 BUG_ON(type > 0xf);
21 BUG_ON(info > (ULONG_MAX >> 4));21 BUG_ON(info > (ULONG_MAX >> 4));
22 SetPagePrivate(page);
23 set_page_private(page, info << 4 | type);22 set_page_private(page, info << 4 | type);
24 page_ref_inc(page);23 page_ref_inc(page);
25}24}
@@ -32,20 +31,15 @@ void put_page_bootmem(struct page *page) · 6 unmodified lines
32 type > MEMORY_HOTPLUG_MAX_BOOTMEM_TYPE);31 type > MEMORY_HOTPLUG_MAX_BOOTMEM_TYPE);
33 32
34 if (page_ref_dec_return(page) == 1) {33 if (page_ref_dec_return(page) == 1) {
35 ClearPagePrivate(page);
36 set_page_private(page, 0);34 set_page_private(page, 0);
37 INIT_LIST_HEAD(&page->lru);
38 kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
39 free_reserved_page(page);35 free_reserved_page(page);
40 }36 }
41}37}
42 38
43static void __init register_page_bootmem_info_section(unsigned long start_pfn)39static void __init register_page_bootmem_info_section(unsigned long start_pfn)
44{40{
45 unsigned long mapsize, section_nr, i;41 unsigned long section_nr;
46 struct mem_section *ms;42 struct mem_section *ms;
47 struct mem_section_usage *usage;
48 struct page *page;
49 43
50 start_pfn = SECTION_ALIGN_DOWN(start_pfn);44 start_pfn = SECTION_ALIGN_DOWN(start_pfn);
51 section_nr = pfn_to_section_nr(start_pfn);45 section_nr = pfn_to_section_nr(start_pfn);
@@ -54,27 +48,12 @@ static void __init register_page_bootmem_info_section(unsigned long start_pfn) · 2 unmodified lines
54 if (!preinited_vmemmap_section(ms))48 if (!preinited_vmemmap_section(ms))
55 register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn),49 register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn),
56 PAGES_PER_SECTION);50 PAGES_PER_SECTION);
57
58 usage = ms->usage;
59 page = virt_to_page(usage);
60
61 mapsize = PAGE_ALIGN(mem_section_usage_size()) >> PAGE_SHIFT;
62
63 for (i = 0; i < mapsize; i++, page++)
64 get_page_bootmem(section_nr, page, MIX_SECTION_INFO);
65}51}
66 52
67void __init register_page_bootmem_info_node(struct pglist_data *pgdat)53void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
68{54{
69 unsigned long i, pfn, end_pfn, nr_pages;55 unsigned long pfn, end_pfn;
70 int node = pgdat->node_id;56 int node = pgdat->node_id;
71 struct page *page;
72
73 nr_pages = PAGE_ALIGN(sizeof(struct pglist_data)) >> PAGE_SHIFT;
74 page = virt_to_page(pgdat);
75
76 for (i = 0; i < nr_pages; i++, page++)
77 get_page_bootmem(node, page, NODE_INFO);
78 57
79 pfn = pgdat->node_start_pfn;58 pfn = pgdat->node_start_pfn;
80 end_pfn = pgdat_end_pfn(pgdat);59 end_pfn = pgdat_end_pfn(pgdat);
mm/compaction.c+8−3
@@ -1123,7 +1123,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · 1122 unmodified lines
1123 * To minimise LRU disruption, the caller can indicate with1123 * To minimise LRU disruption, the caller can indicate with
1124 * ISOLATE_ASYNC_MIGRATE that it only wants to isolate pages1124 * ISOLATE_ASYNC_MIGRATE that it only wants to isolate pages
1125 * it will be able to migrate without blocking - clean pages1125 * it will be able to migrate without blocking - clean pages
1126 * for the most part. PageWriteback would require blocking.1126 * for the most part. Writeback would require blocking.
1127 */1127 */
1128 if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio))1128 if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio))
1129 goto isolate_fail_put;1129 goto isolate_fail_put;
@@ -2340,7 +2340,8 @@ static enum compact_result __compact_finished(struct compact_control *cc) · 1210 unmodified lines
2340 * Job done if allocation would steal freepages from2340 * Job done if allocation would steal freepages from
2341 * other migratetype buddy lists.2341 * other migratetype buddy lists.
2342 */2342 */
2343 if (find_suitable_fallback(area, order, migratetype, true) >= 0)2343 if (find_suitable_fallback(area, order, migratetype, true, NULL)
2344 == FALLBACK_FOUND)
2344 /*2345 /*
2345 * Movable pages are OK in any pageblock. If we are2346 * Movable pages are OK in any pageblock. If we are
2346 * stealing for a non-movable allocation, make sure2347 * stealing for a non-movable allocation, make sure
@@ -2447,7 +2448,7 @@ bool compaction_suitable(struct zone *zone, int order, unsigned long watermark, · 100 unmodified lines
2447 2448
2448/* Used by direct reclaimers */2449/* Used by direct reclaimers */
2449bool compaction_zonelist_suitable(struct alloc_context *ac, int order,2450bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
2450 int alloc_flags)2451 int alloc_flags, gfp_t gfp_mask)
2451{2452{
2452 struct zone *zone;2453 struct zone *zone;
2453 struct zoneref *z;2454 struct zoneref *z;
@@ -2460,6 +2461,10 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, · 6 unmodified lines
2460 ac->highest_zoneidx, ac->nodemask) {2461 ac->highest_zoneidx, ac->nodemask) {
2461 unsigned long available;2462 unsigned long available;
2462 2463
2464 if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
2465 !__cpuset_zone_allowed(zone, gfp_mask))
2466 continue;
2467
2463 /*2468 /*
2464 * Do not consider all the reclaimable memory because we do not2469 * Do not consider all the reclaimable memory because we do not
2465 * want to trash just for a single high order allocation which2470 * want to trash just for a single high order allocation which
mm/damon/core.c+578−77
@@ -13,10 +13,14 @@ · 12 unmodified lines
13#include <linux/memcontrol.h>13#include <linux/memcontrol.h>
14#include <linux/mm.h>14#include <linux/mm.h>
15#include <linux/psi.h>15#include <linux/psi.h>
16#include <linux/sched.h>
16#include <linux/slab.h>17#include <linux/slab.h>
17#include <linux/string.h>18#include <linux/string.h>
18#include <linux/string_choices.h>19#include <linux/string_choices.h>
19 20
21/* for damon_get_folio() used by node eligible memory metrics */
22#include "ops-common.h"
23
20#define CREATE_TRACE_POINTS24#define CREATE_TRACE_POINTS
21#include <trace/events/damon.h>25#include <trace/events/damon.h>
22 26
@@ -109,6 +113,103 @@ int damon_select_ops(struct damon_ctx *ctx, enum damon_ops_id id) · 86 unmodified lines
109 return err;113 return err;
110}114}
111 115
116struct damon_filter *damon_new_filter(enum damon_filter_type type,
117 bool matching, bool allow)
118{
119 struct damon_filter *filter;
120
121 filter = kmalloc_obj(*filter);
122 if (!filter)
123 return NULL;
124 filter->type = type;
125 filter->matching = matching;
126 filter->allow = allow;
127 INIT_LIST_HEAD(&filter->list);
128 return filter;
129}
130
131void damon_add_filter(struct damon_probe *p, struct damon_filter *f)
132{
133 list_add_tail(&f->list, &p->filters);
134}
135
136static void damon_del_filter(struct damon_filter *f)
137{
138 list_del(&f->list);
139}
140
141static void damon_free_filter(struct damon_filter *f)
142{
143 kfree(f);
144}
145
146void damon_destroy_filter(struct damon_filter *f)
147{
148 damon_del_filter(f);
149 damon_free_filter(f);
150}
151
152static struct damon_filter *damon_nth_filter(int n, struct damon_probe *p)
153{
154 struct damon_filter *f;
155 int i = 0;
156
157 damon_for_each_filter(f, p) {
158 if (i++ == n)
159 return f;
160 }
161 return NULL;
162}
163
164struct damon_probe *damon_new_probe(void)
165{
166 struct damon_probe *p;
167
168 p = kmalloc_obj(*p);
169 if (!p)
170 return NULL;
171 INIT_LIST_HEAD(&p->filters);
172 INIT_LIST_HEAD(&p->list);
173 return p;
174}
175
176void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe)
177{
178 list_add_tail(&probe->list, &ctx->probes);
179}
180
181static void damon_del_probe(struct damon_probe *p)
182{
183 list_del(&p->list);
184}
185
186static void damon_free_probe(struct damon_probe *p)
187{
188 struct damon_filter *f, *next;
189
190 damon_for_each_filter_safe(f, next, p)
191 damon_free_filter(f);
192 kfree(p);
193}
194
195static void damon_destroy_probe(struct damon_probe *p)
196{
197 damon_del_probe(p);
198 damon_free_probe(p);
199}
200
201static struct damon_probe *damon_nth_probe(int n, struct damon_ctx *ctx)
202{
203 struct damon_probe *p;
204 int i = 0;
205
206 damon_for_each_probe(p, ctx) {
207 if (i++ == n)
208 return p;
209 }
210 return NULL;
211}
212
112#ifdef CONFIG_DAMON_DEBUG_SANITY213#ifdef CONFIG_DAMON_DEBUG_SANITY
113static void damon_verify_new_region(unsigned long start, unsigned long end)214static void damon_verify_new_region(unsigned long start, unsigned long end)
114{215{
@@ -128,6 +229,7 @@ static void damon_verify_new_region(unsigned long start, unsigned long end) · 13 unmodified lines
128struct damon_region *damon_new_region(unsigned long start, unsigned long end)229struct damon_region *damon_new_region(unsigned long start, unsigned long end)
129{230{
130 struct damon_region *region;231 struct damon_region *region;
232 int i;
131 233
132 damon_verify_new_region(start, end);234 damon_verify_new_region(start, end);
133 region = kmem_cache_alloc(damon_region_cache, GFP_KERNEL);235 region = kmem_cache_alloc(damon_region_cache, GFP_KERNEL);
@@ -138,6 +240,8 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 4 unmodified lines
138 region->ar.end = end;240 region->ar.end = end;
139 region->nr_accesses = 0;241 region->nr_accesses = 0;
140 region->nr_accesses_bp = 0;242 region->nr_accesses_bp = 0;
243 for (i = 0; i < DAMON_MAX_PROBES; i++)
244 region->probe_hits[i] = 0;
141 INIT_LIST_HEAD(&region->list);245 INIT_LIST_HEAD(&region->list);
142 246
143 region->age = 0;247 region->age = 0;
@@ -146,12 +250,23 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 2 unmodified lines
146 return region;250 return region;
147}251}
148 252
149void damon_add_region(struct damon_region *r, struct damon_target *t)253static void damon_add_region(struct damon_region *r, struct damon_target *t)
150{254{
151 list_add_tail(&r->list, &t->regions_list);255 list_add_tail(&r->list, &t->regions_list);
152 t->nr_regions++;256 t->nr_regions++;
153}257}
154 258
259/*
260 * Add a region between two other regions
261 */
262static inline void damon_insert_region(struct damon_region *r,
263 struct damon_region *prev, struct damon_region *next,
264 struct damon_target *t)
265{
266 __list_add(&r->list, &prev->list, &next->list);
267 t->nr_regions++;
268}
269
155#ifdef CONFIG_DAMON_DEBUG_SANITY270#ifdef CONFIG_DAMON_DEBUG_SANITY
156static void damon_verify_del_region(struct damon_target *t)271static void damon_verify_del_region(struct damon_target *t)
157{272{
@@ -176,7 +291,8 @@ static void damon_free_region(struct damon_region *r) · 18 unmodified lines
176 kmem_cache_free(damon_region_cache, r);291 kmem_cache_free(damon_region_cache, r);
177}292}
178 293
179void damon_destroy_region(struct damon_region *r, struct damon_target *t)294static void damon_destroy_region(struct damon_region *r,
295 struct damon_target *t)
180{296{
181 damon_del_region(r, t);297 damon_del_region(r, t);
182 damon_free_region(r);298 damon_free_region(r);
@@ -252,11 +368,25 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 69 unmodified lines
252 damon_destroy_region(r, t);368 damon_destroy_region(r, t);
253 }369 }
254 370
371 if (!damon_nr_regions(t)) {
372 for (i = 0; i < nr_ranges; i++) {
373 r = damon_new_region(
374 ALIGN_DOWN(ranges[i].start,
375 min_region_sz),
376 ALIGN(ranges[i].end, min_region_sz));
377 if (!r)
378 return -ENOMEM;
379 damon_add_region(r, t);
380 }
381 return 0;
382 }
383
255 r = damon_first_region(t);384 r = damon_first_region(t);
256 /* Add new regions or resize existing regions to fit in the ranges */385 /* Add new regions or resize existing regions to fit in the ranges */
257 for (i = 0; i < nr_ranges; i++) {386 for (i = 0; i < nr_ranges; i++) {
258 struct damon_region *first = NULL, *last, *newr;387 struct damon_region *first = NULL, *last, *newr;
259 struct damon_addr_range *range;388 struct damon_addr_range *range;
389 bool insert_before_r = false;
260 390
261 range = &ranges[i];391 range = &ranges[i];
262 /* Get the first/last regions intersecting with the range */392 /* Get the first/last regions intersecting with the range */
@@ -266,8 +396,10 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 3 unmodified lines
266 first = r;396 first = r;
267 last = r;397 last = r;
268 }398 }
269 if (r->ar.start >= range->end)399 if (r->ar.start >= range->end) {
400 insert_before_r = true;
270 break;401 break;
402 }
271 }403 }
272 if (!first) {404 if (!first) {
273 /* no region intersects with this range */405 /* no region intersects with this range */
@@ -277,7 +409,11 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 3 unmodified lines
277 ALIGN(range->end, min_region_sz));409 ALIGN(range->end, min_region_sz));
278 if (!newr)410 if (!newr)
279 return -ENOMEM;411 return -ENOMEM;
280 damon_insert_region(newr, damon_prev_region(r), r, t);412 if (insert_before_r)
413 damon_insert_region(newr, damon_prev_region(r),
414 r, t);
415 else
416 damon_add_region(newr, t);
281 } else {417 } else {
282 /* resize intersecting regions to fit in this range */418 /* resize intersecting regions to fit in this range */
283 first->ar.start = ALIGN_DOWN(range->start,419 first->ar.start = ALIGN_DOWN(range->start,
@@ -550,27 +686,8 @@ void damon_destroy_target(struct damon_target *t, struct damon_ctx *ctx) · 266 unmodified lines
550 damon_free_target(t);686 damon_free_target(t);
551}687}
552 688
553#ifdef CONFIG_DAMON_DEBUG_SANITY
554static void damon_verify_nr_regions(struct damon_target *t)
555{
556 struct damon_region *r;
557 unsigned int count = 0;
558
559 damon_for_each_region(r, t)
560 count++;
561 WARN_ONCE(count != t->nr_regions, "t->nr_regions (%u) != count (%u)\n",
562 t->nr_regions, count);
563}
564#else
565static void damon_verify_nr_regions(struct damon_target *t)
566{
567}
568#endif
569
570unsigned int damon_nr_regions(struct damon_target *t)689unsigned int damon_nr_regions(struct damon_target *t)
571{690{
572 damon_verify_nr_regions(t);
573
574 return t->nr_regions;691 return t->nr_regions;
575}692}
576 693
@@ -601,12 +718,16 @@ struct damon_ctx *damon_new_ctx(void) · 24 unmodified lines
601 ctx->attrs.min_nr_regions = 10;718 ctx->attrs.min_nr_regions = 10;
602 ctx->attrs.max_nr_regions = 1000;719 ctx->attrs.max_nr_regions = 1000;
603 720
721 INIT_LIST_HEAD(&ctx->probes);
722
604 ctx->addr_unit = 1;723 ctx->addr_unit = 1;
605 ctx->min_region_sz = DAMON_MIN_REGION_SZ;724 ctx->min_region_sz = DAMON_MIN_REGION_SZ;
606 725
607 INIT_LIST_HEAD(&ctx->adaptive_targets);726 INIT_LIST_HEAD(&ctx->adaptive_targets);
608 INIT_LIST_HEAD(&ctx->schemes);727 INIT_LIST_HEAD(&ctx->schemes);
609 728
729 prandom_seed_state(&ctx->rnd_state, get_random_u64());
730
610 return ctx;731 return ctx;
611}732}
612 733
@@ -621,12 +742,16 @@ static void damon_destroy_targets(struct damon_ctx *ctx) · 8 unmodified lines
621void damon_destroy_ctx(struct damon_ctx *ctx)742void damon_destroy_ctx(struct damon_ctx *ctx)
622{743{
623 struct damos *s, *next_s;744 struct damos *s, *next_s;
745 struct damon_probe *p, *next_p;
624 746
625 damon_destroy_targets(ctx);747 damon_destroy_targets(ctx);
626 748
627 damon_for_each_scheme_safe(s, next_s, ctx)749 damon_for_each_scheme_safe(s, next_s, ctx)
628 damon_destroy_scheme(s);750 damon_destroy_scheme(s);
629 751
752 damon_for_each_probe_safe(p, next_p, ctx)
753 damon_destroy_probe(p);
754
630 kfree(ctx);755 kfree(ctx);
631}756}
632 757
@@ -797,6 +922,9 @@ int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) · 164 unmodified lines
797 attrs->aggr_interval / sample_interval;922 attrs->aggr_interval / sample_interval;
798 ctx->next_ops_update_sis = ctx->passed_sample_intervals +923 ctx->next_ops_update_sis = ctx->passed_sample_intervals +
799 attrs->ops_update_interval / sample_interval;924 attrs->ops_update_interval / sample_interval;
925 /*
926 * next_intervals_tune_sis will be updated inside kdamond_fn().
927 */
800 928
801 damon_update_monitoring_results(ctx, attrs, aggregating);929 damon_update_monitoring_results(ctx, attrs, aggregating);
802 ctx->attrs = *attrs;930 ctx->attrs = *attrs;
@@ -918,6 +1046,8 @@ static int damos_commit_quota(struct damos_quota *dst, struct damos_quota *src) · 115 unmodified lines
918 if (err)1046 if (err)
919 return err;1047 return err;
920 dst->goal_tuner = src->goal_tuner;1048 dst->goal_tuner = src->goal_tuner;
1049 dst->fail_charge_num = src->fail_charge_num;
1050 dst->fail_charge_denom = src->fail_charge_denom;
921 dst->weight_sz = src->weight_sz;1051 dst->weight_sz = src->weight_sz;
922 dst->weight_nr_accesses = src->weight_nr_accesses;1052 dst->weight_nr_accesses = src->weight_nr_accesses;
923 dst->weight_age = src->weight_age;1053 dst->weight_age = src->weight_age;
@@ -1310,6 +1440,86 @@ static int damon_commit_targets( · 386 unmodified lines
1310 return 0;1440 return 0;
1311}1441}
1312 1442
1443static void damon_commit_filter(struct damon_filter *dst,
1444 struct damon_filter *src)
1445{
1446 dst->type = src->type;
1447 dst->matching = src->matching;
1448 dst->allow = src->allow;
1449 switch (dst->type) {
1450 case DAMON_FILTER_TYPE_MEMCG:
1451 dst->memcg_id = src->memcg_id;
1452 break;
1453 default:
1454 break;
1455 }
1456}
1457
1458static int damon_commit_filters(struct damon_probe *dst,
1459 struct damon_probe *src)
1460{
1461 struct damon_filter *dst_filter, *next, *src_filter, *new_filter;
1462 int i = 0, j = 0;
1463
1464 damon_for_each_filter_safe(dst_filter, next, dst) {
1465 src_filter = damon_nth_filter(i++, src);
1466 if (src_filter)
1467 damon_commit_filter(dst_filter, src_filter);
1468 else
1469 damon_destroy_filter(dst_filter);
1470 }
1471
1472 damon_for_each_filter_safe(src_filter, next, src) {
1473 if (j++ < i)
1474 continue;
1475
1476 new_filter = damon_new_filter(src_filter->type,
1477 src_filter->matching, src_filter->allow);
1478 if (!new_filter)
1479 return -ENOMEM;
1480 switch (src_filter->type) {
1481 case DAMON_FILTER_TYPE_MEMCG:
1482 new_filter->memcg_id = src_filter->memcg_id;
1483 break;
1484 default:
1485 break;
1486 }
1487 damon_add_filter(dst, new_filter);
1488 }
1489 return 0;
1490}
1491
1492static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src)
1493{
1494 struct damon_probe *dst_probe, *next, *src_probe, *new_probe;
1495 int i = 0, j = 0, err;
1496
1497 damon_for_each_probe_safe(dst_probe, next, dst) {
1498 src_probe = damon_nth_probe(i++, src);
1499 if (src_probe) {
1500 err = damon_commit_filters(dst_probe, src_probe);
1501 if (err)
1502 return err;
1503 } else {
1504 damon_destroy_probe(dst_probe);
1505 }
1506 }
1507
1508 damon_for_each_probe_safe(src_probe, next, src) {
1509 if (j++ < i)
1510 continue;
1511
1512 new_probe = damon_new_probe();
1513 if (!new_probe)
1514 return -ENOMEM;
1515 damon_add_probe(dst, new_probe);
1516 err = damon_commit_filters(new_probe, src_probe);
1517 if (err)
1518 return err;
1519 }
1520 return 0;
1521}
1522
1313/**1523/**
1314 * damon_commit_ctx() - Commit parameters of a DAMON context to another.1524 * damon_commit_ctx() - Commit parameters of a DAMON context to another.
1315 * @dst: The commit destination DAMON context.1525 * @dst: The commit destination DAMON context.
@@ -1326,11 +1536,26 @@ static int damon_commit_targets( · 10 unmodified lines
1326int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)1536int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
1327{1537{
1328 int err;1538 int err;
1539 struct damos *scheme;
1540 struct damos_quota_goal *goal;
1329 1541
1330 dst->maybe_corrupted = true;1542 dst->maybe_corrupted = true;
1331 if (!is_power_of_2(src->min_region_sz))1543 if (!is_power_of_2(src->min_region_sz))
1332 return -EINVAL;1544 return -EINVAL;
1333 1545
1546 /* node_eligible_mem_bp metric requires PADDR ops */
1547 if (src->ops.id != DAMON_OPS_PADDR) {
1548 damon_for_each_scheme(scheme, src) {
1549 struct damos_quota *quota = &scheme->quota;
1550
1551 damos_for_each_quota_goal(goal, quota) {
1552 if (goal->metric ==
1553 DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP)
1554 return -EINVAL;
1555 }
1556 }
1557 }
1558
1334 err = damon_commit_schemes(dst, src);1559 err = damon_commit_schemes(dst, src);
1335 if (err)1560 if (err)
1336 return err;1561 return err;
@@ -1349,7 +1574,11 @@ int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src) · 12 unmodified lines
1349 if (err)1574 if (err)
1350 return err;1575 return err;
1351 }1576 }
1577 dst->pause = src->pause;
1352 dst->ops = src->ops;1578 dst->ops = src->ops;
1579 err = damon_commit_probes(dst, src);
1580 if (err)
1581 return err;
1353 dst->addr_unit = src->addr_unit;1582 dst->addr_unit = src->addr_unit;
1354 dst->min_region_sz = src->min_region_sz;1583 dst->min_region_sz = src->min_region_sz;
1355 1584
@@ -1706,15 +1935,28 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) · 350 unmodified lines
1706{1935{
1707 struct damon_target *t;1936 struct damon_target *t;
1708 unsigned int ti = 0; /* target's index */1937 unsigned int ti = 0; /* target's index */
1938 unsigned int nr_probes = 0;
1939 struct damon_probe *probe;
1940
1941 if (trace_damon_region_aggregated_enabled()) {
1942 damon_for_each_probe(probe, c)
1943 nr_probes++;
1944 }
1709 1945
1710 damon_for_each_target(t, c) {1946 damon_for_each_target(t, c) {
1711 struct damon_region *r;1947 struct damon_region *r;
1712 1948
1713 damon_for_each_region(r, t) {1949 damon_for_each_region(r, t) {
1950 int i;
1951
1714 trace_damon_aggregated(ti, r, damon_nr_regions(t));1952 trace_damon_aggregated(ti, r, damon_nr_regions(t));
1953 trace_damon_region_aggregated(ti, r,
1954 damon_nr_regions(t), nr_probes);
1715 damon_warn_fix_nr_accesses_corruption(r);1955 damon_warn_fix_nr_accesses_corruption(r);
1716 r->last_nr_accesses = r->nr_accesses;1956 r->last_nr_accesses = r->nr_accesses;
1717 r->nr_accesses = 0;1957 r->nr_accesses = 0;
1958 for (i = 0; i < DAMON_MAX_PROBES; i++)
1959 r->probe_hits[i] = 0;
1718 damon_verify_reset_aggregated(r, c);1960 damon_verify_reset_aggregated(r, c);
1719 }1961 }
1720 ti++;1962 ti++;
@@ -2046,6 +2288,37 @@ static void damos_walk_cancel(struct damon_ctx *ctx) · 325 unmodified lines
2046 mutex_unlock(&ctx->walk_control_lock);2288 mutex_unlock(&ctx->walk_control_lock);
2047}2289}
2048 2290
2291static void damos_charge_quota(struct damos_quota *quota,
2292 unsigned long sz_region, unsigned long sz_applied)
2293{
2294 /*
2295 * sz_applied could be bigger than sz_region, depending on ops
2296 * implementation of the action, e.g., damos_pa_pageout(). Charge only
2297 * the region size in the case.
2298 */
2299 if (!quota->fail_charge_denom || sz_applied > sz_region)
2300 quota->charged_sz += sz_region;
2301 else
2302 quota->charged_sz += sz_applied + mult_frac(
2303 (sz_region - sz_applied),
2304 quota->fail_charge_num,
2305 quota->fail_charge_denom);
2306}
2307
2308static bool damos_quota_is_full(struct damos_quota *quota,
2309 unsigned long min_region_sz)
2310{
2311 if (!damos_quota_is_set(quota))
2312 return false;
2313 if (quota->charged_sz >= quota->esz)
2314 return true;
2315 /*
2316 * DAMOS action is applied per region, so <min_region_sz remaining
2317 * quota means the quota is effectively full.
2318 */
2319 return quota->esz - quota->charged_sz < min_region_sz;
2320}
2321
2049static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t,2322static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t,
2050 struct damon_region *r, struct damos *s)2323 struct damon_region *r, struct damos *s)
2051{2324{
@@ -2102,11 +2375,10 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 50 unmodified lines
2102 ktime_get_coarse_ts64(&end);2375 ktime_get_coarse_ts64(&end);
2103 quota->total_charged_ns += timespec64_to_ns(&end) -2376 quota->total_charged_ns += timespec64_to_ns(&end) -
2104 timespec64_to_ns(&begin);2377 timespec64_to_ns(&begin);
2105 quota->charged_sz += sz;2378 damos_charge_quota(quota, sz, sz_applied);
2106 if (damos_quota_is_set(quota) &&2379 if (damos_quota_is_full(quota, c->min_region_sz)) {
2107 quota->charged_sz >= quota->esz) {
2108 quota->charge_target_from = t;2380 quota->charge_target_from = t;
2109 quota->charge_addr_from = r->ar.end + 1;2381 quota->charge_addr_from = r->ar.end;
2110 }2382 }
2111 }2383 }
2112 if (s->action != DAMOS_STAT)2384 if (s->action != DAMOS_STAT)
@@ -2132,8 +2404,7 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 19 unmodified lines
2132 continue;2404 continue;
2133 2405
2134 /* Check the quota */2406 /* Check the quota */
2135 if (damos_quota_is_set(quota) &&2407 if (damos_quota_is_full(quota, c->min_region_sz))
2136 quota->charged_sz >= quota->esz)
2137 continue;2408 continue;
2138 2409
2139 if (damos_skip_charged_region(t, r, s, c->min_region_sz))2410 if (damos_skip_charged_region(t, r, s, c->min_region_sz))
@@ -2151,6 +2422,58 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 11 unmodified lines
2151 }2422 }
2152}2423}
2153 2424
2425/*
2426 * damos_apply_target() - Apply DAMOS schemes to a given target.
2427 * @c: monitoring context to apply its DAMOS schemes to..
2428 * @t: monitoring target to apply the schemes to.
2429 * @max_region_sz: maximum region size for @c.
2430 *
2431 * This function could split regions for keeping the quota. To minimize
2432 * overhead from the split operations increased number of regions, this
2433 * function will also merge regions after the schemes applying attempt is done,
2434 * for each region. The merge operation is made only when it doesn't lose the
2435 * monitoring information and not violating @max_region_sz.
2436 *
2437 * Hence, after this function is called, the total number of regions could
2438 * be increased or reduced. The increase could make max_nr_regions temporarily
2439 * be violated, until the next per-aggregation interval regions merge operation
2440 * is executed. The decrease will not violate min_nr_regions though, since it
2441 * keeps @max_region_sz.
2442 */
2443static void damos_apply_target(struct damon_ctx *c, struct damon_target *t,
2444 unsigned long max_region_sz)
2445{
2446 struct damon_region *r;
2447
2448 damon_for_each_region(r, t) {
2449 struct damon_region *prev_r;
2450
2451 damon_do_apply_schemes(c, t, r);
2452 /*
2453 * damon_do_apply_scheems() could split the region for the
2454 * quota. Keeping the new slices is an overhead. Merge back
2455 * the slices into the previous region if it doesn't lose any
2456 * information and not violating the max_region_sz.
2457 */
2458 if (damon_first_region(t) == r)
2459 continue;
2460 prev_r = damon_prev_region(r);
2461 if (prev_r->ar.end != r->ar.start)
2462 continue;
2463 if (prev_r->age != r->age)
2464 continue;
2465 if (prev_r->last_nr_accesses != r->last_nr_accesses)
2466 continue;
2467 if (prev_r->nr_accesses != r->nr_accesses)
2468 continue;
2469 if (r->ar.end - prev_r->ar.start > max_region_sz)
2470 continue;
2471 prev_r->ar.end = r->ar.end;
2472 damon_destroy_region(r, t);
2473 r = prev_r;
2474 }
2475}
2476
2154/*2477/*
2155 * damon_feed_loop_next_input() - get next input to achieve a target score.2478 * damon_feed_loop_next_input() - get next input to achieve a target score.
2156 * @last_input The last input.2479 * @last_input The last input.
@@ -2287,7 +2610,115 @@ static unsigned long damos_get_node_memcg_used_bp( · 130 unmodified lines
2287 numerator = i.totalram - used_pages;2610 numerator = i.totalram - used_pages;
2288 return mult_frac(numerator, 10000, i.totalram);2611 return mult_frac(numerator, 10000, i.totalram);
2289}2612}
2290#else2613
2614#ifdef CONFIG_DAMON_PADDR
2615/*
2616 * damos_calc_eligible_bytes() - Calculate raw eligible bytes per node.
2617 * @c: The DAMON context.
2618 * @s: The scheme.
2619 * @nid: The target NUMA node id.
2620 * @total: Output for total eligible bytes across all nodes.
2621 *
2622 * Iterates through each folio in eligible regions to accurately determine
2623 * which node the memory resides on. Returns eligible bytes on the specified
2624 * node and sets *total to the sum across all nodes.
2625 *
2626 * Note: This function requires damon_get_folio() from ops-common.c, which is
2627 * only available when CONFIG_DAMON_PADDR is enabled. It also requires the
2628 * context to be using PADDR operations for meaningful results.
2629 */
2630static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c,
2631 struct damos *s, int nid, phys_addr_t *total)
2632{
2633 struct damon_target *t;
2634 struct damon_region *r;
2635 phys_addr_t total_eligible = 0;
2636 phys_addr_t node_eligible = 0;
2637
2638 damon_for_each_target(t, c) {
2639 damon_for_each_region(r, t) {
2640 phys_addr_t addr, end_addr;
2641
2642 if (!__damos_valid_target(r, s))
2643 continue;
2644
2645 /* Convert from core address units to physical bytes */
2646 addr = (phys_addr_t)r->ar.start * c->addr_unit;
2647 end_addr = (phys_addr_t)r->ar.end * c->addr_unit;
2648 while (addr < end_addr) {
2649 struct folio *folio;
2650 phys_addr_t folio_start, folio_end;
2651 phys_addr_t overlap_start, overlap_end;
2652 phys_addr_t counted;
2653
2654 folio = damon_get_folio(PHYS_PFN(addr));
2655 if (!folio) {
2656 addr = PAGE_ALIGN_DOWN(addr +
2657 PAGE_SIZE);
2658 if (!addr)
2659 break;
2660 continue;
2661 }
2662
2663 /*
2664 * Calculate exact overlap between the region
2665 * [addr, end_addr) and the folio range.
2666 * The folio may start before addr if addr is
2667 * in the middle of a large folio.
2668 */
2669 folio_start = PFN_PHYS(folio_pfn(folio));
2670 folio_end = folio_start + folio_size(folio);
2671
2672 overlap_start = max(addr, folio_start);
2673 overlap_end = min(end_addr, folio_end);
2674
2675 if (overlap_end > overlap_start) {
2676 counted = overlap_end - overlap_start;
2677 total_eligible += counted;
2678 if (folio_nid(folio) == nid)
2679 node_eligible += counted;
2680 }
2681
2682 /* Advance past the entire folio */
2683 addr = folio_end;
2684 folio_put(folio);
2685 }
2686 cond_resched();
2687 }
2688 }
2689
2690 *total = total_eligible;
2691 return node_eligible;
2692}
2693
2694static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2695 struct damos *s, int nid)
2696{
2697 phys_addr_t total_eligible = 0;
2698 phys_addr_t node_eligible;
2699
2700 if (c->ops.id != DAMON_OPS_PADDR)
2701 return 0;
2702
2703 if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
2704 return 0;
2705
2706 node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
2707
2708 if (!(unsigned long)total_eligible)
2709 return 0;
2710
2711 return mult_frac((unsigned long)node_eligible, 10000,
2712 (unsigned long)total_eligible);
2713}
2714#else /* CONFIG_DAMON_PADDR */
2715static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2716 struct damos *s, int nid)
2717{
2718 return 0;
2719}
2720#endif /* CONFIG_DAMON_PADDR */
2721#else /* CONFIG_NUMA */
2291static __kernel_ulong_t damos_get_node_mem_bp(2722static __kernel_ulong_t damos_get_node_mem_bp(
2292 struct damos_quota_goal *goal)2723 struct damos_quota_goal *goal)
2293{2724{
@@ -2299,7 +2730,13 @@ static unsigned long damos_get_node_memcg_used_bp( · 5 unmodified lines
2299{2730{
2300 return 0;2731 return 0;
2301}2732}
2302#endif2733
2734static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2735 struct damos *s, int nid)
2736{
2737 return 0;
2738}
2739#endif /* CONFIG_NUMA */
2303 2740
2304/*2741/*
2305 * Returns LRU-active or inactive memory to total LRU memory size ratio.2742 * Returns LRU-active or inactive memory to total LRU memory size ratio.
@@ -2319,7 +2756,8 @@ static unsigned int damos_get_in_active_mem_bp(bool active_ratio) · 13 unmodified lines
2319 return mult_frac(inactive, 10000, total);2756 return mult_frac(inactive, 10000, total);
2320}2757}
2321 2758
2322static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal)2759static void damos_set_quota_goal_current_value(struct damon_ctx *c,
2760 struct damos *s, struct damos_quota_goal *goal)
2323{2761{
2324 u64 now_psi_total;2762 u64 now_psi_total;
2325 2763
@@ -2345,19 +2783,24 @@ static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal) · 19 unmodified lines
2345 goal->current_value = damos_get_in_active_mem_bp(2783 goal->current_value = damos_get_in_active_mem_bp(
2346 goal->metric == DAMOS_QUOTA_ACTIVE_MEM_BP);2784 goal->metric == DAMOS_QUOTA_ACTIVE_MEM_BP);
2347 break;2785 break;
2786 case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
2787 goal->current_value = damos_get_node_eligible_mem_bp(c, s,
2788 goal->nid);
2789 break;
2348 default:2790 default:
2349 break;2791 break;
2350 }2792 }
2351}2793}
2352 2794
2353/* Return the highest score since it makes schemes least aggressive */2795/* Return the highest score since it makes schemes least aggressive */
2354static unsigned long damos_quota_score(struct damos_quota *quota)2796static unsigned long damos_quota_score(struct damon_ctx *c, struct damos *s)
2355{2797{
2356 struct damos_quota_goal *goal;2798 struct damos_quota_goal *goal;
2799 struct damos_quota *quota = &s->quota;
2357 unsigned long highest_score = 0;2800 unsigned long highest_score = 0;
2358 2801
2359 damos_for_each_quota_goal(goal, quota) {2802 damos_for_each_quota_goal(goal, quota) {
2360 damos_set_quota_goal_current_value(goal);2803 damos_set_quota_goal_current_value(c, s, goal);
2361 highest_score = max(highest_score,2804 highest_score = max(highest_score,
2362 mult_frac(goal->current_value, 10000,2805 mult_frac(goal->current_value, 10000,
2363 goal->target_value));2806 goal->target_value));
@@ -2366,17 +2809,20 @@ static unsigned long damos_quota_score(struct damos_quota *quota) · 2 unmodified lines
2366 return highest_score;2809 return highest_score;
2367}2810}
2368 2811
2369static void damos_goal_tune_esz_bp_consist(struct damos_quota *quota)2812static void damos_goal_tune_esz_bp_consist(struct damon_ctx *c, struct damos *s)
2370{2813{
2371 unsigned long score = damos_quota_score(quota);2814 struct damos_quota *quota = &s->quota;
2815 unsigned long score = damos_quota_score(c, s);
2372 2816
2373 quota->esz_bp = damon_feed_loop_next_input(2817 quota->esz_bp = damon_feed_loop_next_input(
2374 max(quota->esz_bp, 10000UL), score);2818 max(quota->esz_bp, 10000UL), score);
2375}2819}
2376 2820
2377static void damos_goal_tune_esz_bp_temporal(struct damos_quota *quota)2821static void damos_goal_tune_esz_bp_temporal(struct damon_ctx *c,
2822 struct damos *s)
2378{2823{
2379 unsigned long score = damos_quota_score(quota);2824 struct damos_quota *quota = &s->quota;
2825 unsigned long score = damos_quota_score(c, s);
2380 2826
2381 if (score >= 10000)2827 if (score >= 10000)
2382 quota->esz_bp = 0;2828 quota->esz_bp = 0;
@@ -2389,9 +2835,9 @@ static void damos_goal_tune_esz_bp_temporal(struct damos_quota *quota) · 6 unmodified lines
2389/*2835/*
2390 * Called only if quota->ms, or quota->sz are set, or quota->goals is not empty2836 * Called only if quota->ms, or quota->sz are set, or quota->goals is not empty
2391 */2837 */
2392static void damos_set_effective_quota(struct damos_quota *quota,2838static void damos_set_effective_quota(struct damon_ctx *ctx, struct damos *s)
2393 struct damon_ctx *ctx)
2394{2839{
2840 struct damos_quota *quota = &s->quota;
2395 unsigned long throughput;2841 unsigned long throughput;
2396 unsigned long esz = ULONG_MAX;2842 unsigned long esz = ULONG_MAX;
2397 2843
@@ -2402,9 +2848,9 @@ static void damos_set_effective_quota(struct damos_quota *quota, · 4 unmodified lines
2402 2848
2403 if (!list_empty(&quota->goals)) {2849 if (!list_empty(&quota->goals)) {
2404 if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_CONSIST)2850 if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_CONSIST)
2405 damos_goal_tune_esz_bp_consist(quota);2851 damos_goal_tune_esz_bp_consist(ctx, s);
2406 else if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_TEMPORAL)2852 else if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_TEMPORAL)
2407 damos_goal_tune_esz_bp_temporal(quota);2853 damos_goal_tune_esz_bp_temporal(ctx, s);
2408 esz = quota->esz_bp / 10000;2854 esz = quota->esz_bp / 10000;
2409 }2855 }
2410 2856
@@ -2452,22 +2898,23 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) · 41 unmodified lines
2452 /* First charge window */2898 /* First charge window */
2453 if (!quota->total_charged_sz && !quota->charged_from) {2899 if (!quota->total_charged_sz && !quota->charged_from) {
2454 quota->charged_from = jiffies;2900 quota->charged_from = jiffies;
2455 damos_set_effective_quota(quota, c);2901 damos_set_effective_quota(c, s);
2902 if (trace_damos_esz_enabled())
2903 damos_trace_esz(c, s, quota);
2456 }2904 }
2457 2905
2458 /* New charge window starts */2906 /* New charge window starts */
2459 if (!time_in_range_open(jiffies, quota->charged_from,2907 if (!time_in_range_open(jiffies, quota->charged_from,
2460 quota->charged_from +2908 quota->charged_from +
2461 msecs_to_jiffies(quota->reset_interval))) {2909 msecs_to_jiffies(quota->reset_interval))) {
2462 if (damos_quota_is_set(quota) &&2910 if (damos_quota_is_full(quota, c->min_region_sz))
2463 quota->charged_sz >= quota->esz)
2464 s->stat.qt_exceeds++;2911 s->stat.qt_exceeds++;
2465 quota->total_charged_sz += quota->charged_sz;2912 quota->total_charged_sz += quota->charged_sz;
2466 quota->charged_from = jiffies;2913 quota->charged_from = jiffies;
2467 quota->charged_sz = 0;2914 quota->charged_sz = 0;
2468 if (trace_damos_esz_enabled())2915 if (trace_damos_esz_enabled())
2469 cached_esz = quota->esz;2916 cached_esz = quota->esz;
2470 damos_set_effective_quota(quota, c);2917 damos_set_effective_quota(c, s);
2471 if (trace_damos_esz_enabled() && quota->esz != cached_esz)2918 if (trace_damos_esz_enabled() && quota->esz != cached_esz)
2472 damos_trace_esz(c, s, quota);2919 damos_trace_esz(c, s, quota);
2473 }2920 }
@@ -2521,9 +2968,9 @@ static void damos_trace_stat(struct damon_ctx *c, struct damos *s) · 47 unmodified lines
2521static void kdamond_apply_schemes(struct damon_ctx *c)2968static void kdamond_apply_schemes(struct damon_ctx *c)
2522{2969{
2523 struct damon_target *t;2970 struct damon_target *t;
2524 struct damon_region *r;
2525 struct damos *s;2971 struct damos *s;
2526 bool has_schemes_to_apply = false;2972 bool has_schemes_to_apply = false;
2973 unsigned long max_region_sz;
2527 2974
2528 damon_for_each_scheme(s, c) {2975 damon_for_each_scheme(s, c) {
2529 if (time_before(c->passed_sample_intervals, s->next_apply_sis))2976 if (time_before(c->passed_sample_intervals, s->next_apply_sis))
@@ -2540,13 +2987,12 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 10 unmodified lines
2540 if (!has_schemes_to_apply)2987 if (!has_schemes_to_apply)
2541 return;2988 return;
2542 2989
2990 max_region_sz = damon_region_sz_limit(c);
2543 mutex_lock(&c->walk_control_lock);2991 mutex_lock(&c->walk_control_lock);
2544 damon_for_each_target(t, c) {2992 damon_for_each_target(t, c) {
2545 if (c->ops.target_valid && c->ops.target_valid(t) == false)2993 if (c->ops.target_valid && c->ops.target_valid(t) == false)
2546 continue;2994 continue;
2547 2995 damos_apply_target(c, t, max_region_sz);
2548 damon_for_each_region(r, t)
2549 damon_do_apply_schemes(c, t, r);
2550 }2996 }
2551 2997
2552 damon_for_each_scheme(s, c) {2998 damon_for_each_scheme(s, c) {
@@ -2582,12 +3028,17 @@ static void damon_merge_two_regions(struct damon_target *t, · 29 unmodified lines
2582 struct damon_region *l, struct damon_region *r)3028 struct damon_region *l, struct damon_region *r)
2583{3029{
2584 unsigned long sz_l = damon_sz_region(l), sz_r = damon_sz_region(r);3030 unsigned long sz_l = damon_sz_region(l), sz_r = damon_sz_region(r);
3031 int i;
2585 3032
2586 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /3033 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
2587 (sz_l + sz_r);3034 (sz_l + sz_r);
2588 l->nr_accesses_bp = l->nr_accesses * 10000;3035 l->nr_accesses_bp = l->nr_accesses * 10000;
2589 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);3036 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
2590 l->ar.end = r->ar.end;3037 l->ar.end = r->ar.end;
3038 /* todo: do this for only installed probes */
3039 for (i = 0; i < DAMON_MAX_PROBES; i++)
3040 l->probe_hits[i] = (l->probe_hits[i] * sz_l + r->probe_hits[i]
3041 * sz_r) / (sz_l + sz_r);
2591 damon_verify_merge_two_regions(l, r);3042 damon_verify_merge_two_regions(l, r);
2592 damon_destroy_region(r, t);3043 damon_destroy_region(r, t);
2593}3044}
@@ -2710,13 +3161,16 @@ static void damon_split_region_at(struct damon_target *t, · 116 unmodified lines
2710 new->last_nr_accesses = r->last_nr_accesses;3161 new->last_nr_accesses = r->last_nr_accesses;
2711 new->nr_accesses_bp = r->nr_accesses_bp;3162 new->nr_accesses_bp = r->nr_accesses_bp;
2712 new->nr_accesses = r->nr_accesses;3163 new->nr_accesses = r->nr_accesses;
3164 /* todo: do this for only installed probes */
3165 memcpy(new->probe_hits, r->probe_hits, sizeof(r->probe_hits));
2713 3166
2714 damon_insert_region(new, r, damon_next_region(r), t);3167 damon_insert_region(new, r, damon_next_region(r), t);
2715}3168}
2716 3169
2717/* Split every region in the given target into 'nr_subs' regions */3170/* Split every region in the given target into 'nr_subs' regions */
2718static void damon_split_regions_of(struct damon_target *t, int nr_subs,3171static void damon_split_regions_of(struct damon_ctx *ctx,
2719 unsigned long min_region_sz)3172 struct damon_target *t, int nr_subs,
3173 unsigned long min_region_sz)
2720{3174{
2721 struct damon_region *r, *next;3175 struct damon_region *r, *next;
2722 unsigned long sz_region, sz_sub = 0;3176 unsigned long sz_region, sz_sub = 0;
@@ -2731,7 +3185,7 @@ static void damon_split_regions_of(struct damon_target *t, int nr_subs, · 8 unmodified lines
2731 * Randomly select size of left sub-region to be at3185 * Randomly select size of left sub-region to be at
2732 * least 10 percent and at most 90% of original region3186 * least 10 percent and at most 90% of original region
2733 */3187 */
2734 sz_sub = ALIGN_DOWN(damon_rand(1, 10) *3188 sz_sub = ALIGN_DOWN(damon_rand(ctx, 1, 10) *
2735 sz_region / 10, min_region_sz);3189 sz_region / 10, min_region_sz);
2736 /* Do not allow blank region */3190 /* Do not allow blank region */
2737 if (sz_sub == 0 || sz_sub >= sz_region)3191 if (sz_sub == 0 || sz_sub >= sz_region)
@@ -2772,7 +3226,8 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 34 unmodified lines
2772 nr_subregions = 3;3226 nr_subregions = 3;
2773 3227
2774 damon_for_each_target(t, ctx)3228 damon_for_each_target(t, ctx)
2775 damon_split_regions_of(t, nr_subregions, ctx->min_region_sz);3229 damon_split_regions_of(ctx, t, nr_subregions,
3230 ctx->min_region_sz);
2776 3231
2777 last_nr_regions = nr_regions;3232 last_nr_regions = nr_regions;
2778}3233}
@@ -2857,6 +3312,37 @@ static void kdamond_usleep(unsigned long usecs) · 78 unmodified lines
2857 usleep_range_idle(usecs, usecs + 1);3312 usleep_range_idle(usecs, usecs + 1);
2858}3313}
2859 3314
3315#ifdef CONFIG_DAMON_DEBUG_SANITY
3316static void damon_verify_ctx(struct damon_ctx *c)
3317{
3318 struct damon_target *t;
3319 struct damon_region *r;
3320
3321 damon_for_each_target(t, c) {
3322 struct damon_region *prev_r = NULL;
3323 unsigned int nr_regions = 0;
3324
3325 damon_for_each_region(r, t) {
3326 WARN_ONCE(r->ar.start >= r->ar.end,
3327 "region start (%lu) >= end (%lu)\n",
3328 r->ar.start, r->ar.end);
3329 WARN_ONCE(prev_r && prev_r->ar.end > r->ar.start,
3330 "region overlap (%lu > %lu)\n",
3331 prev_r->ar.end, r->ar.start);
3332 prev_r = r;
3333 nr_regions++;
3334 }
3335 WARN_ONCE(damon_nr_regions(t) != nr_regions,
3336 "nr_regions mismatch: %u != %u\n",
3337 damon_nr_regions(t), nr_regions);
3338 }
3339}
3340#else
3341static void damon_verify_ctx(struct damon_ctx *c)
3342{
3343}
3344#endif
3345
2860/*3346/*
2861 * kdamond_call() - handle damon_call_control objects.3347 * kdamond_call() - handle damon_call_control objects.
2862 * @ctx: The &struct damon_ctx of the kdamond.3348 * @ctx: The &struct damon_ctx of the kdamond.
@@ -2872,6 +3358,8 @@ static void kdamond_call(struct damon_ctx *ctx, bool cancel) · 9 unmodified lines
2872 struct damon_call_control *control, *next;3358 struct damon_call_control *control, *next;
2873 LIST_HEAD(controls);3359 LIST_HEAD(controls);
2874 3360
3361 damon_verify_ctx(ctx);
3362
2875 mutex_lock(&ctx->call_controls_lock);3363 mutex_lock(&ctx->call_controls_lock);
2876 list_splice_tail_init(&ctx->call_controls, &controls);3364 list_splice_tail_init(&ctx->call_controls, &controls);
2877 mutex_unlock(&ctx->call_controls_lock);3365 mutex_unlock(&ctx->call_controls_lock);
@@ -2997,6 +3485,8 @@ static int kdamond_fn(void *data) · 119 unmodified lines
2997 3485
2998 if (ctx->ops.check_accesses)3486 if (ctx->ops.check_accesses)
2999 max_nr_accesses = ctx->ops.check_accesses(ctx);3487 max_nr_accesses = ctx->ops.check_accesses(ctx);
3488 if (ctx->ops.apply_probes)
3489 ctx->ops.apply_probes(ctx);
3000 3490
3001 if (time_after_eq(ctx->passed_sample_intervals,3491 if (time_after_eq(ctx->passed_sample_intervals,
3002 next_aggregation_sis)) {3492 next_aggregation_sis)) {
@@ -3014,6 +3504,14 @@ static int kdamond_fn(void *data) · 11 unmodified lines
3014 kdamond_call(ctx, false);3504 kdamond_call(ctx, false);
3015 if (ctx->maybe_corrupted)3505 if (ctx->maybe_corrupted)
3016 break;3506 break;
3507 while (ctx->pause) {
3508 damos_walk_cancel(ctx);
3509 kdamond_usleep(ctx->attrs.sample_interval);
3510 /* allow caller unset pause via damon_call() */
3511 kdamond_call(ctx, false);
3512 if (kdamond_need_stop(ctx) || ctx->maybe_corrupted)
3513 goto done;
3514 }
3017 if (!list_empty(&ctx->schemes))3515 if (!list_empty(&ctx->schemes))
3018 kdamond_apply_schemes(ctx);3516 kdamond_apply_schemes(ctx);
3019 else3517 else
@@ -3096,14 +3594,20 @@ done: · 76 unmodified lines
3096 return 0;3594 return 0;
3097}3595}
3098 3596
3099static int walk_system_ram(struct resource *res, void *arg)3597struct damon_system_ram_range_walk_arg {
3598 bool walked;
3599 struct resource res;
3600};
3601
3602static int damon_system_ram_walk_fn(struct resource *res, void *arg)
3100{3603{
3101 struct resource *a = arg;3604 struct damon_system_ram_range_walk_arg *a = arg;
3102 3605
3103 if (resource_size(a) < resource_size(res)) {3606 if (!a->walked) {
3104 a->start = res->start;3607 a->walked = true;
3105 a->end = res->end;3608 a->res.start = res->start;
3106 }3609 }
3610 a->res.end = res->end;
3107 return 0;3611 return 0;
3108}3612}
3109 3613
@@ -3120,27 +3624,24 @@ static unsigned long damon_res_to_core_addr(resource_size_t ra, · 10 unmodified lines
3120 return ra / addr_unit;3624 return ra / addr_unit;
3121}3625}
3122 3626
3123/*3627static bool damon_find_system_rams_range(unsigned long *start,
3124 * Find biggest 'System RAM' resource and store its start and end address in
3125 * @start and @end, respectively. If no System RAM is found, returns false.
3126 */
3127static bool damon_find_biggest_system_ram(unsigned long *start,
3128 unsigned long *end, unsigned long addr_unit)3628 unsigned long *end, unsigned long addr_unit)
3129
3130{3629{
3131 struct resource res = {};3630 struct damon_system_ram_range_walk_arg arg = {};
3132 3631
3133 walk_system_ram_res(0, -1, &res, walk_system_ram);3632 walk_system_ram_res(0, -1, &arg, damon_system_ram_walk_fn);
3134 *start = damon_res_to_core_addr(res.start, addr_unit);3633 if (!arg.walked)
3135 *end = damon_res_to_core_addr(res.end + 1, addr_unit);3634 return false;
3635 *start = damon_res_to_core_addr(arg.res.start, addr_unit);
3636 *end = damon_res_to_core_addr(arg.res.end + 1, addr_unit);
3136 if (*end <= *start)3637 if (*end <= *start)
3137 return false;3638 return false;
3138 return true;3639 return true;
3139}3640}
3140 3641
3141/**3642/**
3142 * damon_set_region_biggest_system_ram_default() - Set the region of the given3643 * damon_set_region_system_rams_default() - Set the region of the given
3143 * monitoring target as requested, or biggest 'System RAM'.3644 * monitoring target as requested, or to cover all 'System RAM' resources.
3144 * @t: The monitoring target to set the region.3645 * @t: The monitoring target to set the region.
3145 * @start: The pointer to the start address of the region.3646 * @start: The pointer to the start address of the region.
3146 * @end: The pointer to the end address of the region.3647 * @end: The pointer to the end address of the region.
@@ -3148,14 +3649,14 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 1 unmodified line
3148 * @min_region_sz: Minimum region size.3649 * @min_region_sz: Minimum region size.
3149 *3650 *
3150 * This function sets the region of @t as requested by @start and @end. If the3651 * This function sets the region of @t as requested by @start and @end. If the
3151 * values of @start and @end are zero, however, this function finds the biggest3652 * values of @start and @end are zero, however, this function finds 'System
3152 * 'System RAM' resource and sets the region to cover the resource. In the3653 * RAM' resources and sets the region to cover all the resource. In the latter
3153 * latter case, this function saves the start and end addresses of the resource3654 * case, this function saves the start and the end addresseses of the first and
3154 * in @start and @end, respectively.3655 * the last resources in @start and @end, respectively.
3155 *3656 *
3156 * Return: 0 on success, negative error code otherwise.3657 * Return: 0 on success, negative error code otherwise.
3157 */3658 */
3158int damon_set_region_biggest_system_ram_default(struct damon_target *t,3659int damon_set_region_system_rams_default(struct damon_target *t,
3159 unsigned long *start, unsigned long *end,3660 unsigned long *start, unsigned long *end,
3160 unsigned long addr_unit, unsigned long min_region_sz)3661 unsigned long addr_unit, unsigned long min_region_sz)
3161{3662{
@@ -3165,7 +3666,7 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 3 unmodified lines
3165 return -EINVAL;3666 return -EINVAL;
3166 3667
3167 if (!*start && !*end &&3668 if (!*start && !*end &&
3168 !damon_find_biggest_system_ram(start, end, addr_unit))3669 !damon_find_system_rams_range(start, end, addr_unit))
3169 return -EINVAL;3670 return -EINVAL;
3170 3671
3171 addr_range.start = *start;3672 addr_range.start = *start;
mm/damon/lru_sort.c+49−10
@@ -39,7 +39,6 @@ static bool enabled __read_mostly; · 38 unmodified lines
39 * the re-reading, DAMON_LRU_SORT will be disabled.39 * the re-reading, DAMON_LRU_SORT will be disabled.
40 */40 */
41static bool commit_inputs __read_mostly;41static bool commit_inputs __read_mostly;
42module_param(commit_inputs, bool, 0600);
43 42
44/*43/*
45 * Desired active to [in]active memory ratio in bp (1/10,000).44 * Desired active to [in]active memory ratio in bp (1/10,000).
@@ -140,7 +139,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_lru_sort_mon_attrs); · 94 unmodified lines
140 * Start of the target memory region in physical address.139 * Start of the target memory region in physical address.
141 *140 *
142 * The start physical address of memory region that DAMON_LRU_SORT will do work141 * The start physical address of memory region that DAMON_LRU_SORT will do work
143 * against. By default, biggest System RAM is used as the region.142 * against. By default, the system's entire physical memory is used as the
143 * region.
144 */144 */
145static unsigned long monitor_region_start __read_mostly;145static unsigned long monitor_region_start __read_mostly;
146module_param(monitor_region_start, ulong, 0600);146module_param(monitor_region_start, ulong, 0600);
@@ -149,7 +149,8 @@ module_param(monitor_region_start, ulong, 0600); · 2 unmodified lines
149 * End of the target memory region in physical address.149 * End of the target memory region in physical address.
150 *150 *
151 * The end physical address of memory region that DAMON_LRU_SORT will do work151 * The end physical address of memory region that DAMON_LRU_SORT will do work
152 * against. By default, biggest System RAM is used as the region.152 * against. By default, the system's entire physical memory is used as the
153 * region.
153 */154 */
154static unsigned long monitor_region_end __read_mostly;155static unsigned long monitor_region_end __read_mostly;
155module_param(monitor_region_end, ulong, 0600);156module_param(monitor_region_end, ulong, 0600);
@@ -285,6 +286,11 @@ static int damon_lru_sort_apply_parameters(void) · 129 unmodified lines
285 param_ctx->addr_unit = addr_unit;286 param_ctx->addr_unit = addr_unit;
286 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);287 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
287 288
289 if (!is_power_of_2(param_ctx->min_region_sz)) {
290 err = -EINVAL;
291 goto out;
292 }
293
288 if (!damon_lru_sort_mon_attrs.sample_interval) {294 if (!damon_lru_sort_mon_attrs.sample_interval) {
289 err = -EINVAL;295 err = -EINVAL;
290 goto out;296 goto out;
@@ -327,7 +333,7 @@ static int damon_lru_sort_apply_parameters(void) · 36 unmodified lines
327 if (err)333 if (err)
328 goto out;334 goto out;
329 335
330 err = damon_set_region_biggest_system_ram_default(param_target,336 err = damon_set_region_system_rams_default(param_target,
331 &monitor_region_start,337 &monitor_region_start,
332 &monitor_region_end,338 &monitor_region_end,
333 param_ctx->addr_unit,339 param_ctx->addr_unit,
@@ -340,18 +346,51 @@ out: · 6 unmodified lines
340 return err;346 return err;
341}347}
342 348
343static int damon_lru_sort_handle_commit_inputs(void)349static int damon_lru_sort_commit_inputs_fn(void *arg)
350{
351 return damon_lru_sort_apply_parameters();
352}
353
354static int damon_lru_sort_commit_inputs_store(const char *val,
355 const struct kernel_param *kp)
344{356{
357 bool commit_inputs_request;
345 int err;358 int err;
359 struct damon_call_control control = {
360 .fn = damon_lru_sort_commit_inputs_fn,
361 };
362
363 if (!val) {
364 commit_inputs_request = true;
365 } else {
366 err = kstrtobool(val, &commit_inputs_request);
367 if (err)
368 return err;
369 }
346 370
347 if (!commit_inputs)371 if (!commit_inputs_request)
348 return 0;372 return 0;
349 373
350 err = damon_lru_sort_apply_parameters();374 /*
351 commit_inputs = false;375 * Skip damon_call() if ctx is not initialized to avoid
352 return err;376 * NULL pointer dereference.
377 */
378 if (!ctx)
379 return -EINVAL;
380
381 err = damon_call(ctx, &control);
382
383 return err ? err : control.return_code;
353}384}
354 385
386static const struct kernel_param_ops commit_inputs_param_ops = {
387 .flags = KERNEL_PARAM_OPS_FL_NOARG,
388 .set = damon_lru_sort_commit_inputs_store,
389 .get = param_get_bool,
390};
391
392module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
393
355static int damon_lru_sort_damon_call_fn(void *arg)394static int damon_lru_sort_damon_call_fn(void *arg)
356{395{
357 struct damon_ctx *c = arg;396 struct damon_ctx *c = arg;
@@ -365,7 +404,7 @@ static int damon_lru_sort_damon_call_fn(void *arg) · 7 unmodified lines
365 damon_lru_sort_cold_stat = s->stat;404 damon_lru_sort_cold_stat = s->stat;
366 }405 }
367 406
368 return damon_lru_sort_handle_commit_inputs();407 return 0;
369}408}
370 409
371static struct damon_call_control call_control = {410static struct damon_call_control call_control = {
mm/damon/ops-common.c+6−3
@@ -117,9 +117,12 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, · 116 unmodified lines
117 damon_max_nr_accesses(&c->attrs);117 damon_max_nr_accesses(&c->attrs);
118 118
119 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;119 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
120 for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;120 if (age_in_sec)
121 age_in_log++, age_in_sec >>= 1)121 age_in_log = min_t(int, ilog2(age_in_sec) + 1,
122 ;122 DAMON_MAX_AGE_IN_LOG);
123 else
124 age_in_log = 0;
125
123 126
124 /* If frequency is 0, higher age means it's colder */127 /* If frequency is 0, higher age means it's colder */
125 if (freq_subscore == 0)128 if (freq_subscore == 0)
mm/damon/paddr.c+80−4
@@ -49,11 +49,11 @@ static void damon_pa_mkold(phys_addr_t paddr) · 48 unmodified lines
49}49}
50 50
51static void __damon_pa_prepare_access_check(struct damon_region *r,51static void __damon_pa_prepare_access_check(struct damon_region *r,
52 unsigned long addr_unit)52 struct damon_ctx *ctx)
53{53{
54 r->sampling_addr = damon_rand(r->ar.start, r->ar.end);54 r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
55 55
56 damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, addr_unit));56 damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit));
57}57}
58 58
59static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)59static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
@@ -63,7 +63,7 @@ static void damon_pa_prepare_access_checks(struct damon_ctx *ctx) · 3 unmodified lines
63 63
64 damon_for_each_target(t, ctx) {64 damon_for_each_target(t, ctx) {
65 damon_for_each_region(r, t)65 damon_for_each_region(r, t)
66 __damon_pa_prepare_access_check(r, ctx->addr_unit);66 __damon_pa_prepare_access_check(r, ctx);
67 }67 }
68}68}
69 69
@@ -120,6 +120,81 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx) · 50 unmodified lines
120 return max_nr_accesses;120 return max_nr_accesses;
121}121}
122 122
123static bool damon_pa_filter_match(struct damon_filter *filter,
124 struct folio *folio)
125{
126 bool matched = false;
127 struct mem_cgroup *memcg;
128
129 switch (filter->type) {
130 case DAMON_FILTER_TYPE_ANON:
131 if (!folio) {
132 matched = false;
133 break;
134 }
135 matched = folio_test_anon(folio);
136 break;
137 case DAMON_FILTER_TYPE_MEMCG:
138 if (!folio) {
139 matched = false;
140 break;
141 }
142 rcu_read_lock();
143 memcg = folio_memcg_check(folio);
144 if (!memcg)
145 matched = false;
146 else
147 matched = filter->memcg_id == mem_cgroup_id(memcg);
148 rcu_read_unlock();
149 break;
150 default:
151 break;
152 }
153 return matched == filter->matching;
154}
155
156static bool damon_pa_filter_pass(phys_addr_t pa, struct folio *folio,
157 struct damon_probe *p)
158{
159 struct damon_filter *f;
160 bool pass = true;
161
162 damon_for_each_filter(f, p) {
163 if (damon_pa_filter_match(f, folio)) {
164 pass = f->allow;
165 break;
166 }
167 pass = !f->allow;
168 }
169 return pass;
170}
171
172static void damon_pa_apply_probes(struct damon_ctx *ctx)
173{
174 struct damon_target *t;
175 struct damon_region *r;
176 struct damon_probe *p;
177
178 damon_for_each_target(t, ctx) {
179 damon_for_each_region(r, t) {
180 int i = 0;
181 phys_addr_t pa;
182 struct folio *folio;
183
184 pa = damon_pa_phys_addr(r->sampling_addr,
185 ctx->addr_unit);
186 folio = damon_get_folio(PHYS_PFN(pa));
187 damon_for_each_probe(p, ctx) {
188 if (damon_pa_filter_pass(pa, folio, p))
189 r->probe_hits[i]++;
190 i++;
191 }
192 if (folio)
193 folio_put(folio);
194 }
195 }
196}
197
123/*198/*
124 * damos_pa_filter_out - Return true if the page should be filtered out.199 * damos_pa_filter_out - Return true if the page should be filtered out.
125 */200 */
@@ -371,6 +446,7 @@ static int __init damon_pa_initcall(void) · 245 unmodified lines
371 .update = NULL,446 .update = NULL,
372 .prepare_access_checks = damon_pa_prepare_access_checks,447 .prepare_access_checks = damon_pa_prepare_access_checks,
373 .check_accesses = damon_pa_check_accesses,448 .check_accesses = damon_pa_check_accesses,
449 .apply_probes = damon_pa_apply_probes,
374 .target_valid = NULL,450 .target_valid = NULL,
375 .apply_scheme = damon_pa_apply_scheme,451 .apply_scheme = damon_pa_apply_scheme,
376 .get_scheme_score = damon_pa_scheme_score,452 .get_scheme_score = damon_pa_scheme_score,
mm/damon/reclaim.c+79−19
@@ -39,7 +39,6 @@ static bool enabled __read_mostly; · 38 unmodified lines
39 * re-reading, DAMON_RECLAIM will be disabled.39 * re-reading, DAMON_RECLAIM will be disabled.
40 */40 */
41static bool commit_inputs __read_mostly;41static bool commit_inputs __read_mostly;
42module_param(commit_inputs, bool, 0600);
43 42
44/*43/*
45 * Time threshold for cold memory regions identification in microseconds.44 * Time threshold for cold memory regions identification in microseconds.
@@ -92,6 +91,20 @@ module_param(quota_mem_pressure_us, ulong, 0600); · 46 unmodified lines
92static unsigned long quota_autotune_feedback __read_mostly;91static unsigned long quota_autotune_feedback __read_mostly;
93module_param(quota_autotune_feedback, ulong, 0600);92module_param(quota_autotune_feedback, ulong, 0600);
94 93
94/*
95 * Auto-tune monitoring intervals.
96 *
97 * If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
98 * sampling and aggregation intervals. The auto-tuning aims to capture
99 * meaningful amount of access events in each DAMON-snapshot, while keeping the
100 * sampling intervals 5 milliseconds in minimum, and 10 seconds in maximum.
101 * Setting this as ``N`` disables the auto-tuning.
102 *
103 * Disabled by default.
104 */
105static bool autotune_monitoring_intervals __read_mostly;
106module_param(autotune_monitoring_intervals, bool, 0600);
107
95static struct damos_watermarks damon_reclaim_wmarks = {108static struct damos_watermarks damon_reclaim_wmarks = {
96 .metric = DAMOS_WMARK_FREE_MEM_RATE,109 .metric = DAMOS_WMARK_FREE_MEM_RATE,
97 .interval = 5000000, /* 5 seconds */110 .interval = 5000000, /* 5 seconds */
@@ -114,7 +127,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_reclaim_mon_attrs); · 16 unmodified lines
114 * Start of the target memory region in physical address.127 * Start of the target memory region in physical address.
115 *128 *
116 * The start physical address of memory region that DAMON_RECLAIM will do work129 * The start physical address of memory region that DAMON_RECLAIM will do work
117 * against. By default, biggest System RAM is used as the region.130 * against. By default, the system's entire physical memory is used as the
131 * region.
118 */132 */
119static unsigned long monitor_region_start __read_mostly;133static unsigned long monitor_region_start __read_mostly;
120module_param(monitor_region_start, ulong, 0600);134module_param(monitor_region_start, ulong, 0600);
@@ -123,7 +137,8 @@ module_param(monitor_region_start, ulong, 0600); · 2 unmodified lines
123 * End of the target memory region in physical address.137 * End of the target memory region in physical address.
124 *138 *
125 * The end physical address of memory region that DAMON_RECLAIM will do work139 * The end physical address of memory region that DAMON_RECLAIM will do work
126 * against. By default, biggest System RAM is used as the region.140 * against. By default, the system's entire physical memory is used as the
141 * region.
127 */142 */
128static unsigned long monitor_region_end __read_mostly;143static unsigned long monitor_region_end __read_mostly;
129module_param(monitor_region_end, ulong, 0600);144module_param(monitor_region_end, ulong, 0600);
@@ -151,7 +166,7 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_reclaim_stat, · 21 unmodified lines
151static struct damon_ctx *ctx;166static struct damon_ctx *ctx;
152static struct damon_target *target;167static struct damon_target *target;
153 168
154static struct damos *damon_reclaim_new_scheme(void)169static struct damos *damon_reclaim_new_scheme(unsigned long aggr_interval)
155{170{
156 struct damos_access_pattern pattern = {171 struct damos_access_pattern pattern = {
157 /* Find regions having PAGE_SIZE or larger size */172 /* Find regions having PAGE_SIZE or larger size */
@@ -161,8 +176,7 @@ static struct damos *damon_reclaim_new_scheme(void) · 3 unmodified lines
161 .min_nr_accesses = 0,176 .min_nr_accesses = 0,
162 .max_nr_accesses = 0,177 .max_nr_accesses = 0,
163 /* for min_age or more micro-seconds */178 /* for min_age or more micro-seconds */
164 .min_age_region = min_age /179 .min_age_region = min_age / aggr_interval,
165 damon_reclaim_mon_attrs.aggr_interval,
166 .max_age_region = UINT_MAX,180 .max_age_region = UINT_MAX,
167 };181 };
168 182
@@ -183,6 +197,7 @@ static int damon_reclaim_apply_parameters(void) · 14 unmodified lines
183{197{
184 struct damon_ctx *param_ctx;198 struct damon_ctx *param_ctx;
185 struct damon_target *param_target;199 struct damon_target *param_target;
200 struct damon_attrs attrs;
186 struct damos *scheme;201 struct damos *scheme;
187 struct damos_quota_goal *goal;202 struct damos_quota_goal *goal;
188 struct damos_filter *filter;203 struct damos_filter *filter;
@@ -195,17 +210,31 @@ static int damon_reclaim_apply_parameters(void) · 6 unmodified lines
195 param_ctx->addr_unit = addr_unit;210 param_ctx->addr_unit = addr_unit;
196 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);211 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
197 212
213 if (!is_power_of_2(param_ctx->min_region_sz)) {
214 err = -EINVAL;
215 goto out;
216 }
217
198 if (!damon_reclaim_mon_attrs.aggr_interval) {218 if (!damon_reclaim_mon_attrs.aggr_interval) {
199 err = -EINVAL;219 err = -EINVAL;
200 goto out;220 goto out;
201 }221 }
202 222
203 err = damon_set_attrs(param_ctx, &damon_reclaim_mon_attrs);223 attrs = damon_reclaim_mon_attrs;
224 if (autotune_monitoring_intervals) {
225 attrs.sample_interval = 5000;
226 attrs.aggr_interval = 100000;
227 attrs.intervals_goal.access_bp = 40;
228 attrs.intervals_goal.aggrs = 3;
229 attrs.intervals_goal.min_sample_us = 5000;
230 attrs.intervals_goal.max_sample_us = 10 * 1000 * 1000;
231 }
232 err = damon_set_attrs(param_ctx, &attrs);
204 if (err)233 if (err)
205 goto out;234 goto out;
206 235
207 err = -ENOMEM;236 err = -ENOMEM;
208 scheme = damon_reclaim_new_scheme();237 scheme = damon_reclaim_new_scheme(attrs.aggr_interval);
209 if (!scheme)238 if (!scheme)
210 goto out;239 goto out;
211 damon_set_schemes(param_ctx, &scheme, 1);240 damon_set_schemes(param_ctx, &scheme, 1);
@@ -233,11 +262,9 @@ static int damon_reclaim_apply_parameters(void) · 21 unmodified lines
233 damos_add_filter(scheme, filter);262 damos_add_filter(scheme, filter);
234 }263 }
235 264
236 err = damon_set_region_biggest_system_ram_default(param_target,265 err = damon_set_region_system_rams_default(param_target,
237 &monitor_region_start,266 &monitor_region_start, &monitor_region_end,
238 &monitor_region_end,267 param_ctx->addr_unit, param_ctx->min_region_sz);
239 param_ctx->addr_unit,
240 param_ctx->min_region_sz);
241 if (err)268 if (err)
242 goto out;269 goto out;
243 err = damon_commit_ctx(ctx, param_ctx);270 err = damon_commit_ctx(ctx, param_ctx);
@@ -246,18 +273,51 @@ out: · 2 unmodified lines
246 return err;273 return err;
247}274}
248 275
249static int damon_reclaim_handle_commit_inputs(void)276static int damon_reclaim_commit_inputs_fn(void *arg)
250{277{
278 return damon_reclaim_apply_parameters();
279}
280
281static int damon_reclaim_commit_inputs_store(const char *val,
282 const struct kernel_param *kp)
283{
284 bool commit_inputs_request;
251 int err;285 int err;
286 struct damon_call_control control = {
287 .fn = damon_reclaim_commit_inputs_fn,
288 };
252 289
253 if (!commit_inputs)290 if (!val) {
291 commit_inputs_request = true;
292 } else {
293 err = kstrtobool(val, &commit_inputs_request);
294 if (err)
295 return err;
296 }
297
298 if (!commit_inputs_request)
254 return 0;299 return 0;
255 300
256 err = damon_reclaim_apply_parameters();301 /*
257 commit_inputs = false;302 * Skip damon_call() if ctx is not initialized to avoid
258 return err;303 * NULL pointer dereference.
304 */
305 if (!ctx)
306 return -EINVAL;
307
308 err = damon_call(ctx, &control);
309
310 return err ? err : control.return_code;
259}311}
260 312
313static const struct kernel_param_ops commit_inputs_param_ops = {
314 .flags = KERNEL_PARAM_OPS_FL_NOARG,
315 .set = damon_reclaim_commit_inputs_store,
316 .get = param_get_bool,
317};
318
319module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
320
261static int damon_reclaim_damon_call_fn(void *arg)321static int damon_reclaim_damon_call_fn(void *arg)
262{322{
263 struct damon_ctx *c = arg;323 struct damon_ctx *c = arg;
@@ -267,7 +327,7 @@ static int damon_reclaim_damon_call_fn(void *arg) · 3 unmodified lines
267 damon_for_each_scheme(s, c)327 damon_for_each_scheme(s, c)
268 damon_reclaim_stat = s->stat;328 damon_reclaim_stat = s->stat;
269 329
270 return damon_reclaim_handle_commit_inputs();330 return 0;
271}331}
272 332
273static struct damon_call_control call_control = {333static struct damon_call_control call_control = {
mm/damon/stat.c+42−50
@@ -148,59 +148,12 @@ static int damon_stat_damon_call_fn(void *data) · 147 unmodified lines
148 return 0;148 return 0;
149}149}
150 150
151struct damon_stat_system_ram_range_walk_arg {
152 bool walked;
153 struct resource res;
154};
155
156static int damon_stat_system_ram_walk_fn(struct resource *res, void *arg)
157{
158 struct damon_stat_system_ram_range_walk_arg *a = arg;
159
160 if (!a->walked) {
161 a->walked = true;
162 a->res.start = res->start;
163 }
164 a->res.end = res->end;
165 return 0;
166}
167
168static unsigned long damon_stat_res_to_core_addr(resource_size_t ra,
169 unsigned long addr_unit)
170{
171 /*
172 * Use div_u64() for avoiding linking errors related with __udivdi3,
173 * __aeabi_uldivmod, or similar problems. This should also improve the
174 * performance optimization (read div_u64() comment for the detail).
175 */
176 if (sizeof(ra) == 8 && sizeof(addr_unit) == 4)
177 return div_u64(ra, addr_unit);
178 return ra / addr_unit;
179}
180
181static int damon_stat_set_monitoring_region(struct damon_target *t,
182 unsigned long addr_unit, unsigned long min_region_sz)
183{
184 struct damon_addr_range addr_range;
185 struct damon_stat_system_ram_range_walk_arg arg = {};
186
187 walk_system_ram_res(0, -1, &arg, damon_stat_system_ram_walk_fn);
188 if (!arg.walked)
189 return -EINVAL;
190 addr_range.start = damon_stat_res_to_core_addr(
191 arg.res.start, addr_unit);
192 addr_range.end = damon_stat_res_to_core_addr(
193 arg.res.end + 1, addr_unit);
194 if (addr_range.end <= addr_range.start)
195 return -EINVAL;
196 return damon_set_regions(t, &addr_range, 1, min_region_sz);
197}
198
199static struct damon_ctx *damon_stat_build_ctx(void)151static struct damon_ctx *damon_stat_build_ctx(void)
200{152{
201 struct damon_ctx *ctx;153 struct damon_ctx *ctx;
202 struct damon_attrs attrs;154 struct damon_attrs attrs;
203 struct damon_target *target;155 struct damon_target *target;
156 unsigned long start = 0, end = 0;
204 157
205 ctx = damon_new_ctx();158 ctx = damon_new_ctx();
206 if (!ctx)159 if (!ctx)
@@ -230,8 +183,8 @@ static struct damon_ctx *damon_stat_build_ctx(void) · 23 unmodified lines
230 if (!target)183 if (!target)
231 goto free_out;184 goto free_out;
232 damon_add_target(ctx, target);185 damon_add_target(ctx, target);
233 if (damon_stat_set_monitoring_region(target, ctx->addr_unit,186 if (damon_set_region_system_rams_default(target, &start, &end,
234 ctx->min_region_sz))187 ctx->addr_unit, ctx->min_region_sz))
235 goto free_out;188 goto free_out;
236 return ctx;189 return ctx;
237free_out:190free_out:
@@ -313,6 +266,45 @@ static int damon_stat_enabled_load(char *buffer, const struct kernel_param *kp) · 75 unmodified lines
313 return sprintf(buffer, "%c\n", damon_stat_enabled() ? 'Y' : 'N');266 return sprintf(buffer, "%c\n", damon_stat_enabled() ? 'Y' : 'N');
314}267}
315 268
269static int damon_stat_kdamond_pid_store(
270 const char *val, const struct kernel_param *kp)
271{
272 /*
273 * kdamond_pid is read-only, but kernel command line could write it.
274 * Do nothing here.
275 */
276 return 0;
277}
278
279static int damon_stat_kdamond_pid_load(
280 char *buffer, const struct kernel_param *kp)
281{
282 int pid;
283
284 if (!damon_stat_context) {
285 pid = -1;
286 } else {
287 pid = damon_kdamond_pid(damon_stat_context);
288 if (pid < 1)
289 pid = -1;
290 }
291 return sprintf(buffer, "%d\n", pid);
292}
293
294static const struct kernel_param_ops kdamond_pid_param_ops = {
295 .set = damon_stat_kdamond_pid_store,
296 .get = damon_stat_kdamond_pid_load,
297};
298
299/*
300 * PID of the DAMON thread
301 *
302 * If DAMON_STAT is enabled, this becomes the PID of the worker thread.
303 * Else, -1.
304 */
305module_param_cb(kdamond_pid, &kdamond_pid_param_ops, NULL, 0400);
306MODULE_PARM_DESC(kdamond_pid, "pid of the kdamond");
307
316static int __init damon_stat_init(void)308static int __init damon_stat_init(void)
317{309{
318 int err = 0;310 int err = 0;
mm/damon/sysfs-common.c+41−0
@@ -104,3 +104,44 @@ const struct kobj_type damon_sysfs_ul_range_ktype = { · 103 unmodified lines
104 .default_groups = damon_sysfs_ul_range_groups,104 .default_groups = damon_sysfs_ul_range_groups,
105};105};
106 106
107
108static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
109 char *memcg_path_buf, char *path)
110{
111#ifdef CONFIG_MEMCG
112 cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
113 if (sysfs_streq(memcg_path_buf, path))
114 return true;
115#endif /* CONFIG_MEMCG */
116 return false;
117}
118
119int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
120{
121 struct mem_cgroup *memcg;
122 char *path;
123 bool found = false;
124
125 if (!memcg_path)
126 return -EINVAL;
127
128 path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
129 if (!path)
130 return -ENOMEM;
131
132 for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
133 memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
134 /* skip offlined memcg */
135 if (!mem_cgroup_online(memcg))
136 continue;
137 if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
138 *id = mem_cgroup_id(memcg);
139 found = true;
140 mem_cgroup_iter_break(NULL, memcg);
141 break;
142 }
143 }
144
145 kfree(path);
146 return found ? 0 : -EINVAL;
147}
mm/damon/sysfs-common.h+2−0
@@ -59,3 +59,5 @@ int damos_sysfs_set_quota_scores(struct damon_sysfs_schemes *sysfs_schemes, · 58 unmodified lines
59void damos_sysfs_update_effective_quotas(59void damos_sysfs_update_effective_quotas(
60 struct damon_sysfs_schemes *sysfs_schemes,60 struct damon_sysfs_schemes *sysfs_schemes,
61 struct damon_ctx *ctx);61 struct damon_ctx *ctx);
62
63int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id);
mm/damon/sysfs-schemes.c+244−45
@@ -10,6 +10,140 @@ · 9 unmodified lines
10 10
11#include "sysfs-common.h"11#include "sysfs-common.h"
12 12
13/*
14 * probe directory
15 */
16
17struct damos_sysfs_probe {
18 struct kobject kobj;
19 unsigned char hits;
20};
21
22static struct damos_sysfs_probe *damos_sysfs_probe_alloc(unsigned char hits)
23{
24 struct damos_sysfs_probe *probe;
25
26 probe = kzalloc_obj(*probe);
27 if (!probe)
28 return NULL;
29 probe->hits = hits;
30 return probe;
31}
32
33static ssize_t hits_show(struct kobject *kobj, struct kobj_attribute *attr,
34 char *buf)
35{
36 struct damos_sysfs_probe *probe = container_of(kobj,
37 struct damos_sysfs_probe, kobj);
38
39 return sysfs_emit(buf, "%hhu\n", probe->hits);
40}
41
42static void damos_sysfs_probe_release(struct kobject *kobj)
43{
44 struct damos_sysfs_probe *probe = container_of(kobj,
45 struct damos_sysfs_probe, kobj);
46
47 kfree(probe);
48}
49
50static struct kobj_attribute damos_sysfs_probe_hits_attr =
51 __ATTR_RO_MODE(hits, 0400);
52
53static struct attribute *damos_sysfs_probe_attrs[] = {
54 &damos_sysfs_probe_hits_attr.attr,
55 NULL,
56};
57ATTRIBUTE_GROUPS(damos_sysfs_probe);
58
59static const struct kobj_type damos_sysfs_probe_ktype = {
60 .release = damos_sysfs_probe_release,
61 .sysfs_ops = &kobj_sysfs_ops,
62 .default_groups = damos_sysfs_probe_groups,
63};
64
65/*
66 * probes directory
67 */
68
69struct damos_sysfs_probes {
70 struct kobject kobj;
71 struct damos_sysfs_probe **probes_arr;
72 int nr;
73};
74
75static struct damos_sysfs_probes *damos_sysfs_probes_alloc(void)
76{
77 return kzalloc_obj(struct damos_sysfs_probes);
78}
79
80static void damos_sysfs_probes_rm_dirs(struct damos_sysfs_probes *probes)
81{
82 struct damos_sysfs_probe **probes_arr = probes->probes_arr;
83 int i;
84
85 for (i = 0; i < probes->nr; i++)
86 kobject_put(&probes_arr[i]->kobj);
87 probes->nr = 0;
88 kfree(probes_arr);
89 probes->probes_arr = NULL;
90}
91
92static int damos_sysfs_probes_add_dirs(struct damos_sysfs_probes *probes,
93 struct damon_ctx *ctx, struct damon_region *region)
94{
95 struct damon_probe *probe;
96 struct damos_sysfs_probe **probes_arr;
97 int i = 0;
98
99 damon_for_each_probe(probe, ctx)
100 i++;
101
102 if (!i)
103 return 0;
104
105 probes_arr = kmalloc_objs(*probes_arr, i);
106 if (!probes_arr)
107 return -ENOMEM;
108 probes->probes_arr = probes_arr;
109
110 i = 0;
111 damon_for_each_probe(probe, ctx) {
112 struct damos_sysfs_probe *sys_probe;
113 int err;
114
115 sys_probe = damos_sysfs_probe_alloc(region->probe_hits[i]);
116 if (!sys_probe) {
117 damos_sysfs_probes_rm_dirs(probes);
118 return -ENOMEM;
119 }
120 err = kobject_init_and_add(&sys_probe->kobj,
121 &damos_sysfs_probe_ktype, &probes->kobj, "%d",
122 i);
123 if (err) {
124 kobject_put(&sys_probe->kobj);
125 damos_sysfs_probes_rm_dirs(probes);
126 return err;
127 }
128 probes_arr[i++] = sys_probe;
129 probes->nr++;
130 }
131 return 0;
132}
133
134static void damos_sysfs_probes_release(struct kobject *kobj)
135{
136 struct damos_sysfs_probes *probes = container_of(kobj,
137 struct damos_sysfs_probes, kobj);
138
139 kfree(probes);
140}
141
142static const struct kobj_type damos_sysfs_probes_ktype = {
143 .release = damos_sysfs_probes_release,
144 .sysfs_ops = &kobj_sysfs_ops,
145};
146
13/*147/*
14 * scheme region directory148 * scheme region directory
15 */149 */
@@ -20,6 +154,7 @@ struct damon_sysfs_scheme_region { · 4 unmodified lines
20 unsigned int nr_accesses;154 unsigned int nr_accesses;
21 unsigned int age;155 unsigned int age;
22 unsigned long sz_filter_passed;156 unsigned long sz_filter_passed;
157 struct damos_sysfs_probes *probes;
23 struct list_head list;158 struct list_head list;
24};159};
25 160
@@ -34,10 +169,44 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc( · 8 unmodified lines
34 sysfs_region->ar = region->ar;169 sysfs_region->ar = region->ar;
35 sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;170 sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;
36 sysfs_region->age = region->age;171 sysfs_region->age = region->age;
172 sysfs_region->probes = NULL;
37 INIT_LIST_HEAD(&sysfs_region->list);173 INIT_LIST_HEAD(&sysfs_region->list);
38 return sysfs_region;174 return sysfs_region;
39}175}
40 176
177static int damos_sysfs_region_add_dirs(
178 struct damon_sysfs_scheme_region *region,
179 struct damon_ctx *ctx,
180 struct damon_region *dregion)
181{
182 struct damos_sysfs_probes *probes = damos_sysfs_probes_alloc();
183 int err;
184
185 if (!probes)
186 return -ENOMEM;
187 err = kobject_init_and_add(&probes->kobj, &damos_sysfs_probes_ktype,
188 &region->kobj, "probes");
189 if (err)
190 goto fail;
191 err = damos_sysfs_probes_add_dirs(probes, ctx, dregion);
192 if (err)
193 goto fail;
194
195 region->probes = probes;
196 return 0;
197
198fail:
199 kobject_put(&probes->kobj);
200 return err;
201}
202
203static void damos_sysfs_region_rm_dirs(
204 struct damon_sysfs_scheme_region *region)
205{
206 damos_sysfs_probes_rm_dirs(region->probes);
207 kobject_put(&region->probes->kobj);
208}
209
41static ssize_t start_show(struct kobject *kobj, struct kobj_attribute *attr,210static ssize_t start_show(struct kobject *kobj, struct kobj_attribute *attr,
42 char *buf)211 char *buf)
43{212{
@@ -163,6 +332,7 @@ static void damon_sysfs_scheme_regions_rm_dirs( · 119 unmodified lines
163 struct damon_sysfs_scheme_region *r, *next;332 struct damon_sysfs_scheme_region *r, *next;
164 333
165 list_for_each_entry_safe(r, next, &regions->regions_list, list) {334 list_for_each_entry_safe(r, next, &regions->regions_list, list) {
335 damos_sysfs_region_rm_dirs(r);
166 list_del(&r->list);336 list_del(&r->list);
167 kobject_put(&r->kobj);337 kobject_put(&r->kobj);
168 regions->nr_regions--;338 regions->nr_regions--;
@@ -1093,6 +1263,10 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = { · 924 unmodified lines
1093 .metric = DAMOS_QUOTA_INACTIVE_MEM_BP,1263 .metric = DAMOS_QUOTA_INACTIVE_MEM_BP,
1094 .name = "inactive_mem_bp",1264 .name = "inactive_mem_bp",
1095 },1265 },
1266 {
1267 .metric = DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
1268 .name = "node_eligible_mem_bp",
1269 },
1096};1270};
1097 1271
1098static ssize_t target_metric_show(struct kobject *kobj,1272static ssize_t target_metric_show(struct kobject *kobj,
@@ -1508,6 +1682,8 @@ struct damon_sysfs_quotas { · 409 unmodified lines
1508 unsigned long reset_interval_ms;1682 unsigned long reset_interval_ms;
1509 unsigned long effective_sz; /* Effective size quota in bytes */1683 unsigned long effective_sz; /* Effective size quota in bytes */
1510 enum damos_quota_goal_tuner goal_tuner;1684 enum damos_quota_goal_tuner goal_tuner;
1685 unsigned int fail_charge_num;
1686 unsigned int fail_charge_denom;
1511};1687};
1512 1688
1513static struct damon_sysfs_quotas *damon_sysfs_quotas_alloc(void)1689static struct damon_sysfs_quotas *damon_sysfs_quotas_alloc(void)
@@ -1682,6 +1858,48 @@ static ssize_t goal_tuner_store(struct kobject *kobj, · 168 unmodified lines
1682 return -EINVAL;1858 return -EINVAL;
1683}1859}
1684 1860
1861static ssize_t fail_charge_num_show(struct kobject *kobj,
1862 struct kobj_attribute *attr, char *buf)
1863{
1864 struct damon_sysfs_quotas *quotas = container_of(kobj,
1865 struct damon_sysfs_quotas, kobj);
1866
1867 return sysfs_emit(buf, "%u\n", quotas->fail_charge_num);
1868}
1869
1870static ssize_t fail_charge_num_store(struct kobject *kobj,
1871 struct kobj_attribute *attr, const char *buf, size_t count)
1872{
1873 struct damon_sysfs_quotas *quotas = container_of(kobj,
1874 struct damon_sysfs_quotas, kobj);
1875 int err = kstrtouint(buf, 0, &quotas->fail_charge_num);
1876
1877 if (err)
1878 return -EINVAL;
1879 return count;
1880}
1881
1882static ssize_t fail_charge_denom_show(struct kobject *kobj,
1883 struct kobj_attribute *attr, char *buf)
1884{
1885 struct damon_sysfs_quotas *quotas = container_of(kobj,
1886 struct damon_sysfs_quotas, kobj);
1887
1888 return sysfs_emit(buf, "%u\n", quotas->fail_charge_denom);
1889}
1890
1891static ssize_t fail_charge_denom_store(struct kobject *kobj,
1892 struct kobj_attribute *attr, const char *buf, size_t count)
1893{
1894 struct damon_sysfs_quotas *quotas = container_of(kobj,
1895 struct damon_sysfs_quotas, kobj);
1896 int err = kstrtouint(buf, 0, &quotas->fail_charge_denom);
1897
1898 if (err)
1899 return -EINVAL;
1900 return count;
1901}
1902
1685static void damon_sysfs_quotas_release(struct kobject *kobj)1903static void damon_sysfs_quotas_release(struct kobject *kobj)
1686{1904{
1687 kfree(container_of(kobj, struct damon_sysfs_quotas, kobj));1905 kfree(container_of(kobj, struct damon_sysfs_quotas, kobj));
@@ -1702,12 +1920,20 @@ static struct kobj_attribute damon_sysfs_quotas_effective_bytes_attr = · 14 unmodified lines
1702static struct kobj_attribute damon_sysfs_quotas_goal_tuner_attr =1920static struct kobj_attribute damon_sysfs_quotas_goal_tuner_attr =
1703 __ATTR_RW_MODE(goal_tuner, 0600);1921 __ATTR_RW_MODE(goal_tuner, 0600);
1704 1922
1923static struct kobj_attribute damon_sysfs_quotas_fail_charge_num_attr =
1924 __ATTR_RW_MODE(fail_charge_num, 0600);
1925
1926static struct kobj_attribute damon_sysfs_quotas_fail_charge_denom_attr =
1927 __ATTR_RW_MODE(fail_charge_denom, 0600);
1928
1705static struct attribute *damon_sysfs_quotas_attrs[] = {1929static struct attribute *damon_sysfs_quotas_attrs[] = {
1706 &damon_sysfs_quotas_ms_attr.attr,1930 &damon_sysfs_quotas_ms_attr.attr,
1707 &damon_sysfs_quotas_sz_attr.attr,1931 &damon_sysfs_quotas_sz_attr.attr,
1708 &damon_sysfs_quotas_reset_interval_ms_attr.attr,1932 &damon_sysfs_quotas_reset_interval_ms_attr.attr,
1709 &damon_sysfs_quotas_effective_bytes_attr.attr,1933 &damon_sysfs_quotas_effective_bytes_attr.attr,
1710 &damon_sysfs_quotas_goal_tuner_attr.attr,1934 &damon_sysfs_quotas_goal_tuner_attr.attr,
1935 &damon_sysfs_quotas_fail_charge_num_attr.attr,
1936 &damon_sysfs_quotas_fail_charge_denom_attr.attr,
1711 NULL,1937 NULL,
1712};1938};
1713ATTRIBUTE_GROUPS(damon_sysfs_quotas);1939ATTRIBUTE_GROUPS(damon_sysfs_quotas);
@@ -2060,6 +2286,10 @@ static struct damos_sysfs_action_name damos_sysfs_action_names[] = { · 346 unmodified lines
2060 .action = DAMOS_NOHUGEPAGE,2286 .action = DAMOS_NOHUGEPAGE,
2061 .name = "nohugepage",2287 .name = "nohugepage",
2062 },2288 },
2289 {
2290 .action = DAMOS_COLLAPSE,
2291 .name = "collapse",
2292 },
2063 {2293 {
2064 .action = DAMOS_LRU_PRIO,2294 .action = DAMOS_LRU_PRIO,
2065 .name = "lru_prio",2295 .name = "lru_prio",
@@ -2561,47 +2791,6 @@ const struct kobj_type damon_sysfs_schemes_ktype = { · 495 unmodified lines
2561 .default_groups = damon_sysfs_schemes_groups,2791 .default_groups = damon_sysfs_schemes_groups,
2562};2792};
2563 2793
2564static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
2565 char *memcg_path_buf, char *path)
2566{
2567#ifdef CONFIG_MEMCG
2568 cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
2569 if (sysfs_streq(memcg_path_buf, path))
2570 return true;
2571#endif /* CONFIG_MEMCG */
2572 return false;
2573}
2574
2575static int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
2576{
2577 struct mem_cgroup *memcg;
2578 char *path;
2579 bool found = false;
2580
2581 if (!memcg_path)
2582 return -EINVAL;
2583
2584 path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
2585 if (!path)
2586 return -ENOMEM;
2587
2588 for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
2589 memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
2590 /* skip offlined memcg */
2591 if (!mem_cgroup_online(memcg))
2592 continue;
2593 if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
2594 *id = mem_cgroup_id(memcg);
2595 found = true;
2596 mem_cgroup_iter_break(NULL, memcg);
2597 break;
2598 }
2599 }
2600
2601 kfree(path);
2602 return found ? 0 : -EINVAL;
2603}
2604
2605static int damon_sysfs_add_scheme_filters(struct damos *scheme,2794static int damon_sysfs_add_scheme_filters(struct damos *scheme,
2606 struct damon_sysfs_scheme_filters *sysfs_filters)2795 struct damon_sysfs_scheme_filters *sysfs_filters)
2607{2796{
@@ -2685,6 +2874,9 @@ static int damos_sysfs_add_quota_score( · 77 unmodified lines
2685 }2874 }
2686 goal->nid = sysfs_goal->nid;2875 goal->nid = sysfs_goal->nid;
2687 break;2876 break;
2877 case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
2878 goal->nid = sysfs_goal->nid;
2879 break;
2688 default:2880 default:
2689 break;2881 break;
2690 }2882 }
@@ -2796,6 +2988,8 @@ static struct damos *damon_sysfs_mk_scheme( · 105 unmodified lines
2796 .weight_nr_accesses = sysfs_weights->nr_accesses,2988 .weight_nr_accesses = sysfs_weights->nr_accesses,
2797 .weight_age = sysfs_weights->age,2989 .weight_age = sysfs_weights->age,
2798 .goal_tuner = sysfs_quotas->goal_tuner,2990 .goal_tuner = sysfs_quotas->goal_tuner,
2991 .fail_charge_num = sysfs_quotas->fail_charge_num,
2992 .fail_charge_denom = sysfs_quotas->fail_charge_denom,
2799 };2993 };
2800 struct damos_watermarks wmarks = {2994 struct damos_watermarks wmarks = {
2801 .metric = sysfs_wmarks->metric,2995 .metric = sysfs_wmarks->metric,
@@ -2930,12 +3124,17 @@ void damos_sysfs_populate_region_dir(struct damon_sysfs_schemes *sysfs_schemes, · 128 unmodified lines
2930 if (kobject_init_and_add(&region->kobj,3124 if (kobject_init_and_add(&region->kobj,
2931 &damon_sysfs_scheme_region_ktype,3125 &damon_sysfs_scheme_region_ktype,
2932 &sysfs_regions->kobj, "%d",3126 &sysfs_regions->kobj, "%d",
2933 sysfs_regions->nr_regions++)) {3127 sysfs_regions->nr_regions))
2934 kobject_put(&region->kobj);3128 goto out;
2935 return;3129 if (damos_sysfs_region_add_dirs(region, ctx, r))
2936 }3130 goto out;
3131
2937 list_add_tail(&region->list, &sysfs_regions->regions_list);3132 list_add_tail(&region->list, &sysfs_regions->regions_list);
2938 sysfs_regions->nr_regions++;3133 sysfs_regions->nr_regions++;
3134 return;
3135
3136out:
3137 kobject_put(&region->kobj);
2939}3138}
2940 3139
2941int damon_sysfs_schemes_clear_regions(3140int damon_sysfs_schemes_clear_regions(
mm/damon/sysfs.c+588−0
@@ -747,6 +747,497 @@ static const struct kobj_type damon_sysfs_intervals_ktype = { · 746 unmodified lines
747 .default_groups = damon_sysfs_intervals_groups,747 .default_groups = damon_sysfs_intervals_groups,
748};748};
749 749
750/*
751 * filter directory
752 */
753
754struct damon_sysfs_filter {
755 struct kobject kobj;
756 enum damon_filter_type type;
757 bool matching;
758 bool allow;
759 char *path;
760};
761
762static struct damon_sysfs_filter *damon_sysfs_filter_alloc(void)
763{
764 return kzalloc_obj(struct damon_sysfs_filter);
765}
766
767struct damon_sysfs_filter_type_name {
768 enum damon_filter_type type;
769 char *name;
770};
771
772static const struct damon_sysfs_filter_type_name
773damon_sysfs_filter_type_names[] = {
774 {
775 .type = DAMON_FILTER_TYPE_ANON,
776 .name = "anon",
777 },
778 {
779 .type = DAMON_FILTER_TYPE_MEMCG,
780 .name = "memcg",
781 },
782};
783
784static ssize_t type_show(struct kobject *kobj,
785 struct kobj_attribute *attr, char *buf)
786{
787 struct damon_sysfs_filter *filter = container_of(kobj,
788 struct damon_sysfs_filter, kobj);
789 int i;
790
791 for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
792 const struct damon_sysfs_filter_type_name *type_name;
793
794 type_name = &damon_sysfs_filter_type_names[i];
795 if (type_name->type == filter->type)
796 return sysfs_emit(buf, "%s\n", type_name->name);
797 }
798 return -EINVAL;
799}
800
801static ssize_t type_store(struct kobject *kobj,
802 struct kobj_attribute *attr, const char *buf, size_t count)
803{
804 struct damon_sysfs_filter *filter = container_of(kobj,
805 struct damon_sysfs_filter, kobj);
806 ssize_t ret = -EINVAL;
807 int i;
808
809 for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
810 const struct damon_sysfs_filter_type_name *type_name;
811
812 type_name = &damon_sysfs_filter_type_names[i];
813 if (sysfs_streq(buf, type_name->name)) {
814 filter->type = type_name->type;
815 ret = count;
816 break;
817 }
818 }
819 return ret;
820}
821
822static ssize_t matching_show(struct kobject *kobj,
823 struct kobj_attribute *attr, char *buf)
824{
825 struct damon_sysfs_filter *filter = container_of(kobj,
826 struct damon_sysfs_filter, kobj);
827
828 return sysfs_emit(buf, "%c\n", filter->matching ? 'Y' : 'N');
829}
830
831static ssize_t matching_store(struct kobject *kobj,
832 struct kobj_attribute *attr, const char *buf, size_t count)
833{
834 struct damon_sysfs_filter *filter = container_of(kobj,
835 struct damon_sysfs_filter, kobj);
836 bool matching;
837 int err = kstrtobool(buf, &matching);
838
839 if (err)
840 return err;
841
842 filter->matching = matching;
843 return count;
844}
845
846static ssize_t allow_show(struct kobject *kobj,
847 struct kobj_attribute *attr, char *buf)
848{
849 struct damon_sysfs_filter *filter = container_of(kobj,
850 struct damon_sysfs_filter, kobj);
851
852 return sysfs_emit(buf, "%c\n", filter->allow ? 'Y' : 'N');
853}
854
855static ssize_t allow_store(struct kobject *kobj,
856 struct kobj_attribute *attr, const char *buf, size_t count)
857{
858 struct damon_sysfs_filter *filter = container_of(kobj,
859 struct damon_sysfs_filter, kobj);
860 bool allow;
861 int err = kstrtobool(buf, &allow);
862
863 if (err)
864 return err;
865
866 filter->allow = allow;
867 return count;
868}
869
870static ssize_t path_show(struct kobject *kobj,
871 struct kobj_attribute *attr, char *buf)
872{
873 struct damon_sysfs_filter *filter = container_of(kobj,
874 struct damon_sysfs_filter, kobj);
875 int len;
876
877 if (!mutex_trylock(&damon_sysfs_lock))
878 return -EBUSY;
879 len = sysfs_emit(buf, "%s\n", filter->path ? filter->path : "");
880 mutex_unlock(&damon_sysfs_lock);
881 return len;
882}
883
884static ssize_t path_store(struct kobject *kobj,
885 struct kobj_attribute *attr, const char *buf, size_t count)
886{
887 struct damon_sysfs_filter *filter = container_of(kobj,
888 struct damon_sysfs_filter, kobj);
889 char *path = kmalloc_objs(*path, size_add(count, 1));
890
891 if (!path)
892 return -ENOMEM;
893 strscpy(path, buf, size_add(count, 1));
894 if (!mutex_trylock(&damon_sysfs_lock)) {
895 kfree(path);
896 return -EBUSY;
897 }
898 kfree(filter->path);
899 filter->path = path;
900 mutex_unlock(&damon_sysfs_lock);
901 return count;
902}
903
904static void damon_sysfs_filter_release(struct kobject *kobj)
905{
906 struct damon_sysfs_filter *filter = container_of(kobj,
907 struct damon_sysfs_filter, kobj);
908
909 kfree(filter->path);
910 kfree(filter);
911}
912
913static struct kobj_attribute damon_sysfs_filter_type_attr =
914 __ATTR_RW_MODE(type, 0600);
915
916static struct kobj_attribute damon_sysfs_filter_matching_attr =
917 __ATTR_RW_MODE(matching, 0600);
918
919static struct kobj_attribute damon_sysfs_filter_allow_attr =
920 __ATTR_RW_MODE(allow, 0600);
921
922static struct kobj_attribute damon_sysfs_filter_path_attr =
923 __ATTR_RW_MODE(path, 0600);
924
925static struct attribute *damon_sysfs_filter_attrs[] = {
926 &damon_sysfs_filter_type_attr.attr,
927 &damon_sysfs_filter_matching_attr.attr,
928 &damon_sysfs_filter_allow_attr.attr,
929 &damon_sysfs_filter_path_attr.attr,
930 NULL,
931};
932ATTRIBUTE_GROUPS(damon_sysfs_filter);
933
934static const struct kobj_type damon_sysfs_filter_ktype = {
935 .release = damon_sysfs_filter_release,
936 .sysfs_ops = &kobj_sysfs_ops,
937 .default_groups = damon_sysfs_filter_groups,
938};
939
940/*
941 * filters directory
942 */
943
944struct damon_sysfs_filters {
945 struct kobject kobj;
946 struct damon_sysfs_filter **filters_arr;
947 int nr;
948};
949
950static struct damon_sysfs_filters *damon_sysfs_filters_alloc(void)
951{
952 return kzalloc_obj(struct damon_sysfs_filters);
953}
954
955static void damon_sysfs_filters_rm_dirs(struct damon_sysfs_filters *filters)
956{
957 struct damon_sysfs_filter **filters_arr = filters->filters_arr;
958 int i;
959
960 for (i = 0; i < filters->nr; i++)
961 kobject_put(&filters_arr[i]->kobj);
962 filters->nr = 0;
963 kfree(filters_arr);
964 filters->filters_arr = NULL;
965}
966
967static int damon_sysfs_filters_add_dirs(
968 struct damon_sysfs_filters *filters, int nr_filters)
969{
970 struct damon_sysfs_filter **filters_arr, *filter;
971 int err, i;
972
973 damon_sysfs_filters_rm_dirs(filters);
974 if (!nr_filters)
975 return 0;
976
977 filters_arr = kmalloc_objs(*filters_arr, nr_filters,
978 GFP_KERNEL | __GFP_NOWARN);
979 if (!filters_arr)
980 return -ENOMEM;
981 filters->filters_arr = filters_arr;
982
983 for (i = 0; i < nr_filters; i++) {
984 filter = damon_sysfs_filter_alloc();
985 if (!filter) {
986 damon_sysfs_filters_rm_dirs(filters);
987 return -ENOMEM;
988 }
989
990 err = kobject_init_and_add(&filter->kobj,
991 &damon_sysfs_filter_ktype, &filters->kobj,
992 "%d", i);
993 if (err) {
994 kobject_put(&filter->kobj);
995 damon_sysfs_filters_rm_dirs(filters);
996 return err;
997 }
998
999 filters_arr[i] = filter;
1000 filters->nr++;
1001 }
1002 return 0;
1003}
1004
1005static ssize_t nr_filters_show(struct kobject *kobj,
1006 struct kobj_attribute *attr, char *buf)
1007{
1008 struct damon_sysfs_filters *filters = container_of(kobj,
1009 struct damon_sysfs_filters, kobj);
1010
1011 return sysfs_emit(buf, "%d\n", filters->nr);
1012}
1013
1014static ssize_t nr_filters_store(struct kobject *kobj,
1015 struct kobj_attribute *attr, const char *buf, size_t count)
1016{
1017 struct damon_sysfs_filters *filters;
1018 int nr, err = kstrtoint(buf, 0, &nr);
1019
1020 if (err)
1021 return err;
1022 if (nr < 0)
1023 return -EINVAL;
1024
1025 filters = container_of(kobj, struct damon_sysfs_filters, kobj);
1026
1027 if (!mutex_trylock(&damon_sysfs_lock))
1028 return -EBUSY;
1029 err = damon_sysfs_filters_add_dirs(filters, nr);
1030 mutex_unlock(&damon_sysfs_lock);
1031 if (err)
1032 return err;
1033
1034 return count;
1035}
1036
1037static void damon_sysfs_filters_release(struct kobject *kobj)
1038{
1039 kfree(container_of(kobj, struct damon_sysfs_filters, kobj));
1040}
1041
1042static struct kobj_attribute damon_sysfs_filters_nr_attr =
1043 __ATTR_RW_MODE(nr_filters, 0600);
1044
1045static struct attribute *damon_sysfs_filters_attrs[] = {
1046 &damon_sysfs_filters_nr_attr.attr,
1047 NULL,
1048};
1049ATTRIBUTE_GROUPS(damon_sysfs_filters);
1050
1051static const struct kobj_type damon_sysfs_filters_ktype = {
1052 .release = damon_sysfs_filters_release,
1053 .sysfs_ops = &kobj_sysfs_ops,
1054 .default_groups = damon_sysfs_filters_groups,
1055};
1056
1057/*
1058 * probe directory
1059 */
1060
1061struct damon_sysfs_probe {
1062 struct kobject kobj;
1063 struct damon_sysfs_filters *filters;
1064};
1065
1066static struct damon_sysfs_probe *damon_sysfs_probe_alloc(void)
1067{
1068 return kzalloc_obj(struct damon_sysfs_probe);
1069}
1070
1071static int damon_sysfs_probe_add_dirs(struct damon_sysfs_probe *attr)
1072{
1073 struct damon_sysfs_filters *filters;
1074 int err;
1075
1076 filters = damon_sysfs_filters_alloc();
1077 if (!filters)
1078 return -ENOMEM;
1079 attr->filters = filters;
1080
1081 err = kobject_init_and_add(&filters->kobj, &damon_sysfs_filters_ktype,
1082 &attr->kobj, "filters");
1083 if (err) {
1084 kobject_put(&filters->kobj);
1085 attr->filters = NULL;
1086 }
1087 return err;
1088}
1089
1090static void damon_sysfs_probe_rm_dirs(struct damon_sysfs_probe *attr)
1091{
1092 if (attr->filters) {
1093 damon_sysfs_filters_rm_dirs(attr->filters);
1094 kobject_put(&attr->filters->kobj);
1095 }
1096}
1097
1098static void damon_sysfs_probe_release(struct kobject *kobj)
1099{
1100 kfree(container_of(kobj, struct damon_sysfs_probe, kobj));
1101}
1102
1103static struct attribute *damon_sysfs_probe_attrs[] = {
1104 NULL,
1105};
1106ATTRIBUTE_GROUPS(damon_sysfs_probe);
1107
1108static const struct kobj_type damon_sysfs_probe_ktype = {
1109 .release = damon_sysfs_probe_release,
1110 .sysfs_ops = &kobj_sysfs_ops,
1111 .default_groups = damon_sysfs_probe_groups,
1112};
1113
1114/*
1115 * probes directory
1116 */
1117
1118struct damon_sysfs_probes {
1119 struct kobject kobj;
1120 struct damon_sysfs_probe **probes_arr;
1121 int nr;
1122};
1123
1124static struct damon_sysfs_probes *damon_sysfs_probes_alloc(void)
1125{
1126 return kzalloc_obj(struct damon_sysfs_probes);
1127}
1128
1129static void damon_sysfs_probes_rm_dirs(
1130 struct damon_sysfs_probes *probes)
1131{
1132 struct damon_sysfs_probe **probes_arr = probes->probes_arr;
1133 int i;
1134
1135 for (i = 0; i < probes->nr; i++) {
1136 damon_sysfs_probe_rm_dirs(probes_arr[i]);
1137 kobject_put(&probes_arr[i]->kobj);
1138 }
1139 probes->nr = 0;
1140 kfree(probes_arr);
1141 probes->probes_arr = NULL;
1142}
1143
1144static int damon_sysfs_probes_add_dirs(
1145 struct damon_sysfs_probes *probes, int nr_probes)
1146{
1147 struct damon_sysfs_probe **probes_arr, *probe;
1148 int err, i;
1149
1150 damon_sysfs_probes_rm_dirs(probes);
1151 if (!nr_probes)
1152 return 0;
1153
1154 probes_arr = kmalloc_objs(*probes_arr, nr_probes,
1155 GFP_KERNEL | __GFP_NOWARN);
1156 if (!probes_arr)
1157 return -ENOMEM;
1158 probes->probes_arr = probes_arr;
1159
1160 for (i = 0; i < nr_probes; i++) {
1161 probe = damon_sysfs_probe_alloc();
1162 if (!probe) {
1163 damon_sysfs_probes_rm_dirs(probes);
1164 return -ENOMEM;
1165 }
1166
1167 err = kobject_init_and_add(&probe->kobj,
1168 &damon_sysfs_probe_ktype, &probes->kobj,
1169 "%d", i);
1170 if (err) {
1171 kobject_put(&probe->kobj);
1172 damon_sysfs_probes_rm_dirs(probes);
1173 return err;
1174 }
1175
1176 err = damon_sysfs_probe_add_dirs(probe);
1177 if (err) {
1178 kobject_put(&probe->kobj);
1179 damon_sysfs_probes_rm_dirs(probes);
1180 return err;
1181 }
1182
1183 probes_arr[i] = probe;
1184 probes->nr++;
1185 }
1186 return 0;
1187}
1188
1189static ssize_t nr_probes_show(struct kobject *kobj,
1190 struct kobj_attribute *attr, char *buf)
1191{
1192 struct damon_sysfs_probes *probes = container_of(kobj,
1193 struct damon_sysfs_probes, kobj);
1194
1195 return sysfs_emit(buf, "%d\n", probes->nr);
1196}
1197
1198static ssize_t nr_probes_store(struct kobject *kobj,
1199 struct kobj_attribute *attr, const char *buf, size_t count)
1200{
1201 struct damon_sysfs_probes *probes;
1202 int nr, err = kstrtoint(buf, 0, &nr);
1203
1204 if (err)
1205 return err;
1206 if (nr < 0 || nr > DAMON_MAX_PROBES)
1207 return -EINVAL;
1208
1209 probes = container_of(kobj, struct damon_sysfs_probes, kobj);
1210
1211 if (!mutex_trylock(&damon_sysfs_lock))
1212 return -EBUSY;
1213 err = damon_sysfs_probes_add_dirs(probes, nr);
1214 mutex_unlock(&damon_sysfs_lock);
1215 if (err)
1216 return err;
1217
1218 return count;
1219}
1220
1221static void damon_sysfs_probes_release(struct kobject *kobj)
1222{
1223 kfree(container_of(kobj, struct damon_sysfs_probes, kobj));
1224}
1225
1226static struct kobj_attribute damon_sysfs_probes_nr_probes =
1227 __ATTR_RW_MODE(nr_probes, 0600);
1228
1229static struct attribute *damon_sysfs_probes_attrs[] = {
1230 &damon_sysfs_probes_nr_probes.attr,
1231 NULL,
1232};
1233ATTRIBUTE_GROUPS(damon_sysfs_probes);
1234
1235static const struct kobj_type damon_sysfs_probes_ktype = {
1236 .release = damon_sysfs_probes_release,
1237 .sysfs_ops = &kobj_sysfs_ops,
1238 .default_groups = damon_sysfs_probes_groups,
1239};
1240
750/*1241/*
751 * monitoring_attrs directory1242 * monitoring_attrs directory
752 */1243 */
@@ -755,6 +1246,7 @@ struct damon_sysfs_attrs { · 2 unmodified lines
755 struct kobject kobj;1246 struct kobject kobj;
756 struct damon_sysfs_intervals *intervals;1247 struct damon_sysfs_intervals *intervals;
757 struct damon_sysfs_ul_range *nr_regions_range;1248 struct damon_sysfs_ul_range *nr_regions_range;
1249 struct damon_sysfs_probes *probes;
758};1250};
759 1251
760static struct damon_sysfs_attrs *damon_sysfs_attrs_alloc(void)1252static struct damon_sysfs_attrs *damon_sysfs_attrs_alloc(void)
@@ -771,6 +1263,7 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs) · 10 unmodified lines
771{1263{
772 struct damon_sysfs_intervals *intervals;1264 struct damon_sysfs_intervals *intervals;
773 struct damon_sysfs_ul_range *nr_regions_range;1265 struct damon_sysfs_ul_range *nr_regions_range;
1266 struct damon_sysfs_probes *probes;
774 int err;1267 int err;
775 1268
776 intervals = damon_sysfs_intervals_alloc(5000, 100000, 60000000);1269 intervals = damon_sysfs_intervals_alloc(5000, 100000, 60000000);
@@ -799,8 +1292,22 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs) · 22 unmodified lines
799 if (err)1292 if (err)
800 goto put_nr_regions_intervals_out;1293 goto put_nr_regions_intervals_out;
801 attrs->nr_regions_range = nr_regions_range;1294 attrs->nr_regions_range = nr_regions_range;
1295
1296 probes = damon_sysfs_probes_alloc();
1297 if (!probes) {
1298 err = -ENOMEM;
1299 goto put_nr_regions_intervals_out;
1300 }
1301 err = kobject_init_and_add(&probes->kobj,
1302 &damon_sysfs_probes_ktype, &attrs->kobj, "probes");
1303 if (err)
1304 goto put_probes_out;
1305 attrs->probes = probes;
802 return 0;1306 return 0;
803 1307
1308put_probes_out:
1309 kobject_put(&probes->kobj);
1310 attrs->probes = NULL;
804put_nr_regions_intervals_out:1311put_nr_regions_intervals_out:
805 kobject_put(&nr_regions_range->kobj);1312 kobject_put(&nr_regions_range->kobj);
806 attrs->nr_regions_range = NULL;1313 attrs->nr_regions_range = NULL;
@@ -817,6 +1324,8 @@ static void damon_sysfs_attrs_rm_dirs(struct damon_sysfs_attrs *attrs) · 10 unmodified lines
817 kobject_put(&attrs->nr_regions_range->kobj);1324 kobject_put(&attrs->nr_regions_range->kobj);
818 damon_sysfs_intervals_rm_dirs(attrs->intervals);1325 damon_sysfs_intervals_rm_dirs(attrs->intervals);
819 kobject_put(&attrs->intervals->kobj);1326 kobject_put(&attrs->intervals->kobj);
1327 damon_sysfs_probes_rm_dirs(attrs->probes);
1328 kobject_put(&attrs->probes->kobj);
820}1329}
821 1330
822static void damon_sysfs_attrs_release(struct kobject *kobj)1331static void damon_sysfs_attrs_release(struct kobject *kobj)
@@ -866,6 +1375,7 @@ struct damon_sysfs_context { · 43 unmodified lines
866 struct damon_sysfs_attrs *attrs;1375 struct damon_sysfs_attrs *attrs;
867 struct damon_sysfs_targets *targets;1376 struct damon_sysfs_targets *targets;
868 struct damon_sysfs_schemes *schemes;1377 struct damon_sysfs_schemes *schemes;
1378 bool pause;
869};1379};
870 1380
871static struct damon_sysfs_context *damon_sysfs_context_alloc(1381static struct damon_sysfs_context *damon_sysfs_context_alloc(
@@ -878,6 +1388,7 @@ static struct damon_sysfs_context *damon_sysfs_context_alloc( · 6 unmodified lines
878 context->kobj = (struct kobject){};1388 context->kobj = (struct kobject){};
879 context->ops_id = ops_id;1389 context->ops_id = ops_id;
880 context->addr_unit = 1;1390 context->addr_unit = 1;
1391 context->pause = false;
881 return context;1392 return context;
882}1393}
883 1394
@@ -1053,6 +1564,30 @@ static ssize_t addr_unit_store(struct kobject *kobj, · 169 unmodified lines
1053 return count;1564 return count;
1054}1565}
1055 1566
1567static ssize_t pause_show(struct kobject *kobj, struct kobj_attribute *attr,
1568 char *buf)
1569{
1570 struct damon_sysfs_context *context = container_of(kobj,
1571 struct damon_sysfs_context, kobj);
1572
1573 return sysfs_emit(buf, "%c\n", context->pause ? 'Y' : 'N');
1574}
1575
1576static ssize_t pause_store(struct kobject *kobj, struct kobj_attribute *attr,
1577 const char *buf, size_t count)
1578{
1579 struct damon_sysfs_context *context = container_of(kobj,
1580 struct damon_sysfs_context, kobj);
1581 bool pause;
1582 int err = kstrtobool(buf, &pause);
1583
1584 if (err)
1585 return err;
1586 context->pause = pause;
1587 return count;
1588}
1589
1590
1056static void damon_sysfs_context_release(struct kobject *kobj)1591static void damon_sysfs_context_release(struct kobject *kobj)
1057{1592{
1058 kfree(container_of(kobj, struct damon_sysfs_context, kobj));1593 kfree(container_of(kobj, struct damon_sysfs_context, kobj));
@@ -1067,10 +1602,14 @@ static struct kobj_attribute damon_sysfs_context_operations_attr = · 8 unmodified lines
1067static struct kobj_attribute damon_sysfs_context_addr_unit_attr =1602static struct kobj_attribute damon_sysfs_context_addr_unit_attr =
1068 __ATTR_RW_MODE(addr_unit, 0600);1603 __ATTR_RW_MODE(addr_unit, 0600);
1069 1604
1605static struct kobj_attribute damon_sysfs_context_pause_attr =
1606 __ATTR_RW_MODE(pause, 0600);
1607
1070static struct attribute *damon_sysfs_context_attrs[] = {1608static struct attribute *damon_sysfs_context_attrs[] = {
1071 &damon_sysfs_context_avail_operations_attr.attr,1609 &damon_sysfs_context_avail_operations_attr.attr,
1072 &damon_sysfs_context_operations_attr.attr,1610 &damon_sysfs_context_operations_attr.attr,
1073 &damon_sysfs_context_addr_unit_attr.attr,1611 &damon_sysfs_context_addr_unit_attr.attr,
1612 &damon_sysfs_context_pause_attr.attr,
1074 NULL,1613 NULL,
1075};1614};
1076ATTRIBUTE_GROUPS(damon_sysfs_context);1615ATTRIBUTE_GROUPS(damon_sysfs_context);
@@ -1360,6 +1899,51 @@ static int damon_sysfs_set_attrs(struct damon_ctx *ctx, · 283 unmodified lines
1360 return damon_set_attrs(ctx, &attrs);1899 return damon_set_attrs(ctx, &attrs);
1361}1900}
1362 1901
1902static int damon_sysfs_set_probes(struct damon_ctx *ctx,
1903 struct damon_sysfs_probes *sys_probes)
1904{
1905 int i;
1906
1907 for (i = 0; i < sys_probes->nr; i++) {
1908 struct damon_sysfs_filters *sys_filters =
1909 sys_probes->probes_arr[i]->filters;
1910 struct damon_probe *c;
1911 int j;
1912
1913 if (!sys_filters)
1914 continue;
1915 c = damon_new_probe();
1916 if (!c)
1917 return -ENOMEM;
1918 damon_add_probe(ctx, c);
1919
1920 for (j = 0; j < sys_filters->nr; j++) {
1921 struct damon_sysfs_filter *sys_filter =
1922 sys_filters->filters_arr[j];
1923 struct damon_filter *filter;
1924
1925 filter = damon_new_filter(sys_filter->type,
1926 sys_filter->matching,
1927 sys_filter->allow);
1928 if (!filter)
1929 return -ENOMEM;
1930 if (filter->type == DAMON_FILTER_TYPE_MEMCG) {
1931 int err;
1932
1933 err = damon_sysfs_memcg_path_to_id(
1934 sys_filter->path,
1935 &filter->memcg_id);
1936 if (err) {
1937 damon_destroy_filter(filter);
1938 return err;
1939 }
1940 }
1941 damon_add_filter(c, filter);
1942 }
1943 }
1944 return 0;
1945}
1946
1363static int damon_sysfs_set_regions(struct damon_target *t,1947static int damon_sysfs_set_regions(struct damon_target *t,
1364 struct damon_sysfs_regions *sysfs_regions,1948 struct damon_sysfs_regions *sysfs_regions,
1365 unsigned long min_region_sz)1949 unsigned long min_region_sz)
@@ -1470,7 +2054,11 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 104 unmodified lines
1470 if (sys_ctx->ops_id == DAMON_OPS_PADDR)2054 if (sys_ctx->ops_id == DAMON_OPS_PADDR)
1471 ctx->min_region_sz = max(2055 ctx->min_region_sz = max(
1472 DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1);2056 DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1);
2057 ctx->pause = sys_ctx->pause;
1473 err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);2058 err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
2059 if (err)
2060 return err;
2061 err = damon_sysfs_set_probes(ctx, sys_ctx->attrs->probes);
1474 if (err)2062 if (err)
1475 return err;2063 return err;
1476 err = damon_sysfs_add_targets(ctx, sys_ctx->targets);2064 err = damon_sysfs_add_targets(ctx, sys_ctx->targets);
mm/damon/tests/core-kunit.h+152−28
@@ -273,54 +273,70 @@ static void damon_test_merge_regions_of(struct kunit *test) · 272 unmodified lines
273 273
274static void damon_test_split_regions_of(struct kunit *test)274static void damon_test_split_regions_of(struct kunit *test)
275{275{
276 struct damon_ctx *c;
276 struct damon_target *t;277 struct damon_target *t;
277 struct damon_region *r;278 struct damon_region *r;
278 unsigned long sa[] = {0, 300, 500};279 unsigned long sa[] = {0, 300, 500};
279 unsigned long ea[] = {220, 400, 700};280 unsigned long ea[] = {220, 400, 700};
280 int i;281 int i;
281 282
283 c = damon_new_ctx();
284 if (!c)
285 kunit_skip(test, "ctx alloc fail");
286
282 t = damon_new_target();287 t = damon_new_target();
283 if (!t)288 if (!t) {
289 damon_destroy_ctx(c);
284 kunit_skip(test, "target alloc fail");290 kunit_skip(test, "target alloc fail");
291 }
285 r = damon_new_region(0, 22);292 r = damon_new_region(0, 22);
286 if (!r) {293 if (!r) {
287 damon_free_target(t);294 damon_free_target(t);
295 damon_destroy_ctx(c);
288 kunit_skip(test, "region alloc fail");296 kunit_skip(test, "region alloc fail");
289 }297 }
290 damon_add_region(r, t);298 damon_add_region(r, t);
291 damon_split_regions_of(t, 2, 1);299 damon_split_regions_of(c, t, 2, 1);
292 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 2u);300 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 2u);
293 damon_free_target(t);301 damon_free_target(t);
294 302
295 t = damon_new_target();303 t = damon_new_target();
296 if (!t)304 if (!t) {
305 damon_destroy_ctx(c);
297 kunit_skip(test, "second target alloc fail");306 kunit_skip(test, "second target alloc fail");
307 }
298 r = damon_new_region(0, 220);308 r = damon_new_region(0, 220);
299 if (!r) {309 if (!r) {
300 damon_free_target(t);310 damon_free_target(t);
311 damon_destroy_ctx(c);
301 kunit_skip(test, "second region alloc fail");312 kunit_skip(test, "second region alloc fail");
302 }313 }
303 damon_add_region(r, t);314 damon_add_region(r, t);
304 damon_split_regions_of(t, 4, 1);315 damon_split_regions_of(c, t, 4, 1);
305 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u);316 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u);
306 damon_free_target(t);317 damon_free_target(t);
307 318
308 t = damon_new_target();319 t = damon_new_target();
309 if (!t)320 if (!t) {
321 damon_destroy_ctx(c);
310 kunit_skip(test, "third target alloc fail");322 kunit_skip(test, "third target alloc fail");
323 }
311 for (i = 0; i < ARRAY_SIZE(sa); i++) {324 for (i = 0; i < ARRAY_SIZE(sa); i++) {
312 r = damon_new_region(sa[i], ea[i]);325 r = damon_new_region(sa[i], ea[i]);
313 if (!r) {326 if (!r) {
314 damon_free_target(t);327 damon_free_target(t);
328 damon_destroy_ctx(c);
315 kunit_skip(test, "region alloc fail");329 kunit_skip(test, "region alloc fail");
316 }330 }
317 damon_add_region(r, t);331 damon_add_region(r, t);
318 }332 }
319 damon_split_regions_of(t, 4, 5);333 damon_split_regions_of(c, t, 4, 5);
320 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u);334 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u);
321 damon_for_each_region(r, t)335 damon_for_each_region(r, t)
322 KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul);336 KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul);
323 damon_free_target(t);337 damon_free_target(t);
338
339 damon_destroy_ctx(c);
324}340}
325 341
326static void damon_test_ops_registration(struct kunit *test)342static void damon_test_ops_registration(struct kunit *test)
@@ -374,41 +390,139 @@ static void damon_test_ops_registration(struct kunit *test) · 47 unmodified lines
374 }390 }
375}391}
376 392
377static void damon_test_set_regions(struct kunit *test)393static void damon_test_set_regions_for(struct kunit *test,
394 struct damon_addr_range *old_ranges, int sz_old_ranges,
395 struct damon_addr_range *new_ranges, int sz_new_ranges,
396 unsigned long min_region_sz,
397 struct damon_addr_range *expect_ranges, int sz_expect_ranges)
378{398{
379 struct damon_target *t = damon_new_target();399 struct damon_target *t;
380 struct damon_region *r1, *r2;
381 struct damon_addr_range range = {.start = 8, .end = 28};
382 unsigned long expects[] = {8, 16, 16, 24, 24, 28};
383 int expect_idx = 0;
384 struct damon_region *r;400 struct damon_region *r;
401 int i;
385 402
403 t = damon_new_target();
386 if (!t)404 if (!t)
387 kunit_skip(test, "target alloc fail");405 kunit_skip(test, "target alloc fail");
388 r1 = damon_new_region(4, 16);406 for (i = 0; i < sz_old_ranges; i++) {
389 if (!r1) {407 r = damon_new_region(old_ranges[i].start, old_ranges[i].end);
390 damon_free_target(t);408 if (!r) {
391 kunit_skip(test, "region alloc fail");409 damon_destroy_target(t, NULL);
392 }410 kunit_skip(test, "%d-th r alloc fail\n", i);
393 r2 = damon_new_region(24, 32);411 }
394 if (!r2) {412 damon_add_region(r, t);
395 damon_free_target(t);
396 damon_free_region(r1);
397 kunit_skip(test, "second region alloc fail");
398 }413 }
399 414
400 damon_add_region(r1, t);415 damon_set_regions(t, new_ranges, sz_new_ranges, min_region_sz);
401 damon_add_region(r2, t);
402 damon_set_regions(t, &range, 1, 1);
403 416
404 KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 3);417 KUNIT_EXPECT_EQ(test, damon_nr_regions(t), sz_expect_ranges);
418 if (damon_nr_regions(t) != sz_expect_ranges) {
419 damon_destroy_target(t, NULL);
420 return;
421 }
422 i = 0;
405 damon_for_each_region(r, t) {423 damon_for_each_region(r, t) {
406 KUNIT_EXPECT_EQ(test, r->ar.start, expects[expect_idx++]);424 KUNIT_EXPECT_EQ(test, r->ar.start, expect_ranges[i].start);
407 KUNIT_EXPECT_EQ(test, r->ar.end, expects[expect_idx++]);425 KUNIT_EXPECT_EQ(test, r->ar.end, expect_ranges[i++].end);
408 }426 }
427
409 damon_destroy_target(t, NULL);428 damon_destroy_target(t, NULL);
410}429}
411 430
431static void damon_test_set_regions(struct kunit *test)
432{
433 /* Initial build up on empty target. */
434 damon_test_set_regions_for(test,
435 (struct damon_addr_range[]){}, 0,
436 (struct damon_addr_range[]){
437 {.start = 5, .end = 15},
438 {.start = 15, .end = 25},
439 }, 2,
440 1,
441 (struct damon_addr_range[]){
442 {.start = 5, .end = 15},
443 {.start = 15, .end = 25},
444 }, 2);
445 /* Un-intersecting regions should be removed. */
446 damon_test_set_regions_for(test,
447 (struct damon_addr_range[]){
448 {.start = 4, .end = 16},
449 {.start = 24, .end = 32},
450 }, 2,
451 (struct damon_addr_range[]){
452 {.start = 18, .end = 23},
453 }, 1,
454 1,
455 (struct damon_addr_range[]){
456 {.start = 18, .end = 23},
457 }, 1);
458 /*
459 * Holes should be filled up with new regions.
460 *
461 * old: [4, 16) [24, 32)
462 * new: [8, 28)
463 * expect: [8, 16)[16,24),[24, 28)
464 */
465 damon_test_set_regions_for(test,
466 (struct damon_addr_range[]){
467 {.start = 4, .end = 16},
468 {.start = 24, .end = 32},
469 }, 2,
470 (struct damon_addr_range[]){
471 {.start = 8, .end = 28},
472 }, 1,
473 1,
474 (struct damon_addr_range[]){
475 {.start = 8, .end = 16},
476 {.start = 16, .end = 24},
477 {.start = 24, .end = 28},
478 }, 3);
479 /*
480 * New regions should be able to be appended.
481 *
482 * old: [0, 4)[4, 17)
483 * new: [0, 15) [25, 40)
484 * expect: [0, 4)[4, 15) [25, 40)
485 */
486 damon_test_set_regions_for(test,
487 (struct damon_addr_range[]){
488 {.start = 0, .end = 4},
489 {.start = 4, .end = 17},
490 }, 2,
491 (struct damon_addr_range[]){
492 {.start = 0, .end = 15},
493 {.start = 25, .end = 40},
494 }, 2,
495 1,
496 (struct damon_addr_range[]){
497 {.start = 0, .end = 4},
498 {.start = 4, .end = 15},
499 {.start = 25, .end = 40},
500 }, 3);
501 /*
502 * New regions should be able to be inserted.
503 *
504 * old: [0, 4) [42, 52)
505 * new: [0, 15) [25, 40) [44, 50)
506 * expect: [0, 15) [25, 40) [44, 50)
507 */
508 damon_test_set_regions_for(test,
509 (struct damon_addr_range[]){
510 {.start = 0, .end = 4},
511 {.start = 42, .end = 52},
512 }, 2,
513 (struct damon_addr_range[]){
514 {.start = 0, .end = 15},
515 {.start = 25, .end = 40},
516 {.start = 44, .end = 50},
517 }, 3,
518 1,
519 (struct damon_addr_range[]){
520 {.start = 0, .end = 15},
521 {.start = 25, .end = 40},
522 {.start = 44, .end = 50},
523 }, 3);
524}
525
412static void damon_test_nr_accesses_to_accesses_bp(struct kunit *test)526static void damon_test_nr_accesses_to_accesses_bp(struct kunit *test)
413{527{
414 struct damon_attrs attrs = {528 struct damon_attrs attrs = {
@@ -694,6 +808,8 @@ static void damos_test_commit_quota(struct kunit *test) · 279 unmodified lines
694 .ms = 2,808 .ms = 2,
695 .sz = 3,809 .sz = 3,
696 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST,810 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST,
811 .fail_charge_num = 2,
812 .fail_charge_denom = 3,
697 .weight_sz = 4,813 .weight_sz = 4,
698 .weight_nr_accesses = 5,814 .weight_nr_accesses = 5,
699 .weight_age = 6,815 .weight_age = 6,
@@ -703,6 +819,8 @@ static void damos_test_commit_quota(struct kunit *test) · 3 unmodified lines
703 .ms = 8,819 .ms = 8,
704 .sz = 9,820 .sz = 9,
705 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL,821 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL,
822 .fail_charge_num = 1,
823 .fail_charge_denom = 1024,
706 .weight_sz = 10,824 .weight_sz = 10,
707 .weight_nr_accesses = 11,825 .weight_nr_accesses = 11,
708 .weight_age = 12,826 .weight_age = 12,
@@ -717,6 +835,8 @@ static void damos_test_commit_quota(struct kunit *test) · 8 unmodified lines
717 KUNIT_EXPECT_EQ(test, dst.ms, src.ms);835 KUNIT_EXPECT_EQ(test, dst.ms, src.ms);
718 KUNIT_EXPECT_EQ(test, dst.sz, src.sz);836 KUNIT_EXPECT_EQ(test, dst.sz, src.sz);
719 KUNIT_EXPECT_EQ(test, dst.goal_tuner, src.goal_tuner);837 KUNIT_EXPECT_EQ(test, dst.goal_tuner, src.goal_tuner);
838 KUNIT_EXPECT_EQ(test, dst.fail_charge_num, src.fail_charge_num);
839 KUNIT_EXPECT_EQ(test, dst.fail_charge_denom, src.fail_charge_denom);
720 KUNIT_EXPECT_EQ(test, dst.weight_sz, src.weight_sz);840 KUNIT_EXPECT_EQ(test, dst.weight_sz, src.weight_sz);
721 KUNIT_EXPECT_EQ(test, dst.weight_nr_accesses, src.weight_nr_accesses);841 KUNIT_EXPECT_EQ(test, dst.weight_nr_accesses, src.weight_nr_accesses);
722 KUNIT_EXPECT_EQ(test, dst.weight_age, src.weight_age);842 KUNIT_EXPECT_EQ(test, dst.weight_age, src.weight_age);
@@ -1077,6 +1197,10 @@ static void damon_test_commit_ctx(struct kunit *test) · 354 unmodified lines
1077 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);1197 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
1078 src->min_region_sz = 4095;1198 src->min_region_sz = 4095;
1079 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), -EINVAL);1199 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), -EINVAL);
1200 src->min_region_sz = 4096;
1201 src->pause = true;
1202 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
1203 KUNIT_EXPECT_TRUE(test, dst->pause);
1080 damon_destroy_ctx(src);1204 damon_destroy_ctx(src);
1081 damon_destroy_ctx(dst);1205 damon_destroy_ctx(dst);
1082}1206}
mm/damon/tests/vaddr-kunit.h+20−7
@@ -132,22 +132,35 @@ static void damon_do_test_apply_three_regions(struct kunit *test, · 131 unmodified lines
132 unsigned long *expected, int nr_expected)132 unsigned long *expected, int nr_expected)
133{133{
134 struct damon_target *t;134 struct damon_target *t;
135 struct damon_addr_range *ranges;
135 struct damon_region *r;136 struct damon_region *r;
136 int i;137 int i;
137 138
138 t = damon_new_target();139 t = damon_new_target();
139 if (!t)140 if (!t)
140 kunit_skip(test, "target alloc fail");141 kunit_skip(test, "target alloc fail");
142
143 ranges = kmalloc_array(nr_regions / 2, sizeof(*ranges), GFP_KERNEL);
144 if (!ranges) {
145 damon_destroy_target(t, NULL);
146 kunit_skip(test, "ranges alloc fail");
147 }
141 for (i = 0; i < nr_regions / 2; i++) {148 for (i = 0; i < nr_regions / 2; i++) {
142 r = damon_new_region(regions[i * 2], regions[i * 2 + 1]);149 ranges[i].start = regions[i * 2];
143 if (!r) {150 ranges[i].end = regions[i * 2 + 1];
144 damon_destroy_target(t, NULL);
145 kunit_skip(test, "region alloc fail");
146 }
147 damon_add_region(r, t);
148 }151 }
152 if (damon_set_regions(t, ranges, nr_regions / 2,
153 DAMON_MIN_REGION_SZ)) {
154 kfree(ranges);
155 damon_destroy_target(t, NULL);
156 kunit_skip(test, "damon_set_regions() fail");
157 }
158 kfree(ranges);
149 159
150 damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ);160 if (damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ)) {
161 damon_destroy_target(t, NULL);
162 kunit_skip(test, "second damon_set_regions() fail");
163 }
151 164
152 for (i = 0; i < nr_expected / 2; i++) {165 for (i = 0; i < nr_expected / 2; i++) {
153 r = __nth_region_of(t, i);166 r = __nth_region_of(t, i);
mm/damon/vaddr.c+50−29
@@ -237,6 +237,35 @@ static void damon_va_update(struct damon_ctx *ctx) · 236 unmodified lines
237 }237 }
238}238}
239 239
240static void damon_va_walk_page_range(struct mm_struct *mm, unsigned long start,
241 unsigned long end, struct mm_walk_ops *ops, void *private)
242{
243 struct vm_area_struct *vma;
244
245 vma = lock_vma_under_rcu(mm, start);
246 if (!vma)
247 goto lock_mmap;
248
249 if (end > vma->vm_end) {
250 vma_end_read(vma);
251 goto lock_mmap;
252 }
253
254 if (!(vma->vm_flags & VM_PFNMAP)) {
255 ops->walk_lock = PGWALK_VMA_RDLOCK_VERIFY;
256 walk_page_range_vma(vma, start, end, ops, private);
257 }
258
259 vma_end_read(vma);
260 return;
261
262lock_mmap:
263 mmap_read_lock(mm);
264 ops->walk_lock = PGWALK_RDLOCK;
265 walk_page_range(mm, start, end, ops, private);
266 mmap_read_unlock(mm);
267}
268
240static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,269static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
241 unsigned long next, struct mm_walk *walk)270 unsigned long next, struct mm_walk *walk)
242{271{
@@ -315,17 +344,14 @@ out: · 72 unmodified lines
315#define damon_mkold_hugetlb_entry NULL344#define damon_mkold_hugetlb_entry NULL
316#endif /* CONFIG_HUGETLB_PAGE */345#endif /* CONFIG_HUGETLB_PAGE */
317 346
318static const struct mm_walk_ops damon_mkold_ops = {
319 .pmd_entry = damon_mkold_pmd_entry,
320 .hugetlb_entry = damon_mkold_hugetlb_entry,
321 .walk_lock = PGWALK_RDLOCK,
322};
323
324static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)347static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
325{348{
326 mmap_read_lock(mm);349 struct mm_walk_ops damon_mkold_ops = {
327 walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);350 .pmd_entry = damon_mkold_pmd_entry,
328 mmap_read_unlock(mm);351 .hugetlb_entry = damon_mkold_hugetlb_entry,
352 };
353
354 damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
329}355}
330 356
331/*357/*
@@ -333,9 +359,10 @@ static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) · 1 unmodified line
333 */359 */
334 360
335static void __damon_va_prepare_access_check(struct mm_struct *mm,361static void __damon_va_prepare_access_check(struct mm_struct *mm,
336 struct damon_region *r)362 struct damon_region *r,
363 struct damon_ctx *ctx)
337{364{
338 r->sampling_addr = damon_rand(r->ar.start, r->ar.end);365 r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
339 366
340 damon_va_mkold(mm, r->sampling_addr);367 damon_va_mkold(mm, r->sampling_addr);
341}368}
@@ -351,7 +378,7 @@ static void damon_va_prepare_access_checks(struct damon_ctx *ctx) · 9 unmodified lines
351 if (!mm)378 if (!mm)
352 continue;379 continue;
353 damon_for_each_region(r, t)380 damon_for_each_region(r, t)
354 __damon_va_prepare_access_check(mm, r);381 __damon_va_prepare_access_check(mm, r, ctx);
355 mmput(mm);382 mmput(mm);
356 }383 }
357}384}
@@ -444,12 +471,6 @@ out: · 86 unmodified lines
444#define damon_young_hugetlb_entry NULL471#define damon_young_hugetlb_entry NULL
445#endif /* CONFIG_HUGETLB_PAGE */472#endif /* CONFIG_HUGETLB_PAGE */
446 473
447static const struct mm_walk_ops damon_young_ops = {
448 .pmd_entry = damon_young_pmd_entry,
449 .hugetlb_entry = damon_young_hugetlb_entry,
450 .walk_lock = PGWALK_RDLOCK,
451};
452
453static bool damon_va_young(struct mm_struct *mm, unsigned long addr,474static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
454 unsigned long *folio_sz)475 unsigned long *folio_sz)
455{476{
@@ -458,9 +479,12 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines
458 .young = false,479 .young = false,
459 };480 };
460 481
461 mmap_read_lock(mm);482 struct mm_walk_ops damon_young_ops = {
462 walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);483 .pmd_entry = damon_young_pmd_entry,
463 mmap_read_unlock(mm);484 .hugetlb_entry = damon_young_hugetlb_entry,
485 };
486
487 damon_va_walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);
464 return arg.young;488 return arg.young;
465}489}
466 490
@@ -749,7 +773,6 @@ static unsigned long damos_va_migrate(struct damon_target *target, · 282 unmodified lines
749 struct mm_walk_ops walk_ops = {773 struct mm_walk_ops walk_ops = {
750 .pmd_entry = damos_va_migrate_pmd_entry,774 .pmd_entry = damos_va_migrate_pmd_entry,
751 .pte_entry = NULL,775 .pte_entry = NULL,
752 .walk_lock = PGWALK_RDLOCK,
753 };776 };
754 777
755 use_target_nid = dests->nr_dests == 0;778 use_target_nid = dests->nr_dests == 0;
@@ -767,9 +790,7 @@ static unsigned long damos_va_migrate(struct damon_target *target, · 11 unmodified lines
767 if (!mm)790 if (!mm)
768 goto free_lists;791 goto free_lists;
769 792
770 mmap_read_lock(mm);793 damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
771 walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
772 mmap_read_unlock(mm);
773 mmput(mm);794 mmput(mm);
774 795
775 for (int i = 0; i < nr_dests; i++) {796 for (int i = 0; i < nr_dests; i++) {
@@ -861,7 +882,6 @@ static unsigned long damos_va_stat(struct damon_target *target, · 85 unmodified lines
861 struct mm_struct *mm;882 struct mm_struct *mm;
862 struct mm_walk_ops walk_ops = {883 struct mm_walk_ops walk_ops = {
863 .pmd_entry = damos_va_stat_pmd_entry,884 .pmd_entry = damos_va_stat_pmd_entry,
864 .walk_lock = PGWALK_RDLOCK,
865 };885 };
866 886
867 priv.scheme = s;887 priv.scheme = s;
@@ -874,9 +894,7 @@ static unsigned long damos_va_stat(struct damon_target *target, · 6 unmodified lines
874 if (!mm)894 if (!mm)
875 return 0;895 return 0;
876 896
877 mmap_read_lock(mm);897 damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
878 walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
879 mmap_read_unlock(mm);
880 mmput(mm);898 mmput(mm);
881 return 0;899 return 0;
882}900}
@@ -903,6 +921,9 @@ static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx, · 20 unmodified lines
903 case DAMOS_NOHUGEPAGE:921 case DAMOS_NOHUGEPAGE:
904 madv_action = MADV_NOHUGEPAGE;922 madv_action = MADV_NOHUGEPAGE;
905 break;923 break;
924 case DAMOS_COLLAPSE:
925 madv_action = MADV_COLLAPSE;
926 break;
906 case DAMOS_MIGRATE_HOT:927 case DAMOS_MIGRATE_HOT:
907 case DAMOS_MIGRATE_COLD:928 case DAMOS_MIGRATE_COLD:
908 return damos_va_migrate(t, r, scheme, sz_filter_passed);929 return damos_va_migrate(t, r, scheme, sz_filter_passed);
mm/filemap.c+80−53
@@ -1808,9 +1808,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1807 unmodified lines
1808 pgoff_t index, unsigned long max_scan)1808 pgoff_t index, unsigned long max_scan)
1809{1809{
1810 XA_STATE(xas, &mapping->i_pages, index);1810 XA_STATE(xas, &mapping->i_pages, index);
1811 unsigned long nr = max_scan;
1812 1811
1813 while (nr--) {1812 while (max_scan--) {
1814 void *entry = xas_next(&xas);1813 void *entry = xas_next(&xas);
1815 if (!entry || xa_is_value(entry))1814 if (!entry || xa_is_value(entry))
1816 return xas.xa_index;1815 return xas.xa_index;
@@ -1818,7 +1817,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1 unmodified line
1818 return 0;1817 return 0;
1819 }1818 }
1820 1819
1821 return index + max_scan;1820 /* Return end of the range + 1 when no hole is found */
1821 return xas.xa_index + 1;
1822}1822}
1823EXPORT_SYMBOL(page_cache_next_miss);1823EXPORT_SYMBOL(page_cache_next_miss);
1824 1824
@@ -1849,12 +1849,13 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, · 24 unmodified lines
1849 while (max_scan--) {1849 while (max_scan--) {
1850 void *entry = xas_prev(&xas);1850 void *entry = xas_prev(&xas);
1851 if (!entry || xa_is_value(entry))1851 if (!entry || xa_is_value(entry))
1852 break;1852 return xas.xa_index;
1853 if (xas.xa_index == ULONG_MAX)1853 if (xas.xa_index == ULONG_MAX)
1854 break;1854 return ULONG_MAX;
1855 }1855 }
1856 1856
1857 return xas.xa_index;1857 /* Return start of the range - 1 when no hole is found */
1858 return xas.xa_index - 1;
1858}1859}
1859EXPORT_SYMBOL(page_cache_prev_miss);1860EXPORT_SYMBOL(page_cache_prev_miss);
1860 1861
@@ -2294,8 +2295,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, · 433 unmodified lines
2294 goto put_folio;2295 goto put_folio;
2295 2296
2296 if (!folio_batch_add(fbatch, folio)) {2297 if (!folio_batch_add(fbatch, folio)) {
2297 nr = folio_nr_pages(folio);2298 *start = folio_next_index(folio);
2298 *start = folio->index + nr;
2299 goto out;2299 goto out;
2300 }2300 }
2301 xas_advance(&xas, folio_next_index(folio) - 1);2301 xas_advance(&xas, folio_next_index(folio) - 1);
@@ -2355,8 +2355,7 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, · 53 unmodified lines
2355 if (xa_is_value(folio))2355 if (xa_is_value(folio))
2356 continue;2356 continue;
2357 if (!folio_batch_add(fbatch, folio)) {2357 if (!folio_batch_add(fbatch, folio)) {
2358 unsigned long nr = folio_nr_pages(folio);2358 *start = folio_next_index(folio);
2359 *start = folio->index + nr;
2360 goto out;2359 goto out;
2361 }2360 }
2362 }2361 }
@@ -2414,8 +2413,7 @@ unsigned filemap_get_folios_dirty(struct address_space *mapping, pgoff_t *start, · 51 unmodified lines
2414 }2413 }
2415 }2414 }
2416 if (!folio_batch_add(fbatch, folio)) {2415 if (!folio_batch_add(fbatch, folio)) {
2417 unsigned long nr = folio_nr_pages(folio);2416 *start = folio_next_index(folio);
2418 *start = folio->index + nr;
2419 goto out;2417 goto out;
2420 }2418 }
2421 }2419 }
@@ -3323,12 +3321,26 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 901 unmodified lines
3323 struct file *fpin = NULL;3321 struct file *fpin = NULL;
3324 vm_flags_t vm_flags = vmf->vma->vm_flags;3322 vm_flags_t vm_flags = vmf->vma->vm_flags;
3325 bool force_thp_readahead = false;3323 bool force_thp_readahead = false;
3324 unsigned int thp_order = 0;
3326 unsigned short mmap_miss;3325 unsigned short mmap_miss;
3327 3326
3327 ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
3328
3328 /* Use the readahead code, even if readahead is disabled */3329 /* Use the readahead code, even if readahead is disabled */
3329 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&3330 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
3330 (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)3331 /*
3331 force_thp_readahead = true;3332 * Cap max THP order at 2MB: this is the common PMD-sized
3333 * hugepage size, and it avoids memory pressure from very
3334 * large forced readahead when mapping_max_folio_order() is
3335 * high (for example, 128MB with 64K base pages on arm64).
3336 */
3337 if (mapping_large_folio_support(mapping)) {
3338 force_thp_readahead = true;
3339 thp_order = min_t(unsigned int,
3340 mapping_max_folio_order(mapping),
3341 get_order(SZ_2M));
3342 }
3343 }
3332 3344
3333 if (!force_thp_readahead) {3345 if (!force_thp_readahead) {
3334 /*3346 /*
@@ -3348,7 +3360,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 13 unmodified lines
3348 }3360 }
3349 }3361 }
3350 3362
3351 if (!(vm_flags & VM_SEQ_READ)) {3363 if (!(vm_flags & (VM_SEQ_READ | VM_EXEC))) {
3352 /* Avoid banging the cache line if not needed */3364 /* Avoid banging the cache line if not needed */
3353 mmap_miss = READ_ONCE(ra->mmap_miss);3365 mmap_miss = READ_ONCE(ra->mmap_miss);
3354 if (mmap_miss < MMAP_LOTSAMISS * 10)3366 if (mmap_miss < MMAP_LOTSAMISS * 10)
@@ -3363,17 +3375,19 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 8 unmodified lines
3363 }3375 }
3364 3376
3365 if (force_thp_readahead) {3377 if (force_thp_readahead) {
3378 unsigned long folio_nr_pages = 1UL << thp_order;
3379
3366 fpin = maybe_unlock_mmap_for_io(vmf, fpin);3380 fpin = maybe_unlock_mmap_for_io(vmf, fpin);
3367 ractl._index &= ~((unsigned long)HPAGE_PMD_NR - 1);3381 ractl._index &= ~(folio_nr_pages - 1);
3368 ra->size = HPAGE_PMD_NR;3382 ra->size = folio_nr_pages;
3369 /*3383 /*
3370 * Fetch two PMD folios, so we get the chance to actually3384 * Fetch two folios so we get the chance to actually
3371 * readahead, unless we've been told not to.3385 * readahead, unless we've been told not to.
3372 */3386 */
3373 if (!(vm_flags & VM_RAND_READ))3387 if (!(vm_flags & VM_RAND_READ))
3374 ra->size *= 2;3388 ra->size *= 2;
3375 ra->async_size = HPAGE_PMD_NR;3389 ra->async_size = folio_nr_pages;
3376 ra->order = HPAGE_PMD_ORDER;3390 ra->order = thp_order;
3377 page_cache_ra_order(&ractl, ra);3391 page_cache_ra_order(&ractl, ra);
3378 return fpin;3392 return fpin;
3379 }3393 }
@@ -3407,6 +3421,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 27 unmodified lines
3407 * mmap read-around3421 * mmap read-around
3408 */3422 */
3409 ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);3423 ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);
3424 ra->start = max(ra->start, vmf->vma->vm_pgoff);
3410 ra->size = ra->ra_pages;3425 ra->size = ra->ra_pages;
3411 ra->async_size = ra->ra_pages / 4;3426 ra->async_size = ra->ra_pages / 4;
3412 ra->order = 0;3427 ra->order = 0;
@@ -3441,14 +3456,20 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · 28 unmodified lines
3441 * Don't touch the mmap_miss counter to avoid decreasing it multiple3456 * Don't touch the mmap_miss counter to avoid decreasing it multiple
3442 * times for a single folio and break the balance with mmap_miss3457 * times for a single folio and break the balance with mmap_miss
3443 * increase in do_sync_mmap_readahead().3458 * increase in do_sync_mmap_readahead().
3459 *
3460 * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss increment in
3461 * do_sync_mmap_readahead(), so skip the decrement here as well to
3462 * keep the counter symmetric.
3444 */3463 */
3445 if (likely(!folio_test_locked(folio))) {3464 if (likely(!folio_test_locked(folio)) &&
3465 !(vmf->vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
3446 mmap_miss = READ_ONCE(ra->mmap_miss);3466 mmap_miss = READ_ONCE(ra->mmap_miss);
3447 if (mmap_miss)3467 if (mmap_miss)
3448 WRITE_ONCE(ra->mmap_miss, --mmap_miss);3468 WRITE_ONCE(ra->mmap_miss, --mmap_miss);
3449 }3469 }
3450 3470
3451 if (folio_test_readahead(folio)) {3471 if (folio_test_readahead(folio)) {
3472 ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
3452 fpin = maybe_unlock_mmap_for_io(vmf, fpin);3473 fpin = maybe_unlock_mmap_for_io(vmf, fpin);
3453 page_cache_async_ra(&ractl, folio, ra->ra_pages);3474 page_cache_async_ra(&ractl, folio, ra->ra_pages);
3454 }3475 }
@@ -3758,8 +3779,7 @@ skip: · 303 unmodified lines
3758static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,3779static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,
3759 struct folio *folio, unsigned long start,3780 struct folio *folio, unsigned long start,
3760 unsigned long addr, unsigned int nr_pages,3781 unsigned long addr, unsigned int nr_pages,
3761 unsigned long *rss, unsigned short *mmap_miss,3782 unsigned long *rss, pgoff_t file_end)
3762 pgoff_t file_end)
3763{3783{
3764 struct address_space *mapping = folio->mapping;3784 struct address_space *mapping = folio->mapping;
3765 unsigned int ref_from_caller = 1;3785 unsigned int ref_from_caller = 1;
@@ -3791,16 +3811,6 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, · 25 unmodified lines
3791 if (PageHWPoison(page + count))3811 if (PageHWPoison(page + count))
3792 goto skip;3812 goto skip;
3793 3813
3794 /*
3795 * If there are too many folios that are recently evicted
3796 * in a file, they will probably continue to be evicted.
3797 * In such situation, read-ahead is only a waste of IO.
3798 * Don't decrease mmap_miss in this scenario to make sure
3799 * we can stop read-ahead.
3800 */
3801 if (!folio_test_workingset(folio))
3802 (*mmap_miss)++;
3803
3804 /*3814 /*
3805 * NOTE: If there're PTE markers, we'll leave them to be3815 * NOTE: If there're PTE markers, we'll leave them to be
3806 * handled in the specific fault path, and it'll prohibit the3816 * handled in the specific fault path, and it'll prohibit the
@@ -3847,7 +3857,7 @@ skip: · 40 unmodified lines
3847 3857
3848static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,3858static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
3849 struct folio *folio, unsigned long addr,3859 struct folio *folio, unsigned long addr,
3850 unsigned long *rss, unsigned short *mmap_miss)3860 unsigned long *rss)
3851{3861{
3852 vm_fault_t ret = 0;3862 vm_fault_t ret = 0;
3853 struct page *page = &folio->page;3863 struct page *page = &folio->page;
@@ -3855,10 +3865,6 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, · 1 unmodified line
3855 if (PageHWPoison(page))3865 if (PageHWPoison(page))
3856 goto out;3866 goto out;
3857 3867
3858 /* See comment of filemap_map_folio_range() */
3859 if (!folio_test_workingset(folio))
3860 (*mmap_miss)++;
3861
3862 /*3868 /*
3863 * NOTE: If there're PTE markers, we'll leave them to be3869 * NOTE: If there're PTE markers, we'll leave them to be
3864 * handled in the specific fault path, and it'll prohibit3870 * handled in the specific fault path, and it'll prohibit
@@ -3893,7 +3899,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 28 unmodified lines
3893 vm_fault_t ret = 0;3899 vm_fault_t ret = 0;
3894 unsigned long rss = 0;3900 unsigned long rss = 0;
3895 unsigned int nr_pages = 0, folio_type;3901 unsigned int nr_pages = 0, folio_type;
3896 unsigned short mmap_miss = 0, mmap_miss_saved;
3897 3902
3898 /*3903 /*
3899 * Recalculate end_pgoff based on file_end before calling3904 * Recalculate end_pgoff based on file_end before calling
@@ -3932,6 +3937,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 32 unmodified lines
3932 folio_type = mm_counter_file(folio);3937 folio_type = mm_counter_file(folio);
3933 do {3938 do {
3934 unsigned long end;3939 unsigned long end;
3940 vm_fault_t map_ret;
3935 3941
3936 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;3942 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
3937 vmf->pte += xas.xa_index - last_pgoff;3943 vmf->pte += xas.xa_index - last_pgoff;
@@ -3939,13 +3945,40 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 1 unmodified line
3939 end = folio_next_index(folio) - 1;3945 end = folio_next_index(folio) - 1;
3940 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;3946 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;
3941 3947
3942 if (!folio_test_large(folio))3948 if (!folio_test_large(folio)) {
3943 ret |= filemap_map_order0_folio(vmf,3949 map_ret = filemap_map_order0_folio(vmf, folio, addr,
3944 folio, addr, &rss, &mmap_miss);3950 &rss);
3945 else3951 } else {
3946 ret |= filemap_map_folio_range(vmf, folio,3952 unsigned long start = xas.xa_index - folio->index;
3947 xas.xa_index - folio->index, addr,3953
3948 nr_pages, &rss, &mmap_miss, file_end);3954 map_ret = filemap_map_folio_range(vmf, folio, start,
3955 addr, nr_pages, &rss,
3956 file_end);
3957 }
3958 ret |= map_ret;
3959
3960 /*
3961 * If there are too many folios that are recently evicted
3962 * in a file, they will probably continue to be evicted.
3963 * In such situation, read-ahead is only a waste of IO.
3964 * Don't decrease mmap_miss in this scenario to make sure
3965 * we can stop read-ahead.
3966 *
3967 * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss
3968 * increment in do_sync_mmap_readahead(), so skip the
3969 * decrement here as well to keep the counter symmetric.
3970 */
3971 if ((map_ret & VM_FAULT_NOPAGE) &&
3972 !(vmf->flags & FAULT_FLAG_TRIED) &&
3973 !folio_test_workingset(folio) &&
3974 !(vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
3975 unsigned short mmap_miss;
3976
3977 mmap_miss = READ_ONCE(file->f_ra.mmap_miss);
3978 if (mmap_miss)
3979 WRITE_ONCE(file->f_ra.mmap_miss,
3980 mmap_miss - 1);
3981 }
3949 3982
3950 folio_unlock(folio);3983 folio_unlock(folio);
3951 } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL);3984 } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL);
@@ -3955,12 +3988,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 3 unmodified lines
3955out:3988out:
3956 rcu_read_unlock();3989 rcu_read_unlock();
3957 3990
3958 mmap_miss_saved = READ_ONCE(file->f_ra.mmap_miss);
3959 if (mmap_miss >= mmap_miss_saved)
3960 WRITE_ONCE(file->f_ra.mmap_miss, 0);
3961 else
3962 WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss_saved - mmap_miss);
3963
3964 return ret;3991 return ret;
3965}3992}
3966EXPORT_SYMBOL(filemap_map_pages);3993EXPORT_SYMBOL(filemap_map_pages);
mm/gup.c+4−4
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, · 2864 unmodified lines
2865 if (!folio)2865 if (!folio)
2866 goto pte_unmap;2866 goto pte_unmap;
2867 2867
2868 if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) ||2868 if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
2869 unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {2869 unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
2870 gup_put_folio(folio, 1, flags);2870 gup_put_folio(folio, 1, flags);
2871 goto pte_unmap;2871 goto pte_unmap;
2872 }2872 }
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, · 69 unmodified lines
2942 if (!folio)2942 if (!folio)
2943 return 0;2943 return 0;
2944 2944
2945 if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {2945 if (unlikely(pmd_val(orig) != pmd_val(pmdp_get_lockless(pmdp)))) {
2946 gup_put_folio(folio, refs, flags);2946 gup_put_folio(folio, refs, flags);
2947 return 0;2947 return 0;
2948 }2948 }
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, · 36 unmodified lines
2985 if (!folio)2985 if (!folio)
2986 return 0;2986 return 0;
2987 2987
2988 if (unlikely(pud_val(orig) != pud_val(*pudp))) {2988 if (unlikely(pud_val(orig) != pud_val(pudp_get(pudp)))) {
2989 gup_put_folio(folio, refs, flags);2989 gup_put_folio(folio, refs, flags);
2990 return 0;2990 return 0;
2991 }2991 }
mm/huge_memory.c+190−289
@@ -14,6 +14,7 @@ · 13 unmodified lines
14#include <linux/mmu_notifier.h>14#include <linux/mmu_notifier.h>
15#include <linux/rmap.h>15#include <linux/rmap.h>
16#include <linux/swap.h>16#include <linux/swap.h>
17#include <linux/list_lru.h>
17#include <linux/shrinker.h>18#include <linux/shrinker.h>
18#include <linux/mm_inline.h>19#include <linux/mm_inline.h>
19#include <linux/swapops.h>20#include <linux/swapops.h>
@@ -67,6 +68,8 @@ unsigned long transparent_hugepage_flags __read_mostly = · 47 unmodified lines
67 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|68 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
68 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);69 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
69 70
71static struct lock_class_key deferred_split_key;
72static struct list_lru deferred_split_lru;
70static struct shrinker *deferred_split_shrinker;73static struct shrinker *deferred_split_shrinker;
71static unsigned long deferred_split_count(struct shrinker *shrink,74static unsigned long deferred_split_count(struct shrinker *shrink,
72 struct shrink_control *sc);75 struct shrink_control *sc);
@@ -429,61 +432,75 @@ ssize_t single_hugepage_flag_store(struct kobject *kobj, · 356 unmodified lines
429 return count;432 return count;
430}433}
431 434
435enum defrag_mode {
436 DEFRAG_ALWAYS = 0,
437 DEFRAG_DEFER,
438 DEFRAG_DEFER_MADVISE,
439 DEFRAG_MADVISE,
440 DEFRAG_NEVER,
441};
442
443static const char * const defrag_mode_strings[] = {
444 [DEFRAG_ALWAYS] = "always",
445 [DEFRAG_DEFER] = "defer",
446 [DEFRAG_DEFER_MADVISE] = "defer+madvise",
447 [DEFRAG_MADVISE] = "madvise",
448 [DEFRAG_NEVER] = "never",
449};
450
451static const enum transparent_hugepage_flag defrag_flags[] = {
452 [DEFRAG_ALWAYS] = TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,
453 [DEFRAG_DEFER] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,
454 [DEFRAG_DEFER_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
455 [DEFRAG_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
456};
457
432static ssize_t defrag_show(struct kobject *kobj,458static ssize_t defrag_show(struct kobject *kobj,
433 struct kobj_attribute *attr, char *buf)459 struct kobj_attribute *attr, char *buf)
434{460{
435 const char *output;461 int active = DEFRAG_NEVER;
462 int len = 0;
463 int i;
436 464
437 if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,465 for (i = 0; i < ARRAY_SIZE(defrag_flags); i++) {
438 &transparent_hugepage_flags))466 if (test_bit(defrag_flags[i], &transparent_hugepage_flags)) {
439 output = "[always] defer defer+madvise madvise never";467 active = i;
440 else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,468 break;
441 &transparent_hugepage_flags))469 }
442 output = "always [defer] defer+madvise madvise never";470 }
443 else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
444 &transparent_hugepage_flags))
445 output = "always defer [defer+madvise] madvise never";
446 else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
447 &transparent_hugepage_flags))
448 output = "always defer defer+madvise [madvise] never";
449 else
450 output = "always defer defer+madvise madvise [never]";
451 471
452 return sysfs_emit(buf, "%s\n", output);472 for (i = 0; i < ARRAY_SIZE(defrag_mode_strings); i++) {
473 if (i == active)
474 len += sysfs_emit_at(buf, len, "[%s] ",
475 defrag_mode_strings[i]);
476 else
477 len += sysfs_emit_at(buf, len, "%s ",
478 defrag_mode_strings[i]);
479 }
480
481 /* Replace trailing space with newline */
482 buf[len - 1] = '\n';
483
484 return len;
453}485}
454 486
455static ssize_t defrag_store(struct kobject *kobj,487static ssize_t defrag_store(struct kobject *kobj,
456 struct kobj_attribute *attr,488 struct kobj_attribute *attr,
457 const char *buf, size_t count)489 const char *buf, size_t count)
458{490{
459 if (sysfs_streq(buf, "always")) {491 int mode, m;
460 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);492
461 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);493 mode = sysfs_match_string(defrag_mode_strings, buf);
462 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);494 if (mode < 0)
463 set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
464 } else if (sysfs_streq(buf, "defer+madvise")) {
465 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
466 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
467 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
468 set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
469 } else if (sysfs_streq(buf, "defer")) {
470 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
471 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
472 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
473 set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
474 } else if (sysfs_streq(buf, "madvise")) {
475 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
476 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
477 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
478 set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
479 } else if (sysfs_streq(buf, "never")) {
480 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
481 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
482 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
483 clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
484 } else
485 return -EINVAL;495 return -EINVAL;
486 496
497 for (m = 0; m < ARRAY_SIZE(defrag_flags); m++) {
498 if (m == mode)
499 set_bit(defrag_flags[m], &transparent_hugepage_flags);
500 else
501 clear_bit(defrag_flags[m], &transparent_hugepage_flags);
502 }
503
487 return count;504 return count;
488}505}
489static struct kobj_attribute defrag_attr = __ATTR_RW(defrag);506static struct kobj_attribute defrag_attr = __ATTR_RW(defrag);
@@ -918,15 +935,28 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · 428 unmodified lines
918}935}
919#endif /* CONFIG_SYSFS */936#endif /* CONFIG_SYSFS */
920 937
938int folio_memcg_alloc_deferred(struct folio *folio)
939{
940 if (mem_cgroup_disabled())
941 return 0;
942 return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
943}
944
921static int __init thp_shrinker_init(void)945static int __init thp_shrinker_init(void)
922{946{
923 deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |947 deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
924 SHRINKER_MEMCG_AWARE |948 SHRINKER_MEMCG_AWARE,
925 SHRINKER_NONSLAB,
926 "thp-deferred_split");949 "thp-deferred_split");
927 if (!deferred_split_shrinker)950 if (!deferred_split_shrinker)
928 return -ENOMEM;951 return -ENOMEM;
929 952
953 if (list_lru_init_memcg_key(&deferred_split_lru,
954 deferred_split_shrinker,
955 &deferred_split_key)) {
956 shrinker_free(deferred_split_shrinker);
957 return -ENOMEM;
958 }
959
930 deferred_split_shrinker->count_objects = deferred_split_count;960 deferred_split_shrinker->count_objects = deferred_split_count;
931 deferred_split_shrinker->scan_objects = deferred_split_scan;961 deferred_split_shrinker->scan_objects = deferred_split_scan;
932 shrinker_register(deferred_split_shrinker);962 shrinker_register(deferred_split_shrinker);
@@ -948,6 +978,7 @@ static int __init thp_shrinker_init(void) · 15 unmodified lines
948 huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");978 huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
949 if (!huge_zero_folio_shrinker) {979 if (!huge_zero_folio_shrinker) {
950 shrinker_free(deferred_split_shrinker);980 shrinker_free(deferred_split_shrinker);
981 list_lru_destroy(&deferred_split_lru);
951 return -ENOMEM;982 return -ENOMEM;
952 }983 }
953 984
@@ -962,6 +993,7 @@ static void __init thp_shrinker_exit(void) · 8 unmodified lines
962{993{
963 shrinker_free(huge_zero_folio_shrinker);994 shrinker_free(huge_zero_folio_shrinker);
964 shrinker_free(deferred_split_shrinker);995 shrinker_free(deferred_split_shrinker);
996 list_lru_destroy(&deferred_split_lru);
965}997}
966 998
967static int __init hugepage_init(void)999static int __init hugepage_init(void)
@@ -1141,119 +1173,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) · 173 unmodified lines
1141 return pmd;1173 return pmd;
1142}1174}
1143 1175
1144static struct deferred_split *split_queue_node(int nid)
1145{
1146 struct pglist_data *pgdata = NODE_DATA(nid);
1147
1148 return &pgdata->deferred_split_queue;
1149}
1150
1151#ifdef CONFIG_MEMCG
1152static inline
1153struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
1154 struct deferred_split *queue)
1155{
1156 if (mem_cgroup_disabled())
1157 return NULL;
1158 if (split_queue_node(folio_nid(folio)) == queue)
1159 return NULL;
1160 return container_of(queue, struct mem_cgroup, deferred_split_queue);
1161}
1162
1163static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
1164{
1165 return memcg ? &memcg->deferred_split_queue : split_queue_node(nid);
1166}
1167#else
1168static inline
1169struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
1170 struct deferred_split *queue)
1171{
1172 return NULL;
1173}
1174
1175static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
1176{
1177 return split_queue_node(nid);
1178}
1179#endif
1180
1181static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup *memcg)
1182{
1183 struct deferred_split *queue;
1184
1185retry:
1186 queue = memcg_split_queue(nid, memcg);
1187 spin_lock(&queue->split_queue_lock);
1188 /*
1189 * There is a period between setting memcg to dying and reparenting
1190 * deferred split queue, and during this period the THPs in the deferred
1191 * split queue will be hidden from the shrinker side.
1192 */
1193 if (unlikely(memcg_is_dying(memcg))) {
1194 spin_unlock(&queue->split_queue_lock);
1195 memcg = parent_mem_cgroup(memcg);
1196 goto retry;
1197 }
1198
1199 return queue;
1200}
1201
1202static struct deferred_split *
1203split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flags)
1204{
1205 struct deferred_split *queue;
1206
1207retry:
1208 queue = memcg_split_queue(nid, memcg);
1209 spin_lock_irqsave(&queue->split_queue_lock, *flags);
1210 if (unlikely(memcg_is_dying(memcg))) {
1211 spin_unlock_irqrestore(&queue->split_queue_lock, *flags);
1212 memcg = parent_mem_cgroup(memcg);
1213 goto retry;
1214 }
1215
1216 return queue;
1217}
1218
1219static struct deferred_split *folio_split_queue_lock(struct folio *folio)
1220{
1221 struct deferred_split *queue;
1222
1223 rcu_read_lock();
1224 queue = split_queue_lock(folio_nid(folio), folio_memcg(folio));
1225 /*
1226 * The memcg destruction path is acquiring the split queue lock for
1227 * reparenting. Once you have it locked, it's safe to drop the rcu lock.
1228 */
1229 rcu_read_unlock();
1230
1231 return queue;
1232}
1233
1234static struct deferred_split *
1235folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags)
1236{
1237 struct deferred_split *queue;
1238
1239 rcu_read_lock();
1240 queue = split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
1241 rcu_read_unlock();
1242
1243 return queue;
1244}
1245
1246static inline void split_queue_unlock(struct deferred_split *queue)
1247{
1248 spin_unlock(&queue->split_queue_lock);
1249}
1250
1251static inline void split_queue_unlock_irqrestore(struct deferred_split *queue,
1252 unsigned long flags)
1253{
1254 spin_unlock_irqrestore(&queue->split_queue_lock, flags);
1255}
1256
1257static inline bool is_transparent_hugepage(const struct folio *folio)1176static inline bool is_transparent_hugepage(const struct folio *folio)
1258{1177{
1259 if (!folio_test_large(folio))1178 if (!folio_test_large(folio))
@@ -1354,6 +1273,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, · 94 unmodified lines
1354 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);1273 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
1355 return NULL;1274 return NULL;
1356 }1275 }
1276
1277 if (folio_memcg_alloc_deferred(folio)) {
1278 folio_put(folio);
1279 count_vm_event(THP_FAULT_FALLBACK);
1280 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
1281 return NULL;
1282 }
1283
1357 folio_throttle_swaprate(folio, gfp);1284 folio_throttle_swaprate(folio, gfp);
1358 1285
1359 /*1286 /*
@@ -2638,6 +2565,8 @@ static void change_non_present_huge_pmd(struct mm_struct *mm, · 1278 unmodified lines
2638 } else if (softleaf_is_device_private_write(entry)) {2565 } else if (softleaf_is_device_private_write(entry)) {
2639 entry = make_readable_device_private_entry(swp_offset(entry));2566 entry = make_readable_device_private_entry(swp_offset(entry));
2640 newpmd = swp_entry_to_pmd(entry);2567 newpmd = swp_entry_to_pmd(entry);
2568 if (pmd_swp_uffd_wp(*pmd))
2569 newpmd = pmd_swp_mkuffd_wp(newpmd);
2641 } else {2570 } else {
2642 newpmd = *pmd;2571 newpmd = *pmd;
2643 }2572 }
@@ -3890,34 +3819,43 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n · 1246 unmodified lines
3890 struct folio *end_folio = folio_next(folio);3819 struct folio *end_folio = folio_next(folio);
3891 struct folio *new_folio, *next;3820 struct folio *new_folio, *next;
3892 int old_order = folio_order(folio);3821 int old_order = folio_order(folio);
3822 struct list_lru_one *lru;
3823 bool dequeue_deferred;
3893 int ret = 0;3824 int ret = 0;
3894 struct deferred_split *ds_queue;
3895 3825
3896 VM_WARN_ON_ONCE(!mapping && end);3826 VM_WARN_ON_ONCE(!mapping && end);
3897 /* Prevent deferred_split_scan() touching ->_refcount */3827 /*
3898 ds_queue = folio_split_queue_lock(folio);3828 * If this folio can be on the deferred split queue, lock out
3829 * the shrinker before freezing the ref. If the shrinker sees
3830 * a 0-ref folio, it assumes it beat folio_put() to the list
3831 * lock and must clean up the LRU state - the same dequeue we
3832 * will do below as part of the split.
3833 */
3834 dequeue_deferred = folio_test_anon(folio) && old_order > 1;
3835 if (dequeue_deferred) {
3836 struct mem_cgroup *memcg;
3837
3838 rcu_read_lock();
3839 memcg = folio_memcg(folio);
3840 lru = list_lru_lock(&deferred_split_lru,
3841 folio_nid(folio), &memcg);
3842 }
3899 if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {3843 if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
3900 struct swap_cluster_info *ci = NULL;3844 struct swap_cluster_info *ci = NULL;
3901 struct lruvec *lruvec;3845 struct lruvec *lruvec;
3902 3846
3903 if (old_order > 1) {3847 if (dequeue_deferred) {
3904 if (!list_empty(&folio->_deferred_list)) {3848 __list_lru_del(&deferred_split_lru, lru,
3905 ds_queue->split_queue_len--;3849 &folio->_deferred_list, folio_nid(folio));
3906 /*
3907 * Reinitialize page_deferred_list after removing the
3908 * page from the split_queue, otherwise a subsequent
3909 * split will see list corruption when checking the
3910 * page_deferred_list.
3911 */
3912 list_del_init(&folio->_deferred_list);
3913 }
3914 if (folio_test_partially_mapped(folio)) {3850 if (folio_test_partially_mapped(folio)) {
3915 folio_clear_partially_mapped(folio);3851 folio_clear_partially_mapped(folio);
3916 mod_mthp_stat(old_order,3852 mod_mthp_stat(old_order,
3917 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);3853 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
3918 }3854 }
3855 list_lru_unlock(lru);
3856 rcu_read_unlock();
3919 }3857 }
3920 split_queue_unlock(ds_queue);3858
3921 if (mapping) {3859 if (mapping) {
3922 int nr = folio_nr_pages(folio);3860 int nr = folio_nr_pages(folio);
3923 3861
@@ -4018,7 +3956,10 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n · 94 unmodified lines
4018 if (ci)3956 if (ci)
4019 swap_cluster_unlock(ci);3957 swap_cluster_unlock(ci);
4020 } else {3958 } else {
4021 split_queue_unlock(ds_queue);3959 if (dequeue_deferred) {
3960 list_lru_unlock(lru);
3961 rcu_read_unlock();
3962 }
4022 return -EAGAIN;3963 return -EAGAIN;
4023 }3964 }
4024 3965
@@ -4193,11 +4134,10 @@ fail: · 168 unmodified lines
4193 4134
4194 folio_unlock(new_folio);4135 folio_unlock(new_folio);
4195 /*4136 /*
4196 * Subpages may be freed if there wasn't any mapping4137 * Subpages whose mapping has been zapped may be freed
4197 * like if add_to_swap() is running on a lru page that4138 * earlier, but freeing them requires taking the
4198 * had its mapping zapped. And freeing these pages4139 * lru_lock, so we defer put_page() on tail pages until
4199 * requires taking the lru_lock so we do the put_page4140 * after the split completes.
4200 * of the tail pages after the split is complete.
4201 */4141 */
4202 free_folio_and_swap_cache(new_folio);4142 free_folio_and_swap_cache(new_folio);
4203 }4143 }
@@ -4385,33 +4325,37 @@ int split_folio_to_list(struct folio *folio, struct list_head *list) · 181 unmodified lines
4385 * queueing THP splits, and that list is (racily observed to be) non-empty.4325 * queueing THP splits, and that list is (racily observed to be) non-empty.
4386 *4326 *
4387 * It is unsafe to call folio_unqueue_deferred_split() until folio refcount is4327 * It is unsafe to call folio_unqueue_deferred_split() until folio refcount is
4388 * zero: because even when split_queue_lock is held, a non-empty _deferred_list4328 * zero: because even when the list_lru lock is held, a non-empty
4389 * might be in use on deferred_split_scan()'s unlocked on-stack list.4329 * _deferred_list might be in use on deferred_split_scan()'s unlocked
4330 * on-stack list.
4390 *4331 *
4391 * If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: it is4332 * The list_lru sublist is determined by folio's memcg: it is therefore
4392 * therefore important to unqueue deferred split before changing folio memcg.4333 * important to unqueue deferred split before changing folio memcg.
4393 */4334 */
4394bool __folio_unqueue_deferred_split(struct folio *folio)4335bool __folio_unqueue_deferred_split(struct folio *folio)
4395{4336{
4396 struct deferred_split *ds_queue;4337 struct mem_cgroup *memcg;
4338 struct list_lru_one *lru;
4339 int nid = folio_nid(folio);
4397 unsigned long flags;4340 unsigned long flags;
4398 bool unqueued = false;4341 bool unqueued = false;
4399 4342
4400 WARN_ON_ONCE(folio_ref_count(folio));4343 WARN_ON_ONCE(folio_ref_count(folio));
4401 WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio));4344 WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio));
4402 4345
4403 ds_queue = folio_split_queue_lock_irqsave(folio, &flags);4346 rcu_read_lock();
4404 if (!list_empty(&folio->_deferred_list)) {4347 memcg = folio_memcg(folio);
4405 ds_queue->split_queue_len--;4348 lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
4349 if (__list_lru_del(&deferred_split_lru, lru, &folio->_deferred_list, nid)) {
4406 if (folio_test_partially_mapped(folio)) {4350 if (folio_test_partially_mapped(folio)) {
4407 folio_clear_partially_mapped(folio);4351 folio_clear_partially_mapped(folio);
4408 mod_mthp_stat(folio_order(folio),4352 mod_mthp_stat(folio_order(folio),
4409 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);4353 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
4410 }4354 }
4411 list_del_init(&folio->_deferred_list);
4412 unqueued = true;4355 unqueued = true;
4413 }4356 }
4414 split_queue_unlock_irqrestore(ds_queue, flags);4357 list_lru_unlock_irqrestore(lru, &flags);
4358 rcu_read_unlock();
4415 4359
4416 return unqueued; /* useful for debug warnings */4360 return unqueued; /* useful for debug warnings */
4417}4361}
@@ -4419,7 +4363,9 @@ bool __folio_unqueue_deferred_split(struct folio *folio) · 1 unmodified line
4419/* partially_mapped=false won't clear PG_partially_mapped folio flag */4363/* partially_mapped=false won't clear PG_partially_mapped folio flag */
4420void deferred_split_folio(struct folio *folio, bool partially_mapped)4364void deferred_split_folio(struct folio *folio, bool partially_mapped)
4421{4365{
4422 struct deferred_split *ds_queue;4366 struct list_lru_one *lru;
4367 int nid;
4368 struct mem_cgroup *memcg;
4423 unsigned long flags;4369 unsigned long flags;
4424 4370
4425 /*4371 /*
@@ -4434,7 +4380,7 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 8 unmodified lines
4434 4380
4435 /*4381 /*
4436 * Exclude swapcache: originally to avoid a corrupt deferred split4382 * Exclude swapcache: originally to avoid a corrupt deferred split
4437 * queue. Nowadays that is fully prevented by memcg1_swapout();4383 * queue. Nowadays that is fully prevented by __memcg1_swapout();
4438 * but if page reclaim is already handling the same folio, it is4384 * but if page reclaim is already handling the same folio, it is
4439 * unnecessary to handle it again in the shrinker, so excluding4385 * unnecessary to handle it again in the shrinker, so excluding
4440 * swapcache here may still be a useful optimization.4386 * swapcache here may still be a useful optimization.
@@ -4442,7 +4388,11 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 1 unmodified line
4442 if (folio_test_swapcache(folio))4388 if (folio_test_swapcache(folio))
4443 return;4389 return;
4444 4390
4445 ds_queue = folio_split_queue_lock_irqsave(folio, &flags);4391 nid = folio_nid(folio);
4392
4393 rcu_read_lock();
4394 memcg = folio_memcg(folio);
4395 lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
4446 if (partially_mapped) {4396 if (partially_mapped) {
4447 if (!folio_test_partially_mapped(folio)) {4397 if (!folio_test_partially_mapped(folio)) {
4448 folio_set_partially_mapped(folio);4398 folio_set_partially_mapped(folio);
@@ -4450,36 +4400,23 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 1 unmodified line
4450 count_vm_event(THP_DEFERRED_SPLIT_PAGE);4400 count_vm_event(THP_DEFERRED_SPLIT_PAGE);
4451 count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);4401 count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
4452 mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);4402 mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);
4453
4454 }4403 }
4455 } else {4404 } else {
4456 /* partially mapped folios cannot become non-partially mapped */4405 /* partially mapped folios cannot become non-partially mapped */
4457 VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);4406 VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);
4458 }4407 }
4459 if (list_empty(&folio->_deferred_list)) {4408 __list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, memcg);
4460 struct mem_cgroup *memcg;4409 list_lru_unlock_irqrestore(lru, &flags);
4461 4410 rcu_read_unlock();
4462 memcg = folio_split_queue_memcg(folio, ds_queue);
4463 list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
4464 ds_queue->split_queue_len++;
4465 if (memcg)
4466 set_shrinker_bit(memcg, folio_nid(folio),
4467 shrinker_id(deferred_split_shrinker));
4468 }
4469 split_queue_unlock_irqrestore(ds_queue, flags);
4470}4411}
4471 4412
4472static unsigned long deferred_split_count(struct shrinker *shrink,4413static unsigned long deferred_split_count(struct shrinker *shrink,
4473 struct shrink_control *sc)4414 struct shrink_control *sc)
4474{4415{
4475 struct pglist_data *pgdata = NODE_DATA(sc->nid);4416 unsigned long count;
4476 struct deferred_split *ds_queue = &pgdata->deferred_split_queue;
4477 4417
4478#ifdef CONFIG_MEMCG4418 count = list_lru_shrink_count(&deferred_split_lru, sc);
4479 if (sc->memcg)4419 return count ?: SHRINK_EMPTY;
4480 ds_queue = &sc->memcg->deferred_split_queue;
4481#endif
4482 return READ_ONCE(ds_queue->split_queue_len);
4483}4420}
4484 4421
4485static bool thp_underused(struct folio *folio)4422static bool thp_underused(struct folio *folio)
@@ -4509,45 +4446,49 @@ static bool thp_underused(struct folio *folio) · 23 unmodified lines
4509 return false;4446 return false;
4510}4447}
4511 4448
4449static enum lru_status deferred_split_isolate(struct list_head *item,
4450 struct list_lru_one *lru,
4451 void *cb_arg)
4452{
4453 struct folio *folio = container_of(item, struct folio, _deferred_list);
4454 struct list_head *freeable = cb_arg;
4455
4456 if (folio_try_get(folio)) {
4457 list_lru_isolate_move(lru, item, freeable);
4458 return LRU_REMOVED;
4459 }
4460
4461 /*
4462 * We lost race with folio_put(). Read folio state before the
4463 * isolate: folio_unqueue_deferred_split() checks list_empty()
4464 * locklessly, so once removed the folio can be freed any time.
4465 */
4466 if (folio_test_partially_mapped(folio)) {
4467 folio_clear_partially_mapped(folio);
4468 mod_mthp_stat(folio_order(folio),
4469 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
4470 }
4471 list_lru_isolate(lru, item);
4472 return LRU_REMOVED;
4473}
4474
4512static unsigned long deferred_split_scan(struct shrinker *shrink,4475static unsigned long deferred_split_scan(struct shrinker *shrink,
4513 struct shrink_control *sc)4476 struct shrink_control *sc)
4514{4477{
4515 struct deferred_split *ds_queue;4478 LIST_HEAD(dispose);
4516 unsigned long flags;
4517 struct folio *folio, *next;4479 struct folio *folio, *next;
4518 int split = 0, i;4480 int split = 0;
4519 struct folio_batch fbatch;4481 unsigned long isolated;
4520
4521 folio_batch_init(&fbatch);
4522 4482
4523retry:4483 isolated = list_lru_shrink_walk_irq(&deferred_split_lru, sc,
4524 ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);4484 deferred_split_isolate, &dispose);
4525 /* Take pin on all head pages to avoid freeing them under us */
4526 list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
4527 _deferred_list) {
4528 if (folio_try_get(folio)) {
4529 folio_batch_add(&fbatch, folio);
4530 } else if (folio_test_partially_mapped(folio)) {
4531 /* We lost race with folio_put() */
4532 folio_clear_partially_mapped(folio);
4533 mod_mthp_stat(folio_order(folio),
4534 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
4535 }
4536 list_del_init(&folio->_deferred_list);
4537 ds_queue->split_queue_len--;
4538 if (!--sc->nr_to_scan)
4539 break;
4540 if (!folio_batch_space(&fbatch))
4541 break;
4542 }
4543 split_queue_unlock_irqrestore(ds_queue, flags);
4544 4485
4545 for (i = 0; i < folio_batch_count(&fbatch); i++) {4486 list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
4546 bool did_split = false;4487 bool did_split = false;
4547 bool underused = false;4488 bool underused = false;
4548 struct deferred_split *fqueue;
4549 4489
4550 folio = fbatch.folios[i];4490 list_del_init(&folio->_deferred_list);
4491
4551 if (!folio_test_partially_mapped(folio)) {4492 if (!folio_test_partially_mapped(folio)) {
4552 /*4493 /*
4553 * See try_to_map_unused_to_zeropage(): we cannot4494 * See try_to_map_unused_to_zeropage(): we cannot
@@ -4576,63 +4517,23 @@ next: · 22 unmodified lines
4576 * underused, then consider it used and don't add it back to4517 * underused, then consider it used and don't add it back to
4577 * split_queue.4518 * split_queue.
4578 */4519 */
4579 if (did_split || !folio_test_partially_mapped(folio))4520 if (!did_split && folio_test_partially_mapped(folio)) {
4580 continue;
4581requeue:4521requeue:
4582 /*4522 rcu_read_lock();
4583 * Add back partially mapped folios, or underused folios that4523 list_lru_add_irq(&deferred_split_lru,
4584 * we could not lock this round.4524 &folio->_deferred_list,
4585 */4525 folio_nid(folio),
4586 fqueue = folio_split_queue_lock_irqsave(folio, &flags);4526 folio_memcg(folio));
4587 if (list_empty(&folio->_deferred_list)) {4527 rcu_read_unlock();
4588 list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
4589 fqueue->split_queue_len++;
4590 }4528 }
4591 split_queue_unlock_irqrestore(fqueue, flags);4529 folio_put(folio);
4592 }
4593 folios_put(&fbatch);
4594
4595 if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
4596 cond_resched();
4597 goto retry;
4598 }4530 }
4599 4531
4600 /*4532 if (!split && !isolated)
4601 * Stop shrinker if we didn't split any page, but the queue is empty.
4602 * This can happen if pages were freed under us.
4603 */
4604 if (!split && list_empty(&ds_queue->split_queue))
4605 return SHRINK_STOP;4533 return SHRINK_STOP;
4606 return split;4534 return split;
4607}4535}
4608 4536
4609#ifdef CONFIG_MEMCG
4610void reparent_deferred_split_queue(struct mem_cgroup *memcg)
4611{
4612 struct mem_cgroup *parent = parent_mem_cgroup(memcg);
4613 struct deferred_split *ds_queue = &memcg->deferred_split_queue;
4614 struct deferred_split *parent_ds_queue = &parent->deferred_split_queue;
4615 int nid;
4616
4617 spin_lock_irq(&ds_queue->split_queue_lock);
4618 spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING);
4619
4620 if (!ds_queue->split_queue_len)
4621 goto unlock;
4622
4623 list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_queue);
4624 parent_ds_queue->split_queue_len += ds_queue->split_queue_len;
4625 ds_queue->split_queue_len = 0;
4626
4627 for_each_node(nid)
4628 set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker));
4629
4630unlock:
4631 spin_unlock(&parent_ds_queue->split_queue_lock);
4632 spin_unlock_irq(&ds_queue->split_queue_lock);
4633}
4634#endif
4635
4636#ifdef CONFIG_DEBUG_FS4537#ifdef CONFIG_DEBUG_FS
4637static void split_huge_pages_all(void)4538static void split_huge_pages_all(void)
4638{4539{
mm/hugetlb.c+4−3
@@ -2862,6 +2862,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 2861 unmodified lines
2862 map_chg_state map_chg;2862 map_chg_state map_chg;
2863 int ret, idx;2863 int ret, idx;
2864 struct hugetlb_cgroup *h_cg = NULL;2864 struct hugetlb_cgroup *h_cg = NULL;
2865 struct hugetlb_cgroup *h_cg_rsvd = NULL;
2865 gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;2866 gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
2866 2867
2867 idx = hstate_index(h);2868 idx = hstate_index(h);
@@ -2912,7 +2913,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 44 unmodified lines
2912 */2913 */
2913 if (map_chg) {2914 if (map_chg) {
2914 ret = hugetlb_cgroup_charge_cgroup_rsvd(2915 ret = hugetlb_cgroup_charge_cgroup_rsvd(
2915 idx, pages_per_huge_page(h), &h_cg);2916 idx, pages_per_huge_page(h), &h_cg_rsvd);
2916 if (ret)2917 if (ret)
2917 goto out_subpool_put;2918 goto out_subpool_put;
2918 }2919 }
@@ -2954,7 +2955,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 35 unmodified lines
2954 */2955 */
2955 if (map_chg) {2956 if (map_chg) {
2956 hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h),2957 hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h),
2957 h_cg, folio);2958 h_cg_rsvd, folio);
2958 }2959 }
2959 2960
2960 spin_unlock_irq(&hugetlb_lock);2961 spin_unlock_irq(&hugetlb_lock);
@@ -3006,7 +3007,7 @@ out_uncharge_cgroup: · 45 unmodified lines
3006out_uncharge_cgroup_reservation:3007out_uncharge_cgroup_reservation:
3007 if (map_chg)3008 if (map_chg)
3008 hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),3009 hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
3009 h_cg);3010 h_cg_rsvd);
3010out_subpool_put:3011out_subpool_put:
3011 /*3012 /*
3012 * put page to subpool iff the quota of subpool's rsv_hpages is used3013 * put page to subpool iff the quota of subpool's rsv_hpages is used
mm/hugetlb_cma.c+16−19
@@ -142,7 +142,7 @@ unsigned int __weak arch_hugetlb_cma_order(void) · 141 unmodified lines
142 142
143void __init hugetlb_cma_reserve(void)143void __init hugetlb_cma_reserve(void)
144{144{
145 unsigned long size, reserved, per_node, order;145 unsigned long size, reserved, per_node, order, gigantic_page_size;
146 bool node_specific_cma_alloc = false;146 bool node_specific_cma_alloc = false;
147 int nid;147 int nid;
148 148
@@ -162,37 +162,36 @@ void __init hugetlb_cma_reserve(void) · 13 unmodified lines
162 * breaking this assumption.162 * breaking this assumption.
163 */163 */
164 VM_WARN_ON(order <= MAX_PAGE_ORDER);164 VM_WARN_ON(order <= MAX_PAGE_ORDER);
165 gigantic_page_size = PAGE_SIZE << order;
165 166
166 hugetlb_bootmem_set_nodes();167 hugetlb_bootmem_set_nodes();
167 168
168 for (nid = 0; nid < MAX_NUMNODES; nid++) {169 for (nid = 0; nid < MAX_NUMNODES; nid++) {
169 if (hugetlb_cma_size_in_node[nid] == 0)170 size = hugetlb_cma_size_in_node[nid];
171 if (size == 0)
170 continue;172 continue;
171 173
172 if (!node_isset(nid, hugetlb_bootmem_nodes)) {174 if (!node_isset(nid, hugetlb_bootmem_nodes)) {
173 pr_warn("hugetlb_cma: invalid node %d specified\n", nid);175 pr_warn("hugetlb_cma: invalid node %d specified\n", nid);
174 hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];176 } else if (!IS_ALIGNED(size, gigantic_page_size)) {
175 hugetlb_cma_size_in_node[nid] = 0;177 pr_warn("hugetlb_cma: cma area of node %d must be a multiple of %lu MiB\n",
176 continue;178 nid, gigantic_page_size / SZ_1M);
177 }
178
179 if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) {
180 pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n",
181 nid, (PAGE_SIZE << order) / SZ_1M);
182 hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
183 hugetlb_cma_size_in_node[nid] = 0;
184 } else {179 } else {
185 node_specific_cma_alloc = true;180 node_specific_cma_alloc = true;
181 continue;
186 }182 }
183
184 hugetlb_cma_size -= size;
185 hugetlb_cma_size_in_node[nid] = 0;
187 }186 }
188 187
189 /* Validate the CMA size again in case some invalid nodes specified. */188 /* Validate the CMA size again in case some invalid nodes specified. */
190 if (!hugetlb_cma_size)189 if (!hugetlb_cma_size)
191 return;190 return;
192 191
193 if (hugetlb_cma_size < (PAGE_SIZE << order)) {192 if (!IS_ALIGNED(hugetlb_cma_size, gigantic_page_size)) {
194 pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n",193 pr_warn("hugetlb_cma: cma area must be a multiple of %lu MiB\n",
195 (PAGE_SIZE << order) / SZ_1M);194 gigantic_page_size / SZ_1M);
196 hugetlb_cma_size = 0;195 hugetlb_cma_size = 0;
197 return;196 return;
198 }197 }
@@ -204,7 +203,7 @@ void __init hugetlb_cma_reserve(void) · 5 unmodified lines
204 */203 */
205 per_node = DIV_ROUND_UP(hugetlb_cma_size,204 per_node = DIV_ROUND_UP(hugetlb_cma_size,
206 nodes_weight(hugetlb_bootmem_nodes));205 nodes_weight(hugetlb_bootmem_nodes));
207 per_node = round_up(per_node, PAGE_SIZE << order);206 per_node = round_up(per_node, gigantic_page_size);
208 pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",207 pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
209 hugetlb_cma_size / SZ_1M, per_node / SZ_1M);208 hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
210 }209 }
@@ -223,15 +222,13 @@ void __init hugetlb_cma_reserve(void) · 12 unmodified lines
223 size = min(per_node, hugetlb_cma_size - reserved);222 size = min(per_node, hugetlb_cma_size - reserved);
224 }223 }
225 224
226 size = round_up(size, PAGE_SIZE << order);
227
228 snprintf(name, sizeof(name), "hugetlb%d", nid);225 snprintf(name, sizeof(name), "hugetlb%d", nid);
229 /*226 /*
230 * Note that 'order per bit' is based on smallest size that227 * Note that 'order per bit' is based on smallest size that
231 * may be returned to CMA allocator in the case of228 * may be returned to CMA allocator in the case of
232 * huge page demotion.229 * huge page demotion.
233 */230 */
234 res = cma_declare_contiguous_multi(size, PAGE_SIZE << order,231 res = cma_declare_contiguous_multi(size, gigantic_page_size,
235 HUGETLB_PAGE_ORDER, name,232 HUGETLB_PAGE_ORDER, name,
236 &hugetlb_cma[nid], nid);233 &hugetlb_cma[nid], nid);
237 if (res) {234 if (res) {
mm/internal.h+13−14
@@ -17,7 +17,6 @@ · 16 unmodified lines
17#include <linux/rmap.h>17#include <linux/rmap.h>
18#include <linux/swap.h>18#include <linux/swap.h>
19#include <linux/leafops.h>19#include <linux/leafops.h>
20#include <linux/swap_cgroup.h>
21#include <linux/tracepoint-defs.h>20#include <linux/tracepoint-defs.h>
22 21
23/* Internal core VMA manipulation functions. */22/* Internal core VMA manipulation functions. */
@@ -451,24 +450,16 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) · 427 unmodified lines
451{450{
452 pte_t expected_pte = pte_next_swp_offset(pte);451 pte_t expected_pte = pte_next_swp_offset(pte);
453 const pte_t *end_ptep = start_ptep + max_nr;452 const pte_t *end_ptep = start_ptep + max_nr;
454 const softleaf_t entry = softleaf_from_pte(pte);
455 pte_t *ptep = start_ptep + 1;453 pte_t *ptep = start_ptep + 1;
456 unsigned short cgroup_id;
457 454
458 VM_WARN_ON(max_nr < 1);455 VM_WARN_ON(max_nr < 1);
459 VM_WARN_ON(!softleaf_is_swap(entry));456 VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte)));
460 457
461 cgroup_id = lookup_swap_cgroup_id(entry);
462 while (ptep < end_ptep) {458 while (ptep < end_ptep) {
463 softleaf_t entry;
464
465 pte = ptep_get(ptep);459 pte = ptep_get(ptep);
466 460
467 if (!pte_same(pte, expected_pte))461 if (!pte_same(pte, expected_pte))
468 break;462 break;
469 entry = softleaf_from_pte(pte);
470 if (lookup_swap_cgroup_id(entry) != cgroup_id)
471 break;
472 expected_pte = pte_next_swp_offset(expected_pte);463 expected_pte = pte_next_swp_offset(expected_pte);
473 ptep++;464 ptep++;
474 }465 }
@@ -861,7 +852,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio) · 386 unmodified lines
861 /*852 /*
862 * At this point, there is no one trying to add the folio to853 * At this point, there is no one trying to add the folio to
863 * deferred_list. If folio is not in deferred_list, it's safe854 * deferred_list. If folio is not in deferred_list, it's safe
864 * to check without acquiring the split_queue_lock.855 * to check without acquiring the list_lru lock.
865 */856 */
866 if (data_race(list_empty(&folio->_deferred_list)))857 if (data_race(list_empty(&folio->_deferred_list)))
867 return false;858 return false;
@@ -1104,9 +1095,17 @@ static inline void init_cma_pageblock(struct page *page) · 236 unmodified lines
1104}1095}
1105#endif1096#endif
1106 1097
1107 1098enum fallback_result {
1108int find_suitable_fallback(struct free_area *area, unsigned int order,1099 /* Found suitable migratetype, *mt_out is valid. */
1109 int migratetype, bool claimable);1100 FALLBACK_FOUND,
1101 /* No fallback found in requested order. */
1102 FALLBACK_EMPTY,
1103 /* Passed @claimable, but claiming whole block is a bad idea. */
1104 FALLBACK_NOCLAIM,
1105};
1106enum fallback_result
1107find_suitable_fallback(struct free_area *area, unsigned int order,
1108 int migratetype, bool claimable, int *mt_out);
1110 1109
1111static inline bool free_area_empty(struct free_area *area, int migratetype)1110static inline bool free_area_empty(struct free_area *area, int migratetype)
1112{1111{
mm/kasan/kasan_test_c.c+10−0
@@ -874,6 +874,16 @@ static void kmalloc_double_kzfree(struct kunit *test) · 873 unmodified lines
874 char *ptr;874 char *ptr;
875 size_t size = 16;875 size_t size = 16;
876 876
877 /*
878 * With the tag-based KASAN modes, if the memory happens to be
879 * reallocated between the two frees and the new allocation tag happens
880 * to match the old one, the second free will cause a memory corruption.
881 * Resolving https://bugzilla.kernel.org/show_bug.cgi?id=212177 would
882 * help to deal with this. With Generic KASAN, it's effectively
883 * impossible for the memory to get reallocated due to the quarantine.
884 */
885 KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
886
877 ptr = kmalloc(size, GFP_KERNEL);887 ptr = kmalloc(size, GFP_KERNEL);
878 KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);888 KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
879 889
mm/kfence/kfence_test.c+1−1
@@ -263,7 +263,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat · 262 unmodified lines
263 break;263 break;
264 }264 }
265 265
266 kunit_info(test, "%s: size=%zu, gfp=%x, policy=%s, cache=%i\n", __func__, size, gfp,266 kunit_info(test, "%s: size=%zu, gfp=%pGg, policy=%s, cache=%i\n", __func__, size, &gfp,
267 policy_name, !!test_cache);267 policy_name, !!test_cache);
268 268
269 /*269 /*
mm/khugepaged.c+15−6
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm) · 436 unmodified lines
437 437
438 /* __khugepaged_exit() must not run from under us */438 /* __khugepaged_exit() must not run from under us */
439 VM_BUG_ON_MM(collapse_test_exit(mm), mm);439 VM_BUG_ON_MM(collapse_test_exit(mm), mm);
440 if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
441 return;
442 440
443 slot = mm_slot_alloc(mm_slot_cache);441 slot = mm_slot_alloc(mm_slot_cache);
444 if (!slot)442 if (!slot)
445 return;443 return;
446 444
445 if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) {
446 mm_slot_free(mm_slot_cache, slot);
447 return;
448 }
449
447 spin_lock(&khugepaged_mm_lock);450 spin_lock(&khugepaged_mm_lock);
448 mm_slot_insert(mm_slots_hash, mm, slot);451 mm_slot_insert(mm_slots_hash, mm, slot);
449 /*452 /*
@@ -1120,6 +1123,11 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a · 670 unmodified lines
1120 if (result != SCAN_SUCCEED)1123 if (result != SCAN_SUCCEED)
1121 goto out_nolock;1124 goto out_nolock;
1122 1125
1126 if (folio_memcg_alloc_deferred(folio)) {
1127 result = SCAN_ALLOC_HUGE_PAGE_FAIL;
1128 goto out_nolock;
1129 }
1130
1123 mmap_read_lock(mm);1131 mmap_read_lock(mm);
1124 result = hugepage_vma_revalidate(mm, address, true, &vma, cc);1132 result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
1125 if (result != SCAN_SUCCEED) {1133 if (result != SCAN_SUCCEED) {
@@ -2528,8 +2536,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, · 1402 unmodified lines
2528 cc->progress++;2536 cc->progress++;
2529 continue;2537 continue;
2530 }2538 }
2531 hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE);2539 hstart = ALIGN(vma->vm_start, HPAGE_PMD_SIZE);
2532 hend = round_down(vma->vm_end, HPAGE_PMD_SIZE);2540 hend = ALIGN_DOWN(vma->vm_end, HPAGE_PMD_SIZE);
2533 if (khugepaged_scan.address > hend) {2541 if (khugepaged_scan.address > hend) {
2534 cc->progress++;2542 cc->progress++;
2535 continue;2543 continue;
@@ -2808,6 +2816,7 @@ static int madvise_collapse_errno(enum scan_result r) · 272 unmodified lines
2808 case SCAN_PAGE_LRU:2816 case SCAN_PAGE_LRU:
2809 case SCAN_DEL_PAGE_LRU:2817 case SCAN_DEL_PAGE_LRU:
2810 case SCAN_PAGE_FILLED:2818 case SCAN_PAGE_FILLED:
2819 case SCAN_PAGE_HAS_PRIVATE:
2811 case SCAN_PAGE_DIRTY_OR_WRITEBACK:2820 case SCAN_PAGE_DIRTY_OR_WRITEBACK:
2812 return -EAGAIN;2821 return -EAGAIN;
2813 /*2822 /*
@@ -2845,8 +2854,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · 31 unmodified lines
2845 mmgrab(mm);2854 mmgrab(mm);
2846 lru_add_drain_all();2855 lru_add_drain_all();
2847 2856
2848 hstart = (start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;2857 hstart = ALIGN(start, HPAGE_PMD_SIZE);
2849 hend = end & HPAGE_PMD_MASK;2858 hend = ALIGN_DOWN(end, HPAGE_PMD_SIZE);
2850 2859
2851 for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {2860 for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
2852 enum scan_result result = SCAN_FAIL;2861 enum scan_result result = SCAN_FAIL;
mm/kmemleak.c+140−8
@@ -92,6 +92,7 @@ · 91 unmodified lines
92#include <linux/nodemask.h>92#include <linux/nodemask.h>
93#include <linux/mm.h>93#include <linux/mm.h>
94#include <linux/workqueue.h>94#include <linux/workqueue.h>
95#include <linux/xarray.h>
95#include <linux/crc32.h>96#include <linux/crc32.h>
96 97
97#include <asm/sections.h>98#include <asm/sections.h>
@@ -157,6 +158,8 @@ struct kmemleak_object { · 59 unmodified lines
157 struct hlist_head area_list;158 struct hlist_head area_list;
158 unsigned long jiffies; /* creation timestamp */159 unsigned long jiffies; /* creation timestamp */
159 pid_t pid; /* pid of the current task */160 pid_t pid; /* pid of the current task */
161 /* per-scan dedup count, valid only while in scan-local dedup xarray */
162 unsigned int dup_count;
160 char comm[TASK_COMM_LEN]; /* executable name */163 char comm[TASK_COMM_LEN]; /* executable name */
161};164};
162 165
@@ -360,8 +363,9 @@ static const char *__object_type_str(struct kmemleak_object *object) · 197 unmodified lines
360 * Printing of the unreferenced objects information to the seq file. The363 * Printing of the unreferenced objects information to the seq file. The
361 * print_unreferenced function must be called with the object->lock held.364 * print_unreferenced function must be called with the object->lock held.
362 */365 */
363static void print_unreferenced(struct seq_file *seq,366static void __print_unreferenced(struct seq_file *seq,
364 struct kmemleak_object *object)367 struct kmemleak_object *object,
368 bool hex_dump)
365{369{
366 int i;370 int i;
367 unsigned long *entries;371 unsigned long *entries;
@@ -373,7 +377,8 @@ static void print_unreferenced(struct seq_file *seq, · 5 unmodified lines
373 object->pointer, object->size);377 object->pointer, object->size);
374 warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu\n",378 warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu\n",
375 object->comm, object->pid, object->jiffies);379 object->comm, object->pid, object->jiffies);
376 hex_dump_object(seq, object);380 if (hex_dump)
381 hex_dump_object(seq, object);
377 warn_or_seq_printf(seq, " backtrace (crc %x):\n", object->checksum);382 warn_or_seq_printf(seq, " backtrace (crc %x):\n", object->checksum);
378 383
379 for (i = 0; i < nr_entries; i++) {384 for (i = 0; i < nr_entries; i++) {
@@ -382,6 +387,12 @@ static void print_unreferenced(struct seq_file *seq, · 2 unmodified lines
382 }387 }
383}388}
384 389
390static void print_unreferenced(struct seq_file *seq,
391 struct kmemleak_object *object)
392{
393 __print_unreferenced(seq, object, true);
394}
395
385/*396/*
386 * Print the kmemleak_object information. This function is used mainly for397 * Print the kmemleak_object information. This function is used mainly for
387 * debugging special cases when kmemleak operations. It must be called with398 * debugging special cases when kmemleak operations. It must be called with
@@ -1684,6 +1695,103 @@ unlock_put: · 1296 unmodified lines
1684 put_object(object);1695 put_object(object);
1685}1696}
1686 1697
1698/*
1699 * Print one leak inline. The hex dump is gated on OBJECT_ALLOCATED so it
1700 * does not touch user memory that was freed concurrently; the rest of the
1701 * report (backtrace, comm, pid) is always emitted since the kmemleak_object
1702 * metadata is pinned by the caller.
1703 */
1704static void print_leak_locked(struct kmemleak_object *object, bool hex_dump)
1705{
1706 raw_spin_lock_irq(&object->lock);
1707 __print_unreferenced(NULL, object,
1708 hex_dump && (object->flags & OBJECT_ALLOCATED));
1709 raw_spin_unlock_irq(&object->lock);
1710}
1711
1712/*
1713 * Per-scan dedup table for verbose leak printing. The xarray is keyed by
1714 * stackdepot trace_handle and stores a pointer to the representative
1715 * kmemleak_object. The per-scan repeat count lives in object->dup_count.
1716 *
1717 * dedup_record() must run outside object->lock: xa_store() may take
1718 * mutexes (xa_node slab allocation) which lockdep would flag against the
1719 * raw spinlock object->lock.
1720 */
1721static void dedup_record(struct xarray *dedup, struct kmemleak_object *object,
1722 depot_stack_handle_t trace_handle)
1723{
1724 struct kmemleak_object *rep;
1725 void *old;
1726
1727 /*
1728 * No stack trace to dedup against: early-boot allocation tracked
1729 * before kmemleak_init() set up object_cache, or stack_depot_save()
1730 * failure under memory pressure.
1731 */
1732 if (!trace_handle) {
1733 print_leak_locked(object, true);
1734 return;
1735 }
1736
1737 /* stack is available, now we can de-dup */
1738 rep = xa_load(dedup, trace_handle);
1739 if (rep) {
1740 rep->dup_count++;
1741 return;
1742 }
1743
1744 /*
1745 * Object is being torn down (use_count already hit zero); the
1746 * tracked memory at object->pointer is unsafe to read, so skip.
1747 */
1748 if (!get_object(object))
1749 return;
1750
1751 object->dup_count = 1;
1752 old = xa_store(dedup, trace_handle, object, GFP_ATOMIC);
1753 if (xa_is_err(old)) {
1754 /* xa_node allocation failed; fall back to inline print. */
1755 print_leak_locked(object, true);
1756 put_object(object);
1757 return;
1758 }
1759 /*
1760 * scan_mutex serialises all writers to the dedup xarray, so xa_store()
1761 * after a NULL xa_load() must always overwrite an empty slot.
1762 */
1763 WARN_ON_ONCE(old);
1764}
1765
1766/*
1767 * Drain the dedup table. Re-acquires object->lock and re-checks
1768 * OBJECT_ALLOCATED before printing: while get_object() pins the
1769 * kmemleak_object metadata, the underlying tracked allocation may have
1770 * been freed since the scan walked it (kmemleak_free clears
1771 * OBJECT_ALLOCATED under object->lock before the user memory goes away).
1772 * The hex dump is skipped for coalesced entries since the bytes would
1773 * differ across objects anyway.
1774 */
1775static void dedup_flush(struct xarray *dedup)
1776{
1777 struct kmemleak_object *object;
1778 unsigned long idx;
1779 unsigned int dup;
1780 bool coalesced;
1781
1782 xa_for_each(dedup, idx, object) {
1783 dup = object->dup_count;
1784 coalesced = dup > 1;
1785
1786 print_leak_locked(object, !coalesced);
1787 if (coalesced)
1788 pr_warn(" ... and %u more object(s) with the same backtrace\n",
1789 dup - 1);
1790 put_object(object);
1791 xa_erase(dedup, idx);
1792 }
1793}
1794
1687/*1795/*
1688 * Scan data sections and all the referenced memory blocks allocated via the1796 * Scan data sections and all the referenced memory blocks allocated via the
1689 * kernel's standard allocators. This function must be called with the1797 * kernel's standard allocators. This function must be called with the
@@ -1694,6 +1802,7 @@ static void kmemleak_scan(void) · 4 unmodified lines
1694 struct kmemleak_object *object;1802 struct kmemleak_object *object;
1695 struct zone *zone;1803 struct zone *zone;
1696 int __maybe_unused i;1804 int __maybe_unused i;
1805 struct xarray dedup;
1697 int new_leaks = 0;1806 int new_leaks = 0;
1698 1807
1699 jiffies_last_scan = jiffies;1808 jiffies_last_scan = jiffies;
@@ -1834,10 +1943,18 @@ static void kmemleak_scan(void) · 134 unmodified lines
1834 return;1943 return;
1835 1944
1836 /*1945 /*
1837 * Scanning result reporting.1946 * Scanning result reporting. When verbose printing is enabled, dedupe
1947 * by stackdepot trace_handle so each unique backtrace is logged once
1948 * per scan, annotated with the number of objects that share it. The
1949 * per-leak count below still reflects every object, and
1950 * /sys/kernel/debug/kmemleak still lists them individually.
1838 */1951 */
1952 xa_init(&dedup);
1839 rcu_read_lock();1953 rcu_read_lock();
1840 list_for_each_entry_rcu(object, &object_list, object_list) {1954 list_for_each_entry_rcu(object, &object_list, object_list) {
1955 depot_stack_handle_t trace_handle;
1956 bool dedup_print;
1957
1841 if (need_resched())1958 if (need_resched())
1842 kmemleak_cond_resched(object);1959 kmemleak_cond_resched(object);
1843 1960
@@ -1849,18 +1966,33 @@ static void kmemleak_scan(void) · 5 unmodified lines
1849 if (!color_white(object))1966 if (!color_white(object))
1850 continue;1967 continue;
1851 raw_spin_lock_irq(&object->lock);1968 raw_spin_lock_irq(&object->lock);
1969 trace_handle = 0;
1970 dedup_print = false;
1852 if (unreferenced_object(object) &&1971 if (unreferenced_object(object) &&
1853 !(object->flags & OBJECT_REPORTED)) {1972 !(object->flags & OBJECT_REPORTED)) {
1854 object->flags |= OBJECT_REPORTED;1973 object->flags |= OBJECT_REPORTED;
1855 1974 if (kmemleak_verbose) {
1856 if (kmemleak_verbose)1975 trace_handle = object->trace_handle;
1857 print_unreferenced(NULL, object);1976 dedup_print = true;
1858 1977 }
1859 new_leaks++;1978 new_leaks++;
1860 }1979 }
1861 raw_spin_unlock_irq(&object->lock);1980 raw_spin_unlock_irq(&object->lock);
1981
1982 /*
1983 * Defer the verbose print outside object->lock: xa_store()
1984 * may take xa_node slab locks at a higher wait-context level
1985 * which lockdep would flag against the raw_spinlock_t
1986 * object->lock. rcu_read_lock() keeps the kmemleak_object
1987 * alive across the call.
1988 */
1989 if (dedup_print)
1990 dedup_record(&dedup, object, trace_handle);
1862 }1991 }
1863 rcu_read_unlock();1992 rcu_read_unlock();
1993 /* Flush'em all */
1994 dedup_flush(&dedup);
1995 xa_destroy(&dedup);
1864 1996
1865 if (new_leaks) {1997 if (new_leaks) {
1866 kmemleak_found_leaks = true;1998 kmemleak_found_leaks = true;
mm/list_lru.c+162−76
@@ -15,6 +15,28 @@ · 14 unmodified lines
15#include "slab.h"15#include "slab.h"
16#include "internal.h"16#include "internal.h"
17 17
18static inline void lock_list_lru(struct list_lru_one *l, bool irq,
19 unsigned long *irq_flags)
20{
21 if (irq_flags)
22 spin_lock_irqsave(&l->lock, *irq_flags);
23 else if (irq)
24 spin_lock_irq(&l->lock);
25 else
26 spin_lock(&l->lock);
27}
28
29static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
30 unsigned long *irq_flags)
31{
32 if (irq_flags)
33 spin_unlock_irqrestore(&l->lock, *irq_flags);
34 else if (irq_off)
35 spin_unlock_irq(&l->lock);
36 else
37 spin_unlock(&l->lock);
38}
39
18#ifdef CONFIG_MEMCG40#ifdef CONFIG_MEMCG
19static LIST_HEAD(memcg_list_lrus);41static LIST_HEAD(memcg_list_lrus);
20static DEFINE_MUTEX(list_lrus_mutex);42static DEFINE_MUTEX(list_lrus_mutex);
@@ -60,34 +82,23 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 39 unmodified lines
60 return &lru->node[nid].lru;82 return &lru->node[nid].lru;
61}83}
62 84
63static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
64{
65 if (irq)
66 spin_lock_irq(&l->lock);
67 else
68 spin_lock(&l->lock);
69 if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
70 if (irq)
71 spin_unlock_irq(&l->lock);
72 else
73 spin_unlock(&l->lock);
74 return false;
75 }
76 return true;
77}
78
79static inline struct list_lru_one *85static inline struct list_lru_one *
80lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,86lock_list_lru_of_memcg(struct list_lru *lru, int nid,
81 bool irq, bool skip_empty)87 struct mem_cgroup **memcg, bool irq,
88 unsigned long *irq_flags, bool skip_empty)
82{89{
83 struct list_lru_one *l;90 struct list_lru_one *l;
84 91
85 rcu_read_lock();92 rcu_read_lock();
86again:93again:
87 l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));94 l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(*memcg));
88 if (likely(l) && lock_list_lru(l, irq)) {95 if (likely(l)) {
89 rcu_read_unlock();96 lock_list_lru(l, irq, irq_flags);
90 return l;97 if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
98 rcu_read_unlock();
99 return l;
100 }
101 unlock_list_lru(l, irq, irq_flags);
91 }102 }
92 /*103 /*
93 * Caller may simply bail out if raced with reparenting or104 * Caller may simply bail out if raced with reparenting or
@@ -97,18 +108,10 @@ again: · 3 unmodified lines
97 rcu_read_unlock();108 rcu_read_unlock();
98 return NULL;109 return NULL;
99 }110 }
100 VM_WARN_ON(!css_is_dying(&memcg->css));111 VM_WARN_ON(!css_is_dying(&(*memcg)->css));
101 memcg = parent_mem_cgroup(memcg);112 *memcg = parent_mem_cgroup(*memcg);
102 goto again;113 goto again;
103}114}
104
105static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
106{
107 if (irq_off)
108 spin_unlock_irq(&l->lock);
109 else
110 spin_unlock(&l->lock);
111}
112#else115#else
113static void list_lru_register(struct list_lru *lru)116static void list_lru_register(struct list_lru *lru)
114{117{
@@ -135,52 +138,112 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 20 unmodified lines
135}138}
136 139
137static inline struct list_lru_one *140static inline struct list_lru_one *
138lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,141lock_list_lru_of_memcg(struct list_lru *lru, int nid,
139 bool irq, bool skip_empty)142 struct mem_cgroup **memcg, bool irq,
143 unsigned long *irq_flags, bool skip_empty)
140{144{
141 struct list_lru_one *l = &lru->node[nid].lru;145 struct list_lru_one *l = &lru->node[nid].lru;
142 146
143 if (irq)147 lock_list_lru(l, irq, irq_flags);
144 spin_lock_irq(&l->lock);
145 else
146 spin_lock(&l->lock);
147 148
148 return l;149 return l;
149}150}
151#endif /* CONFIG_MEMCG */
150 152
151static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)153struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
154 struct mem_cgroup **memcg)
152{155{
153 if (irq_off)156 return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
154 spin_unlock_irq(&l->lock);157 /*irq_flags=*/NULL, /*skip_empty=*/false);
155 else
156 spin_unlock(&l->lock);
157}158}
158#endif /* CONFIG_MEMCG */
159 159
160/* The caller must ensure the memcg lifetime. */160void list_lru_unlock(struct list_lru_one *l)
161bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
162 struct mem_cgroup *memcg)
163{161{
164 struct list_lru_node *nlru = &lru->node[nid];162 unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
165 struct list_lru_one *l;163}
166 164
167 l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);165struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
168 if (!l)166 struct mem_cgroup **memcg)
169 return false;167{
168 return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
169 /*irq_flags=*/NULL, /*skip_empty=*/false);
170}
171
172void list_lru_unlock_irq(struct list_lru_one *l)
173{
174 unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
175}
176
177struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
178 struct mem_cgroup **memcg,
179 unsigned long *flags)
180{
181 return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
182 /*irq_flags=*/flags, /*skip_empty=*/false);
183}
184
185void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
186{
187 unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
188}
189
190bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
191 struct list_head *item, int nid,
192 struct mem_cgroup *memcg)
193{
170 if (list_empty(item)) {194 if (list_empty(item)) {
171 list_add_tail(item, &l->list);195 list_add_tail(item, &l->list);
172 /* Set shrinker bit if the first element was added */196 /*
197 * Set shrinker bit on the memcg that owns the locked
198 * sublist - lock_list_lru_of_memcg() may have walked up
199 * past a dying memcg, and the bit must be set there.
200 */
173 if (!l->nr_items++)201 if (!l->nr_items++)
174 set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));202 set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
175 unlock_list_lru(l, false);203 atomic_long_inc(&lru->node[nid].nr_items);
176 atomic_long_inc(&nlru->nr_items);
177 return true;204 return true;
178 }205 }
179 unlock_list_lru(l, false);
180 return false;206 return false;
181}207}
182EXPORT_SYMBOL_GPL(list_lru_add);208EXPORT_SYMBOL_GPL(list_lru_add);
183 209
210bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
211 struct list_head *item, int nid)
212{
213 if (!list_empty(item)) {
214 list_del_init(item);
215 l->nr_items--;
216 atomic_long_dec(&lru->node[nid].nr_items);
217 return true;
218 }
219 return false;
220}
221
222/* The caller must ensure the memcg lifetime. */
223bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
224 struct mem_cgroup *memcg)
225{
226 struct list_lru_one *l;
227 bool ret;
228
229 l = list_lru_lock(lru, nid, &memcg);
230 ret = __list_lru_add(lru, l, item, nid, memcg);
231 list_lru_unlock(l);
232 return ret;
233}
234
235bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
236 int nid, struct mem_cgroup *memcg)
237{
238 struct list_lru_one *l;
239 bool ret;
240
241 l = list_lru_lock_irq(lru, nid, &memcg);
242 ret = __list_lru_add(lru, l, item, nid, memcg);
243 list_lru_unlock_irq(l);
244 return ret;
245}
246
184bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)247bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
185{248{
186 bool ret;249 bool ret;
@@ -202,20 +265,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); · 15 unmodified lines
202bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,265bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
203 struct mem_cgroup *memcg)266 struct mem_cgroup *memcg)
204{267{
205 struct list_lru_node *nlru = &lru->node[nid];
206 struct list_lru_one *l;268 struct list_lru_one *l;
207 l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);269 bool ret;
208 if (!l)270
209 return false;271 l = list_lru_lock(lru, nid, &memcg);
210 if (!list_empty(item)) {272 ret = __list_lru_del(lru, l, item, nid);
211 list_del_init(item);273 list_lru_unlock(l);
212 l->nr_items--;274 return ret;
213 unlock_list_lru(l, false);
214 atomic_long_dec(&nlru->nr_items);
215 return true;
216 }
217 unlock_list_lru(l, false);
218 return false;
219}275}
220 276
221bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)277bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
@@ -288,7 +344,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, · 66 unmodified lines
288 unsigned long isolated = 0;344 unsigned long isolated = 0;
289 345
290restart:346restart:
291 l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);347 l = lock_list_lru_of_memcg(lru, nid, &memcg, /*irq=*/irq_off,
348 /*irq_flags=*/NULL, /*skip_empty=*/true);
292 if (!l)349 if (!l)
293 return isolated;350 return isolated;
294 list_for_each_safe(item, n, &l->list) {351 list_for_each_safe(item, n, &l->list) {
@@ -329,7 +386,7 @@ restart: · 34 unmodified lines
329 BUG();386 BUG();
330 }387 }
331 }388 }
332 unlock_list_lru(l, irq_off);389 unlock_list_lru(l, irq_off, NULL);
333out:390out:
334 return isolated;391 return isolated;
335}392}
@@ -514,17 +571,14 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg, · 178 unmodified lines
514 return idx < 0 || xa_load(&lru->xa, idx);571 return idx < 0 || xa_load(&lru->xa, idx);
515}572}
516 573
517int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,574static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
518 gfp_t gfp)575 struct list_lru *lru, gfp_t gfp)
519{576{
520 unsigned long flags;577 unsigned long flags;
521 struct list_lru_memcg *mlru = NULL;578 struct list_lru_memcg *mlru = NULL;
522 struct mem_cgroup *pos, *parent;579 struct mem_cgroup *pos, *parent;
523 XA_STATE(xas, &lru->xa, 0);580 XA_STATE(xas, &lru->xa, 0);
524 581
525 if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
526 return 0;
527
528 gfp &= GFP_RECLAIM_MASK;582 gfp &= GFP_RECLAIM_MASK;
529 /*583 /*
530 * Because the list_lru can be reparented to the parent cgroup's584 * Because the list_lru can be reparented to the parent cgroup's
@@ -565,6 +619,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, · 34 unmodified lines
565 619
566 return xas_error(&xas);620 return xas_error(&xas);
567}621}
622
623int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
624 gfp_t gfp)
625{
626 if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
627 return 0;
628 return __memcg_list_lru_alloc(memcg, lru, gfp);
629}
630
631int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
632 gfp_t gfp)
633{
634 struct mem_cgroup *memcg;
635 int res;
636
637 if (!list_lru_memcg_aware(lru))
638 return 0;
639
640 /* Fast path when list_lru heads already exist */
641 rcu_read_lock();
642 memcg = folio_memcg(folio);
643 res = memcg_list_lru_allocated(memcg, lru);
644 rcu_read_unlock();
645 if (likely(res))
646 return 0;
647
648 /* Allocation may block, pin the memcg */
649 memcg = get_mem_cgroup_from_folio(folio);
650 res = __memcg_list_lru_alloc(memcg, lru, gfp);
651 mem_cgroup_put(memcg);
652 return res;
653}
568#else654#else
569static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware)655static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware)
570{656{
mm/madvise.c+25−35
@@ -1834,50 +1834,29 @@ static void madvise_finish_tlb(struct madvise_behavior *madv_behavior) · 1833 unmodified lines
1834 tlb_finish_mmu(madv_behavior->tlb);1834 tlb_finish_mmu(madv_behavior->tlb);
1835}1835}
1836 1836
1837static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior)1837/**
1838 * check_input_range() - Check if the requested range is valid.
1839 * @start: Start address of madvise-requested address range.
1840 * @len_in: Length of madvise-requested address range.
1841 *
1842 * Returns: 0 if the input range is valid, otherwise an error code.
1843 */
1844static int check_input_range(unsigned long start, size_t len_in)
1838{1845{
1839 size_t len;1846 size_t len;
1840 1847
1841 if (!madvise_behavior_valid(behavior))
1842 return false;
1843
1844 if (!PAGE_ALIGNED(start))1848 if (!PAGE_ALIGNED(start))
1845 return false;1849 return -EINVAL;
1846 len = PAGE_ALIGN(len_in);1850 len = PAGE_ALIGN(len_in);
1847 1851
1848 /* Check to see whether len was rounded up from small -ve to zero */1852 /* Check to see whether len was rounded up from small -ve to zero */
1849 if (len_in && !len)1853 if (len_in && !len)
1850 return false;1854 return -EINVAL;
1851 1855
1852 if (start + len < start)1856 if (start + len < start)
1853 return false;1857 return -EINVAL;
1854
1855 return true;
1856}
1857 1858
1858/*1859 return 0;
1859 * madvise_should_skip() - Return if the request is invalid or nothing.
1860 * @start: Start address of madvise-requested address range.
1861 * @len_in: Length of madvise-requested address range.
1862 * @behavior: Requested madvise behavior.
1863 * @err: Pointer to store an error code from the check.
1864 *
1865 * If the specified behaviour is invalid or nothing would occur, we skip the
1866 * operation. This function returns true in the cases, otherwise false. In
1867 * the former case we store an error on @err.
1868 */
1869static bool madvise_should_skip(unsigned long start, size_t len_in,
1870 int behavior, int *err)
1871{
1872 if (!is_valid_madvise(start, len_in, behavior)) {
1873 *err = -EINVAL;
1874 return true;
1875 }
1876 if (start + PAGE_ALIGN(len_in) == start) {
1877 *err = 0;
1878 return true;
1879 }
1880 return false;
1881}1860}
1882 1861
1883static bool is_madvise_populate(struct madvise_behavior *madv_behavior)1862static bool is_madvise_populate(struct madvise_behavior *madv_behavior)
@@ -2013,8 +1992,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh · 129 unmodified lines
2013 .tlb = &tlb,1992 .tlb = &tlb,
2014 };1993 };
2015 1994
2016 if (madvise_should_skip(start, len_in, behavior, &error))1995 if (!madvise_behavior_valid(behavior))
1996 return -EINVAL;
1997
1998 error = check_input_range(start, len_in);
1999 if (error || !len_in)
2017 return error;2000 return error;
2001
2018 error = madvise_lock(&madv_behavior);2002 error = madvise_lock(&madv_behavior);
2019 if (error)2003 if (error)
2020 return error;2004 return error;
@@ -2056,7 +2040,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · 35 unmodified lines
2056 size_t len_in = iter_iov_len(iter);2040 size_t len_in = iter_iov_len(iter);
2057 int error;2041 int error;
2058 2042
2059 if (madvise_should_skip(start, len_in, behavior, &error))2043 error = check_input_range(start, len_in);
2044 if (error || !len_in)
2060 ret = error;2045 ret = error;
2061 else2046 else
2062 ret = madvise_do_behavior(start, len_in, &madv_behavior);2047 ret = madvise_do_behavior(start, len_in, &madv_behavior);
@@ -2131,6 +2116,11 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, · 68 unmodified lines
2131 goto release_task;2116 goto release_task;
2132 }2117 }
2133 2118
2119 if (!madvise_behavior_valid(behavior)) {
2120 ret = -EINVAL;
2121 goto release_mm;
2122 }
2123
2134 /*2124 /*
2135 * We need only perform this check if we are attempting to manipulate a2125 * We need only perform this check if we are attempting to manipulate a
2136 * remote process's address space.2126 * remote process's address space.
mm/memcontrol-v1.c+44−24
@@ -5,7 +5,6 @@ · 4 unmodified lines
5#include <linux/mm_inline.h>5#include <linux/mm_inline.h>
6#include <linux/pagewalk.h>6#include <linux/pagewalk.h>
7#include <linux/backing-dev.h>7#include <linux/backing-dev.h>
8#include <linux/swap_cgroup.h>
9#include <linux/eventfd.h>8#include <linux/eventfd.h>
10#include <linux/poll.h>9#include <linux/poll.h>
11#include <linux/sort.h>10#include <linux/sort.h>
@@ -14,6 +13,7 @@ · 2 unmodified lines
14 13
15#include "internal.h"14#include "internal.h"
16#include "swap.h"15#include "swap.h"
16#include "swap_table.h"
17#include "memcontrol-v1.h"17#include "memcontrol-v1.h"
18 18
19/*19/*
@@ -603,19 +603,26 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 583 unmodified lines
603 local_irq_restore(flags);603 local_irq_restore(flags);
604}604}
605 605
606#ifdef CONFIG_SWAP
606/**607/**
607 * memcg1_swapout - transfer a memsw charge to swap608 * __memcg1_swapout - transfer a memsw charge to swap
608 * @folio: folio whose memsw charge to transfer609 * @folio: folio whose memsw charge to transfer
609 * @entry: swap entry to move the charge to610 * @ci: the locked swap cluster holding the swap entries
611 *
612 * Transfer the memsw charge of @folio to the swap entry stored in
613 * folio->swap.
610 *614 *
611 * Transfer the memsw charge of @folio to @entry.615 * Context: folio must be isolated, unmapped, locked and is just about to
616 * be freed, and caller must disable IRQs and hold the swap cluster lock.
612 */617 */
613void memcg1_swapout(struct folio *folio, swp_entry_t entry)618void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
614{619{
615 struct mem_cgroup *memcg, *swap_memcg;620 struct mem_cgroup *memcg, *swap_memcg;
616 struct obj_cgroup *objcg;621 struct obj_cgroup *objcg;
617 unsigned int nr_entries;622 unsigned int nr_entries;
618 623
624 VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
625 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
619 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);626 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
620 VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);627 VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);
621 628
@@ -641,7 +648,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 19 unmodified lines
641 swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries);648 swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries);
642 mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);649 mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);
643 650
644 swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry);651 __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries,
652 mem_cgroup_private_id(swap_memcg));
645 653
646 folio_unqueue_deferred_split(folio);654 folio_unqueue_deferred_split(folio);
647 folio->memcg_data = 0;655 folio->memcg_data = 0;
@@ -656,8 +664,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 8 unmodified lines
656 }664 }
657 665
658 /*666 /*
659 * Interrupts should be disabled here because the caller holds the667 * The caller must hold the swap cluster lock with IRQ off. It is
660 * i_pages lock which is taken with interrupts-off. It is
661 * important here to have the interrupts disabled because it is the668 * important here to have the interrupts disabled because it is the
662 * only synchronisation we have for updating the per-CPU variables.669 * only synchronisation we have for updating the per-CPU variables.
663 */670 */
@@ -671,18 +678,24 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 7 unmodified lines
671 obj_cgroup_put(objcg);678 obj_cgroup_put(objcg);
672}679}
673 680
674/*681/**
675 * memcg1_swapin - uncharge swap slot682 * memcg1_swapin - uncharge swap slot on swapin
676 * @entry: the first swap entry for which the pages are charged683 * @folio: folio being swapped in
677 * @nr_pages: number of pages which will be uncharged
678 *684 *
679 * Call this function after successfully adding the charged page to swapcache.685 * Call this function after successfully adding the charged
686 * folio to swapcache.
680 *687 *
681 * Note: This function assumes the page for which swap slot is being uncharged688 * Context: The folio has to be in swap cache and locked.
682 * is order 0 page.
683 */689 */
684void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)690void memcg1_swapin(struct folio *folio)
685{691{
692 struct swap_cluster_info *ci;
693 unsigned long nr_pages;
694 unsigned short id;
695
696 VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
697 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
698
686 /*699 /*
687 * Cgroup1's unified memory+swap counter has been charged with the700 * Cgroup1's unified memory+swap counter has been charged with the
688 * new swapcache page, finish the transfer by uncharging the swap701 * new swapcache page, finish the transfer by uncharging the swap
@@ -695,15 +708,22 @@ void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages) · 6 unmodified lines
695 * correspond 1:1 to page and swap slot lifetimes: we charge the708 * correspond 1:1 to page and swap slot lifetimes: we charge the
696 * page to memory here, and uncharge swap when the slot is freed.709 * page to memory here, and uncharge swap when the slot is freed.
697 */710 */
698 if (do_memsw_account()) {711 if (!do_memsw_account())
699 /*712 return;
700 * The swap entry might not get freed for a long time,713
701 * let's not wait for it. The page already received a714 /*
702 * memory+swap charge, drop the swap entry duplicate.715 * The swap entry might not get freed for a long time,
703 */716 * let's not wait for it. The page already received a
704 mem_cgroup_uncharge_swap(entry, nr_pages);717 * memory+swap charge, drop the swap entry duplicate.
705 }718 */
719 nr_pages = folio_nr_pages(folio);
720 ci = swap_cluster_get_and_lock(folio);
721 id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
722 nr_pages);
723 swap_cluster_unlock(ci);
724 mem_cgroup_uncharge_swap(id, nr_pages);
706}725}
726#endif
707 727
708void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,728void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
709 unsigned long nr_memory, int nid)729 unsigned long nr_memory, int nid)
mm/memcontrol.c+197−97
@@ -54,7 +54,6 @@ · 53 unmodified lines
54#include <linux/vmpressure.h>54#include <linux/vmpressure.h>
55#include <linux/memremap.h>55#include <linux/memremap.h>
56#include <linux/mm_inline.h>56#include <linux/mm_inline.h>
57#include <linux/swap_cgroup.h>
58#include <linux/cpu.h>57#include <linux/cpu.h>
59#include <linux/oom.h>58#include <linux/oom.h>
60#include <linux/lockdep.h>59#include <linux/lockdep.h>
@@ -64,6 +63,7 @@ · 3 unmodified lines
64#include <linux/sched/isolation.h>63#include <linux/sched/isolation.h>
65#include <linux/kmemleak.h>64#include <linux/kmemleak.h>
66#include "internal.h"65#include "internal.h"
66#include "swap_table.h"
67#include <net/sock.h>67#include <net/sock.h>
68#include <net/ip.h>68#include <net/ip.h>
69#include "slab.h"69#include "slab.h"
@@ -150,15 +150,15 @@ static void obj_cgroup_release(struct percpu_ref *ref) · 80 unmodified lines
150 * However, it can be PAGE_SIZE or (x * PAGE_SIZE).150 * However, it can be PAGE_SIZE or (x * PAGE_SIZE).
151 *151 *
152 * The following sequence can lead to it:152 * The following sequence can lead to it:
153 * 1) CPU0: objcg == stock->cached_objcg153 * 1) CPU0: objcg cached in one of stock->cached[i]
154 * 2) CPU1: we do a small allocation (e.g. 92 bytes),154 * 2) CPU1: we do a small allocation (e.g. 92 bytes),
155 * PAGE_SIZE bytes are charged155 * PAGE_SIZE bytes are charged
156 * 3) CPU1: a process from another memcg is allocating something,156 * 3) CPU1: a process from another memcg is allocating something,
157 * the stock if flushed,157 * the stock if flushed,
158 * objcg->nr_charged_bytes = PAGE_SIZE - 92158 * objcg->nr_charged_bytes = PAGE_SIZE - 92
159 * 5) CPU0: we do release this object,159 * 4) CPU0: we do release this object,
160 * 92 bytes are added to stock->nr_bytes160 * 92 bytes are added to stock->nr_bytes[i]
161 * 6) CPU0: stock is flushed,161 * 5) CPU0: stock is flushed,
162 * 92 bytes are added to objcg->nr_charged_bytes162 * 92 bytes are added to objcg->nr_charged_bytes
163 *163 *
164 * In the result, nr_charged_bytes == PAGE_SIZE.164 * In the result, nr_charged_bytes == PAGE_SIZE.
@@ -2018,24 +2018,49 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { · 1853 unmodified lines
2018 .lock = INIT_LOCAL_TRYLOCK(lock),2018 .lock = INIT_LOCAL_TRYLOCK(lock),
2019};2019};
2020 2020
2021/*
2022 * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
2023 * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
2024 * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
2025 * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
2026 * + pad(6) + cached(40) == 64 bytes.
2027 * A CPU can thus consume/refill/account against five different objcgs
2028 * (typically per-node variants of the same memcg) while incurring at
2029 * most one cache miss on the stock.
2030 */
2031#define NR_OBJ_STOCK 5
2021struct obj_stock_pcp {2032struct obj_stock_pcp {
2022 local_trylock_t lock;2033 local_trylock_t lock;
2023 unsigned int nr_bytes;2034 int8_t index;
2024 struct obj_cgroup *cached_objcg;2035 int16_t node_id;
2025 struct pglist_data *cached_pgdat;2036 int16_t nr_slab_reclaimable_b;
2026 int nr_slab_reclaimable_b;2037 int16_t nr_slab_unreclaimable_b;
2027 int nr_slab_unreclaimable_b;2038#if PAGE_SHIFT > 16
2039 /*
2040 * On rare archs with 256KiB base page size (hexagon and powerpc 44x)
2041 * keep nr_bytes to unsigned int as uint16_t cannot represent the full
2042e patches/memcg-uint16_t-for-nr_bytes-in-obj_stock_pcp.patch * sub-page remainder. Such archs are not cacheline optimization target.
2043 */
2044 unsigned int nr_bytes[NR_OBJ_STOCK];
2045#else
2046 uint16_t nr_bytes[NR_OBJ_STOCK];
2047#endif
2048 struct obj_cgroup *cached[NR_OBJ_STOCK];
2028 2049
2029 struct work_struct work;2050 struct work_struct work;
2030 unsigned long flags;2051 unsigned long flags;
2052 uint8_t drain_idx;
2031};2053};
2032 2054
2033static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {2055static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
2034 .lock = INIT_LOCAL_TRYLOCK(lock),2056 .lock = INIT_LOCAL_TRYLOCK(lock),
2057 .index = -1,
2058 .node_id = NUMA_NO_NODE,
2035};2059};
2036 2060
2037static DEFINE_MUTEX(percpu_charge_mutex);2061static DEFINE_MUTEX(percpu_charge_mutex);
2038 2062
2063static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
2039static void drain_obj_stock(struct obj_stock_pcp *stock);2064static void drain_obj_stock(struct obj_stock_pcp *stock);
2040static bool obj_stock_flush_required(struct obj_stock_pcp *stock,2065static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
2041 struct mem_cgroup *root_memcg);2066 struct mem_cgroup *root_memcg);
@@ -3155,54 +3180,73 @@ static void unlock_stock(struct obj_stock_pcp *stock) · 1113 unmodified lines
3155 local_unlock(&obj_stock.lock);3180 local_unlock(&obj_stock.lock);
3156}3181}
3157 3182
3158/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */3183/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
3159static void __account_obj_stock(struct obj_cgroup *objcg,3184static void __account_obj_stock(struct obj_cgroup *objcg,
3160 struct obj_stock_pcp *stock, int nr,3185 struct obj_stock_pcp *stock, int nr,
3161 struct pglist_data *pgdat, enum node_stat_item idx)3186 struct pglist_data *pgdat, enum node_stat_item idx)
3162{3187{
3163 int *bytes;3188 int16_t *bytes;
3189 int i;
3164 3190
3165 if (!stock || READ_ONCE(stock->cached_objcg) != objcg)3191 /*
3192 * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
3193 * sure it does not exceed S16_MAX otherwise we need to fix node_id type
3194 * in struct obj_stock_pcp.
3195 */
3196 BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
3197
3198 if (!stock)
3199 goto direct;
3200
3201 for (i = 0; i < NR_OBJ_STOCK; ++i) {
3202 if (READ_ONCE(stock->cached[i]) == objcg)
3203 break;
3204 }
3205 if (i == NR_OBJ_STOCK)
3166 goto direct;3206 goto direct;
3167 3207
3168 /*3208 /*
3169 * Save vmstat data in stock and skip vmstat array update unless3209 * Save vmstat data in stock and skip vmstat array update unless
3170 * accumulating over a page of vmstat data or when pgdat changes.3210 * accumulating over a page of vmstat data or when the objcg slot or
3211 * pgdat the stats belong to changes.
3171 */3212 */
3172 if (stock->cached_pgdat != pgdat) {3213 if (stock->index < 0) {
3173 /* Flush the existing cached vmstat data */3214 stock->index = i;
3174 struct pglist_data *oldpg = stock->cached_pgdat;3215 stock->node_id = pgdat->node_id;
3216 } else if (stock->index != i || stock->node_id != pgdat->node_id) {
3217 struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
3218 struct pglist_data *oldpg = NODE_DATA(stock->node_id);
3175 3219
3176 if (stock->nr_slab_reclaimable_b) {3220 if (stock->nr_slab_reclaimable_b) {
3177 mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,3221 mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
3178 stock->nr_slab_reclaimable_b);3222 stock->nr_slab_reclaimable_b);
3179 stock->nr_slab_reclaimable_b = 0;3223 stock->nr_slab_reclaimable_b = 0;
3180 }3224 }
3181 if (stock->nr_slab_unreclaimable_b) {3225 if (stock->nr_slab_unreclaimable_b) {
3182 mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,3226 mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
3183 stock->nr_slab_unreclaimable_b);3227 stock->nr_slab_unreclaimable_b);
3184 stock->nr_slab_unreclaimable_b = 0;3228 stock->nr_slab_unreclaimable_b = 0;
3185 }3229 }
3186 stock->cached_pgdat = pgdat;3230 stock->index = i;
3231 stock->node_id = pgdat->node_id;
3187 }3232 }
3188 3233
3189 bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b3234 bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
3190 : &stock->nr_slab_unreclaimable_b;3235 : &stock->nr_slab_unreclaimable_b;
3236
3191 /*3237 /*
3192 * Even for large object >= PAGE_SIZE, the vmstat data will still be3238 * Fold @nr into the cached value and decide whether to keep it cached
3193 * cached locally at least once before pushing it out.3239 * or flush it directly. Cache the combined value when it fits in the
3240 * int16_t storage and either the cache was empty (so even a value
3241 * above PAGE_SIZE gets a chance to be canceled by a paired delta) or
3242 * the combined value is within the PAGE_SIZE flush threshold.
3194 */3243 */
3195 if (!*bytes) {3244 nr += *bytes;
3245 if (abs(nr) <= S16_MAX && (!*bytes || abs(nr) <= PAGE_SIZE)) {
3196 *bytes = nr;3246 *bytes = nr;
3197 nr = 0;3247 nr = 0;
3198 } else {3248 } else {
3199 *bytes += nr;3249 *bytes = 0;
3200 if (abs(*bytes) > PAGE_SIZE) {
3201 nr = *bytes;
3202 *bytes = 0;
3203 } else {
3204 nr = 0;
3205 }
3206 }3250 }
3207direct:3251direct:
3208 if (nr)3252 if (nr)
@@ -3213,10 +3257,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg, · 4 unmodified lines
3213 struct obj_stock_pcp *stock,3257 struct obj_stock_pcp *stock,
3214 unsigned int nr_bytes)3258 unsigned int nr_bytes)
3215{3259{
3216 if (objcg == READ_ONCE(stock->cached_objcg) &&3260 int i;
3217 stock->nr_bytes >= nr_bytes) {3261
3218 stock->nr_bytes -= nr_bytes;3262 for (i = 0; i < NR_OBJ_STOCK; ++i) {
3219 return true;3263 if (READ_ONCE(stock->cached[i]) != objcg)
3264 continue;
3265 if (stock->nr_bytes[i] >= nr_bytes) {
3266 stock->nr_bytes[i] -= nr_bytes;
3267 return true;
3268 }
3269 return false;
3220 }3270 }
3221 3271
3222 return false;3272 return false;
@@ -3237,16 +3287,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) · 14 unmodified lines
3237 return ret;3287 return ret;
3238}3288}
3239 3289
3240static void drain_obj_stock(struct obj_stock_pcp *stock)3290/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
3291static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
3241{3292{
3242 struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);3293 struct obj_cgroup *old;
3294 struct pglist_data *oldpg;
3295
3296 if (stock->index < 0)
3297 return;
3298
3299 old = READ_ONCE(stock->cached[stock->index]);
3300 oldpg = NODE_DATA(stock->node_id);
3301
3302 if (stock->nr_slab_reclaimable_b) {
3303 mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
3304 stock->nr_slab_reclaimable_b);
3305 stock->nr_slab_reclaimable_b = 0;
3306 }
3307 if (stock->nr_slab_unreclaimable_b) {
3308 mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
3309 stock->nr_slab_unreclaimable_b);
3310 stock->nr_slab_unreclaimable_b = 0;
3311 }
3312 stock->index = -1;
3313 stock->node_id = NUMA_NO_NODE;
3314}
3315
3316static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
3317{
3318 struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
3243 3319
3244 if (!old)3320 if (!old)
3245 return;3321 return;
3246 3322
3247 if (stock->nr_bytes) {3323 if (stock->nr_bytes[i]) {
3248 unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;3324 unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
3249 unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);3325 unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
3250 3326
3251 if (nr_pages) {3327 if (nr_pages) {
3252 struct mem_cgroup *memcg;3328 struct mem_cgroup *memcg;
@@ -3272,44 +3348,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock) · 19 unmodified lines
3272 * so it might be changed in the future.3348 * so it might be changed in the future.
3273 */3349 */
3274 atomic_add(nr_bytes, &old->nr_charged_bytes);3350 atomic_add(nr_bytes, &old->nr_charged_bytes);
3275 stock->nr_bytes = 0;3351 stock->nr_bytes[i] = 0;
3276 }3352 }
3277 3353
3278 /*3354 /* Flush vmstat data when its owning slot is being drained. */
3279 * Flush the vmstat data in current stock3355 if (stock->index == i)
3280 */3356 drain_obj_stock_stats(stock);
3281 if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
3282 if (stock->nr_slab_reclaimable_b) {
3283 mod_objcg_mlstate(old, stock->cached_pgdat,
3284 NR_SLAB_RECLAIMABLE_B,
3285 stock->nr_slab_reclaimable_b);
3286 stock->nr_slab_reclaimable_b = 0;
3287 }
3288 if (stock->nr_slab_unreclaimable_b) {
3289 mod_objcg_mlstate(old, stock->cached_pgdat,
3290 NR_SLAB_UNRECLAIMABLE_B,
3291 stock->nr_slab_unreclaimable_b);
3292 stock->nr_slab_unreclaimable_b = 0;
3293 }
3294 stock->cached_pgdat = NULL;
3295 }
3296 3357
3297 WRITE_ONCE(stock->cached_objcg, NULL);3358 WRITE_ONCE(stock->cached[i], NULL);
3298 obj_cgroup_put(old);3359 obj_cgroup_put(old);
3299}3360}
3300 3361
3362static void drain_obj_stock(struct obj_stock_pcp *stock)
3363{
3364 int i;
3365
3366 for (i = 0; i < NR_OBJ_STOCK; ++i)
3367 drain_obj_stock_slot(stock, i);
3368}
3369
3301static bool obj_stock_flush_required(struct obj_stock_pcp *stock,3370static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
3302 struct mem_cgroup *root_memcg)3371 struct mem_cgroup *root_memcg)
3303{3372{
3304 struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);3373 struct obj_cgroup *objcg;
3305 struct mem_cgroup *memcg;3374 struct mem_cgroup *memcg;
3306 bool flush = false;3375 bool flush = false;
3376 int i;
3307 3377
3308 rcu_read_lock();3378 rcu_read_lock();
3309 if (objcg) {3379 for (i = 0; i < NR_OBJ_STOCK; ++i) {
3380 objcg = READ_ONCE(stock->cached[i]);
3381 if (!objcg)
3382 continue;
3310 memcg = obj_cgroup_memcg(objcg);3383 memcg = obj_cgroup_memcg(objcg);
3311 if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))3384 if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
3312 flush = true;3385 flush = true;
3386 break;
3387 }
3313 }3388 }
3314 rcu_read_unlock();3389 rcu_read_unlock();
3315 3390
@@ -3322,6 +3397,8 @@ static void __refill_obj_stock(struct obj_cgroup *objcg, · 6 unmodified lines
3322 bool allow_uncharge)3397 bool allow_uncharge)
3323{3398{
3324 unsigned int nr_pages = 0;3399 unsigned int nr_pages = 0;
3400 unsigned int stock_nr_bytes;
3401 int i, slot = -1, empty_slot = -1;
3325 3402
3326 if (!stock) {3403 if (!stock) {
3327 nr_pages = nr_bytes >> PAGE_SHIFT;3404 nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3330,21 +3407,52 @@ static void __refill_obj_stock(struct obj_cgroup *objcg, · 2 unmodified lines
3330 goto out;3407 goto out;
3331 }3408 }
3332 3409
3333 if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */3410 for (i = 0; i < NR_OBJ_STOCK; ++i) {
3334 drain_obj_stock(stock);3411 struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
3412
3413 if (!cached) {
3414 if (empty_slot == -1)
3415 empty_slot = i;
3416 continue;
3417 }
3418 if (cached == objcg) {
3419 slot = i;
3420 break;
3421 }
3422 }
3423
3424 if (slot == -1) {
3425 slot = empty_slot;
3426 if (slot == -1) {
3427 slot = stock->drain_idx++;
3428 if (stock->drain_idx == NR_OBJ_STOCK)
3429 stock->drain_idx = 0;
3430 drain_obj_stock_slot(stock, slot);
3431 }
3335 obj_cgroup_get(objcg);3432 obj_cgroup_get(objcg);
3336 stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)3433 /*
3434 * Keep the xchg result in the unsigned int local; storing
3435 * it directly into stock->nr_bytes[slot] (uint16_t) would
3436 * silently truncate values >= U16_MAX and bypass the flush
3437 * guard below, leaking page-counter charges.
3438 */
3439 stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
3337 ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;3440 ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
3338 WRITE_ONCE(stock->cached_objcg, objcg);3441 WRITE_ONCE(stock->cached[slot], objcg);
3339 3442
3340 allow_uncharge = true; /* Allow uncharge when objcg changes */3443 allow_uncharge = true; /* Allow uncharge when objcg changes */
3444 } else {
3445 stock_nr_bytes = stock->nr_bytes[slot];
3341 }3446 }
3342 stock->nr_bytes += nr_bytes;
3343 3447
3344 if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {3448 stock_nr_bytes += nr_bytes;
3345 nr_pages = stock->nr_bytes >> PAGE_SHIFT;3449
3346 stock->nr_bytes &= (PAGE_SIZE - 1);3450 if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
3451 stock_nr_bytes > U16_MAX) {
3452 nr_pages = stock_nr_bytes >> PAGE_SHIFT;
3453 stock_nr_bytes &= (PAGE_SIZE - 1);
3347 }3454 }
3455 stock->nr_bytes[slot] = stock_nr_bytes;
3348 3456
3349out:3457out:
3350 if (nr_pages)3458 if (nr_pages)
@@ -4005,11 +4113,10 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 654 unmodified lines
4005 if (!memcg1_alloc_events(memcg))4113 if (!memcg1_alloc_events(memcg))
4006 goto fail;4114 goto fail;
4007 4115
4116 pstatc_pcpu = parent ? parent->vmstats_percpu : NULL;
4008 for_each_possible_cpu(cpu) {4117 for_each_possible_cpu(cpu) {
4009 if (parent)
4010 pstatc_pcpu = parent->vmstats_percpu;
4011 statc = per_cpu_ptr(memcg->vmstats_percpu, cpu);4118 statc = per_cpu_ptr(memcg->vmstats_percpu, cpu);
4012 statc->parent_pcpu = parent ? pstatc_pcpu : NULL;4119 statc->parent_pcpu = pstatc_pcpu;
4013 statc->vmstats = memcg->vmstats;4120 statc->vmstats = memcg->vmstats;
4014 }4121 }
4015 4122
@@ -4036,11 +4143,6 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 20 unmodified lines
4036 for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)4143 for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)
4037 memcg->cgwb_frn[i].done =4144 memcg->cgwb_frn[i].done =
4038 __WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq);4145 __WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq);
4039#endif
4040#ifdef CONFIG_TRANSPARENT_HUGEPAGE
4041 spin_lock_init(&memcg->deferred_split_queue.split_queue_lock);
4042 INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue);
4043 memcg->deferred_split_queue.split_queue_len = 0;
4044#endif4146#endif
4045 lru_gen_init_memcg(memcg);4147 lru_gen_init_memcg(memcg);
4046 return memcg;4148 return memcg;
@@ -4192,11 +4294,10 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) · 145 unmodified lines
4192 zswap_memcg_offline_cleanup(memcg);4294 zswap_memcg_offline_cleanup(memcg);
4193 4295
4194 memcg_offline_kmem(memcg);4296 memcg_offline_kmem(memcg);
4195 reparent_deferred_split_queue(memcg);
4196 /*4297 /*
4197 * The reparenting of objcg must be after the reparenting of the4298 * The reparenting of objcg must be after the reparenting of
4198 * list_lru and deferred_split_queue above, which ensures that they will4299 * the list_lru in memcg_offline_kmem(), which ensures that
4199 * not mistakenly get the parent list_lru and deferred_split_queue.4300 * they will not mistakenly get the parent list_lru.
4200 */4301 */
4201 memcg_reparent_objcgs(memcg);4302 memcg_reparent_objcgs(memcg);
4202 reparent_shrinker_deferred(memcg);4303 reparent_shrinker_deferred(memcg);
@@ -5080,27 +5181,25 @@ out: · 877 unmodified lines
5080 5181
5081/**5182/**
5082 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.5183 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
5083 * @folio: folio to charge.5184 * @folio: the folio to charge
5185 * @id: memory cgroup id
5084 * @mm: mm context of the victim5186 * @mm: mm context of the victim
5085 * @gfp: reclaim mode5187 * @gfp: reclaim mode
5086 * @entry: swap entry for which the folio is allocated
5087 *5188 *
5088 * This function charges a folio allocated for swapin. Please call this before5189 * This function charges a folio allocated for swapin. Please call this before
5089 * adding the folio to the swapcache.5190 * adding the folio to the swapcache.
5090 *5191 *
5091 * Returns 0 on success. Otherwise, an error code is returned.5192 * Returns 0 on success. Otherwise, an error code is returned.
5092 */5193 */
5093int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,5194int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
5094 gfp_t gfp, swp_entry_t entry)5195 struct mm_struct *mm, gfp_t gfp)
5095{5196{
5096 struct mem_cgroup *memcg;5197 struct mem_cgroup *memcg;
5097 unsigned short id;
5098 int ret;5198 int ret;
5099 5199
5100 if (mem_cgroup_disabled())5200 if (mem_cgroup_disabled())
5101 return 0;5201 return 0;
5102 5202
5103 id = lookup_swap_cgroup_id(entry);
5104 rcu_read_lock();5203 rcu_read_lock();
5105 memcg = mem_cgroup_from_private_id(id);5204 memcg = mem_cgroup_from_private_id(id);
5106 if (!memcg || !css_tryget_online(&memcg->css))5205 if (!memcg || !css_tryget_online(&memcg->css))
@@ -5474,15 +5573,15 @@ int __init mem_cgroup_init(void) · 367 unmodified lines
5474/**5573/**
5475 * __mem_cgroup_try_charge_swap - try charging swap space for a folio5574 * __mem_cgroup_try_charge_swap - try charging swap space for a folio
5476 * @folio: folio being added to swap5575 * @folio: folio being added to swap
5477 * @entry: swap entry to charge
5478 *5576 *
5479 * Try to charge @folio's memcg for the swap space at @entry.5577 * Try to charge @folio's memcg for the swap space at folio->swap.
5480 *5578 *
5481 * Returns 0 on success, -ENOMEM on failure.5579 * Returns 0 on success, -ENOMEM on failure.
5482 */5580 */
5483int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)5581int __mem_cgroup_try_charge_swap(struct folio *folio)
5484{5582{
5485 unsigned int nr_pages = folio_nr_pages(folio);5583 unsigned int nr_pages = folio_nr_pages(folio);
5584 struct swap_cluster_info *ci;
5486 struct page_counter *counter;5585 struct page_counter *counter;
5487 struct mem_cgroup *memcg;5586 struct mem_cgroup *memcg;
5488 struct obj_cgroup *objcg;5587 struct obj_cgroup *objcg;
@@ -5497,7 +5596,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) · 8 unmodified lines
5497 5596
5498 rcu_read_lock();5597 rcu_read_lock();
5499 memcg = obj_cgroup_memcg(objcg);5598 memcg = obj_cgroup_memcg(objcg);
5500 if (!entry.val) {5599 if (!folio_test_swapcache(folio)) {
5501 memcg_memory_event(memcg, MEMCG_SWAP_FAIL);5600 memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
5502 rcu_read_unlock();5601 rcu_read_unlock();
5503 return 0;5602 return 0;
@@ -5516,22 +5615,23 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) · 12 unmodified lines
5516 }5615 }
5517 mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);5616 mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
5518 5617
5519 swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry);5618 ci = swap_cluster_get_and_lock(folio);
5619 __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
5620 mem_cgroup_private_id(memcg));
5621 swap_cluster_unlock(ci);
5520 5622
5521 return 0;5623 return 0;
5522}5624}
5523 5625
5524/**5626/**
5525 * __mem_cgroup_uncharge_swap - uncharge swap space5627 * __mem_cgroup_uncharge_swap - uncharge swap space
5526 * @entry: swap entry to uncharge5628 * @id: cgroup id to uncharge
5527 * @nr_pages: the amount of swap space to uncharge5629 * @nr_pages: the amount of swap space to uncharge
5528 */5630 */
5529void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)5631void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
5530{5632{
5531 struct mem_cgroup *memcg;5633 struct mem_cgroup *memcg;
5532 unsigned short id;
5533 5634
5534 id = swap_cgroup_clear(entry, nr_pages);
5535 rcu_read_lock();5635 rcu_read_lock();
5536 memcg = mem_cgroup_from_private_id(id);5636 memcg = mem_cgroup_from_private_id(id);
5537 if (memcg) {5637 if (memcg) {
mm/memory-failure.c+18−29
@@ -172,23 +172,11 @@ static int __page_handle_poison(struct page *page) · 171 unmodified lines
172{172{
173 int ret;173 int ret;
174 174
175 /*175 zone_pcp_disable(page_zone(page));
176 * zone_pcp_disable() can't be used here. It will
177 * hold pcp_batch_high_lock and dissolve_free_hugetlb_folio() might hold
178 * cpu_hotplug_lock via static_key_slow_dec() when hugetlb vmemmap
179 * optimization is enabled. This will break current lock dependency
180 * chain and leads to deadlock.
181 * Disabling pcp before dissolving the page was a deterministic
182 * approach because we made sure that those pages cannot end up in any
183 * PCP list. Draining PCP lists expels those pages to the buddy system,
184 * but nothing guarantees that those pages do not get back to a PCP
185 * queue if we need to refill those.
186 */
187 ret = dissolve_free_hugetlb_folio(page_folio(page));176 ret = dissolve_free_hugetlb_folio(page_folio(page));
188 if (!ret) {177 if (!ret)
189 drain_all_pages(page_zone(page));
190 ret = take_page_off_buddy(page);178 ret = take_page_off_buddy(page);
191 }179 zone_pcp_enable(page_zone(page));
192 180
193 return ret;181 return ret;
194}182}
@@ -459,7 +447,7 @@ void add_to_kill_ksm(struct task_struct *tsk, const struct page *p, · 264 unmodified lines
459 * Only do anything when FORCEKILL is set, otherwise just free the447 * Only do anything when FORCEKILL is set, otherwise just free the
460 * list (this is used for clean pages which do not need killing)448 * list (this is used for clean pages which do not need killing)
461 */449 */
462static void kill_procs(struct list_head *to_kill, int forcekill,450static void kill_procs(struct list_head *to_kill, bool forcekill,
463 unsigned long pfn, int flags)451 unsigned long pfn, int flags)
464{452{
465 struct to_kill *tk, *next;453 struct to_kill *tk, *next;
@@ -1418,7 +1406,7 @@ try_again: · 952 unmodified lines
1418 * We raced with (possibly temporary) unhandlable1406 * We raced with (possibly temporary) unhandlable
1419 * page, retry.1407 * page, retry.
1420 */1408 */
1421 if (pass++ < 3) {1409 if (pass++ < GET_PAGE_MAX_RETRY_NUM) {
1422 shake_page(p);1410 shake_page(p);
1423 goto try_again;1411 goto try_again;
1424 }1412 }
@@ -1582,7 +1570,7 @@ static bool hwpoison_user_mappings(struct folio *folio, struct page *p, · 157 unmodified lines
1582{1570{
1583 LIST_HEAD(tokill);1571 LIST_HEAD(tokill);
1584 bool unmap_success;1572 bool unmap_success;
1585 int forcekill;1573 bool forcekill;
1586 bool mlocked = folio_test_mlocked(folio);1574 bool mlocked = folio_test_mlocked(folio);
1587 1575
1588 /*1576 /*
@@ -1703,7 +1691,7 @@ static void unmap_and_kill(struct list_head *to_kill, unsigned long pfn, · 114 unmodified lines
1703 unmap_mapping_range(mapping, start, size, 0);1691 unmap_mapping_range(mapping, start, size, 0);
1704 }1692 }
1705 1693
1706 kill_procs(to_kill, flags & MF_MUST_KILL, pfn, flags);1694 kill_procs(to_kill, !!(flags & MF_MUST_KILL), pfn, flags);
1707}1695}
1708 1696
1709/*1697/*
@@ -2027,13 +2015,14 @@ out_unlock: · 317 unmodified lines
2027 * So some of prechecks for hwpoison (pinning, and testing/setting2015 * So some of prechecks for hwpoison (pinning, and testing/setting
2028 * PageHWPoison) should be done in single hugetlb_lock range.2016 * PageHWPoison) should be done in single hugetlb_lock range.
2029 * Returns:2017 * Returns:
2030 * 0 - not hugetlb, or recovered2018 * 0 - recovered
2019 * -ENOENT - no hugetlb page
2031 * -EBUSY - not recovered2020 * -EBUSY - not recovered
2032 * -EOPNOTSUPP - hwpoison_filter'ed2021 * -EOPNOTSUPP - hwpoison_filter'ed
2033 * -EHWPOISON - folio or exact page already poisoned2022 * -EHWPOISON - folio or exact page already poisoned
2034 * -EFAULT - kill_accessing_process finds current->mm null2023 * -EFAULT - kill_accessing_process finds current->mm null
2035 */2024 */
2036static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb)2025static int try_memory_failure_hugetlb(unsigned long pfn, int flags)
2037{2026{
2038 int res, rv;2027 int res, rv;
2039 struct page *p = pfn_to_page(pfn);2028 struct page *p = pfn_to_page(pfn);
@@ -2041,13 +2030,11 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb · 1 unmodified line
2041 unsigned long page_flags;2030 unsigned long page_flags;
2042 bool migratable_cleared = false;2031 bool migratable_cleared = false;
2043 2032
2044 *hugetlb = 1;
2045retry:2033retry:
2046 res = get_huge_page_for_hwpoison(pfn, flags, &migratable_cleared);2034 res = get_huge_page_for_hwpoison(pfn, flags, &migratable_cleared);
2047 switch (res) {2035 switch (res) {
2048 case MF_HUGETLB_NON_HUGEPAGE: /* fallback to normal page handling */2036 case MF_HUGETLB_NON_HUGEPAGE: /* fallback to normal page handling */
2049 *hugetlb = 0;2037 return -ENOENT;
2050 return 0;
2051 case MF_HUGETLB_RETRY:2038 case MF_HUGETLB_RETRY:
2052 if (!(flags & MF_NO_RETRY)) {2039 if (!(flags & MF_NO_RETRY)) {
2053 flags |= MF_NO_RETRY;2040 flags |= MF_NO_RETRY;
@@ -2108,9 +2095,9 @@ retry: · 54 unmodified lines
2108}2095}
2109 2096
2110#else2097#else
2111static inline int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb)2098static inline int try_memory_failure_hugetlb(unsigned long pfn, int flags)
2112{2099{
2113 return 0;2100 return -ENOENT;
2114}2101}
2115 2102
2116static inline unsigned long folio_free_raw_hwp(struct folio *folio, bool flag)2103static inline unsigned long folio_free_raw_hwp(struct folio *folio, bool flag)
@@ -2348,7 +2335,6 @@ int memory_failure(unsigned long pfn, int flags) · 231 unmodified lines
2348 int res = 0;2335 int res = 0;
2349 unsigned long page_flags;2336 unsigned long page_flags;
2350 bool retry = true;2337 bool retry = true;
2351 int hugetlb = 0;
2352 2338
2353 if (!sysctl_memory_failure_recovery)2339 if (!sysctl_memory_failure_recovery)
2354 panic("Memory failure on page %lx", pfn);2340 panic("Memory failure on page %lx", pfn);
@@ -2387,8 +2373,11 @@ int memory_failure(unsigned long pfn, int flags) · 32 unmodified lines
2387 }2373 }
2388 2374
2389try_again:2375try_again:
2390 res = try_memory_failure_hugetlb(pfn, flags, &hugetlb);2376 res = try_memory_failure_hugetlb(pfn, flags);
2391 if (hugetlb)2377 /*
2378 * -ENOENT means the page we found is not hugetlb, so proceed with normal page handling
2379 */
2380 if (res != -ENOENT)
2392 goto unlock_mutex;2381 goto unlock_mutex;
2393 2382
2394 if (TestSetPageHWPoison(p)) {2383 if (TestSetPageHWPoison(p)) {
mm/memory.c+79−127
@@ -3837,8 +3837,8 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf) · 3836 unmodified lines
3837 * Handle the case of a page which we actually need to copy to a new page,3837 * Handle the case of a page which we actually need to copy to a new page,
3838 * either due to COW or unsharing.3838 * either due to COW or unsharing.
3839 *3839 *
3840 * Called with mmap_lock locked and the old page referenced, but3840 * Called with either the VMA lock or the mmap_lock held (see FAULT_FLAG_VMA_LOCK)
3841 * without the ptl held.3841 * and the old page referenced, but without the ptl held.
3842 *3842 *
3843 * High level logic flow:3843 * High level logic flow:
3844 *3844 *
@@ -4237,9 +4237,9 @@ static bool wp_can_reuse_anon_folio(struct folio *folio, · 392 unmodified lines
4237 * though the page will change only once the write actually happens. This4237 * though the page will change only once the write actually happens. This
4238 * avoids a few races, and potentially makes it more efficient.4238 * avoids a few races, and potentially makes it more efficient.
4239 *4239 *
4240 * We enter with non-exclusive mmap_lock (to exclude vma changes,4240 * We enter with either the VMA lock or the mmap_lock held (see
4241 * but allow concurrent faults), with pte both mapped and locked.4241 * FAULT_FLAG_VMA_LOCK) and pte both mapped and locked. We return with
4242 * We return with mmap_lock still held, but pte unmapped and unlocked.4242 * the same lock still held, but pte unmapped and unlocked.
4243 */4243 */
4244static vm_fault_t do_wp_page(struct vm_fault *vmf)4244static vm_fault_t do_wp_page(struct vm_fault *vmf)
4245 __releases(vmf->ptl)4245 __releases(vmf->ptl)
@@ -4609,35 +4609,13 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · 363 unmodified lines
4609 return VM_FAULT_SIGBUS;4609 return VM_FAULT_SIGBUS;
4610}4610}
4611 4611
4612static struct folio *__alloc_swap_folio(struct vm_fault *vmf)
4613{
4614 struct vm_area_struct *vma = vmf->vma;
4615 struct folio *folio;
4616 softleaf_t entry;
4617
4618 folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address);
4619 if (!folio)
4620 return NULL;
4621
4622 entry = softleaf_from_pte(vmf->orig_pte);
4623 if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
4624 GFP_KERNEL, entry)) {
4625 folio_put(folio);
4626 return NULL;
4627 }
4628
4629 return folio;
4630}
4631
4632#ifdef CONFIG_TRANSPARENT_HUGEPAGE4612#ifdef CONFIG_TRANSPARENT_HUGEPAGE
4633/*4613/*
4634 * Check if the PTEs within a range are contiguous swap entries4614 * Check if the PTEs within a range are contiguous swap entries.
4635 * and have consistent swapcache, zeromap.
4636 */4615 */
4637static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)4616static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
4638{4617{
4639 unsigned long addr;4618 unsigned long addr;
4640 softleaf_t entry;
4641 int idx;4619 int idx;
4642 pte_t pte;4620 pte_t pte;
4643 4621
@@ -4647,20 +4625,13 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages) · 3 unmodified lines
4647 4625
4648 if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))4626 if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
4649 return false;4627 return false;
4650 entry = softleaf_from_pte(pte);
4651 if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
4652 return false;
4653
4654 /*4628 /*
4655 * swap_read_folio() can't handle the case a large folio is hybridly4629 * swap_read_folio() can't handle the case a large folio is hybridly
4656 * from different backends. And they are likely corner cases. Similar4630 * from different backends. And they are likely corner cases. Similar
4657 * things might be added once zswap support large folios.4631 * things might be added once zswap support large folios.
4658 */4632 */
4659 if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))4633 if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
4660 return false;
4661 if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
4662 return false;4634 return false;
4663
4664 return true;4635 return true;
4665}4636}
4666 4637
@@ -4687,16 +4658,14 @@ static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset, · 20 unmodified lines
4687 return orders;4658 return orders;
4688}4659}
4689 4660
4690static struct folio *alloc_swap_folio(struct vm_fault *vmf)4661static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
4691{4662{
4692 struct vm_area_struct *vma = vmf->vma;4663 struct vm_area_struct *vma = vmf->vma;
4693 unsigned long orders;4664 unsigned long orders;
4694 struct folio *folio;
4695 unsigned long addr;4665 unsigned long addr;
4696 softleaf_t entry;4666 softleaf_t entry;
4697 spinlock_t *ptl;4667 spinlock_t *ptl;
4698 pte_t *pte;4668 pte_t *pte;
4699 gfp_t gfp;
4700 int order;4669 int order;
4701 4670
4702 /*4671 /*
@@ -4704,7 +4673,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 1 unmodified line
4704 * maintain the uffd semantics.4673 * maintain the uffd semantics.
4705 */4674 */
4706 if (unlikely(userfaultfd_armed(vma)))4675 if (unlikely(userfaultfd_armed(vma)))
4707 goto fallback;4676 return 0;
4708 4677
4709 /*4678 /*
4710 * A large swapped out folio could be partially or fully in zswap. We4679 * A large swapped out folio could be partially or fully in zswap. We
@@ -4712,7 +4681,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 1 unmodified line
4712 * folio.4681 * folio.
4713 */4682 */
4714 if (!zswap_never_enabled())4683 if (!zswap_never_enabled())
4715 goto fallback;4684 return 0;
4716 4685
4717 entry = softleaf_from_pte(vmf->orig_pte);4686 entry = softleaf_from_pte(vmf->orig_pte);
4718 /*4687 /*
@@ -4726,12 +4695,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 7 unmodified lines
4726 vmf->address, orders);4695 vmf->address, orders);
4727 4696
4728 if (!orders)4697 if (!orders)
4729 goto fallback;4698 return 0;
4730 4699
4731 pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,4700 pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
4732 vmf->address & PMD_MASK, &ptl);4701 vmf->address & PMD_MASK, &ptl);
4733 if (unlikely(!pte))4702 if (unlikely(!pte))
4734 goto fallback;4703 return 0;
4735 4704
4736 /*4705 /*
4737 * For do_swap_page, find the highest order where the aligned range is4706 * For do_swap_page, find the highest order where the aligned range is
@@ -4747,29 +4716,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 9 unmodified lines
4747 4716
4748 pte_unmap_unlock(pte, ptl);4717 pte_unmap_unlock(pte, ptl);
4749 4718
4750 /* Try allocating the highest of the remaining orders. */4719 return orders;
4751 gfp = vma_thp_gfp_mask(vma);
4752 while (orders) {
4753 addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
4754 folio = vma_alloc_folio(gfp, order, vma, addr);
4755 if (folio) {
4756 if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
4757 gfp, entry))
4758 return folio;
4759 count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
4760 folio_put(folio);
4761 }
4762 count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
4763 order = next_order(&orders, order);
4764 }
4765
4766fallback:
4767 return __alloc_swap_folio(vmf);
4768}4720}
4769#else /* !CONFIG_TRANSPARENT_HUGEPAGE */4721#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
4770static struct folio *alloc_swap_folio(struct vm_fault *vmf)4722static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
4771{4723{
4772 return __alloc_swap_folio(vmf);4724 return 0;
4773}4725}
4774#endif /* CONFIG_TRANSPARENT_HUGEPAGE */4726#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
4775 4727
@@ -4785,12 +4737,12 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · 9 unmodified lines
4785}4737}
4786 4738
4787/*4739/*
4788 * We enter with non-exclusive mmap_lock (to exclude vma changes,4740 * We enter with either the VMA lock or the mmap_lock held (see
4789 * but allow concurrent faults), and pte mapped but not yet locked.4741 * FAULT_FLAG_VMA_LOCK), and pte mapped but not yet locked.
4790 * We return with pte unmapped and unlocked.4742 * We return with pte unmapped and unlocked.
4791 *4743 *
4792 * We return with the mmap_lock locked or unlocked in the same cases4744 * When returning, the lock may have been released in the same cases
4793 * as does filemap_fault().4745 * as done by filemap_fault().
4794 */4746 */
4795vm_fault_t do_swap_page(struct vm_fault *vmf)4747vm_fault_t do_swap_page(struct vm_fault *vmf)
4796{4748{
@@ -4875,23 +4827,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 78 unmodified lines
4875 if (folio)4827 if (folio)
4876 swap_update_readahead(folio, vma, vmf->address);4828 swap_update_readahead(folio, vma, vmf->address);
4877 if (!folio) {4829 if (!folio) {
4878 if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {4830 /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
4879 folio = alloc_swap_folio(vmf);4831 if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
4880 if (folio) {4832 folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
4881 /*4833 thp_swapin_suitable_orders(vmf) | BIT(0),
4882 * folio is charged, so swapin can only fail due4834 vmf, NULL, 0);
4883 * to raced swapin and return NULL.4835 else
4884 */
4885 swapcache = swapin_folio(entry, folio);
4886 if (swapcache != folio)
4887 folio_put(folio);
4888 folio = swapcache;
4889 }
4890 } else {
4891 folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);4836 folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
4892 }
4893 4837
4894 if (!folio) {4838 if (IS_ERR_OR_NULL(folio)) {
4895 /*4839 /*
4896 * Back out if somebody else faulted in this pte4840 * Back out if somebody else faulted in this pte
4897 * while we released the pte lock.4841 * while we released the pte lock.
@@ -4901,6 +4845,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 3 unmodified lines
4901 if (likely(vmf->pte &&4845 if (likely(vmf->pte &&
4902 pte_same(ptep_get(vmf->pte), vmf->orig_pte)))4846 pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
4903 ret = VM_FAULT_OOM;4847 ret = VM_FAULT_OOM;
4848 folio = NULL;
4904 goto unlock;4849 goto unlock;
4905 }4850 }
4906 4851
@@ -5270,24 +5215,28 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) · 363 unmodified lines
5270 while (orders) {5215 while (orders) {
5271 addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);5216 addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
5272 folio = vma_alloc_folio(gfp, order, vma, addr);5217 folio = vma_alloc_folio(gfp, order, vma, addr);
5273 if (folio) {5218 if (!folio)
5274 if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {5219 goto next;
5275 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);5220 if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
5276 folio_put(folio);5221 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
5277 goto next;5222 folio_put(folio);
5278 }5223 goto next;
5279 folio_throttle_swaprate(folio, gfp);
5280 /*
5281 * When a folio is not zeroed during allocation
5282 * (__GFP_ZERO not used) or user folios require special
5283 * handling, folio_zero_user() is used to make sure
5284 * that the page corresponding to the faulting address
5285 * will be hot in the cache after zeroing.
5286 */
5287 if (user_alloc_needs_zeroing())
5288 folio_zero_user(folio, vmf->address);
5289 return folio;
5290 }5224 }
5225 if (order > 1 && folio_memcg_alloc_deferred(folio)) {
5226 folio_put(folio);
5227 goto fallback;
5228 }
5229 folio_throttle_swaprate(folio, gfp);
5230 /*
5231 * When a folio is not zeroed during allocation
5232 * (__GFP_ZERO not used) or user folios require special
5233 * handling, folio_zero_user() is used to make sure
5234 * that the page corresponding to the faulting address
5235 * will be hot in the cache after zeroing.
5236 */
5237 if (user_alloc_needs_zeroing())
5238 folio_zero_user(folio, vmf->address);
5239 return folio;
5291next:5240next:
5292 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);5241 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
5293 order = next_order(&orders, order);5242 order = next_order(&orders, order);
@@ -5330,9 +5279,10 @@ static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte, · 36 unmodified lines
5330}5279}
5331 5280
5332/*5281/*
5333 * We enter with non-exclusive mmap_lock (to exclude vma changes,5282 * We enter with either the VMA lock or the mmap_lock held (see
5334 * but allow concurrent faults), and pte mapped but not yet locked.5283 * FAULT_FLAG_VMA_LOCK), and pte unmapped and unlocked.
5335 * We return with mmap_lock still held, but pte unmapped and unlocked.5284 * We return with the lock still held, but pte unmapped and unlocked.
5285 * If VM_FAULT_RETRY is returned, the lock may have been released.
5336 */5286 */
5337static vm_fault_t do_anonymous_page(struct vm_fault *vmf)5287static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
5338{5288{
@@ -5440,9 +5390,10 @@ oom: · 101 unmodified lines
5440}5390}
5441 5391
5442/*5392/*
5443 * The mmap_lock must have been held on entry, and may have been5393 * Either the VMA lock or the mmap_lock must have been held on entry
5444 * released depending on flags and vma->vm_ops->fault() return value.5394 * (see FAULT_FLAG_VMA_LOCK) and may have been released depending on
5445 * See filemap_fault() and __lock_page_retry().5395 * flags and vma->vm_ops->fault() return value.
5396 * See filemap_fault() and __folio_lock_or_retry().
5446 */5397 */
5447static vm_fault_t __do_fault(struct vm_fault *vmf)5398static vm_fault_t __do_fault(struct vm_fault *vmf)
5448{5399{
@@ -5451,18 +5402,18 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) · 2 unmodified lines
5451 vm_fault_t ret;5402 vm_fault_t ret;
5452 5403
5453 /*5404 /*
5454 * Preallocate pte before we take page_lock because this might lead to5405 * Preallocate pte before we take folio lock because this might lead to
5455 * deadlocks for memcg reclaim which waits for pages under writeback:5406 * deadlocks for memcg reclaim which waits for folios under writeback:
5456 * lock_page(A)5407 * folio_lock(A)
5457 * SetPageWriteback(A)5408 * folio_set_writeback(A)
5458 * unlock_page(A)5409 * folio_unlock(A)
5459 * lock_page(B)5410 * folio_lock(B)
5460 * lock_page(B)5411 * folio_lock(B)
5461 * pte_alloc_one5412 * pte_alloc_one
5462 * shrink_folio_list5413 * shrink_folio_list
5463 * wait_on_page_writeback(A)5414 * folio_wait_writeback(A)
5464 * SetPageWriteback(B)5415 * folio_set_writeback(B)
5465 * unlock_page(B)5416 * folio_unlock(B)
5466 * # flush A, B to clear the writeback5417 * # flush A, B to clear the writeback
5467 */5418 */
5468 if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) {5419 if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) {
@@ -5480,7 +5431,7 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) · 11 unmodified lines
5480 if (unlikely(PageHWPoison(vmf->page))) {5431 if (unlikely(PageHWPoison(vmf->page))) {
5481 vm_fault_t poisonret = VM_FAULT_HWPOISON;5432 vm_fault_t poisonret = VM_FAULT_HWPOISON;
5482 if (ret & VM_FAULT_LOCKED) {5433 if (ret & VM_FAULT_LOCKED) {
5483 if (page_mapped(vmf->page))5434 if (folio_mapped(folio))
5484 unmap_mapping_folio(folio);5435 unmap_mapping_folio(folio);
5485 /* Retry if a clean folio was removed from the cache. */5436 /* Retry if a clean folio was removed from the cache. */
5486 if (mapping_evict_folio(folio->mapping, folio))5437 if (mapping_evict_folio(folio->mapping, folio))
@@ -6003,11 +5954,11 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) · 516 unmodified lines
6003}5954}
6004 5955
6005/*5956/*
6006 * We enter with non-exclusive mmap_lock (to exclude vma changes,5957 * We enter with either the VMA lock or the mmap_lock held (see
6007 * but allow concurrent faults).5958 * FAULT_FLAG_VMA_LOCK).
6008 * The mmap_lock may have been released depending on flags and our5959 * The lock may have been released depending on flags and our
6009 * return value. See filemap_fault() and __folio_lock_or_retry().5960 * return value. See filemap_fault() and __folio_lock_or_retry().
6010 * If mmap_lock is released, vma may become invalid (for example5961 * If the lock is released, vma may become invalid (for example
6011 * by other thread calling munmap()).5962 * by other thread calling munmap()).
6012 */5963 */
6013static vm_fault_t do_fault(struct vm_fault *vmf)5964static vm_fault_t do_fault(struct vm_fault *vmf)
@@ -6374,10 +6325,11 @@ static void fix_spurious_fault(struct vm_fault *vmf, · 360 unmodified lines
6374 * with external mmu caches can use to update those (ie the Sparc or6325 * with external mmu caches can use to update those (ie the Sparc or
6375 * PowerPC hashed page tables that act as extended TLBs).6326 * PowerPC hashed page tables that act as extended TLBs).
6376 *6327 *
6377 * We enter with non-exclusive mmap_lock (to exclude vma changes, but allow6328 * On entry, we hold either the VMA lock or the mmap_lock
6378 * concurrent faults).6329 * (see FAULT_FLAG_VMA_LOCK).
6379 *6330 *
6380 * The mmap_lock may have been released depending on flags and our return value.6331 * The mmap_lock or VMA lock may have been released depending on flags
6332 * and our return value.
6381 * See filemap_fault() and __folio_lock_or_retry().6333 * See filemap_fault() and __folio_lock_or_retry().
6382 */6334 */
6383static vm_fault_t handle_pte_fault(struct vm_fault *vmf)6335static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
@@ -6458,8 +6410,8 @@ unlock: · 74 unmodified lines
6458 6410
6459/*6411/*
6460 * On entry, we hold either the VMA lock or the mmap_lock6412 * On entry, we hold either the VMA lock or the mmap_lock
6461 * (FAULT_FLAG_VMA_LOCK tells you which). If VM_FAULT_RETRY is set in6413 * (see FAULT_FLAG_VMA_LOCK). If VM_FAULT_RETRY is set in
6462 * the result, the mmap_lock is not held on exit. See filemap_fault()6414 * the result, the lock is not held on exit. See filemap_fault()
6463 * and __folio_lock_or_retry().6415 * and __folio_lock_or_retry().
6464 */6416 */
6465static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,6417static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
@@ -6691,9 +6643,9 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, · 225 unmodified lines
6691 6643
6692/*6644/*
6693 * By the time we get here, we already hold either the VMA lock or the6645 * By the time we get here, we already hold either the VMA lock or the
6694 * mmap_lock (FAULT_FLAG_VMA_LOCK tells you which).6646 * mmap_lock (see FAULT_FLAG_VMA_LOCK).
6695 *6647 *
6696 * The mmap_lock may have been released depending on flags and our6648 * The lock may have been released depending on flags and our
6697 * return value. See filemap_fault() and __folio_lock_or_retry().6649 * return value. See filemap_fault() and __folio_lock_or_retry().
6698 */6650 */
6699vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,6651vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
mm/memory_hotplug.c+18−16
@@ -576,6 +576,7 @@ void remove_pfn_range_from_zone(struct zone *zone, · 575 unmodified lines
576 * @pfn: starting pageframe (must be aligned to start of a section)576 * @pfn: starting pageframe (must be aligned to start of a section)
577 * @nr_pages: number of pages to remove (must be multiple of section size)577 * @nr_pages: number of pages to remove (must be multiple of section size)
578 * @altmap: alternative device page map or %NULL if default memmap is used578 * @altmap: alternative device page map or %NULL if default memmap is used
579 * @pgmap: device page map or %NULL if not ZONE_DEVICE
579 *580 *
580 * Generic helper function to remove section mappings and sysfs entries581 * Generic helper function to remove section mappings and sysfs entries
581 * for the section of the memory we are removing. Caller needs to make582 * for the section of the memory we are removing. Caller needs to make
@@ -583,7 +584,7 @@ void remove_pfn_range_from_zone(struct zone *zone, · 1 unmodified line
583 * calling offline_pages().584 * calling offline_pages().
584 */585 */
585void __remove_pages(unsigned long pfn, unsigned long nr_pages,586void __remove_pages(unsigned long pfn, unsigned long nr_pages,
586 struct vmem_altmap *altmap)587 struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
587{588{
588 const unsigned long end_pfn = pfn + nr_pages;589 const unsigned long end_pfn = pfn + nr_pages;
589 unsigned long cur_nr_pages;590 unsigned long cur_nr_pages;
@@ -598,7 +599,7 @@ void __remove_pages(unsigned long pfn, unsigned long nr_pages, · 8 unmodified lines
598 /* Select all remaining pages up to the next section boundary */599 /* Select all remaining pages up to the next section boundary */
599 cur_nr_pages = min(end_pfn - pfn,600 cur_nr_pages = min(end_pfn - pfn,
600 SECTION_ALIGN_UP(pfn + 1) - pfn);601 SECTION_ALIGN_UP(pfn + 1) - pfn);
601 sparse_remove_section(pfn, cur_nr_pages, altmap);602 sparse_remove_section(pfn, cur_nr_pages, altmap, pgmap);
602 }603 }
603}604}
604 605
@@ -1402,6 +1403,12 @@ bool mhp_supports_memmap_on_memory(void) · 797 unmodified lines
1402}1403}
1403EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);1404EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
1404 1405
1406static void altmap_free(struct vmem_altmap *altmap)
1407{
1408 WARN_ONCE(altmap->alloc, "Altmap not fully unmapped");
1409 kfree(altmap);
1410}
1411
1405static void remove_memory_blocks_and_altmaps(u64 start, u64 size)1412static void remove_memory_blocks_and_altmaps(u64 start, u64 size)
1406{1413{
1407 unsigned long memblock_size = memory_block_size_bytes();1414 unsigned long memblock_size = memory_block_size_bytes();
@@ -1416,22 +1423,17 @@ static void remove_memory_blocks_and_altmaps(u64 start, u64 size) · 8 unmodified lines
1416 struct vmem_altmap *altmap = NULL;1423 struct vmem_altmap *altmap = NULL;
1417 struct memory_block *mem;1424 struct memory_block *mem;
1418 1425
1419 mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(cur_start)));1426 mem = memory_block_get(phys_to_block_id(cur_start));
1420 if (WARN_ON_ONCE(!mem))1427 if (WARN_ON_ONCE(!mem))
1421 continue;1428 continue;
1422 1429
1423 altmap = mem->altmap;1430 altmap = mem->altmap;
1424 mem->altmap = NULL;1431 mem->altmap = NULL;
1425 /* drop the ref. we got via find_memory_block() */1432 memory_block_put(mem);
1426 put_device(&mem->dev);
1427 1433
1428 remove_memory_block_devices(cur_start, memblock_size);1434 remove_memory_block_devices(cur_start, memblock_size);
1429 1435 arch_remove_memory(cur_start, memblock_size, altmap, NULL);
1430 arch_remove_memory(cur_start, memblock_size, altmap);1436 altmap_free(altmap);
1431
1432 /* Verify that all vmemmap pages have actually been freed. */
1433 WARN(altmap->alloc, "Altmap not fully unmapped");
1434 kfree(altmap);
1435 }1437 }
1436}1438}
1437 1439
@@ -1462,7 +1464,7 @@ static int create_altmaps_and_memory_blocks(int nid, struct memory_group *group, · 24 unmodified lines
1462 /* call arch's memory hotadd */1464 /* call arch's memory hotadd */
1463 ret = arch_add_memory(nid, cur_start, memblock_size, &params);1465 ret = arch_add_memory(nid, cur_start, memblock_size, &params);
1464 if (ret < 0) {1466 if (ret < 0) {
1465 kfree(params.altmap);1467 altmap_free(params.altmap);
1466 goto out;1468 goto out;
1467 }1469 }
1468 1470
@@ -1470,8 +1472,8 @@ static int create_altmaps_and_memory_blocks(int nid, struct memory_group *group, · 1 unmodified line
1470 ret = create_memory_block_devices(cur_start, memblock_size, nid,1472 ret = create_memory_block_devices(cur_start, memblock_size, nid,
1471 params.altmap, group);1473 params.altmap, group);
1472 if (ret) {1474 if (ret) {
1473 arch_remove_memory(cur_start, memblock_size, NULL);1475 arch_remove_memory(cur_start, memblock_size, params.altmap, NULL);
1474 kfree(params.altmap);1476 altmap_free(params.altmap);
1475 goto out;1477 goto out;
1476 }1478 }
1477 }1479 }
@@ -1556,7 +1558,7 @@ int add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags) · 78 unmodified lines
1556 /* create memory block devices after memory was added */1558 /* create memory block devices after memory was added */
1557 ret = create_memory_block_devices(start, size, nid, NULL, group);1559 ret = create_memory_block_devices(start, size, nid, NULL, group);
1558 if (ret) {1560 if (ret) {
1559 arch_remove_memory(start, size, params.altmap);1561 arch_remove_memory(start, size, params.altmap, NULL);
1560 goto error;1562 goto error;
1561 }1563 }
1562 }1564 }
@@ -2268,7 +2270,7 @@ static int try_remove_memory(u64 start, u64 size) · 705 unmodified lines
2268 * No altmaps present, do the removal directly2270 * No altmaps present, do the removal directly
2269 */2271 */
2270 remove_memory_block_devices(start, size);2272 remove_memory_block_devices(start, size);
2271 arch_remove_memory(start, size, NULL);2273 arch_remove_memory(start, size, NULL, NULL);
2272 } else {2274 } else {
2273 /* all memblocks in the range have altmaps */2275 /* all memblocks in the range have altmaps */
2274 remove_memory_blocks_and_altmaps(start, size);2276 remove_memory_blocks_and_altmaps(start, size);
mm/mempolicy.c+1−1
@@ -2865,7 +2865,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 2864 unmodified lines
2865 case MPOL_PREFERRED:2865 case MPOL_PREFERRED:
2866 case MPOL_PREFERRED_MANY:2866 case MPOL_PREFERRED_MANY:
2867 case MPOL_WEIGHTED_INTERLEAVE:2867 case MPOL_WEIGHTED_INTERLEAVE:
2868 return !!nodes_equal(a->nodes, b->nodes);2868 return nodes_equal(a->nodes, b->nodes);
2869 case MPOL_LOCAL:2869 case MPOL_LOCAL:
2870 return true;2870 return true;
2871 default:2871 default:
mm/memremap.c+2−2
@@ -97,10 +97,10 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) · 96 unmodified lines
97 PHYS_PFN(range_len(range)));97 PHYS_PFN(range_len(range)));
98 if (pgmap->type == MEMORY_DEVICE_PRIVATE) {98 if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
99 __remove_pages(PHYS_PFN(range->start),99 __remove_pages(PHYS_PFN(range->start),
100 PHYS_PFN(range_len(range)), NULL);100 PHYS_PFN(range_len(range)), NULL, pgmap);
101 } else {101 } else {
102 arch_remove_memory(range->start, range_len(range),102 arch_remove_memory(range->start, range_len(range),
103 pgmap_altmap(pgmap));103 pgmap_altmap(pgmap), pgmap);
104 kasan_remove_zero_shadow(__va(range->start), range_len(range));104 kasan_remove_zero_shadow(__va(range->start), range_len(range));
105 }105 }
106 mem_hotplug_done();106 mem_hotplug_done();
mm/migrate.c+33−30
@@ -1135,26 +1135,24 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 1134 unmodified lines
1135 * This is safe because nobody is using it except us.1135 * This is safe because nobody is using it except us.
1136 */1136 */
1137enum {1137enum {
1138 PAGE_WAS_MAPPED = BIT(0),1138 FOLIO_WAS_MAPPED = BIT(0),
1139 PAGE_WAS_MLOCKED = BIT(1),1139 FOLIO_WAS_MLOCKED = BIT(1),
1140 PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,1140 FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
1141};1141};
1142 1142
1143static void __migrate_folio_record(struct folio *dst,1143static void __migrate_folio_record(struct folio *dst,
1144 int old_page_state,1144 int old_folio_state, struct anon_vma *anon_vma)
1145 struct anon_vma *anon_vma)
1146{1145{
1147 dst->private = (void *)anon_vma + old_page_state;1146 dst->private = (void *)anon_vma + old_folio_state;
1148}1147}
1149 1148
1150static void __migrate_folio_extract(struct folio *dst,1149static void __migrate_folio_extract(struct folio *dst,
1151 int *old_page_state,1150 int *old_folio_state, struct anon_vma **anon_vmap)
1152 struct anon_vma **anon_vmap)
1153{1151{
1154 unsigned long private = (unsigned long)dst->private;1152 unsigned long private = (unsigned long)dst->private;
1155 1153
1156 *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES);1154 *anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
1157 *old_page_state = private & PAGE_OLD_STATES;1155 *old_folio_state = private & FOLIO_OLD_STATES;
1158 dst->private = NULL;1156 dst->private = NULL;
1159}1157}
1160 1158
@@ -1209,7 +1207,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 48 unmodified lines
1209{1207{
1210 struct folio *dst;1208 struct folio *dst;
1211 int rc = -EAGAIN;1209 int rc = -EAGAIN;
1212 int old_page_state = 0;1210 int old_folio_state = 0;
1213 struct anon_vma *anon_vma = NULL;1211 struct anon_vma *anon_vma = NULL;
1214 bool locked = false;1212 bool locked = false;
1215 bool dst_locked = false;1213 bool dst_locked = false;
@@ -1253,12 +1251,12 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 37 unmodified lines
1253 }1251 }
1254 locked = true;1252 locked = true;
1255 if (folio_test_mlocked(src))1253 if (folio_test_mlocked(src))
1256 old_page_state |= PAGE_WAS_MLOCKED;1254 old_folio_state |= FOLIO_WAS_MLOCKED;
1257 1255
1258 if (folio_test_writeback(src)) {1256 if (folio_test_writeback(src)) {
1259 /*1257 /*
1260 * Only in the case of a full synchronous migration is it1258 * Only in the case of a full synchronous migration is it
1261 * necessary to wait for PageWriteback. In the async case,1259 * necessary to wait for writeback. In the async case,
1262 * the retry loop is too short and in the sync-light case,1260 * the retry loop is too short and in the sync-light case,
1263 * the overhead of stalling is too much1261 * the overhead of stalling is too much
1264 */1262 */
@@ -1302,7 +1300,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 37 unmodified lines
1302 dst_locked = true;1300 dst_locked = true;
1303 1301
1304 if (unlikely(page_has_movable_ops(&src->page))) {1302 if (unlikely(page_has_movable_ops(&src->page))) {
1305 __migrate_folio_record(dst, old_page_state, anon_vma);1303 __migrate_folio_record(dst, old_folio_state, anon_vma);
1306 return 0;1304 return 0;
1307 }1305 }
1308 1306
@@ -1328,11 +1326,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 19 unmodified lines
1328 VM_BUG_ON_FOLIO(folio_test_anon(src) &&1326 VM_BUG_ON_FOLIO(folio_test_anon(src) &&
1329 !folio_test_ksm(src) && !anon_vma, src);1327 !folio_test_ksm(src) && !anon_vma, src);
1330 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);1328 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
1331 old_page_state |= PAGE_WAS_MAPPED;1329 old_folio_state |= FOLIO_WAS_MAPPED;
1332 }1330 }
1333 1331
1334 if (!folio_mapped(src)) {1332 if (!folio_mapped(src)) {
1335 __migrate_folio_record(dst, old_page_state, anon_vma);1333 __migrate_folio_record(dst, old_folio_state, anon_vma);
1336 return 0;1334 return 0;
1337 }1335 }
1338 1336
@@ -1344,7 +1342,7 @@ out: · 5 unmodified lines
1344 if (rc == -EAGAIN)1342 if (rc == -EAGAIN)
1345 ret = NULL;1343 ret = NULL;
1346 1344
1347 migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,1345 migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
1348 anon_vma, locked, ret);1346 anon_vma, locked, ret);
1349 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);1347 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
1350 1348
@@ -1358,13 +1356,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 7 unmodified lines
1358 struct list_head *ret)1356 struct list_head *ret)
1359{1357{
1360 int rc;1358 int rc;
1361 int old_page_state = 0;1359 int old_folio_state = 0;
1362 struct anon_vma *anon_vma = NULL;1360 struct anon_vma *anon_vma = NULL;
1363 bool src_deferred_split = false;1361 bool src_deferred_split = false;
1364 bool src_partially_mapped = false;1362 bool src_partially_mapped = false;
1365 struct list_head *prev;1363 struct list_head *prev;
1366 1364
1367 __migrate_folio_extract(dst, &old_page_state, &anon_vma);1365 __migrate_folio_extract(dst, &old_folio_state, &anon_vma);
1368 prev = dst->lru.prev;1366 prev = dst->lru.prev;
1369 list_del(&dst->lru);1367 list_del(&dst->lru);
1370 1368
@@ -1404,10 +1402,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 33 unmodified lines
1404 * isolated from the unevictable LRU: but this case is the easiest.1402 * isolated from the unevictable LRU: but this case is the easiest.
1405 */1403 */
1406 folio_add_lru(dst);1404 folio_add_lru(dst);
1407 if (old_page_state & PAGE_WAS_MLOCKED)1405 if (old_folio_state & FOLIO_WAS_MLOCKED)
1408 lru_add_drain();1406 lru_add_drain();
1409 1407
1410 if (old_page_state & PAGE_WAS_MAPPED)1408 if (old_folio_state & FOLIO_WAS_MAPPED)
1411 remove_migration_ptes(src, dst, 0);1409 remove_migration_ptes(src, dst, 0);
1412 1410
1413out_unlock_both:1411out_unlock_both:
@@ -1439,11 +1437,11 @@ out: · 25 unmodified lines
1439 */1437 */
1440 if (rc == -EAGAIN) {1438 if (rc == -EAGAIN) {
1441 list_add(&dst->lru, prev);1439 list_add(&dst->lru, prev);
1442 __migrate_folio_record(dst, old_page_state, anon_vma);1440 __migrate_folio_record(dst, old_folio_state, anon_vma);
1443 return rc;1441 return rc;
1444 }1442 }
1445 1443
1446 migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,1444 migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
1447 anon_vma, true, ret);1445 anon_vma, true, ret);
1448 migrate_folio_undo_dst(dst, true, put_new_folio, private);1446 migrate_folio_undo_dst(dst, true, put_new_folio, private);
1449 1447
@@ -1777,11 +1775,11 @@ static void migrate_folios_undo(struct list_head *src_folios, · 327 unmodified lines
1777 dst = list_first_entry(dst_folios, struct folio, lru);1775 dst = list_first_entry(dst_folios, struct folio, lru);
1778 dst2 = list_next_entry(dst, lru);1776 dst2 = list_next_entry(dst, lru);
1779 list_for_each_entry_safe(folio, folio2, src_folios, lru) {1777 list_for_each_entry_safe(folio, folio2, src_folios, lru) {
1780 int old_page_state = 0;1778 int old_folio_state = 0;
1781 struct anon_vma *anon_vma = NULL;1779 struct anon_vma *anon_vma = NULL;
1782 1780
1783 __migrate_folio_extract(dst, &old_page_state, &anon_vma);1781 __migrate_folio_extract(dst, &old_folio_state, &anon_vma);
1784 migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,1782 migrate_folio_undo_src(folio, old_folio_state & FOLIO_WAS_MAPPED,
1785 anon_vma, true, ret_folios);1783 anon_vma, true, ret_folios);
1786 list_del(&dst->lru);1784 list_del(&dst->lru);
1787 migrate_folio_undo_dst(dst, true, put_new_folio, private);1785 migrate_folio_undo_dst(dst, true, put_new_folio, private);
@@ -2557,24 +2555,29 @@ static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) · 769 unmodified lines
2557 }2555 }
2558 2556
2559 task = find_get_task_by_vpid(pid);2557 task = find_get_task_by_vpid(pid);
2560 if (!task) {2558 if (!task)
2561 return ERR_PTR(-ESRCH);2559 return ERR_PTR(-ESRCH);
2562 }
2563 2560
2561 if (down_read_killable(&task->signal->exec_update_lock)) {
2562 mm = ERR_PTR(-EINTR);
2563 goto out;
2564 }
2564 /*2565 /*
2565 * Check if this process has the right to modify the specified2566 * Check if this process has the right to modify the specified
2566 * process. Use the regular "ptrace_may_access()" checks.2567 * process. Use the regular "ptrace_may_access()" checks.
2567 */2568 */
2568 if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {2569 if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {
2569 mm = ERR_PTR(-EPERM);2570 mm = ERR_PTR(-EPERM);
2570 goto out;2571 goto unlock;
2571 }2572 }
2572 2573
2573 mm = ERR_PTR(security_task_movememory(task));2574 mm = ERR_PTR(security_task_movememory(task));
2574 if (IS_ERR(mm))2575 if (IS_ERR(mm))
2575 goto out;2576 goto unlock;
2576 *mem_nodes = cpuset_mems_allowed(task);2577 *mem_nodes = cpuset_mems_allowed(task);
2577 mm = get_task_mm(task);2578 mm = get_task_mm(task);
2579unlock:
2580 up_read(&task->signal->exec_update_lock);
2578out:2581out:
2579 put_task_struct(task);2582 put_task_struct(task);
2580 if (!mm)2583 if (!mm)
mm/migrate_device.c+3−6
@@ -801,8 +801,7 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, · 800 unmodified lines
801 bool flush = false;801 bool flush = false;
802 unsigned long i;802 unsigned long i;
803 803
804 VM_WARN_ON_FOLIO(!folio, folio);804 VM_WARN_ON_ONCE(!folio);
805 VM_WARN_ON_ONCE(!pmd_none(*pmdp) && !is_huge_zero_pmd(*pmdp));
806 805
807 if (!thp_vma_suitable_order(vma, addr, HPAGE_PMD_ORDER))806 if (!thp_vma_suitable_order(vma, addr, HPAGE_PMD_ORDER))
808 return -EINVAL;807 return -EINVAL;
@@ -859,11 +858,9 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, · 50 unmodified lines
859 if (userfaultfd_missing(vma))858 if (userfaultfd_missing(vma))
860 goto unlock_abort;859 goto unlock_abort;
861 860
862 if (!pmd_none(*pmdp)) {861 if (is_huge_zero_pmd(*pmdp))
863 if (!is_huge_zero_pmd(*pmdp))
864 goto unlock_abort;
865 flush = true;862 flush = true;
866 } else if (!pmd_none(*pmdp))863 else if (!pmd_none(*pmdp))
867 goto unlock_abort;864 goto unlock_abort;
868 865
869 add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);866 add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
mm/mm_init.c+36−37
@@ -674,6 +674,20 @@ static inline void fixup_hashdist(void) · 673 unmodified lines
674static inline void fixup_hashdist(void) {}674static inline void fixup_hashdist(void) {}
675#endif /* CONFIG_NUMA */675#endif /* CONFIG_NUMA */
676 676
677#ifdef CONFIG_ZONE_DEVICE
678static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
679 unsigned long nr_pages, int migratetype)
680{
681 const unsigned long end = pfn + nr_pages;
682
683 for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) {
684 init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false);
685 if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
686 cond_resched();
687 }
688}
689#endif
690
677/*691/*
678 * Initialize a reserved page unconditionally, finding its zone first.692 * Initialize a reserved page unconditionally, finding its zone first.
679 */693 */
@@ -1011,21 +1025,6 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, · 331 unmodified lines
1011 page_folio(page)->pgmap = pgmap;1025 page_folio(page)->pgmap = pgmap;
1012 page->zone_device_data = NULL;1026 page->zone_device_data = NULL;
1013 1027
1014 /*
1015 * Mark the block movable so that blocks are reserved for
1016 * movable at startup. This will force kernel allocations
1017 * to reserve their blocks rather than leaking throughout
1018 * the address space during boot when many long-lived
1019 * kernel allocations are made.
1020 *
1021 * Please note that MEMINIT_HOTPLUG path doesn't clear memmap
1022 * because this is done early in section_activate()
1023 */
1024 if (pageblock_aligned(pfn)) {
1025 init_pageblock_migratetype(page, MIGRATE_MOVABLE, false);
1026 cond_resched();
1027 }
1028
1029 /*1028 /*
1030 * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released1029 * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released
1031 * directly to the driver page allocator which will set the page count1030 * directly to the driver page allocator which will set the page count
@@ -1056,10 +1055,17 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, · 24 unmodified lines
1056 * of how the sparse_vmemmap internals handle compound pages in the lack1055 * of how the sparse_vmemmap internals handle compound pages in the lack
1057 * of an altmap. See vmemmap_populate_compound_pages().1056 * of an altmap. See vmemmap_populate_compound_pages().
1058 */1057 */
1059static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,1058static inline unsigned long compound_nr_pages(unsigned long pfn,
1059 struct vmem_altmap *altmap,
1060 struct dev_pagemap *pgmap)1060 struct dev_pagemap *pgmap)
1061{1061{
1062 if (!vmemmap_can_optimize(altmap, pgmap))1062 /*
1063 * If DAX memory is hot-plugged into an unoccupied subsection
1064 * of an early section, the unoptimized boot memmap is reused.
1065 * See section_activate().
1066 */
1067 if (early_section(__pfn_to_section(pfn)) ||
1068 !vmemmap_can_optimize(altmap, pgmap))
1063 return pgmap_vmemmap_nr(pgmap);1069 return pgmap_vmemmap_nr(pgmap);
1064 1070
1065 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page));1071 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page));
@@ -1122,13 +1128,18 @@ void __ref memmap_init_zone_device(struct zone *zone, · 56 unmodified lines
1122 1128
1123 __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);1129 __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
1124 1130
1131 if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
1132 cond_resched();
1133
1125 if (pfns_per_compound == 1)1134 if (pfns_per_compound == 1)
1126 continue;1135 continue;
1127 1136
1128 memmap_init_compound(page, pfn, zone_idx, nid, pgmap,1137 memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
1129 compound_nr_pages(altmap, pgmap));1138 compound_nr_pages(pfn, altmap, pgmap));
1130 }1139 }
1131 1140
1141 pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE);
1142
1132 pr_debug("%s initialised %lu pages in %ums\n", __func__,1143 pr_debug("%s initialised %lu pages in %ums\n", __func__,
1133 nr_pages, jiffies_to_msecs(jiffies - start));1144 nr_pages, jiffies_to_msecs(jiffies - start));
1134}1145}
@@ -1362,19 +1373,6 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, · 227 unmodified lines
1362 pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages);1373 pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages);
1363}1374}
1364 1375
1365#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1366static void pgdat_init_split_queue(struct pglist_data *pgdat)
1367{
1368 struct deferred_split *ds_queue = &pgdat->deferred_split_queue;
1369
1370 spin_lock_init(&ds_queue->split_queue_lock);
1371 INIT_LIST_HEAD(&ds_queue->split_queue);
1372 ds_queue->split_queue_len = 0;
1373}
1374#else
1375static void pgdat_init_split_queue(struct pglist_data *pgdat) {}
1376#endif
1377
1378#ifdef CONFIG_COMPACTION1376#ifdef CONFIG_COMPACTION
1379static void pgdat_init_kcompactd(struct pglist_data *pgdat)1377static void pgdat_init_kcompactd(struct pglist_data *pgdat)
1380{1378{
@@ -1390,8 +1388,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) · 9 unmodified lines
1390 1388
1391 pgdat_resize_init(pgdat);1389 pgdat_resize_init(pgdat);
1392 pgdat_kswapd_lock_init(pgdat);1390 pgdat_kswapd_lock_init(pgdat);
1393
1394 pgdat_init_split_queue(pgdat);
1395 pgdat_init_kcompactd(pgdat);1391 pgdat_init_kcompactd(pgdat);
1396 1392
1397 init_waitqueue_head(&pgdat->kswapd_wait);1393 init_waitqueue_head(&pgdat->kswapd_wait);
@@ -1418,11 +1414,14 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, · 20 unmodified lines
1418 1414
1419static void __meminit zone_init_free_lists(struct zone *zone)1415static void __meminit zone_init_free_lists(struct zone *zone)
1420{1416{
1421 unsigned int order, t;1417 struct list_head *list;
1422 for_each_migratetype_order(order, t) {1418 unsigned int order;
1423 INIT_LIST_HEAD(&zone->free_area[order].free_list[t]);1419
1420 for_each_free_list(list, zone, order)
1421 INIT_LIST_HEAD(list);
1422
1423 for (order = 0; order < NR_PAGE_ORDERS; order++)
1424 zone->free_area[order].nr_free = 0;1424 zone->free_area[order].nr_free = 0;
1425 }
1426 1425
1427#ifdef CONFIG_UNACCEPTED_MEMORY1426#ifdef CONFIG_UNACCEPTED_MEMORY
1428 INIT_LIST_HEAD(&zone->unaccepted_pages);1427 INIT_LIST_HEAD(&zone->unaccepted_pages);
mm/mmap.c+1−1
@@ -504,7 +504,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, · 503 unmodified lines
504 break;504 break;
505 case MAP_DROPPABLE:505 case MAP_DROPPABLE:
506 if (VM_DROPPABLE == VM_NONE)506 if (VM_DROPPABLE == VM_NONE)
507 return -ENOTSUPP;507 return -EOPNOTSUPP;
508 /*508 /*
509 * A locked or stack area makes no sense to be droppable.509 * A locked or stack area makes no sense to be droppable.
510 *510 *
mm/mseal.c+3−2
@@ -8,6 +8,7 @@ · 7 unmodified lines
8 */8 */
9 9
10#include <linux/mempolicy.h>10#include <linux/mempolicy.h>
11#include <linux/minmax.h>
11#include <linux/mman.h>12#include <linux/mman.h>
12#include <linux/mm.h>13#include <linux/mm.h>
13#include <linux/mm_inline.h>14#include <linux/mm_inline.h>
@@ -65,8 +66,8 @@ static int mseal_apply(struct mm_struct *mm, · 51 unmodified lines
65 prev = vma;66 prev = vma;
66 67
67 for_each_vma_range(vmi, vma, end) {68 for_each_vma_range(vmi, vma, end) {
68 const unsigned long curr_start = MAX(vma->vm_start, start);69 const unsigned long curr_start = max(vma->vm_start, start);
69 const unsigned long curr_end = MIN(vma->vm_end, end);70 const unsigned long curr_end = min(vma->vm_end, end);
70 71
71 if (!vma_test(vma, VMA_SEALED_BIT)) {72 if (!vma_test(vma, VMA_SEALED_BIT)) {
72 vma_flags_t vma_flags = vma->flags;73 vma_flags_t vma_flags = vma->flags;
mm/page_alloc.c+277−106
@@ -90,6 +90,9 @@ typedef int __bitwise fpi_t; · 89 unmodified lines
90/* Free the page without taking locks. Rely on trylock only. */90/* Free the page without taking locks. Rely on trylock only. */
91#define FPI_TRYLOCK ((__force fpi_t)BIT(2))91#define FPI_TRYLOCK ((__force fpi_t)BIT(2))
92 92
93/* free_pages_prepare() has already been called for page(s) being freed. */
94#define FPI_PREPARED ((__force fpi_t)BIT(3))
95
93/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */96/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */
94static DEFINE_MUTEX(pcp_batch_high_lock);97static DEFINE_MUTEX(pcp_batch_high_lock);
95#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)98#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)
@@ -282,6 +285,14 @@ EXPORT_SYMBOL(nr_node_ids); · 186 unmodified lines
282EXPORT_SYMBOL(nr_online_nodes);285EXPORT_SYMBOL(nr_online_nodes);
283#endif286#endif
284 287
288/*
289 * When page allocations stall for longer than a threshold,
290 * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log. Only one warning
291 * will be printed during this duration for the entire system.
292 */
293#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
294static unsigned long alloc_stall_warn_jiffies = INITIAL_JIFFIES;
295
285static bool page_contains_unaccepted(struct page *page, unsigned int order);296static bool page_contains_unaccepted(struct page *page, unsigned int order);
286static bool cond_accept_memory(struct zone *zone, unsigned int order,297static bool cond_accept_memory(struct zone *zone, unsigned int order,
287 int alloc_flags);298 int alloc_flags);
@@ -353,7 +364,7 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn, · 65 unmodified lines
353#else364#else
354 BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4);365 BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4);
355#endif366#endif
356 BUILD_BUG_ON(__MIGRATE_TYPE_END > MIGRATETYPE_MASK);367 BUILD_BUG_ON(__MIGRATE_TYPE_END > PAGEBLOCK_MIGRATETYPE_MASK);
357 VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page);368 VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page);
358 369
359 bitmap = get_pageblock_bitmap(page, pfn);370 bitmap = get_pageblock_bitmap(page, pfn);
@@ -423,10 +434,10 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · 63 unmodified lines
423 * Use get_pfnblock_migratetype() if caller already has both @page and @pfn434 * Use get_pfnblock_migratetype() if caller already has both @page and @pfn
424 * to save a call to page_to_pfn().435 * to save a call to page_to_pfn().
425 */436 */
426__always_inline enum migratetype437enum migratetype
427get_pfnblock_migratetype(const struct page *page, unsigned long pfn)438get_pfnblock_migratetype(const struct page *page, unsigned long pfn)
428{439{
429 unsigned long mask = MIGRATETYPE_AND_ISO_MASK;440 unsigned long mask = PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK;
430 unsigned long flags;441 unsigned long flags;
431 442
432 flags = __get_pfnblock_flags_mask(page, pfn, mask);443 flags = __get_pfnblock_flags_mask(page, pfn, mask);
@@ -435,7 +446,7 @@ get_pfnblock_migratetype(const struct page *page, unsigned long pfn) · 2 unmodified lines
435 if (flags & BIT(PB_migrate_isolate))446 if (flags & BIT(PB_migrate_isolate))
436 return MIGRATE_ISOLATE;447 return MIGRATE_ISOLATE;
437#endif448#endif
438 return flags & MIGRATETYPE_MASK;449 return flags & PAGEBLOCK_MIGRATETYPE_MASK;
439}450}
440 451
441/**452/**
@@ -523,11 +534,11 @@ static void set_pageblock_migratetype(struct page *page, · 81 unmodified lines
523 }534 }
524 VM_WARN_ONCE(get_pageblock_isolate(page),535 VM_WARN_ONCE(get_pageblock_isolate(page),
525 "Use clear_pageblock_isolate() to unisolate pageblock");536 "Use clear_pageblock_isolate() to unisolate pageblock");
526 /* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */537 /* PAGEBLOCK_ISO_MASK clears PB_migrate_isolate if it is set */
527#endif538#endif
528 __set_pfnblock_flags_mask(page, page_to_pfn(page),539 __set_pfnblock_flags_mask(page, page_to_pfn(page),
529 (unsigned long)migratetype,540 (unsigned long)migratetype,
530 MIGRATETYPE_AND_ISO_MASK);541 PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK);
531}542}
532 543
533void __meminit init_pageblock_migratetype(struct page *page,544void __meminit init_pageblock_migratetype(struct page *page,
@@ -553,7 +564,7 @@ void __meminit init_pageblock_migratetype(struct page *page, · 19 unmodified lines
553 flags |= BIT(PB_migrate_isolate);564 flags |= BIT(PB_migrate_isolate);
554#endif565#endif
555 __set_pfnblock_flags_mask(page, page_to_pfn(page), flags,566 __set_pfnblock_flags_mask(page, page_to_pfn(page), flags,
556 MIGRATETYPE_AND_ISO_MASK);567 PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK);
557}568}
558 569
559#ifdef CONFIG_DEBUG_VM570#ifdef CONFIG_DEBUG_VM
@@ -639,19 +650,12 @@ out: · 79 unmodified lines
639 650
640static inline unsigned int order_to_pindex(int migratetype, int order)651static inline unsigned int order_to_pindex(int migratetype, int order)
641{652{
653 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
654 bool movable = migratetype == MIGRATE_MOVABLE;
642 655
643#ifdef CONFIG_TRANSPARENT_HUGEPAGE656 if (order > PAGE_ALLOC_COSTLY_ORDER)
644 bool movable;657 return NR_LOWORDER_PCP_LISTS + movable;
645 if (order > PAGE_ALLOC_COSTLY_ORDER) {
646 VM_BUG_ON(!is_pmd_order(order));
647
648 movable = migratetype == MIGRATE_MOVABLE;
649
650 return NR_LOWORDER_PCP_LISTS + movable;
651 }658 }
652#else
653 VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
654#endif
655 659
656 return (MIGRATE_PCPTYPES * order) + migratetype;660 return (MIGRATE_PCPTYPES * order) + migratetype;
657}661}
@@ -660,12 +664,10 @@ static inline int pindex_to_order(unsigned int pindex) · 2 unmodified lines
660{664{
661 int order = pindex / MIGRATE_PCPTYPES;665 int order = pindex / MIGRATE_PCPTYPES;
662 666
663#ifdef CONFIG_TRANSPARENT_HUGEPAGE667 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
664 if (pindex >= NR_LOWORDER_PCP_LISTS)668 if (pindex >= NR_LOWORDER_PCP_LISTS)
665 order = HPAGE_PMD_ORDER;669 order = HPAGE_PMD_ORDER;
666#else670 }
667 VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
668#endif
669 671
670 return order;672 return order;
671}673}
@@ -1211,14 +1213,18 @@ static inline bool should_skip_kasan_poison(struct page *page) · 539 unmodified lines
1211 return page_kasan_tag(page) == KASAN_TAG_KERNEL;1213 return page_kasan_tag(page) == KASAN_TAG_KERNEL;
1212}1214}
1213 1215
1214static void kernel_init_pages(struct page *page, int numpages)1216static void clear_highpages_kasan_tagged(struct page *page, int numpages)
1215{1217{
1216 int i;
1217
1218 /* s390's use of memset() could override KASAN redzones. */1218 /* s390's use of memset() could override KASAN redzones. */
1219 kasan_disable_current();1219 kasan_disable_current();
1220 for (i = 0; i < numpages; i++)1220 if (!IS_ENABLED(CONFIG_HIGHMEM)) {
1221 clear_highpage_kasan_tagged(page + i);1221 clear_pages(kasan_reset_tag(page_address(page)), numpages);
1222 } else {
1223 int i;
1224
1225 for (i = 0; i < numpages; i++)
1226 clear_highpage_kasan_tagged(page + i);
1227 }
1222 kasan_enable_current();1228 kasan_enable_current();
1223}1229}
1224 1230
@@ -1303,8 +1309,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) · 78 unmodified lines
1303 1309
1304#endif /* CONFIG_MEM_ALLOC_PROFILING */1310#endif /* CONFIG_MEM_ALLOC_PROFILING */
1305 1311
1306__always_inline bool __free_pages_prepare(struct page *page,1312static __always_inline bool __free_pages_prepare(struct page *page,
1307 unsigned int order, fpi_t fpi_flags)1313 unsigned int order, fpi_t fpi_flags)
1308{1314{
1309 int bad = 0;1315 int bad = 0;
1310 bool skip_kasan_poison = should_skip_kasan_poison(page);1316 bool skip_kasan_poison = should_skip_kasan_poison(page);
@@ -1312,6 +1318,9 @@ __always_inline bool __free_pages_prepare(struct page *page, · 1 unmodified line
1312 bool compound = PageCompound(page);1318 bool compound = PageCompound(page);
1313 struct folio *folio = page_folio(page);1319 struct folio *folio = page_folio(page);
1314 1320
1321 if (fpi_flags & FPI_PREPARED)
1322 return true;
1323
1315 VM_BUG_ON_PAGE(PageTail(page), page);1324 VM_BUG_ON_PAGE(PageTail(page), page);
1316 1325
1317 trace_mm_page_free(page, order);1326 trace_mm_page_free(page, order);
@@ -1423,7 +1432,7 @@ __always_inline bool __free_pages_prepare(struct page *page, · 105 unmodified lines
1423 init = false;1432 init = false;
1424 }1433 }
1425 if (init)1434 if (init)
1426 kernel_init_pages(page, 1 << order);1435 clear_highpages_kasan_tagged(page, 1 << order);
1427 1436
1428 /*1437 /*
1429 * arch_free_page() can make the page's contents inaccessible. s3901438 * arch_free_page() can make the page's contents inaccessible. s390
@@ -1451,7 +1460,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 21 unmodified lines
1451 struct per_cpu_pages *pcp,1460 struct per_cpu_pages *pcp,
1452 int pindex)1461 int pindex)
1453{1462{
1454 unsigned long flags;
1455 unsigned int order;1463 unsigned int order;
1456 struct page *page;1464 struct page *page;
1457 1465
@@ -1464,7 +1472,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 6 unmodified lines
1464 /* Ensure requested pindex is drained first. */1472 /* Ensure requested pindex is drained first. */
1465 pindex = pindex - 1;1473 pindex = pindex - 1;
1466 1474
1467 spin_lock_irqsave(&zone->lock, flags);1475 guard(spinlock_irqsave)(&zone->lock);
1468 1476
1469 while (count > 0) {1477 while (count > 0) {
1470 struct list_head *list;1478 struct list_head *list;
@@ -1496,8 +1504,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 25 unmodified lines
1496 trace_mm_page_pcpu_drain(page, order, mt);1504 trace_mm_page_pcpu_drain(page, order, mt);
1497 } while (count > 0 && !list_empty(list));1505 } while (count > 0 && !list_empty(list));
1498 }1506 }
1499
1500 spin_unlock_irqrestore(&zone->lock, flags);
1501}1507}
1502 1508
1503/* Split a multi-block free page into its individual pageblocks. */1509/* Split a multi-block free page into its individual pageblocks. */
@@ -1848,7 +1854,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, · 344 unmodified lines
1848 }1854 }
1849 /* If memory is still not initialized, initialize it now. */1855 /* If memory is still not initialized, initialize it now. */
1850 if (init)1856 if (init)
1851 kernel_init_pages(page, 1 << order);1857 clear_highpages_kasan_tagged(page, 1 << order);
1852 1858
1853 set_page_owner(page, order, gfp_flags);1859 set_page_owner(page, order, gfp_flags);
1854 page_table_check_alloc(page, order);1860 page_table_check_alloc(page, order);
@@ -2125,15 +2131,15 @@ static bool __move_freepages_block_isolate(struct zone *zone, · 270 unmodified lines
2125 }2131 }
2126 2132
2127move:2133move:
2128 /* Use MIGRATETYPE_MASK to get non-isolate migratetype */2134 /* Use PAGEBLOCK_MIGRATETYPE_MASK to get non-isolate migratetype */
2129 if (isolate) {2135 if (isolate) {
2130 from_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),2136 from_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),
2131 MIGRATETYPE_MASK);2137 PAGEBLOCK_MIGRATETYPE_MASK);
2132 to_mt = MIGRATE_ISOLATE;2138 to_mt = MIGRATE_ISOLATE;
2133 } else {2139 } else {
2134 from_mt = MIGRATE_ISOLATE;2140 from_mt = MIGRATE_ISOLATE;
2135 to_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),2141 to_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),
2136 MIGRATETYPE_MASK);2142 PAGEBLOCK_MIGRATETYPE_MASK);
2137 }2143 }
2138 2144
2139 __move_freepages_block(zone, start_pfn, from_mt, to_mt);2145 __move_freepages_block(zone, start_pfn, from_mt, to_mt);
@@ -2244,25 +2250,29 @@ static bool should_try_claim_block(unsigned int order, int start_mt) · 104 unmodified lines
2244 * we would do this whole-block claiming. This would help to reduce2250 * we would do this whole-block claiming. This would help to reduce
2245 * fragmentation due to mixed migratetype pages in one pageblock.2251 * fragmentation due to mixed migratetype pages in one pageblock.
2246 */2252 */
2247int find_suitable_fallback(struct free_area *area, unsigned int order,2253enum fallback_result
2248 int migratetype, bool claimable)2254find_suitable_fallback(struct free_area *area, unsigned int order,
2255 int migratetype, bool claimable, int *mt_out)
2249{2256{
2250 int i;2257 int i;
2251 2258
2252 if (claimable && !should_try_claim_block(order, migratetype))2259 if (claimable && !should_try_claim_block(order, migratetype))
2253 return -2;2260 return FALLBACK_NOCLAIM;
2254 2261
2255 if (area->nr_free == 0)2262 if (area->nr_free == 0)
2256 return -1;2263 return FALLBACK_EMPTY;
2257 2264
2258 for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) {2265 for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) {
2259 int fallback_mt = fallbacks[migratetype][i];2266 int fallback_mt = fallbacks[migratetype][i];
2260 2267
2261 if (!free_area_empty(area, fallback_mt))2268 if (!free_area_empty(area, fallback_mt)) {
2262 return fallback_mt;2269 if (mt_out)
2270 *mt_out = fallback_mt;
2271 return FALLBACK_FOUND;
2272 }
2263 }2273 }
2264 2274
2265 return -1;2275 return FALLBACK_EMPTY;
2266}2276}
2267 2277
2268/*2278/*
@@ -2372,16 +2382,16 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype, · 103 unmodified lines
2372 */2382 */
2373 for (current_order = MAX_PAGE_ORDER; current_order >= min_order;2383 for (current_order = MAX_PAGE_ORDER; current_order >= min_order;
2374 --current_order) {2384 --current_order) {
2385 enum fallback_result result;
2386
2375 area = &(zone->free_area[current_order]);2387 area = &(zone->free_area[current_order]);
2376 fallback_mt = find_suitable_fallback(area, current_order,2388 result = find_suitable_fallback(area, current_order,
2377 start_migratetype, true);2389 start_migratetype, true, &fallback_mt);
2378 2390
2379 /* No block in that order */2391 if (result == FALLBACK_EMPTY)
2380 if (fallback_mt == -1)
2381 continue;2392 continue;
2382 2393
2383 /* Advanced into orders too low to claim, abort */2394 if (result == FALLBACK_NOCLAIM)
2384 if (fallback_mt == -2)
2385 break;2395 break;
2386 2396
2387 page = get_page_from_free_area(area, fallback_mt);2397 page = get_page_from_free_area(area, fallback_mt);
@@ -2411,10 +2421,12 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype) · 23 unmodified lines
2411 int fallback_mt;2421 int fallback_mt;
2412 2422
2413 for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) {2423 for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) {
2424 enum fallback_result result;
2425
2414 area = &(zone->free_area[current_order]);2426 area = &(zone->free_area[current_order]);
2415 fallback_mt = find_suitable_fallback(area, current_order,2427 result = find_suitable_fallback(area, current_order, start_migratetype,
2416 start_migratetype, false);2428 false, &fallback_mt);
2417 if (fallback_mt == -1)2429 if (result == FALLBACK_EMPTY)
2418 continue;2430 continue;
2419 2431
2420 page = get_page_from_free_area(area, fallback_mt);2432 page = get_page_from_free_area(area, fallback_mt);
@@ -3424,7 +3436,7 @@ static void reserve_highatomic_pageblock(struct page *page, int order, · 1003 unmodified lines
3424 struct zone *zone)3436 struct zone *zone)
3425{3437{
3426 int mt;3438 int mt;
3427 unsigned long max_managed, flags;3439 unsigned long max_managed;
3428 3440
3429 /*3441 /*
3430 * The number reserved as: minimum is 1 pageblock, maximum is3442 * The number reserved as: minimum is 1 pageblock, maximum is
@@ -3438,29 +3450,26 @@ static void reserve_highatomic_pageblock(struct page *page, int order, · 7 unmodified lines
3438 if (zone->nr_reserved_highatomic >= max_managed)3450 if (zone->nr_reserved_highatomic >= max_managed)
3439 return;3451 return;
3440 3452
3441 spin_lock_irqsave(&zone->lock, flags);3453 guard(spinlock_irqsave)(&zone->lock);
3442 3454
3443 /* Recheck the nr_reserved_highatomic limit under the lock */3455 /* Recheck the nr_reserved_highatomic limit under the lock */
3444 if (zone->nr_reserved_highatomic >= max_managed)3456 if (zone->nr_reserved_highatomic >= max_managed)
3445 goto out_unlock;3457 return;
3446 3458
3447 /* Yoink! */3459 /* Yoink! */
3448 mt = get_pageblock_migratetype(page);3460 mt = get_pageblock_migratetype(page);
3449 /* Only reserve normal pageblocks (i.e., they can merge with others) */3461 /* Only reserve normal pageblocks (i.e., they can merge with others) */
3450 if (!migratetype_is_mergeable(mt))3462 if (!migratetype_is_mergeable(mt))
3451 goto out_unlock;3463 return;
3452 3464
3453 if (order < pageblock_order) {3465 if (order < pageblock_order) {
3454 if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) == -1)3466 if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) == -1)
3455 goto out_unlock;3467 return;
3456 zone->nr_reserved_highatomic += pageblock_nr_pages;3468 zone->nr_reserved_highatomic += pageblock_nr_pages;
3457 } else {3469 } else {
3458 change_pageblock_range(page, order, MIGRATE_HIGHATOMIC);3470 change_pageblock_range(page, order, MIGRATE_HIGHATOMIC);
3459 zone->nr_reserved_highatomic += 1 << order;3471 zone->nr_reserved_highatomic += 1 << order;
3460 }3472 }
3461
3462out_unlock:
3463 spin_unlock_irqrestore(&zone->lock, flags);
3464}3473}
3465 3474
3466/*3475/*
@@ -3476,7 +3485,6 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 9 unmodified lines
3476 bool force)3485 bool force)
3477{3486{
3478 struct zonelist *zonelist = ac->zonelist;3487 struct zonelist *zonelist = ac->zonelist;
3479 unsigned long flags;
3480 struct zoneref *z;3488 struct zoneref *z;
3481 struct zone *zone;3489 struct zone *zone;
3482 struct page *page;3490 struct page *page;
@@ -3493,7 +3501,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 10 unmodified lines
3493 pageblock_nr_pages)3501 pageblock_nr_pages)
3494 continue;3502 continue;
3495 3503
3496 spin_lock_irqsave(&zone->lock, flags);3504 guard(spinlock_irqsave)(&zone->lock);
3497 for (order = 0; order < NR_PAGE_ORDERS; order++) {3505 for (order = 0; order < NR_PAGE_ORDERS; order++) {
3498 struct free_area *area = &(zone->free_area[order]);3506 struct free_area *area = &(zone->free_area[order]);
3499 unsigned long size;3507 unsigned long size;
@@ -3540,12 +3548,9 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 40 unmodified lines
3540 * so this should not fail on zone boundaries.3548 * so this should not fail on zone boundaries.
3541 */3549 */
3542 WARN_ON_ONCE(ret == -1);3550 WARN_ON_ONCE(ret == -1);
3543 if (ret > 0) {3551 if (ret > 0)
3544 spin_unlock_irqrestore(&zone->lock, flags);
3545 return ret;3552 return ret;
3546 }
3547 }3553 }
3548 spin_unlock_irqrestore(&zone->lock, flags);
3549 }3554 }
3550 3555
3551 return false;3556 return false;
@@ -4156,7 +4161,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 604 unmodified lines
4156 psi_memstall_leave(&pflags);4161 psi_memstall_leave(&pflags);
4157 delayacct_compact_end();4162 delayacct_compact_end();
4158 4163
4159 if (*compact_result == COMPACT_SKIPPED)4164 if (*compact_result == COMPACT_SKIPPED ||
4165 *compact_result == COMPACT_DEFERRED)
4160 return NULL;4166 return NULL;
4161 /*4167 /*
4162 * At least in one zone compaction wasn't deferred or skipped, so let's4168 * At least in one zone compaction wasn't deferred or skipped, so let's
@@ -4193,7 +4199,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 30 unmodified lines
4193}4199}
4194 4200
4195static inline bool4201static inline bool
4196should_compact_retry(struct alloc_context *ac, int order, int alloc_flags,4202should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order,
4203 int alloc_flags,
4197 enum compact_result compact_result,4204 enum compact_result compact_result,
4198 enum compact_priority *compact_priority,4205 enum compact_priority *compact_priority,
4199 int *compaction_retries)4206 int *compaction_retries)
@@ -4215,7 +4222,8 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, · 15 unmodified lines
4215 * migration targets. Continue if reclaim can help.4222 * migration targets. Continue if reclaim can help.
4216 */4223 */
4217 if (compact_result == COMPACT_SKIPPED) {4224 if (compact_result == COMPACT_SKIPPED) {
4218 ret = compaction_zonelist_suitable(ac, order, alloc_flags);4225 ret = compaction_zonelist_suitable(ac, order, alloc_flags,
4226 gfp_mask);
4219 goto out;4227 goto out;
4220 }4228 }
4221 4229
@@ -4268,7 +4276,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 46 unmodified lines
4268}4276}
4269 4277
4270static inline bool4278static inline bool
4271should_compact_retry(struct alloc_context *ac, int order, int alloc_flags,4279should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order,
4280 int alloc_flags,
4272 enum compact_result compact_result,4281 enum compact_result compact_result,
4273 enum compact_priority *compact_priority,4282 enum compact_priority *compact_priority,
4274 int *compaction_retries)4283 int *compaction_retries)
@@ -4678,6 +4687,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · 403 unmodified lines
4678 return false;4687 return false;
4679}4688}
4680 4689
4690static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
4691 unsigned int order, unsigned long alloc_start_time)
4692{
4693 static DEFINE_SPINLOCK(alloc_stall_lock);
4694 unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
4695
4696 if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
4697 return;
4698 if (time_is_after_jiffies(READ_ONCE(alloc_stall_warn_jiffies)))
4699 return;
4700 if (gfp_mask & __GFP_NOWARN)
4701 return;
4702
4703 if (!spin_trylock(&alloc_stall_lock))
4704 return;
4705
4706 /* Check again, this time under the lock */
4707 if (time_is_after_jiffies(alloc_stall_warn_jiffies)) {
4708 spin_unlock(&alloc_stall_lock);
4709 return;
4710 }
4711
4712 WRITE_ONCE(alloc_stall_warn_jiffies, jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
4713 spin_unlock(&alloc_stall_lock);
4714
4715 pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
4716 current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
4717 nodemask_pr_args(nodemask));
4718 cpuset_print_current_mems_allowed();
4719 pr_cont("\n");
4720 dump_stack();
4721 warn_alloc_show_mem(gfp_mask, nodemask);
4722}
4723
4681static inline struct page *4724static inline struct page *
4682__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,4725__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
4683 struct alloc_context *ac)4726 struct alloc_context *ac)
@@ -4698,6 +4741,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 14 unmodified lines
4698 int reserve_flags;4741 int reserve_flags;
4699 bool compact_first = false;4742 bool compact_first = false;
4700 bool can_retry_reserves = true;4743 bool can_retry_reserves = true;
4744 unsigned long alloc_start_time = jiffies;
4701 4745
4702 if (unlikely(nofail)) {4746 if (unlikely(nofail)) {
4703 /*4747 /*
@@ -4806,13 +4850,27 @@ retry: · 102 unmodified lines
4806 }4850 }
4807 4851
4808 /* Caller is not willing to reclaim, we can't balance anything */4852 /* Caller is not willing to reclaim, we can't balance anything */
4809 if (!can_direct_reclaim)4853 if (!can_direct_reclaim) {
4854 /*
4855 * Reclaim/compaction cannot run, so defrag_mode's strategy
4856 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
4857 * fallbacks rather than failing the allocation outright.
4858 */
4859 if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
4860 (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
4861 alloc_flags &= ~ALLOC_NOFRAGMENT;
4862 goto retry;
4863 }
4810 goto nopage;4864 goto nopage;
4865 }
4811 4866
4812 /* Avoid recursion of direct reclaim */4867 /* Avoid recursion of direct reclaim */
4813 if (current->flags & PF_MEMALLOC)4868 if (current->flags & PF_MEMALLOC)
4814 goto nopage;4869 goto nopage;
4815 4870
4871 /* If allocation has taken excessively long, warn about it */
4872 check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
4873
4816 /* Try direct reclaim and then allocating */4874 /* Try direct reclaim and then allocating */
4817 if (!compact_first) {4875 if (!compact_first) {
4818 page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,4876 page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
@@ -4886,9 +4944,9 @@ retry: · 67 unmodified lines
4886 * of free memory (see __compaction_suitable)4944 * of free memory (see __compaction_suitable)
4887 */4945 */
4888 if (did_some_progress > 0 && can_compact &&4946 if (did_some_progress > 0 && can_compact &&
4889 should_compact_retry(ac, order, alloc_flags,4947 should_compact_retry(gfp_mask, ac, order, alloc_flags,
4890 compact_result, &compact_priority,4948 compact_result, &compact_priority,
4891 &compaction_retries))4949 &compaction_retries))
4892 goto retry;4950 goto retry;
4893 4951
4894 /* Reclaim/compaction failed to prevent the fallback */4952 /* Reclaim/compaction failed to prevent the fallback */
@@ -5044,7 +5102,6 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 149 unmodified lines
5044 struct per_cpu_pages *pcp;5102 struct per_cpu_pages *pcp;
5045 struct list_head *pcp_list;5103 struct list_head *pcp_list;
5046 struct alloc_context ac;5104 struct alloc_context ac;
5047 gfp_t alloc_gfp;
5048 unsigned int alloc_flags = ALLOC_WMARK_LOW;5105 unsigned int alloc_flags = ALLOC_WMARK_LOW;
5049 int nr_populated = 0, nr_account = 0;5106 int nr_populated = 0, nr_account = 0;
5050 5107
@@ -5085,10 +5142,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 34 unmodified lines
5085 5142
5086 /* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */5143 /* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
5087 gfp &= gfp_allowed_mask;5144 gfp &= gfp_allowed_mask;
5088 alloc_gfp = gfp;5145 if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
5089 if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &alloc_gfp, &alloc_flags))
5090 goto out;5146 goto out;
5091 gfp = alloc_gfp;
5092 5147
5093 /* Find an allowed local zone that meets the low watermark. */5148 /* Find an allowed local zone that meets the low watermark. */
5094 z = ac.preferred_zoneref;5149 z = ac.preferred_zoneref;
@@ -5179,6 +5234,34 @@ failed: · 84 unmodified lines
5179}5234}
5180EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);5235EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
5181 5236
5237/*
5238 * free_pages_bulk - Free an array of order-0 pages
5239 * @page_array: Array of pages to free
5240 * @nr_pages: The number of pages in the array
5241 *
5242 * Free the order-0 pages. Adjacent entries whose PFNs form a contiguous
5243 * run are released with a single __free_contig_range() call.
5244 *
5245 * This assumes page_array is sorted in ascending PFN order. Without that,
5246 * the function still frees all pages, but contiguous runs may not be
5247 * detected and the freeing pattern can degrade to freeing one page at a
5248 * time.
5249 *
5250 * Context: Sleepable process context only; calls cond_resched()
5251 */
5252void free_pages_bulk(struct page **page_array, unsigned long nr_pages)
5253{
5254 while (nr_pages) {
5255 unsigned long nr_contig = num_pages_contiguous(page_array, nr_pages);
5256
5257 __free_contig_range(page_to_pfn(*page_array), nr_contig);
5258
5259 nr_pages -= nr_contig;
5260 page_array += nr_contig;
5261 cond_resched();
5262 }
5263}
5264
5182/*5265/*
5183 * This is the 'heart' of the zoned buddy allocator.5266 * This is the 'heart' of the zoned buddy allocator.
5184 */5267 */
@@ -6758,6 +6841,105 @@ void __init page_alloc_sysctl_init(void) · 1573 unmodified lines
6758 register_sysctl_init("vm", page_alloc_sysctl_table);6841 register_sysctl_init("vm", page_alloc_sysctl_table);
6759}6842}
6760 6843
6844static void free_prepared_contig_range(struct page *page,
6845 unsigned long nr_pages)
6846{
6847 unsigned long pfn = page_to_pfn(page);
6848
6849 while (nr_pages) {
6850 unsigned int order;
6851
6852 /* We are limited by the largest buddy order. */
6853 order = pfn ? __ffs(pfn) : MAX_PAGE_ORDER;
6854 /* Don't exceed the number of pages to free. */
6855 order = min_t(unsigned int, order, ilog2(nr_pages));
6856 order = min_t(unsigned int, order, MAX_PAGE_ORDER);
6857
6858 /*
6859 * Free the chunk as a single block. Our caller has already
6860 * called free_pages_prepare() for each order-0 page.
6861 */
6862 __free_frozen_pages(page, order, FPI_PREPARED);
6863
6864 pfn += 1UL << order;
6865 page += 1UL << order;
6866 nr_pages -= 1UL << order;
6867 }
6868}
6869
6870static void __free_contig_range_common(unsigned long pfn, unsigned long nr_pages,
6871 bool is_frozen)
6872{
6873 struct page *page, *start = NULL;
6874 unsigned long nr_start = 0;
6875 unsigned long start_sec;
6876 unsigned long i;
6877
6878 for (i = 0; i < nr_pages; i++) {
6879 bool can_free = true;
6880
6881 /*
6882 * Contiguous PFNs might not have contiguous "struct pages"
6883 * in some kernel configs: page++ across a section boundary
6884 * is undefined. Use pfn_to_page() for each PFN.
6885 */
6886 page = pfn_to_page(pfn + i);
6887
6888 VM_WARN_ON_ONCE(PageHead(page));
6889 VM_WARN_ON_ONCE(PageTail(page));
6890
6891 if (!is_frozen)
6892 can_free = put_page_testzero(page);
6893
6894 if (can_free)
6895 can_free = free_pages_prepare(page, 0);
6896
6897 if (!can_free) {
6898 if (start) {
6899 free_prepared_contig_range(start, i - nr_start);
6900 start = NULL;
6901 }
6902 continue;
6903 }
6904
6905 if (start && memdesc_section(page->flags) != start_sec) {
6906 free_prepared_contig_range(start, i - nr_start);
6907 start = page;
6908 nr_start = i;
6909 start_sec = memdesc_section(page->flags);
6910 } else if (!start) {
6911 start = page;
6912 nr_start = i;
6913 start_sec = memdesc_section(page->flags);
6914 }
6915 }
6916
6917 if (start)
6918 free_prepared_contig_range(start, nr_pages - nr_start);
6919}
6920
6921/**
6922 * __free_contig_range - Free contiguous range of order-0 pages.
6923 * @pfn: Page frame number of the first page in the range.
6924 * @nr_pages: Number of pages to free.
6925 *
6926 * For each order-0 struct page in the physically contiguous range, put a
6927 * reference. Free any page who's reference count falls to zero. The
6928 * implementation is functionally equivalent to, but significantly faster than
6929 * calling __free_page() for each struct page in a loop.
6930 *
6931 * Memory allocated with alloc_pages(order>=1) then subsequently split to
6932 * order-0 with split_page() is an example of appropriate contiguous pages that
6933 * can be freed with this API.
6934 *
6935 * Context: May be called in interrupt context or while holding a normal
6936 * spinlock, but not in NMI context or while holding a raw spinlock.
6937 */
6938void __free_contig_range(unsigned long pfn, unsigned long nr_pages)
6939{
6940 __free_contig_range_common(pfn, nr_pages, /* is_frozen= */ false);
6941}
6942
6761#ifdef CONFIG_CONTIG_ALLOC6943#ifdef CONFIG_CONTIG_ALLOC
6762/* Usage: See admin-guide/dynamic-debug-howto.rst */6944/* Usage: See admin-guide/dynamic-debug-howto.rst */
6763static void alloc_contig_dump_pages(struct list_head *page_list)6945static void alloc_contig_dump_pages(struct list_head *page_list)
@@ -6895,8 +7077,7 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 131 unmodified lines
6895 7077
6896static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)7078static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)
6897{7079{
6898 for (; nr_pages--; pfn++)7080 __free_contig_range_common(pfn, nr_pages, /* is_frozen= */ true);
6899 free_frozen_pages(pfn_to_page(pfn), 0);
6900}7081}
6901 7082
6902/**7083/**
@@ -7304,8 +7485,7 @@ void free_contig_range(unsigned long pfn, unsigned long nr_pages) · 401 unmodified lines
7304 if (WARN_ON_ONCE(PageHead(pfn_to_page(pfn))))7485 if (WARN_ON_ONCE(PageHead(pfn_to_page(pfn))))
7305 return;7486 return;
7306 7487
7307 for (; nr_pages--; pfn++)7488 __free_contig_range(pfn, nr_pages);
7308 __free_page(pfn_to_page(pfn));
7309}7489}
7310EXPORT_SYMBOL(free_contig_range);7490EXPORT_SYMBOL(free_contig_range);
7311#endif /* CONFIG_CONTIG_ALLOC */7491#endif /* CONFIG_CONTIG_ALLOC */
@@ -7363,7 +7543,7 @@ void zone_pcp_reset(struct zone *zone) · 51 unmodified lines
7363unsigned long __offline_isolated_pages(unsigned long start_pfn,7543unsigned long __offline_isolated_pages(unsigned long start_pfn,
7364 unsigned long end_pfn)7544 unsigned long end_pfn)
7365{7545{
7366 unsigned long already_offline = 0, flags;7546 unsigned long already_offline = 0;
7367 unsigned long pfn = start_pfn;7547 unsigned long pfn = start_pfn;
7368 struct page *page;7548 struct page *page;
7369 struct zone *zone;7549 struct zone *zone;
@@ -7371,7 +7551,7 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn, · 1 unmodified line
7371 7551
7372 offline_mem_sections(pfn, end_pfn);7552 offline_mem_sections(pfn, end_pfn);
7373 zone = page_zone(pfn_to_page(pfn));7553 zone = page_zone(pfn_to_page(pfn));
7374 spin_lock_irqsave(&zone->lock, flags);7554 guard(spinlock_irqsave)(&zone->lock);
7375 while (pfn < end_pfn) {7555 while (pfn < end_pfn) {
7376 page = pfn_to_page(pfn);7556 page = pfn_to_page(pfn);
7377 /*7557 /*
@@ -7401,7 +7581,6 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn, · 23 unmodified lines
7401 del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE);7581 del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE);
7402 pfn += (1 << order);7582 pfn += (1 << order);
7403 }7583 }
7404 spin_unlock_irqrestore(&zone->lock, flags);
7405 7584
7406 return end_pfn - start_pfn - already_offline;7585 return end_pfn - start_pfn - already_offline;
7407}7586}
@@ -7473,11 +7652,9 @@ bool take_page_off_buddy(struct page *page) · 65 unmodified lines
7473{7652{
7474 struct zone *zone = page_zone(page);7653 struct zone *zone = page_zone(page);
7475 unsigned long pfn = page_to_pfn(page);7654 unsigned long pfn = page_to_pfn(page);
7476 unsigned long flags;
7477 unsigned int order;7655 unsigned int order;
7478 bool ret = false;
7479 7656
7480 spin_lock_irqsave(&zone->lock, flags);7657 guard(spinlock_irqsave)(&zone->lock);
7481 for (order = 0; order < NR_PAGE_ORDERS; order++) {7658 for (order = 0; order < NR_PAGE_ORDERS; order++) {
7482 struct page *page_head = page - (pfn & ((1 << order) - 1));7659 struct page *page_head = page - (pfn & ((1 << order) - 1));
7483 int page_order = buddy_order(page_head);7660 int page_order = buddy_order(page_head);
@@ -7492,14 +7669,12 @@ bool take_page_off_buddy(struct page *page) · 8 unmodified lines
7492 break_down_buddy_pages(zone, page_head, page, 0,7669 break_down_buddy_pages(zone, page_head, page, 0,
7493 page_order, migratetype);7670 page_order, migratetype);
7494 SetPageHWPoisonTakenOff(page);7671 SetPageHWPoisonTakenOff(page);
7495 ret = true;7672 return true;
7496 break;
7497 }7673 }
7498 if (page_count(page_head) > 0)7674 if (page_count(page_head) > 0)
7499 break;7675 break;
7500 }7676 }
7501 spin_unlock_irqrestore(&zone->lock, flags);7677 return false;
7502 return ret;
7503}7678}
7504 7679
7505/*7680/*
@@ -7508,23 +7683,19 @@ bool take_page_off_buddy(struct page *page) · 2 unmodified lines
7508bool put_page_back_buddy(struct page *page)7683bool put_page_back_buddy(struct page *page)
7509{7684{
7510 struct zone *zone = page_zone(page);7685 struct zone *zone = page_zone(page);
7511 unsigned long flags;
7512 bool ret = false;
7513 7686
7514 spin_lock_irqsave(&zone->lock, flags);7687 guard(spinlock_irqsave)(&zone->lock);
7515 if (put_page_testzero(page)) {7688 if (put_page_testzero(page)) {
7516 unsigned long pfn = page_to_pfn(page);7689 unsigned long pfn = page_to_pfn(page);
7517 int migratetype = get_pfnblock_migratetype(page, pfn);7690 int migratetype = get_pfnblock_migratetype(page, pfn);
7518 7691
7519 ClearPageHWPoisonTakenOff(page);7692 ClearPageHWPoisonTakenOff(page);
7520 __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE);7693 __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE);
7521 if (TestClearPageHWPoison(page)) {7694 if (TestClearPageHWPoison(page))
7522 ret = true;7695 return true;
7523 }
7524 }7696 }
7525 spin_unlock_irqrestore(&zone->lock, flags);
7526 7697
7527 return ret;7698 return false;
7528}7699}
7529#endif7700#endif
7530 7701
@@ -7774,8 +7945,8 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned · 243 unmodified lines
7774 * @order: allocation order size7945 * @order: allocation order size
7775 *7946 *
7776 * Allocates pages of a given order from the given node. This is safe to7947 * Allocates pages of a given order from the given node. This is safe to
7777 * call from any context (from atomic, NMI, and also reentrant7948 * call from any context where RCU is watching (from atomic, NMI, and also
7778 * allocator -> tracepoint -> alloc_pages_nolock_noprof).7949 * reentrant allocator -> tracepoint -> alloc_pages_nolock_noprof).
7779 * Allocation is best effort and to be expected to fail easily so nobody should7950 * Allocation is best effort and to be expected to fail easily so nobody should
7780 * rely on the success. Failures are not reported via warn_alloc().7951 * rely on the success. Failures are not reported via warn_alloc().
7781 * See always fail conditions below.7952 * See always fail conditions below.
mm/page_io.c+73−30
@@ -26,6 +26,7 @@ · 25 unmodified lines
26#include <linux/delayacct.h>26#include <linux/delayacct.h>
27#include <linux/zswap.h>27#include <linux/zswap.h>
28#include "swap.h"28#include "swap.h"
29#include "swap_table.h"
29 30
30static void __end_swap_bio_write(struct bio *bio)31static void __end_swap_bio_write(struct bio *bio)
31{32{
@@ -204,15 +205,20 @@ static bool is_folio_zero_filled(struct folio *folio) · 172 unmodified lines
204static void swap_zeromap_folio_set(struct folio *folio)205static void swap_zeromap_folio_set(struct folio *folio)
205{206{
206 struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);207 struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
207 struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
208 int nr_pages = folio_nr_pages(folio);208 int nr_pages = folio_nr_pages(folio);
209 struct swap_cluster_info *ci;
209 swp_entry_t entry;210 swp_entry_t entry;
210 unsigned int i;211 unsigned int i;
211 212
213 VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
214 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
215
216 ci = swap_cluster_get_and_lock(folio);
212 for (i = 0; i < folio_nr_pages(folio); i++) {217 for (i = 0; i < folio_nr_pages(folio); i++) {
213 entry = page_swap_entry(folio_page(folio, i));218 entry = page_swap_entry(folio_page(folio, i));
214 set_bit(swp_offset(entry), sis->zeromap);219 __swap_table_set_zero(ci, swp_cluster_offset(entry));
215 }220 }
221 swap_cluster_unlock(ci);
216 222
217 count_vm_events(SWPOUT_ZERO, nr_pages);223 count_vm_events(SWPOUT_ZERO, nr_pages);
218 if (objcg) {224 if (objcg) {
@@ -223,14 +229,19 @@ static void swap_zeromap_folio_set(struct folio *folio) · 4 unmodified lines
223 229
224static void swap_zeromap_folio_clear(struct folio *folio)230static void swap_zeromap_folio_clear(struct folio *folio)
225{231{
226 struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);232 struct swap_cluster_info *ci;
227 swp_entry_t entry;233 swp_entry_t entry;
228 unsigned int i;234 unsigned int i;
229 235
236 VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
237 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
238
239 ci = swap_cluster_get_and_lock(folio);
230 for (i = 0; i < folio_nr_pages(folio); i++) {240 for (i = 0; i < folio_nr_pages(folio); i++) {
231 entry = page_swap_entry(folio_page(folio, i));241 entry = page_swap_entry(folio_page(folio, i));
232 clear_bit(swp_offset(entry), sis->zeromap);242 __swap_table_clear_zero(ci, swp_cluster_offset(entry));
233 }243 }
244 swap_cluster_unlock(ci);
234}245}
235 246
236/*247/*
@@ -255,10 +266,9 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug) · 18 unmodified lines
255 }266 }
256 267
257 /*268 /*
258 * Use a bitmap (zeromap) to avoid doing IO for zero-filled pages.269 * Use the swap table zero mark to avoid doing IO for zero-filled
259 * The bits in zeromap are protected by the locked swapcache folio270 * pages. The zero mark is protected by the cluster lock, which is
260 * and atomic updates are used to protect against read-modify-write271 * acquired internally by swap_zeromap_folio_set/clear.
261 * corruption due to other zero swap entries seeing concurrent updates.
262 */272 */
263 if (is_folio_zero_filled(folio)) {273 if (is_folio_zero_filled(folio)) {
264 swap_zeromap_folio_set(folio);274 swap_zeromap_folio_set(folio);
@@ -326,8 +336,8 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) · 61 unmodified lines
326 336
327struct swap_iocb {337struct swap_iocb {
328 struct kiocb iocb;338 struct kiocb iocb;
329 struct bio_vec bvec[SWAP_CLUSTER_MAX];339 struct bio_vec bvecs[SWAP_CLUSTER_MAX];
330 int pages;340 int nr_bvecs;
331 int len;341 int len;
332};342};
333static mempool_t *sio_pool;343static mempool_t *sio_pool;
@@ -348,7 +358,7 @@ int sio_pool_init(void) · 14 unmodified lines
348static void sio_write_complete(struct kiocb *iocb, long ret)358static void sio_write_complete(struct kiocb *iocb, long ret)
349{359{
350 struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);360 struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
351 struct page *page = sio->bvec[0].bv_page;361 struct page *page = sio->bvecs[0].bv_page;
352 int p;362 int p;
353 363
354 if (ret != sio->len) {364 if (ret != sio->len) {
@@ -362,15 +372,15 @@ static void sio_write_complete(struct kiocb *iocb, long ret) · 7 unmodified lines
362 */372 */
363 pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",373 pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
364 ret, swap_dev_pos(page_swap_entry(page)));374 ret, swap_dev_pos(page_swap_entry(page)));
365 for (p = 0; p < sio->pages; p++) {375 for (p = 0; p < sio->nr_bvecs; p++) {
366 page = sio->bvec[p].bv_page;376 page = sio->bvecs[p].bv_page;
367 set_page_dirty(page);377 set_page_dirty(page);
368 ClearPageReclaim(page);378 ClearPageReclaim(page);
369 }379 }
370 }380 }
371 381
372 for (p = 0; p < sio->pages; p++)382 for (p = 0; p < sio->nr_bvecs; p++)
373 end_page_writeback(sio->bvec[p].bv_page);383 end_page_writeback(sio->bvecs[p].bv_page);
374 384
375 mempool_free(sio, sio_pool);385 mempool_free(sio, sio_pool);
376}386}
@@ -397,13 +407,13 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) · 20 unmodified lines
397 init_sync_kiocb(&sio->iocb, swap_file);407 init_sync_kiocb(&sio->iocb, swap_file);
398 sio->iocb.ki_complete = sio_write_complete;408 sio->iocb.ki_complete = sio_write_complete;
399 sio->iocb.ki_pos = pos;409 sio->iocb.ki_pos = pos;
400 sio->pages = 0;410 sio->nr_bvecs = 0;
401 sio->len = 0;411 sio->len = 0;
402 }412 }
403 bvec_set_folio(&sio->bvec[sio->pages], folio, folio_size(folio), 0);413 bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
404 sio->len += folio_size(folio);414 sio->len += folio_size(folio);
405 sio->pages += 1;415 sio->nr_bvecs += 1;
406 if (sio->pages == ARRAY_SIZE(sio->bvec) || !swap_plug) {416 if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !swap_plug) {
407 swap_write_unplug(sio);417 swap_write_unplug(sio);
408 sio = NULL;418 sio = NULL;
409 }419 }
@@ -477,7 +487,7 @@ void swap_write_unplug(struct swap_iocb *sio) · 67 unmodified lines
477 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;487 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
478 int ret;488 int ret;
479 489
480 iov_iter_bvec(&from, ITER_SOURCE, sio->bvec, sio->pages, sio->len);490 iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len);
481 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);491 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
482 if (ret != -EIOCBQUEUED)492 if (ret != -EIOCBQUEUED)
483 sio_write_complete(&sio->iocb, ret);493 sio_write_complete(&sio->iocb, ret);
@@ -489,8 +499,8 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 5 unmodified lines
489 int p;499 int p;
490 500
491 if (ret == sio->len) {501 if (ret == sio->len) {
492 for (p = 0; p < sio->pages; p++) {502 for (p = 0; p < sio->nr_bvecs; p++) {
493 struct folio *folio = bvec_folio(&sio->bvec[p]);503 struct folio *folio = bvec_folio(&sio->bvecs[p]);
494 504
495 count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);505 count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
496 count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));506 count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
@@ -499,8 +509,8 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 2 unmodified lines
499 }509 }
500 count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);510 count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
501 } else {511 } else {
502 for (p = 0; p < sio->pages; p++) {512 for (p = 0; p < sio->nr_bvecs; p++) {
503 struct folio *folio = bvec_folio(&sio->bvec[p]);513 struct folio *folio = bvec_folio(&sio->bvecs[p]);
504 514
505 folio_unlock(folio);515 folio_unlock(folio);
506 }516 }
@@ -509,19 +519,52 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 2 unmodified lines
509 mempool_free(sio, sio_pool);519 mempool_free(sio, sio_pool);
510}520}
511 521
522/*
523 * Return the count of contiguous swap entries that share the same
524 * zeromap status as the starting entry. If is_zerop is not NULL,
525 * it will return the zeromap status of the starting entry.
526 *
527 * Context: Caller must ensure the cluster containing the entries
528 * that will be checked won't be freed.
529 */
530static int swap_zeromap_batch(swp_entry_t entry, int max_nr,
531 bool *is_zerop)
532{
533 int i;
534 bool is_zero;
535 unsigned int ci_start = swp_cluster_offset(entry);
536 struct swap_cluster_info *ci = __swap_entry_to_cluster(entry);
537
538 VM_WARN_ON_ONCE(ci_start + max_nr > SWAPFILE_CLUSTER);
539
540 rcu_read_lock();
541 is_zero = __swap_table_test_zero(ci, ci_start);
542 for (i = 1; i < max_nr; i++)
543 if (is_zero != __swap_table_test_zero(ci, ci_start + i))
544 break;
545 rcu_read_unlock();
546 if (is_zerop)
547 *is_zerop = is_zero;
548
549 return i;
550}
551
512static bool swap_read_folio_zeromap(struct folio *folio)552static bool swap_read_folio_zeromap(struct folio *folio)
513{553{
514 int nr_pages = folio_nr_pages(folio);554 int nr_pages = folio_nr_pages(folio);
515 struct obj_cgroup *objcg;555 struct obj_cgroup *objcg;
516 bool is_zeromap;556 bool is_zeromap;
517 557
558 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
559
518 /*560 /*
519 * Swapping in a large folio that is partially in the zeromap is not561 * Swapping in a large folio that is partially in the zeromap is not
520 * currently handled. Return true without marking the folio uptodate so562 * currently handled. Return true without marking the folio uptodate so
521 * that an IO error is emitted (e.g. do_swap_page() will sigbus).563 * that an IO error is emitted (e.g. do_swap_page() will sigbus).
564 * Folio lock stabilizes the cluster and map, so the check is safe.
522 */565 */
523 if (WARN_ON_ONCE(swap_zeromap_batch(folio->swap, nr_pages,566 if (WARN_ON_ONCE(swap_zeromap_batch(folio->swap, nr_pages,
524 &is_zeromap) != nr_pages))567 &is_zeromap) != nr_pages))
525 return true;568 return true;
526 569
527 if (!is_zeromap)570 if (!is_zeromap)
@@ -559,13 +602,13 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug) · 31 unmodified lines
559 init_sync_kiocb(&sio->iocb, sis->swap_file);602 init_sync_kiocb(&sio->iocb, sis->swap_file);
560 sio->iocb.ki_pos = pos;603 sio->iocb.ki_pos = pos;
561 sio->iocb.ki_complete = sio_read_complete;604 sio->iocb.ki_complete = sio_read_complete;
562 sio->pages = 0;605 sio->nr_bvecs = 0;
563 sio->len = 0;606 sio->len = 0;
564 }607 }
565 bvec_set_folio(&sio->bvec[sio->pages], folio, folio_size(folio), 0);608 bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
566 sio->len += folio_size(folio);609 sio->len += folio_size(folio);
567 sio->pages += 1;610 sio->nr_bvecs += 1;
568 if (sio->pages == ARRAY_SIZE(sio->bvec) || !plug) {611 if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !plug) {
569 swap_read_unplug(sio);612 swap_read_unplug(sio);
570 sio = NULL;613 sio = NULL;
571 }614 }
@@ -666,7 +709,7 @@ void __swap_read_unplug(struct swap_iocb *sio) · 94 unmodified lines
666 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;709 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
667 int ret;710 int ret;
668 711
669 iov_iter_bvec(&from, ITER_DEST, sio->bvec, sio->pages, sio->len);712 iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len);
670 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);713 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
671 if (ret != -EIOCBQUEUED)714 if (ret != -EIOCBQUEUED)
672 sio_read_complete(&sio->iocb, ret);715 sio_read_complete(&sio->iocb, ret);
mm/page_isolation.c+28−39
@@ -167,48 +167,40 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode, · 166 unmodified lines
167{167{
168 struct zone *zone = page_zone(page);168 struct zone *zone = page_zone(page);
169 struct page *unmovable;169 struct page *unmovable;
170 unsigned long flags;
171 unsigned long check_unmovable_start, check_unmovable_end;170 unsigned long check_unmovable_start, check_unmovable_end;
172 171
173 if (PageUnaccepted(page))172 if (PageUnaccepted(page))
174 accept_page(page);173 accept_page(page);
175 174
176 spin_lock_irqsave(&zone->lock, flags);175 scoped_guard(spinlock_irqsave, &zone->lock) {
177 176 /*
178 /*177 * We assume the caller intended to SET migrate type to
179 * We assume the caller intended to SET migrate type to isolate.178 * isolate. If it is already set, then someone else must have
180 * If it is already set, then someone else must have raced and179 * raced and set it before us.
181 * set it before us.180 */
182 */181 if (is_migrate_isolate_page(page))
183 if (is_migrate_isolate_page(page)) {
184 spin_unlock_irqrestore(&zone->lock, flags);
185 return -EBUSY;
186 }
187
188 /*
189 * FIXME: Now, memory hotplug doesn't call shrink_slab() by itself.
190 * We just check MOVABLE pages.
191 *
192 * Pass the intersection of [start_pfn, end_pfn) and the page's pageblock
193 * to avoid redundant checks.
194 */
195 check_unmovable_start = max(page_to_pfn(page), start_pfn);
196 check_unmovable_end = min(pageblock_end_pfn(page_to_pfn(page)),
197 end_pfn);
198
199 unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end,
200 mode);
201 if (!unmovable) {
202 if (!pageblock_isolate_and_move_free_pages(zone, page)) {
203 spin_unlock_irqrestore(&zone->lock, flags);
204 return -EBUSY;182 return -EBUSY;
183
184 /*
185 * FIXME: Now, memory hotplug doesn't call shrink_slab() by
186 * itself. We just check MOVABLE pages.
187 *
188 * Pass the intersection of [start_pfn, end_pfn) and the page's
189 * pageblock to avoid redundant checks.
190 */
191 check_unmovable_start = max(page_to_pfn(page), start_pfn);
192 check_unmovable_end = min(pageblock_end_pfn(page_to_pfn(page)),
193 end_pfn);
194
195 unmovable = has_unmovable_pages(check_unmovable_start,
196 check_unmovable_end, mode);
197 if (!unmovable) {
198 if (!pageblock_isolate_and_move_free_pages(zone, page))
199 return -EBUSY;
200 zone->nr_isolate_pageblock++;
201 return 0;
205 }202 }
206 zone->nr_isolate_pageblock++;
207 spin_unlock_irqrestore(&zone->lock, flags);
208 return 0;
209 }203 }
210
211 spin_unlock_irqrestore(&zone->lock, flags);
212 if (mode == PB_ISOLATE_MODE_MEM_OFFLINE) {204 if (mode == PB_ISOLATE_MODE_MEM_OFFLINE) {
213 /*205 /*
214 * printk() with zone->lock held will likely trigger a206 * printk() with zone->lock held will likely trigger a
@@ -223,15 +215,14 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode, · 8 unmodified lines
223static void unset_migratetype_isolate(struct page *page)215static void unset_migratetype_isolate(struct page *page)
224{216{
225 struct zone *zone;217 struct zone *zone;
226 unsigned long flags;
227 bool isolated_page = false;218 bool isolated_page = false;
228 unsigned int order;219 unsigned int order;
229 struct page *buddy;220 struct page *buddy;
230 221
231 zone = page_zone(page);222 zone = page_zone(page);
232 spin_lock_irqsave(&zone->lock, flags);223 guard(spinlock_irqsave)(&zone->lock);
233 if (!is_migrate_isolate_page(page))224 if (!is_migrate_isolate_page(page))
234 goto out;225 return;
235 226
236 /*227 /*
237 * Because freepage with more than pageblock_order on isolated228 * Because freepage with more than pageblock_order on isolated
@@ -279,8 +270,6 @@ static void unset_migratetype_isolate(struct page *page) · 41 unmodified lines
279 __putback_isolated_page(page, order, get_pageblock_migratetype(page));270 __putback_isolated_page(page, order, get_pageblock_migratetype(page));
280 }271 }
281 zone->nr_isolate_pageblock--;272 zone->nr_isolate_pageblock--;
282out:
283 spin_unlock_irqrestore(&zone->lock, flags);
284}273}
285 274
286static inline struct page *275static inline struct page *
mm/page_owner.c+1−1
@@ -573,7 +573,7 @@ print_page_owner(char __user *buf, size_t count, unsigned long pfn, · 572 unmodified lines
573 migratetype_names[page_mt],573 migratetype_names[page_mt],
574 pfn >> pageblock_order,574 pfn >> pageblock_order,
575 migratetype_names[pageblock_mt],575 migratetype_names[pageblock_mt],
576 &page->flags);576 &page->flags.f);
577 577
578 ret += stack_depot_snprint(handle, kbuf + ret, count - ret, 0);578 ret += stack_depot_snprint(handle, kbuf + ret, count - ret, 0);
579 if (ret >= count)579 if (ret >= count)
mm/page_vma_mapped.c+7−2
@@ -41,7 +41,7 @@ again: · 40 unmodified lines
41 if (!pvmw->pte)41 if (!pvmw->pte)
42 return false;42 return false;
43 43
44 ptent = ptep_get(pvmw->pte);44 ptent = ptep_get_lockless(pvmw->pte);
45 45
46 if (pte_none(ptent)) {46 if (pte_none(ptent)) {
47 return false;47 return false;
@@ -183,6 +183,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · 135 unmodified lines
183 struct mm_struct *mm = vma->vm_mm;183 struct mm_struct *mm = vma->vm_mm;
184 unsigned long end;184 unsigned long end;
185 spinlock_t *ptl;185 spinlock_t *ptl;
186 pte_t pteval;
186 pgd_t *pgd;187 pgd_t *pgd;
187 p4d_t *p4d;188 p4d_t *p4d;
188 pud_t *pud;189 pud_t *pud;
@@ -310,7 +311,11 @@ next_pte: · 121 unmodified lines
310 goto restart;311 goto restart;
311 }312 }
312 pvmw->pte++;313 pvmw->pte++;
313 } while (pte_none(ptep_get(pvmw->pte)));314 if (!pvmw->ptl)
315 pteval = ptep_get_lockless(pvmw->pte);
316 else
317 pteval = ptep_get(pvmw->pte);
318 } while (pte_none(pteval));
314 319
315 if (!pvmw->ptl) {320 if (!pvmw->ptl) {
316 spin_lock(ptl);321 spin_lock(ptl);
mm/percpu-internal.h+3−3
@@ -77,13 +77,13 @@ struct pcpu_chunk { · 76 unmodified lines
77 int end_offset; /* additional area required to77 int end_offset; /* additional area required to
78 have the region end page78 have the region end page
79 aligned */79 aligned */
80 int nr_pages; /* # of pages served by this chunk */
81 int nr_populated; /* # of populated pages */
82 int nr_empty_pop_pages; /* # of empty populated pages */
80#ifdef NEED_PCPUOBJ_EXT83#ifdef NEED_PCPUOBJ_EXT
81 struct pcpuobj_ext *obj_exts; /* vector of object cgroups */84 struct pcpuobj_ext *obj_exts; /* vector of object cgroups */
82#endif85#endif
83 86
84 int nr_pages; /* # of pages served by this chunk */
85 int nr_populated; /* # of populated pages */
86 int nr_empty_pop_pages; /* # of empty populated pages */
87 unsigned long populated[]; /* populated bitmap */87 unsigned long populated[]; /* populated bitmap */
88};88};
89 89
mm/readahead.c+30−6
@@ -146,6 +146,17 @@ file_ra_state_init(struct file_ra_state *ra, struct address_space *mapping) · 145 unmodified lines
146}146}
147EXPORT_SYMBOL_GPL(file_ra_state_init);147EXPORT_SYMBOL_GPL(file_ra_state_init);
148 148
149/**
150 * read_pages() - Start IO for a contiguous range of allocated folios in the
151 * page cache.
152 * @rac: Readahead control.
153 *
154 * When read_pages() returns, it is guaranteed that all of the folios will have
155 * been processed or removed so that ``readahead_count(rac) == 0``. However,
156 * that does not imply that ``readahead_index(rac)`` will be updated to point
157 * to the end of the originally requested range because, for example, the
158 * filesystem may expand the range upwards.
159 */
149static void read_pages(struct readahead_control *rac)160static void read_pages(struct readahead_control *rac)
150{161{
151 const struct address_space_operations *aops = rac->mapping->a_ops;162 const struct address_space_operations *aops = rac->mapping->a_ops;
@@ -270,7 +281,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, · 118 unmodified lines
270 */281 */
271 read_pages(ractl);282 read_pages(ractl);
272 ractl->_index += min_nrpages;283 ractl->_index += min_nrpages;
273 i = ractl->_index + ractl->_nr_pages - index;284 i = ractl->_index - index;
274 continue;285 continue;
275 }286 }
276 287
@@ -286,7 +297,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, · 9 unmodified lines
286 break;297 break;
287 read_pages(ractl);298 read_pages(ractl);
288 ractl->_index += min_nrpages;299 ractl->_index += min_nrpages;
289 i = ractl->_index + ractl->_nr_pages - index;300 i = ractl->_index - index;
290 continue;301 continue;
291 }302 }
292 if (i == mark)303 if (i == mark)
@@ -324,11 +335,16 @@ static void do_page_cache_ra(struct readahead_control *ractl, · 31 unmodified lines
324 return;335 return;
325 336
326 end_index = (isize - 1) >> PAGE_SHIFT;337 end_index = (isize - 1) >> PAGE_SHIFT;
338 if (end_index > ractl->_max_index)
339 end_index = ractl->_max_index;
327 if (index > end_index)340 if (index > end_index)
328 return;341 return;
329 /* Don't read past the page containing the last byte of the file */342 /* Don't read past the page containing the last byte of the file */
330 if (nr_to_read > end_index - index)343 if (nr_to_read > end_index - index) {
331 nr_to_read = end_index - index + 1;344 nr_to_read = end_index - index + 1;
345 /* We've reached the end, so don't set a readahead marker. */
346 lookahead_size = 0;
347 }
332 348
333 filemap_invalidate_lock_shared(mapping);349 filemap_invalidate_lock_shared(mapping);
334 page_cache_ra_unbounded(ractl, nr_to_read, lookahead_size);350 page_cache_ra_unbounded(ractl, nr_to_read, lookahead_size);
@@ -471,8 +487,8 @@ void page_cache_ra_order(struct readahead_control *ractl, · 136 unmodified lines
471 pgoff_t start = readahead_index(ractl);487 pgoff_t start = readahead_index(ractl);
472 pgoff_t index = start;488 pgoff_t index = start;
473 unsigned int min_order = mapping_min_folio_order(mapping);489 unsigned int min_order = mapping_min_folio_order(mapping);
474 pgoff_t limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;490 pgoff_t limit;
475 pgoff_t mark = index + ra->size - ra->async_size;491 pgoff_t mark;
476 unsigned int nofs;492 unsigned int nofs;
477 int err = 0;493 int err = 0;
478 gfp_t gfp = readahead_gfp_mask(mapping);494 gfp_t gfp = readahead_gfp_mask(mapping);
@@ -484,7 +500,15 @@ void page_cache_ra_order(struct readahead_control *ractl, · 5 unmodified lines
484 goto fallback;500 goto fallback;
485 }501 }
486 502
487 limit = min(limit, index + ra->size - 1);503 limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
504 limit = min(limit, ractl->_max_index);
505 if (limit > index + ra->size - 1) {
506 limit = index + ra->size - 1;
507 mark = index + ra->size - ra->async_size;
508 } else {
509 /* We've reached the end, so don't set a readahead marker. */
510 mark = ULONG_MAX;
511 }
488 512
489 new_order = min(mapping_max_folio_order(mapping), new_order);513 new_order = min(mapping_max_folio_order(mapping), new_order);
490 new_order = min_t(unsigned int, new_order, ilog2(ra->size));514 new_order = min_t(unsigned int, new_order, ilog2(ra->size));
mm/rmap.c+4−4
@@ -571,7 +571,7 @@ void __init anon_vma_init(void) · 570 unmodified lines
571 * In case it was remapped to a different anon_vma, the new anon_vma will be a571 * In case it was remapped to a different anon_vma, the new anon_vma will be a
572 * child of the old anon_vma, and the anon_vma lifetime rules will therefore572 * child of the old anon_vma, and the anon_vma lifetime rules will therefore
573 * ensure that any anon_vma obtained from the page will still be valid for as573 * ensure that any anon_vma obtained from the page will still be valid for as
574 * long as we observe page_mapped() [ hence all those page_mapped() tests ].574 * long as we observe folio_mapped() [ hence all those folio_mapped() tests ].
575 *575 *
576 * All users of this function must be very careful when walking the anon_vma576 * All users of this function must be very careful when walking the anon_vma
577 * chain and verify that the page in question is indeed mapped in it577 * chain and verify that the page in question is indeed mapped in it
@@ -1999,7 +1999,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · 1421 unmodified lines
1999 /*1999 /*
2000 * When racing against e.g. zap_pte_range() on another cpu,2000 * When racing against e.g. zap_pte_range() on another cpu,
2001 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),2001 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),
2002 * try_to_unmap() may return before page_mapped() has become false,2002 * try_to_unmap() may return before folio_mapped() has become false,
2003 * if page table locking is skipped: use TTU_SYNC to wait for that.2003 * if page table locking is skipped: use TTU_SYNC to wait for that.
2004 */2004 */
2005 if (flags & TTU_SYNC)2005 if (flags & TTU_SYNC)
@@ -2428,7 +2428,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · 422 unmodified lines
2428 /*2428 /*
2429 * When racing against e.g. zap_pte_range() on another cpu,2429 * When racing against e.g. zap_pte_range() on another cpu,
2430 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),2430 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),
2431 * try_to_migrate() may return before page_mapped() has become false,2431 * try_to_migrate() may return before folio_mapped() has become false,
2432 * if page table locking is skipped: use TTU_SYNC to wait for that.2432 * if page table locking is skipped: use TTU_SYNC to wait for that.
2433 */2433 */
2434 if (flags & TTU_SYNC)2434 if (flags & TTU_SYNC)
@@ -2929,7 +2929,7 @@ static struct anon_vma *rmap_walk_anon_lock(const struct folio *folio, · 494 unmodified lines
2929 2929
2930 /*2930 /*
2931 * Note: remove_migration_ptes() cannot use folio_lock_anon_vma_read()2931 * Note: remove_migration_ptes() cannot use folio_lock_anon_vma_read()
2932 * because that depends on page_mapped(); but not all its usages2932 * because that depends on folio_mapped(); but not all its usages
2933 * are holding mmap_lock. Users without mmap_lock are required to2933 * are holding mmap_lock. Users without mmap_lock are required to
2934 * take a reference count to prevent the anon_vma disappearing2934 * take a reference count to prevent the anon_vma disappearing
2935 */2935 */
mm/shmem.c+119−159
@@ -159,7 +159,7 @@ static unsigned long shmem_default_max_inodes(void) · 158 unmodified lines
159 159
160static int shmem_swapin_folio(struct inode *inode, pgoff_t index,160static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
161 struct folio **foliop, enum sgp_type sgp, gfp_t gfp,161 struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
162 struct vm_area_struct *vma, vm_fault_t *fault_type);162 struct vm_fault *vmf, vm_fault_t *fault_type);
163 163
164static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)164static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
165{165{
@@ -1789,30 +1789,6 @@ static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, · 1623 unmodified lines
1789 return folio;1789 return folio;
1790}1790}
1791 1791
1792/*
1793 * Make sure huge_gfp is always more limited than limit_gfp.
1794 * Some of the flags set permissions, while others set limitations.
1795 */
1796static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
1797{
1798 gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
1799 gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
1800 gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
1801 gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
1802
1803 /* Allow allocations only from the originally specified zones. */
1804 result |= zoneflags;
1805
1806 /*
1807 * Minimize the result gfp by taking the union with the deny flags,
1808 * and the intersection of the allow flags.
1809 */
1810 result |= (limit_gfp & denyflags);
1811 result |= (huge_gfp & limit_gfp) & allowflags;
1812
1813 return result;
1814}
1815
1816#ifdef CONFIG_TRANSPARENT_HUGEPAGE1792#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1817bool shmem_hpage_pmd_enabled(void)1793bool shmem_hpage_pmd_enabled(void)
1818{1794{
@@ -2039,68 +2015,32 @@ unlock: · 220 unmodified lines
2039}2015}
2040 2016
2041static struct folio *shmem_swap_alloc_folio(struct inode *inode,2017static struct folio *shmem_swap_alloc_folio(struct inode *inode,
2042 struct vm_area_struct *vma, pgoff_t index,2018 struct vm_fault *vmf, pgoff_t index,
2043 swp_entry_t entry, int order, gfp_t gfp)2019 swp_entry_t entry, int order, gfp_t gfp)
2044{2020{
2021 pgoff_t ilx;
2022 struct folio *folio;
2023 struct mempolicy *mpol;
2045 struct shmem_inode_info *info = SHMEM_I(inode);2024 struct shmem_inode_info *info = SHMEM_I(inode);
2046 struct folio *new, *swapcache;
2047 int nr_pages = 1 << order;
2048 gfp_t alloc_gfp = gfp;
2049
2050 if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
2051 if (WARN_ON_ONCE(order))
2052 return ERR_PTR(-EINVAL);
2053 } else if (order) {
2054 /*
2055 * If uffd is active for the vma, we need per-page fault
2056 * fidelity to maintain the uffd semantics, then fallback
2057 * to swapin order-0 folio, as well as for zswap case.
2058 * Any existing sub folio in the swap cache also blocks
2059 * mTHP swapin.
2060 */
2061 if ((vma && unlikely(userfaultfd_armed(vma))) ||
2062 !zswap_never_enabled() ||
2063 non_swapcache_batch(entry, nr_pages) != nr_pages)
2064 goto fallback;
2065 2025
2066 alloc_gfp = limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);2026 if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
2067 }2027 !zswap_never_enabled())
2068retry:2028 order = 0;
2069 new = shmem_alloc_folio(alloc_gfp, order, info, index);
2070 if (!new) {
2071 new = ERR_PTR(-ENOMEM);
2072 goto fallback;
2073 }
2074 2029
2075 if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,2030again:
2076 alloc_gfp, entry)) {2031 mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
2077 folio_put(new);2032 folio = swapin_sync(entry, gfp, BIT(order), vmf, mpol, ilx);
2078 new = ERR_PTR(-ENOMEM);2033 mpol_cond_put(mpol);
2079 goto fallback;
2080 }
2081 2034
2082 swapcache = swapin_folio(entry, new);2035 if (!IS_ERR(folio))
2083 if (swapcache != new) {2036 return folio;
2084 folio_put(new);2037
2085 if (!swapcache) {2038 if (order) {
2086 /*2039 order = 0;
2087 * The new folio is charged already, swapin can2040 goto again;
2088 * only fail due to another raced swapin.
2089 */
2090 new = ERR_PTR(-EEXIST);
2091 goto fallback;
2092 }
2093 }2041 }
2094 return swapcache;2042
2095fallback:2043 return folio;
2096 /* Order 0 swapin failed, nothing to fallback to, abort */
2097 if (!order)
2098 return new;
2099 entry.val += index - round_down(index, nr_pages);
2100 alloc_gfp = gfp;
2101 nr_pages = 1;
2102 order = 0;
2103 goto retry;
2104}2044}
2105 2045
2106/*2046/*
@@ -2139,7 +2079,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, · 32 unmodified lines
2139 if (nr_pages > 1) {2079 if (nr_pages > 1) {
2140 gfp_t huge_gfp = vma_thp_gfp_mask(vma);2080 gfp_t huge_gfp = vma_thp_gfp_mask(vma);
2141 2081
2142 gfp = limit_gfp_mask(huge_gfp, gfp);2082 gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
2143 }2083 }
2144#endif2084#endif
2145 2085
@@ -2287,11 +2227,12 @@ unlock: · 141 unmodified lines
2287 */2227 */
2288static int shmem_swapin_folio(struct inode *inode, pgoff_t index,2228static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
2289 struct folio **foliop, enum sgp_type sgp,2229 struct folio **foliop, enum sgp_type sgp,
2290 gfp_t gfp, struct vm_area_struct *vma,2230 gfp_t gfp, struct vm_fault *vmf,
2291 vm_fault_t *fault_type)2231 vm_fault_t *fault_type)
2292{2232{
2293 struct address_space *mapping = inode->i_mapping;2233 struct address_space *mapping = inode->i_mapping;
2294 struct mm_struct *fault_mm = vma ? vma->vm_mm : NULL;2234 struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
2235 struct mm_struct *fault_mm = vmf ? vmf->vma->vm_mm : NULL;
2295 struct shmem_inode_info *info = SHMEM_I(inode);2236 struct shmem_inode_info *info = SHMEM_I(inode);
2296 swp_entry_t swap;2237 swp_entry_t swap;
2297 softleaf_t index_entry;2238 softleaf_t index_entry;
@@ -2332,20 +2273,19 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 34 unmodified lines
2332 if (!folio) {2273 if (!folio) {
2333 if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {2274 if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
2334 /* Direct swapin skipping swap cache & readahead */2275 /* Direct swapin skipping swap cache & readahead */
2335 folio = shmem_swap_alloc_folio(inode, vma, index,2276 folio = shmem_swap_alloc_folio(inode, vmf, index,
2336 index_entry, order, gfp);2277 swap, order, gfp);
2337 if (IS_ERR(folio)) {
2338 error = PTR_ERR(folio);
2339 folio = NULL;
2340 goto failed;
2341 }
2342 } else {2278 } else {
2343 /* Cached swapin only supports order 0 folio */2279 /* Cached swapin only supports order 0 folio */
2344 folio = shmem_swapin_cluster(swap, gfp, info, index);2280 folio = shmem_swapin_cluster(swap, gfp, info, index);
2345 if (!folio) {2281 }
2282 if (IS_ERR_OR_NULL(folio)) {
2283 if (IS_ERR(folio))
2284 error = PTR_ERR(folio);
2285 else
2346 error = -ENOMEM;2286 error = -ENOMEM;
2347 goto failed;2287 folio = NULL;
2348 }2288 goto failed;
2349 }2289 }
2350 if (fault_type) {2290 if (fault_type) {
2351 *fault_type |= VM_FAULT_MAJOR;2291 *fault_type |= VM_FAULT_MAJOR;
@@ -2493,7 +2433,7 @@ repeat: · 141 unmodified lines
2493 2433
2494 if (xa_is_value(folio)) {2434 if (xa_is_value(folio)) {
2495 error = shmem_swapin_folio(inode, index, &folio,2435 error = shmem_swapin_folio(inode, index, &folio,
2496 sgp, gfp, vma, fault_type);2436 sgp, gfp, vmf, fault_type);
2497 if (error == -EEXIST)2437 if (error == -EEXIST)
2498 goto repeat;2438 goto repeat;
2499 2439
@@ -2546,7 +2486,7 @@ repeat: · 46 unmodified lines
2546 gfp_t huge_gfp;2486 gfp_t huge_gfp;
2547 2487
2548 huge_gfp = vma_thp_gfp_mask(vma);2488 huge_gfp = vma_thp_gfp_mask(vma);
2549 huge_gfp = limit_gfp_mask(huge_gfp, gfp);2489 huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
2550 folio = shmem_alloc_and_add_folio(vmf, huge_gfp,2490 folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
2551 inode, index, fault_mm, orders);2491 inode, index, fault_mm, orders);
2552 if (!IS_ERR(folio)) {2492 if (!IS_ERR(folio)) {
@@ -3100,10 +3040,7 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap, · 547 unmodified lines
3100 cache_no_acl(inode);3040 cache_no_acl(inode);
3101 if (sbinfo->noswap)3041 if (sbinfo->noswap)
3102 mapping_set_unevictable(inode->i_mapping);3042 mapping_set_unevictable(inode->i_mapping);
3103 3043 mapping_set_large_folios(inode->i_mapping);
3104 /* Don't consider 'deny' for emergencies and 'force' for testing */
3105 if (sbinfo->huge)
3106 mapping_set_large_folios(inode->i_mapping);
3107 3044
3108 switch (mode & S_IFMT) {3045 switch (mode & S_IFMT) {
3109 default:3046 default:
@@ -5510,24 +5447,74 @@ static ssize_t shmem_enabled_store(struct kobject *kobj, · 2400 unmodified lines
5510struct kobj_attribute shmem_enabled_attr = __ATTR_RW(shmem_enabled);5447struct kobj_attribute shmem_enabled_attr = __ATTR_RW(shmem_enabled);
5511static DEFINE_SPINLOCK(huge_shmem_orders_lock);5448static DEFINE_SPINLOCK(huge_shmem_orders_lock);
5512 5449
5450enum huge_mode {
5451 HUGE_SHMEM_ENABLED_ALWAYS = 0,
5452 HUGE_SHMEM_ENABLED_INHERIT,
5453 HUGE_SHMEM_ENABLED_WITHIN_SIZE,
5454 HUGE_SHMEM_ENABLED_ADVISE,
5455 HUGE_SHMEM_ENABLED_NEVER,
5456};
5457
5458static const char * const huge_mode_strings[] = {
5459 [HUGE_SHMEM_ENABLED_ALWAYS] = "always",
5460 [HUGE_SHMEM_ENABLED_INHERIT] = "inherit",
5461 [HUGE_SHMEM_ENABLED_WITHIN_SIZE] = "within_size",
5462 [HUGE_SHMEM_ENABLED_ADVISE] = "advise",
5463 [HUGE_SHMEM_ENABLED_NEVER] = "never",
5464};
5465
5466static unsigned long * const huge_mode_orders[] = {
5467 [HUGE_SHMEM_ENABLED_ALWAYS] = &huge_shmem_orders_always,
5468 [HUGE_SHMEM_ENABLED_INHERIT] = &huge_shmem_orders_inherit,
5469 [HUGE_SHMEM_ENABLED_WITHIN_SIZE] = &huge_shmem_orders_within_size,
5470 [HUGE_SHMEM_ENABLED_ADVISE] = &huge_shmem_orders_madvise,
5471};
5472
5513static ssize_t thpsize_shmem_enabled_show(struct kobject *kobj,5473static ssize_t thpsize_shmem_enabled_show(struct kobject *kobj,
5514 struct kobj_attribute *attr, char *buf)5474 struct kobj_attribute *attr, char *buf)
5515{5475{
5516 int order = to_thpsize(kobj)->order;5476 int order = to_thpsize(kobj)->order;
5517 const char *output;5477 int active = HUGE_SHMEM_ENABLED_NEVER;
5518 5478 int len = 0;
5519 if (test_bit(order, &huge_shmem_orders_always))5479 int i;
5520 output = "[always] inherit within_size advise never";5480
5521 else if (test_bit(order, &huge_shmem_orders_inherit))5481 for (i = 0; i < ARRAY_SIZE(huge_mode_orders); i++) {
5522 output = "always [inherit] within_size advise never";5482 if (test_bit(order, huge_mode_orders[i])) {
5523 else if (test_bit(order, &huge_shmem_orders_within_size))5483 active = i;
5524 output = "always inherit [within_size] advise never";5484 break;
5525 else if (test_bit(order, &huge_shmem_orders_madvise))5485 }
5526 output = "always inherit within_size [advise] never";5486 }
5527 else5487
5528 output = "always inherit within_size advise [never]";5488 for (i = 0; i < ARRAY_SIZE(huge_mode_strings); i++) {
5489 if (i == active)
5490 len += sysfs_emit_at(buf, len, "[%s] ",
5491 huge_mode_strings[i]);
5492 else
5493 len += sysfs_emit_at(buf, len, "%s ",
5494 huge_mode_strings[i]);
5495 }
5496
5497 /* Replace trailing space with newline */
5498 buf[len - 1] = '\n';
5499
5500 return len;
5501}
5502
5503static bool set_shmem_enabled_mode(int order, enum huge_mode mode)
5504{
5505 bool changed = false;
5506 enum huge_mode idx;
5507
5508 spin_lock(&huge_shmem_orders_lock);
5509 for (idx = 0; idx < ARRAY_SIZE(huge_mode_orders); idx++) {
5510 if (idx == mode)
5511 changed |= !__test_and_set_bit(order, huge_mode_orders[idx]);
5512 else
5513 changed |= __test_and_clear_bit(order, huge_mode_orders[idx]);
5514 }
5515 spin_unlock(&huge_shmem_orders_lock);
5529 5516
5530 return sysfs_emit(buf, "%s\n", output);5517 return changed;
5531}5518}
5532 5519
5533static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,5520static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,
@@ -5535,58 +5522,31 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj, · 1 unmodified line
5535 const char *buf, size_t count)5522 const char *buf, size_t count)
5536{5523{
5537 int order = to_thpsize(kobj)->order;5524 int order = to_thpsize(kobj)->order;
5538 ssize_t ret = count;5525 int mode;
5539
5540 if (sysfs_streq(buf, "always")) {
5541 spin_lock(&huge_shmem_orders_lock);
5542 clear_bit(order, &huge_shmem_orders_inherit);
5543 clear_bit(order, &huge_shmem_orders_madvise);
5544 clear_bit(order, &huge_shmem_orders_within_size);
5545 set_bit(order, &huge_shmem_orders_always);
5546 spin_unlock(&huge_shmem_orders_lock);
5547 } else if (sysfs_streq(buf, "inherit")) {
5548 /* Do not override huge allocation policy with non-PMD sized mTHP */
5549 if (shmem_huge == SHMEM_HUGE_FORCE && !is_pmd_order(order))
5550 return -EINVAL;
5551 5526
5552 spin_lock(&huge_shmem_orders_lock);5527 mode = sysfs_match_string(huge_mode_strings, buf);
5553 clear_bit(order, &huge_shmem_orders_always);5528 if (mode < 0)
5554 clear_bit(order, &huge_shmem_orders_madvise);5529 return mode;
5555 clear_bit(order, &huge_shmem_orders_within_size);
5556 set_bit(order, &huge_shmem_orders_inherit);
5557 spin_unlock(&huge_shmem_orders_lock);
5558 } else if (sysfs_streq(buf, "within_size")) {
5559 spin_lock(&huge_shmem_orders_lock);
5560 clear_bit(order, &huge_shmem_orders_always);
5561 clear_bit(order, &huge_shmem_orders_inherit);
5562 clear_bit(order, &huge_shmem_orders_madvise);
5563 set_bit(order, &huge_shmem_orders_within_size);
5564 spin_unlock(&huge_shmem_orders_lock);
5565 } else if (sysfs_streq(buf, "advise")) {
5566 spin_lock(&huge_shmem_orders_lock);
5567 clear_bit(order, &huge_shmem_orders_always);
5568 clear_bit(order, &huge_shmem_orders_inherit);
5569 clear_bit(order, &huge_shmem_orders_within_size);
5570 set_bit(order, &huge_shmem_orders_madvise);
5571 spin_unlock(&huge_shmem_orders_lock);
5572 } else if (sysfs_streq(buf, "never")) {
5573 spin_lock(&huge_shmem_orders_lock);
5574 clear_bit(order, &huge_shmem_orders_always);
5575 clear_bit(order, &huge_shmem_orders_inherit);
5576 clear_bit(order, &huge_shmem_orders_within_size);
5577 clear_bit(order, &huge_shmem_orders_madvise);
5578 spin_unlock(&huge_shmem_orders_lock);
5579 } else {
5580 ret = -EINVAL;
5581 }
5582 5530
5583 if (ret > 0) {5531 /* Do not override huge allocation policy with non-PMD sized mTHP */
5584 int err = start_stop_khugepaged();5532 if (mode == HUGE_SHMEM_ENABLED_INHERIT &&
5533 shmem_huge == SHMEM_HUGE_FORCE && !is_pmd_order(order))
5534 return -EINVAL;
5585 5535
5536 if (set_shmem_enabled_mode(order, mode)) {
5537 int err = start_stop_khugepaged();
5586 if (err)5538 if (err)
5587 ret = err;5539 return err;
5540 } else {
5541 /*
5542 * Recalculate watermarks even when the mode hasn't changed
5543 * to preserve the legacy behavior, as this is always called
5544 * inside start_stop_khugepaged().
5545 */
5546 set_recommended_min_free_kbytes();
5588 }5547 }
5589 return ret;5548
5549 return count;
5590}5550}
5591 5551
5592struct kobj_attribute thpsize_shmem_enabled_attr =5552struct kobj_attribute thpsize_shmem_enabled_attr =
mm/shrinker.c+8−10
@@ -197,12 +197,13 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) · 196 unmodified lines
197{197{
198 if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {198 if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {
199 struct shrinker_info *info;199 struct shrinker_info *info;
200 struct shrinker_info_unit *unit;
201 200
202 rcu_read_lock();201 rcu_read_lock();
203 info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);202 info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
204 unit = info->unit[shrinker_id_to_index(shrinker_id)];
205 if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {203 if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {
204 struct shrinker_info_unit *unit;
205
206 unit = info->unit[shrinker_id_to_index(shrinker_id)];
206 /* Pairs with smp mb in shrink_slab() */207 /* Pairs with smp mb in shrink_slab() */
207 smp_mb__before_atomic();208 smp_mb__before_atomic();
208 set_bit(shrinker_id_to_offset(shrinker_id), unit->map);209 set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
@@ -215,29 +216,26 @@ static DEFINE_IDR(shrinker_idr); · 6 unmodified lines
215 216
216static int shrinker_memcg_alloc(struct shrinker *shrinker)217static int shrinker_memcg_alloc(struct shrinker *shrinker)
217{218{
218 int id, ret = -ENOMEM;219 int id;
219 220
220 if (mem_cgroup_disabled())221 if (mem_cgroup_disabled())
221 return -ENOSYS;222 return -ENOSYS;
222 if (mem_cgroup_kmem_disabled() && !(shrinker->flags & SHRINKER_NONSLAB))223 if (mem_cgroup_kmem_disabled() && !(shrinker->flags & SHRINKER_NONSLAB))
223 return -ENOSYS;224 return -ENOSYS;
224 225
225 mutex_lock(&shrinker_mutex);226 guard(mutex)(&shrinker_mutex);
226 id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL);227 id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL);
227 if (id < 0)228 if (id < 0)
228 goto unlock;229 return id;
229 230
230 if (id >= shrinker_nr_max) {231 if (id >= shrinker_nr_max) {
231 if (expand_shrinker_info(id)) {232 if (expand_shrinker_info(id)) {
232 idr_remove(&shrinker_idr, id);233 idr_remove(&shrinker_idr, id);
233 goto unlock;234 return -ENOMEM;
234 }235 }
235 }236 }
236 shrinker->id = id;237 shrinker->id = id;
237 ret = 0;238 return 0;
238unlock:
239 mutex_unlock(&shrinker_mutex);
240 return ret;
241}239}
242 240
243static void shrinker_memcg_remove(struct shrinker *shrinker)241static void shrinker_memcg_remove(struct shrinker *shrinker)
mm/sparse-vmemmap.c+56−26
@@ -87,15 +87,10 @@ static void * __meminit altmap_alloc_block_buf(unsigned long size, · 86 unmodified lines
87void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node,87void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node,
88 struct vmem_altmap *altmap)88 struct vmem_altmap *altmap)
89{89{
90 void *ptr;
91
92 if (altmap)90 if (altmap)
93 return altmap_alloc_block_buf(size, altmap);91 return altmap_alloc_block_buf(size, altmap);
94 92
95 ptr = sparse_buffer_alloc(size);93 return vmemmap_alloc_block(size, node);
96 if (!ptr)
97 ptr = vmemmap_alloc_block(size, node);
98 return ptr;
99}94}
100 95
101static unsigned long __meminit vmem_altmap_next_pfn(struct vmem_altmap *altmap)96static unsigned long __meminit vmem_altmap_next_pfn(struct vmem_altmap *altmap)
@@ -151,7 +146,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, · 49 unmodified lines
151 start, end - 1);146 start, end - 1);
152}147}
153 148
154pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,149static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
155 struct vmem_altmap *altmap,150 struct vmem_altmap *altmap,
156 unsigned long ptpfn, unsigned long flags)151 unsigned long ptpfn, unsigned long flags)
157{152{
@@ -195,7 +190,7 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) · 37 unmodified lines
195 return p;190 return p;
196}191}
197 192
198pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)193static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
199{194{
200 pmd_t *pmd = pmd_offset(pud, addr);195 pmd_t *pmd = pmd_offset(pud, addr);
201 if (pmd_none(*pmd)) {196 if (pmd_none(*pmd)) {
@@ -208,7 +203,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) · 6 unmodified lines
208 return pmd;203 return pmd;
209}204}
210 205
211pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)206static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
212{207{
213 pud_t *pud = pud_offset(p4d, addr);208 pud_t *pud = pud_offset(p4d, addr);
214 if (pud_none(*pud)) {209 if (pud_none(*pud)) {
@@ -221,7 +216,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node) · 6 unmodified lines
221 return pud;216 return pud;
222}217}
223 218
224p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)219static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
225{220{
226 p4d_t *p4d = p4d_offset(pgd, addr);221 p4d_t *p4d = p4d_offset(pgd, addr);
227 if (p4d_none(*p4d)) {222 if (p4d_none(*p4d)) {
@@ -234,7 +229,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node) · 6 unmodified lines
234 return p4d;229 return p4d;
235}230}
236 231
237pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)232static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
238{233{
239 pgd_t *pgd = pgd_offset_k(addr);234 pgd_t *pgd = pgd_offset_k(addr);
240 if (pgd_none(*pgd)) {235 if (pgd_none(*pgd)) {
@@ -391,12 +386,17 @@ int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, · 150 unmodified lines
391void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,386void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
392 unsigned long addr, unsigned long next)387 unsigned long addr, unsigned long next)
393{388{
389 WARN_ON_ONCE(!pmd_set_huge(pmd, virt_to_phys(p), PAGE_KERNEL));
394}390}
395 391
396int __weak __meminit vmemmap_check_pmd(pmd_t *pmd, int node,392int __weak __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
397 unsigned long addr, unsigned long next)393 unsigned long addr, unsigned long next)
398{394{
399 return 0;395 if (!pmd_leaf(pmdp_get(pmd)))
396 return 0;
397 vmemmap_verify((pte_t *)pmd, node, addr, next);
398
399 return 1;
400}400}
401 401
402int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,402int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,
@@ -652,26 +652,61 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) · 249 unmodified lines
652 }652 }
653}653}
654 654
655static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages,
656 struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
657{
658 const unsigned int order = pgmap ? pgmap->vmemmap_shift : 0;
659 const unsigned long pages_per_compound = 1UL << order;
660
661 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION));
662 VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION);
663
664 if (!vmemmap_can_optimize(altmap, pgmap))
665 return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE);
666
667 if (order < PFN_SECTION_SHIFT) {
668 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound));
669 return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound;
670 }
671
672 VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION));
673
674 if (IS_ALIGNED(pfn, pages_per_compound))
675 return VMEMMAP_RESERVE_NR;
676
677 return 0;
678}
679
655static struct page * __meminit populate_section_memmap(unsigned long pfn,680static struct page * __meminit populate_section_memmap(unsigned long pfn,
656 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,681 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
657 struct dev_pagemap *pgmap)682 struct dev_pagemap *pgmap)
658{683{
659 return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap);684 struct page *page = __populate_section_memmap(pfn, nr_pages, nid, altmap,
685 pgmap);
686
687 memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap));
688
689 return page;
660}690}
661 691
662static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages,692static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages,
663 struct vmem_altmap *altmap)693 struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
664{694{
665 unsigned long start = (unsigned long) pfn_to_page(pfn);695 unsigned long start = (unsigned long) pfn_to_page(pfn);
666 unsigned long end = start + nr_pages * sizeof(struct page);696 unsigned long end = start + nr_pages * sizeof(struct page);
667 697
698 memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap));
668 vmemmap_free(start, end, altmap);699 vmemmap_free(start, end, altmap);
669}700}
701
670static void free_map_bootmem(struct page *memmap)702static void free_map_bootmem(struct page *memmap)
671{703{
672 unsigned long start = (unsigned long)memmap;704 unsigned long start = (unsigned long)memmap;
673 unsigned long end = (unsigned long)(memmap + PAGES_PER_SECTION);705 unsigned long end = (unsigned long)(memmap + PAGES_PER_SECTION);
706 unsigned long pfn = page_to_pfn(memmap);
674 707
708 memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION,
709 NULL, NULL));
675 vmemmap_free(start, end, NULL);710 vmemmap_free(start, end, NULL);
676}711}
677 712
@@ -737,7 +772,7 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) · 59 unmodified lines
737 * usage map, but still need to free the vmemmap range.772 * usage map, but still need to free the vmemmap range.
738 */773 */
739static void section_deactivate(unsigned long pfn, unsigned long nr_pages,774static void section_deactivate(unsigned long pfn, unsigned long nr_pages,
740 struct vmem_altmap *altmap)775 struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
741{776{
742 struct mem_section *ms = __pfn_to_section(pfn);777 struct mem_section *ms = __pfn_to_section(pfn);
743 bool section_is_early = early_section(ms);778 bool section_is_early = early_section(ms);
@@ -774,14 +809,10 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, · 30 unmodified lines
774 * The memmap of early sections is always fully populated. See809 * The memmap of early sections is always fully populated. See
775 * section_activate() and pfn_valid() .810 * section_activate() and pfn_valid() .
776 */811 */
777 if (!section_is_early) {812 if (!section_is_early)
778 memmap_pages_add(-1L * (DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE)));813 depopulate_section_memmap(pfn, nr_pages, altmap, pgmap);
779 depopulate_section_memmap(pfn, nr_pages, altmap);814 else if (memmap)
780 } else if (memmap) {
781 memmap_boot_pages_add(-1L * (DIV_ROUND_UP(nr_pages * sizeof(struct page),
782 PAGE_SIZE)));
783 free_map_bootmem(memmap);815 free_map_bootmem(memmap);
784 }
785 816
786 if (empty)817 if (empty)
787 ms->section_mem_map = (unsigned long)NULL;818 ms->section_mem_map = (unsigned long)NULL;
@@ -823,10 +854,9 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, · 35 unmodified lines
823 854
824 memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap);855 memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap);
825 if (!memmap) {856 if (!memmap) {
826 section_deactivate(pfn, nr_pages, altmap);857 section_deactivate(pfn, nr_pages, altmap, pgmap);
827 return ERR_PTR(-ENOMEM);858 return ERR_PTR(-ENOMEM);
828 }859 }
829 memmap_pages_add(DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE));
830 860
831 return memmap;861 return memmap;
832}862}
@@ -885,13 +915,13 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, · 52 unmodified lines
885}915}
886 916
887void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,917void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
888 struct vmem_altmap *altmap)918 struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
889{919{
890 struct mem_section *ms = __pfn_to_section(pfn);920 struct mem_section *ms = __pfn_to_section(pfn);
891 921
892 if (WARN_ON_ONCE(!valid_section(ms)))922 if (WARN_ON_ONCE(!valid_section(ms)))
893 return;923 return;
894 924
895 section_deactivate(pfn, nr_pages, altmap);925 section_deactivate(pfn, nr_pages, altmap, pgmap);
896}926}
897#endif /* CONFIG_MEMORY_HOTPLUG */927#endif /* CONFIG_MEMORY_HOTPLUG */
mm/sparse.c+5−63
@@ -201,13 +201,11 @@ static void __init memblocks_present(void) · 200 unmodified lines
201 int i, nid;201 int i, nid;
202 202
203#ifdef CONFIG_SPARSEMEM_EXTREME203#ifdef CONFIG_SPARSEMEM_EXTREME
204 if (unlikely(!mem_section)) {204 unsigned long size, align;
205 unsigned long size, align;
206 205
207 size = sizeof(struct mem_section *) * NR_SECTION_ROOTS;206 size = sizeof(struct mem_section *) * NR_SECTION_ROOTS;
208 align = 1 << (INTERNODE_CACHE_SHIFT);207 align = 1 << (INTERNODE_CACHE_SHIFT);
209 mem_section = memblock_alloc_or_panic(size, align);208 mem_section = memblock_alloc_or_panic(size, align);
210 }
211#endif209#endif
212 210
213 for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid)211 for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid)
@@ -241,12 +239,9 @@ struct page __init *__populate_section_memmap(unsigned long pfn, · 27 unmodified lines
241 struct dev_pagemap *pgmap)239 struct dev_pagemap *pgmap)
242{240{
243 unsigned long size = section_map_size();241 unsigned long size = section_map_size();
244 struct page *map = sparse_buffer_alloc(size);242 struct page *map;
245 phys_addr_t addr = __pa(MAX_DMA_ADDRESS);243 phys_addr_t addr = __pa(MAX_DMA_ADDRESS);
246 244
247 if (map)
248 return map;
249
250 map = memmap_alloc(size, size, addr, nid, false);245 map = memmap_alloc(size, size, addr, nid, false);
251 if (!map)246 if (!map)
252 panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%pa\n",247 panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%pa\n",
@@ -256,55 +251,6 @@ struct page __init *__populate_section_memmap(unsigned long pfn, · 3 unmodified lines
256}251}
257#endif /* !CONFIG_SPARSEMEM_VMEMMAP */252#endif /* !CONFIG_SPARSEMEM_VMEMMAP */
258 253
259static void *sparsemap_buf __meminitdata;
260static void *sparsemap_buf_end __meminitdata;
261
262static inline void __meminit sparse_buffer_free(unsigned long size)
263{
264 WARN_ON(!sparsemap_buf || size == 0);
265 memblock_free(sparsemap_buf, size);
266}
267
268static void __init sparse_buffer_init(unsigned long size, int nid)
269{
270 phys_addr_t addr = __pa(MAX_DMA_ADDRESS);
271 WARN_ON(sparsemap_buf); /* forgot to call sparse_buffer_fini()? */
272 /*
273 * Pre-allocated buffer is mainly used by __populate_section_memmap
274 * and we want it to be properly aligned to the section size - this is
275 * especially the case for VMEMMAP which maps memmap to PMDs
276 */
277 sparsemap_buf = memmap_alloc(size, section_map_size(), addr, nid, true);
278 sparsemap_buf_end = sparsemap_buf + size;
279}
280
281static void __init sparse_buffer_fini(void)
282{
283 unsigned long size = sparsemap_buf_end - sparsemap_buf;
284
285 if (sparsemap_buf && size > 0)
286 sparse_buffer_free(size);
287 sparsemap_buf = NULL;
288}
289
290void * __meminit sparse_buffer_alloc(unsigned long size)
291{
292 void *ptr = NULL;
293
294 if (sparsemap_buf) {
295 ptr = (void *) roundup((unsigned long)sparsemap_buf, size);
296 if (ptr + size > sparsemap_buf_end)
297 ptr = NULL;
298 else {
299 /* Free redundant aligned space */
300 if ((unsigned long)(ptr - sparsemap_buf) > 0)
301 sparse_buffer_free((unsigned long)(ptr - sparsemap_buf));
302 sparsemap_buf = ptr + size;
303 }
304 }
305 return ptr;
306}
307
308void __weak __meminit vmemmap_populate_print_last(void)254void __weak __meminit vmemmap_populate_print_last(void)
309{255{
310}256}
@@ -362,8 +308,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 51 unmodified lines
362 goto failed;308 goto failed;
363 }309 }
364 310
365 sparse_buffer_init(map_count * section_map_size(), nid);
366
367 sparse_vmemmap_init_nid_early(nid);311 sparse_vmemmap_init_nid_early(nid);
368 312
369 for_each_present_section_nr(pnum_begin, pnum) {313 for_each_present_section_nr(pnum_begin, pnum) {
@@ -381,7 +325,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 11 unmodified lines
381 __func__, nid);325 __func__, nid);
382 pnum_begin = pnum;326 pnum_begin = pnum;
383 sparse_usage_fini();327 sparse_usage_fini();
384 sparse_buffer_fini();
385 goto failed;328 goto failed;
386 }329 }
387 memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page),330 memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page),
@@ -390,7 +333,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 2 unmodified lines
390 }333 }
391 }334 }
392 sparse_usage_fini();335 sparse_usage_fini();
393 sparse_buffer_fini();
394 return;336 return;
395failed:337failed:
396 /*338 /*
mm/swap.c+53−4
@@ -160,13 +160,41 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · 159 unmodified lines
160 int i;160 int i;
161 struct lruvec *lruvec = NULL;161 struct lruvec *lruvec = NULL;
162 unsigned long flags = 0;162 unsigned long flags = 0;
163 struct folio_batch free_fbatch;
164 bool is_lru_add = (move_fn == lru_add);
165
166 /*
167 * If we're adding to the LRU, preemptively filter dead folios. Use
168 * this dedicated folio batch for temp storage and deferred cleanup.
169 */
170 if (is_lru_add)
171 folio_batch_init(&free_fbatch);
163 172
164 for (i = 0; i < folio_batch_count(fbatch); i++) {173 for (i = 0; i < folio_batch_count(fbatch); i++) {
165 struct folio *folio = fbatch->folios[i];174 struct folio *folio = fbatch->folios[i];
166 175
167 /* block memcg migration while the folio moves between lru */176 /* block memcg migration while the folio moves between lru */
168 if (move_fn != lru_add && !folio_test_clear_lru(folio))177 if (!is_lru_add && !folio_test_clear_lru(folio))
178 continue;
179
180 /*
181 * Filter dead folios by moving them from the add batch to the temp
182 * batch for freeing after this loop.
183 *
184 * We're bypassing normal cleanup. Clear flags that are not
185 * applicable to dead folios.
186 *
187 * Since the folio may be part of a huge page, unqueue from
188 * deferred split list to avoid a dangling list entry.
189 */
190 if (is_lru_add && folio_ref_freeze(folio, 1)) {
191 __folio_clear_active(folio);
192 __folio_clear_unevictable(folio);
193 folio_unqueue_deferred_split(folio);
194 fbatch->folios[i] = NULL;
195 folio_batch_add(&free_fbatch, folio);
169 continue;196 continue;
197 }
170 198
171 folio_lruvec_relock_irqsave(folio, &lruvec, &flags);199 folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
172 move_fn(lruvec, folio);200 move_fn(lruvec, folio);
@@ -176,6 +204,13 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · 3 unmodified lines
176 204
177 if (lruvec)205 if (lruvec)
178 lruvec_unlock_irqrestore(lruvec, flags);206 lruvec_unlock_irqrestore(lruvec, flags);
207
208 /* Cleanup filtered dead folios. */
209 if (is_lru_add) {
210 mem_cgroup_uncharge_folios(&free_fbatch);
211 free_unref_folios(&free_fbatch);
212 }
213
179 folios_put(fbatch);214 folios_put(fbatch);
180}215}
181 216
@@ -509,10 +544,20 @@ void folio_add_lru(struct folio *folio) · 327 unmodified lines
509 folio_test_unevictable(folio), folio);544 folio_test_unevictable(folio), folio);
510 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);545 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
511 546
512 /* see the comment in lru_gen_folio_seq() */547 /*
548 * For refaulted workingset folios, set PG_active so they
549 * can be added to active generations.
550 * For prefaulted file folios, folio_mark_accessed() sets
551 * PG_referenced so lru_gen_folio_seq() places them into
552 * the second oldest generation.
553 */
513 if (lru_gen_enabled() && !folio_test_unevictable(folio) &&554 if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
514 lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))555 lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
515 folio_set_active(folio);556 if (folio_test_workingset(folio))
557 folio_set_active(folio);
558 else if (!folio_test_referenced(folio))
559 folio_mark_accessed(folio);
560 }
516 561
517 folio_batch_add_and_move(folio, lru_add);562 folio_batch_add_and_move(folio, lru_add);
518}563}
@@ -964,6 +1009,10 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) · 445 unmodified lines
964 struct folio *folio = folios->folios[i];1009 struct folio *folio = folios->folios[i];
965 unsigned int nr_refs = refs ? refs[i] : 1;1010 unsigned int nr_refs = refs ? refs[i] : 1;
966 1011
1012 /* Folio batch entry may have been preemptively removed during drain. */
1013 if (!folio)
1014 continue;
1015
967 if (is_huge_zero_folio(folio))1016 if (is_huge_zero_folio(folio))
968 continue;1017 continue;
969 1018
mm/swap.h+32−58
@@ -3,11 +3,29 @@ · 2 unmodified lines
3#define _MM_SWAP_H3#define _MM_SWAP_H
4 4
5#include <linux/atomic.h> /* for atomic_long_t */5#include <linux/atomic.h> /* for atomic_long_t */
6#include <linux/mm.h> /* for PAGE_SHIFT */
6struct mempolicy;7struct mempolicy;
7struct swap_iocb;8struct swap_iocb;
9struct swap_memcg_table;
8 10
9extern int page_cluster;11extern int page_cluster;
10 12
13#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
14#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
15#elif defined(MAX_PHYSMEM_BITS)
16#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
17#else
18#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
19#endif
20
21/* Swap table marker, 0x1 means shadow, 0x2 means PFN (SWP_TB_PFN_MARK) */
22#define SWAP_CACHE_PFN_MARK_BITS 2
23/* At least 2 bits are needed to distinguish SWP_TB_COUNT_MAX, 1 and 0 */
24#define SWAP_COUNT_MIN_BITS 2
25/* If there are enough bits besides PFN and marker, store zero flag inline */
26#define SWAP_TABLE_HAS_ZEROFLAG ((BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - \
27 SWAP_CACHE_PFN_BITS) > SWAP_COUNT_MIN_BITS)
28
11#ifdef CONFIG_THP_SWAP29#ifdef CONFIG_THP_SWAP
12#define SWAPFILE_CLUSTER HPAGE_PMD_NR30#define SWAPFILE_CLUSTER HPAGE_PMD_NR
13#define swap_entry_order(order) (order)31#define swap_entry_order(order) (order)
@@ -38,6 +56,12 @@ struct swap_cluster_info { · 24 unmodified lines
38 u8 order;56 u8 order;
39 atomic_long_t __rcu *table; /* Swap table entries, see mm/swap_table.h */57 atomic_long_t __rcu *table; /* Swap table entries, see mm/swap_table.h */
40 unsigned int *extend_table; /* For large swap count, protected by ci->lock */58 unsigned int *extend_table; /* For large swap count, protected by ci->lock */
59#ifdef CONFIG_MEMCG
60 struct swap_memcg_table *memcg_table; /* Swap table entries' cgroup record */
61#endif
62#if !SWAP_TABLE_HAS_ZEROFLAG
63 unsigned long *zero_bitmap;
64#endif
41 struct list_head list;65 struct list_head list;
42};66};
43 67
@@ -280,9 +304,9 @@ bool swap_cache_has_folio(swp_entry_t entry); · 236 unmodified lines
280struct folio *swap_cache_get_folio(swp_entry_t entry);304struct folio *swap_cache_get_folio(swp_entry_t entry);
281void *swap_cache_get_shadow(swp_entry_t entry);305void *swap_cache_get_shadow(swp_entry_t entry);
282void swap_cache_del_folio(struct folio *folio);306void swap_cache_del_folio(struct folio *folio);
283struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,307struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
284 struct mempolicy *mpol, pgoff_t ilx,308 unsigned long orders, struct vm_fault *vmf,
285 bool *alloced);309 struct mempolicy *mpol, pgoff_t ilx);
286/* Below helpers require the caller to lock and pass in the swap cluster. */310/* Below helpers require the caller to lock and pass in the swap cluster. */
287void __swap_cache_add_folio(struct swap_cluster_info *ci,311void __swap_cache_add_folio(struct swap_cluster_info *ci,
288 struct folio *folio, swp_entry_t entry);312 struct folio *folio, swp_entry_t entry);
@@ -300,7 +324,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, · 11 unmodified lines
300 struct mempolicy *mpol, pgoff_t ilx);324 struct mempolicy *mpol, pgoff_t ilx);
301struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,325struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
302 struct vm_fault *vmf);326 struct vm_fault *vmf);
303struct folio *swapin_folio(swp_entry_t entry, struct folio *folio);327struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders,
328 struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx);
304void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,329void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
305 unsigned long addr);330 unsigned long addr);
306 331
@@ -309,49 +334,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio) · 2 unmodified lines
309 return __swap_entry_to_info(folio->swap)->flags;334 return __swap_entry_to_info(folio->swap)->flags;
310}335}
311 336
312/*
313 * Return the count of contiguous swap entries that share the same
314 * zeromap status as the starting entry. If is_zeromap is not NULL,
315 * it will return the zeromap status of the starting entry.
316 */
317static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
318 bool *is_zeromap)
319{
320 struct swap_info_struct *sis = __swap_entry_to_info(entry);
321 unsigned long start = swp_offset(entry);
322 unsigned long end = start + max_nr;
323 bool first_bit;
324
325 first_bit = test_bit(start, sis->zeromap);
326 if (is_zeromap)
327 *is_zeromap = first_bit;
328
329 if (max_nr <= 1)
330 return max_nr;
331 if (first_bit)
332 return find_next_zero_bit(sis->zeromap, end, start) - start;
333 else
334 return find_next_bit(sis->zeromap, end, start) - start;
335}
336
337static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
338{
339 int i;
340
341 /*
342 * While allocating a large folio and doing mTHP swapin, we need to
343 * ensure all entries are not cached, otherwise, the mTHP folio will
344 * be in conflict with the folio in swap cache.
345 */
346 for (i = 0; i < max_nr; i++) {
347 if (swap_cache_has_folio(entry))
348 return i;
349 entry.val++;
350 }
351
352 return i;
353}
354
355#else /* CONFIG_SWAP */337#else /* CONFIG_SWAP */
356struct swap_iocb;338struct swap_iocb;
357static inline struct swap_cluster_info *swap_cluster_lock(339static inline struct swap_cluster_info *swap_cluster_lock(
@@ -433,7 +415,9 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, · 75 unmodified lines
433 return NULL;415 return NULL;
434}416}
435 417
436static inline struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)418static inline struct folio *swapin_sync(
419 swp_entry_t entry, gfp_t flag, unsigned long orders,
420 struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
437{421{
438 return NULL;422 return NULL;
439}423}
@@ -488,15 +472,5 @@ static inline unsigned int folio_swap_flags(struct folio *folio) · 48 unmodified lines
488 return 0;472 return 0;
489}473}
490 474
491static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
492 bool *has_zeromap)
493{
494 return 0;
495}
496
497static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
498{
499 return 0;
500}
501#endif /* CONFIG_SWAP */475#endif /* CONFIG_SWAP */
502#endif /* _MM_SWAP_H */476#endif /* _MM_SWAP_H */
mm/swap_cgroup.c+0−172
@@ -1,172 +0,0 @@
1// SPDX-License-Identifier: GPL-2.0
2#include <linux/swap_cgroup.h>
3#include <linux/vmalloc.h>
4#include <linux/mm.h>
5
6#include <linux/swapops.h> /* depends on mm.h include */
7
8static DEFINE_MUTEX(swap_cgroup_mutex);
9
10/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t) */
11#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short))
12#define ID_SHIFT (BITS_PER_TYPE(unsigned short))
13#define ID_MASK (BIT(ID_SHIFT) - 1)
14struct swap_cgroup {
15 atomic_t ids;
16};
17
18struct swap_cgroup_ctrl {
19 struct swap_cgroup *map;
20};
21
22static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES];
23
24static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map,
25 pgoff_t offset)
26{
27 unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
28 unsigned int old_ids = atomic_read(&map[offset / ID_PER_SC].ids);
29
30 BUILD_BUG_ON(!is_power_of_2(ID_PER_SC));
31 BUILD_BUG_ON(sizeof(struct swap_cgroup) != sizeof(atomic_t));
32
33 return (old_ids >> shift) & ID_MASK;
34}
35
36static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map,
37 pgoff_t offset,
38 unsigned short new_id)
39{
40 unsigned short old_id;
41 struct swap_cgroup *sc = &map[offset / ID_PER_SC];
42 unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
43 unsigned int new_ids, old_ids = atomic_read(&sc->ids);
44
45 do {
46 old_id = (old_ids >> shift) & ID_MASK;
47 new_ids = (old_ids & ~(ID_MASK << shift));
48 new_ids |= ((unsigned int)new_id) << shift;
49 } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids));
50
51 return old_id;
52}
53
54/**
55 * swap_cgroup_record - record mem_cgroup for a set of swap entries.
56 * These entries must belong to one single folio, and that folio
57 * must be being charged for swap space (swap out), and these
58 * entries must not have been charged
59 *
60 * @folio: the folio that the swap entry belongs to
61 * @id: mem_cgroup ID to be recorded
62 * @ent: the first swap entry to be recorded
63 */
64void swap_cgroup_record(struct folio *folio, unsigned short id,
65 swp_entry_t ent)
66{
67 unsigned int nr_ents = folio_nr_pages(folio);
68 struct swap_cgroup *map;
69 pgoff_t offset, end;
70 unsigned short old;
71
72 offset = swp_offset(ent);
73 end = offset + nr_ents;
74 map = swap_cgroup_ctrl[swp_type(ent)].map;
75
76 do {
77 old = __swap_cgroup_id_xchg(map, offset, id);
78 VM_BUG_ON(old);
79 } while (++offset != end);
80}
81
82/**
83 * swap_cgroup_clear - clear mem_cgroup for a set of swap entries.
84 * These entries must be being uncharged from swap. They either
85 * belongs to one single folio in the swap cache (swap in for
86 * cgroup v1), or no longer have any users (slot freeing).
87 *
88 * @ent: the first swap entry to be recorded into
89 * @nr_ents: number of swap entries to be recorded
90 *
91 * Returns the existing old value.
92 */
93unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
94{
95 pgoff_t offset, end;
96 struct swap_cgroup *map;
97 unsigned short old, iter = 0;
98
99 offset = swp_offset(ent);
100 end = offset + nr_ents;
101 map = swap_cgroup_ctrl[swp_type(ent)].map;
102
103 do {
104 old = __swap_cgroup_id_xchg(map, offset, 0);
105 if (!iter)
106 iter = old;
107 VM_BUG_ON(iter != old);
108 } while (++offset != end);
109
110 return old;
111}
112
113/**
114 * lookup_swap_cgroup_id - lookup mem_cgroup id tied to swap entry
115 * @ent: swap entry to be looked up.
116 *
117 * Returns ID of mem_cgroup at success. 0 at failure. (0 is invalid ID)
118 */
119unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
120{
121 struct swap_cgroup_ctrl *ctrl;
122
123 if (mem_cgroup_disabled())
124 return 0;
125
126 ctrl = &swap_cgroup_ctrl[swp_type(ent)];
127 return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent));
128}
129
130int swap_cgroup_swapon(int type, unsigned long max_pages)
131{
132 struct swap_cgroup *map;
133 struct swap_cgroup_ctrl *ctrl;
134
135 if (mem_cgroup_disabled())
136 return 0;
137
138 BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=
139 sizeof(struct swap_cgroup));
140 map = vzalloc(DIV_ROUND_UP(max_pages, ID_PER_SC) *
141 sizeof(struct swap_cgroup));
142 if (!map)
143 goto nomem;
144
145 ctrl = &swap_cgroup_ctrl[type];
146 mutex_lock(&swap_cgroup_mutex);
147 ctrl->map = map;
148 mutex_unlock(&swap_cgroup_mutex);
149
150 return 0;
151nomem:
152 pr_info("couldn't allocate enough memory for swap_cgroup\n");
153 pr_info("swap_cgroup can be disabled by swapaccount=0 boot option\n");
154 return -ENOMEM;
155}
156
157void swap_cgroup_swapoff(int type)
158{
159 struct swap_cgroup *map;
160 struct swap_cgroup_ctrl *ctrl;
161
162 if (mem_cgroup_disabled())
163 return;
164
165 mutex_lock(&swap_cgroup_mutex);
166 ctrl = &swap_cgroup_ctrl[type];
167 map = ctrl->map;
168 ctrl->map = NULL;
169 mutex_unlock(&swap_cgroup_mutex);
170
171 vfree(map);
172}
mm/swap_state.c+308−225
@@ -137,8 +137,71 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 136 unmodified lines
137 return NULL;137 return NULL;
138}138}
139 139
140void __swap_cache_add_folio(struct swap_cluster_info *ci,140/**
141 struct folio *folio, swp_entry_t entry)141 * __swap_cache_add_check - Check if a range is suitable for adding a folio.
142 * @ci: The locked swap cluster
143 * @targ_entry: The target swap entry to check, will be rounded down by @nr
144 * @nr: Number of slots to check, must be a power of 2
145 * @shadowp: Returns the shadow value if one exists in the range
146 * @memcg_id: Returns the memory cgroup id, NULL to ignore cgroup check
147 *
148 * Check if all slots covered by given range have a swap count >= 1.
149 * Retrieves the shadow if there is one. If @memcg_id is not NULL, also
150 * checks if all slots belong to the same cgroup and return the cgroup
151 * private id.
152 *
153 * Context: Caller must lock the cluster.
154 * Return: 0 if success, error code if failed.
155 */
156static int __swap_cache_add_check(struct swap_cluster_info *ci,
157 swp_entry_t targ_entry,
158 unsigned long nr, void **shadowp,
159 unsigned short *memcg_id)
160{
161 unsigned int ci_off, ci_end;
162 unsigned long old_tb;
163 bool is_zero;
164
165 lockdep_assert_held(&ci->lock);
166
167 /*
168 * If the target slot is not swapped out or already cached, return
169 * -ENOENT or -EEXIST. If the batch is not suitable, could be a
170 * race with concurrent free or cache add, return -EBUSY.
171 */
172 if (unlikely(!ci->table))
173 return -ENOENT;
174 ci_off = swp_cluster_offset(targ_entry);
175 old_tb = __swap_table_get(ci, ci_off);
176 if (swp_tb_is_folio(old_tb))
177 return -EEXIST;
178 if (!__swp_tb_get_count(old_tb))
179 return -ENOENT;
180 if (shadowp && swp_tb_is_shadow(old_tb))
181 *shadowp = swp_tb_to_shadow(old_tb);
182 if (memcg_id)
183 *memcg_id = __swap_cgroup_get(ci, ci_off);
184
185 if (nr == 1)
186 return 0;
187
188 is_zero = __swap_table_test_zero(ci, ci_off);
189 ci_off = round_down(ci_off, nr);
190 ci_end = ci_off + nr;
191 do {
192 old_tb = __swap_table_get(ci, ci_off);
193 if (unlikely(swp_tb_is_folio(old_tb) ||
194 !__swp_tb_get_count(old_tb) ||
195 is_zero != __swap_table_test_zero(ci, ci_off) ||
196 (memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
197 return -EBUSY;
198 } while (++ci_off < ci_end);
199
200 return 0;
201}
202
203static void __swap_cache_do_add_folio(struct swap_cluster_info *ci,
204 struct folio *folio, swp_entry_t entry)
142{205{
143 unsigned int ci_off = swp_cluster_offset(entry), ci_end;206 unsigned int ci_off = swp_cluster_offset(entry), ci_end;
144 unsigned long nr_pages = folio_nr_pages(folio);207 unsigned long nr_pages = folio_nr_pages(folio);
@@ -153,88 +216,42 @@ void __swap_cache_add_folio(struct swap_cluster_info *ci, · 8 unmodified lines
153 do {216 do {
154 old_tb = __swap_table_get(ci, ci_off);217 old_tb = __swap_table_get(ci, ci_off);
155 VM_WARN_ON_ONCE(swp_tb_is_folio(old_tb));218 VM_WARN_ON_ONCE(swp_tb_is_folio(old_tb));
156 __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));219 __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
157 } while (++ci_off < ci_end);220 } while (++ci_off < ci_end);
158 221
159 folio_ref_add(folio, nr_pages);222 folio_ref_add(folio, nr_pages);
160 folio_set_swapcache(folio);223 folio_set_swapcache(folio);
161 folio->swap = entry;224 folio->swap = entry;
162
163 node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
164 lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
165}225}
166 226
167/**227/**
168 * swap_cache_add_folio - Add a folio into the swap cache.228 * __swap_cache_add_folio - Add a folio to the swap cache and update stats.
229 * @ci: The locked swap cluster.
169 * @folio: The folio to be added.230 * @folio: The folio to be added.
170 * @entry: The swap entry corresponding to the folio.231 * @entry: The swap entry corresponding to the folio.
171 * @gfp: gfp_mask for XArray node allocation.
172 * @shadowp: If a shadow is found, return the shadow.
173 *232 *
174 * Context: Caller must ensure @entry is valid and protect the swap device233 * Unconditionally add a folio to the swap cache. The caller must ensure
175 * with reference count or locks.234 * all slots are usable and have no conflicts. This assigns entry to
235 * @folio->swap, increases folio refcount by the number of pages, and
236 * updates swap cache stats.
237 *
238 * Context: Caller must ensure the folio is locked and lock the cluster
239 * that holds the entries.
176 */240 */
177static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,241void __swap_cache_add_folio(struct swap_cluster_info *ci,
178 void **shadowp)242 struct folio *folio, swp_entry_t entry)
179{243{
180 int err;
181 void *shadow = NULL;
182 unsigned long old_tb;
183 struct swap_info_struct *si;
184 struct swap_cluster_info *ci;
185 unsigned int ci_start, ci_off, ci_end;
186 unsigned long nr_pages = folio_nr_pages(folio);244 unsigned long nr_pages = folio_nr_pages(folio);
187 245
188 si = __swap_entry_to_info(entry);246 __swap_cache_do_add_folio(ci, folio, entry);
189 ci_start = swp_cluster_offset(entry);247 node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
190 ci_end = ci_start + nr_pages;248 lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
191 ci_off = ci_start;
192 ci = swap_cluster_lock(si, swp_offset(entry));
193 if (unlikely(!ci->table)) {
194 err = -ENOENT;
195 goto failed;
196 }
197 do {
198 old_tb = __swap_table_get(ci, ci_off);
199 if (unlikely(swp_tb_is_folio(old_tb))) {
200 err = -EEXIST;
201 goto failed;
202 }
203 if (unlikely(!__swp_tb_get_count(old_tb))) {
204 err = -ENOENT;
205 goto failed;
206 }
207 if (swp_tb_is_shadow(old_tb))
208 shadow = swp_tb_to_shadow(old_tb);
209 } while (++ci_off < ci_end);
210 __swap_cache_add_folio(ci, folio, entry);
211 swap_cluster_unlock(ci);
212 if (shadowp)
213 *shadowp = shadow;
214 return 0;
215
216failed:
217 swap_cluster_unlock(ci);
218 return err;
219}249}
220 250
221/**251static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
222 * __swap_cache_del_folio - Removes a folio from the swap cache.252 struct folio *folio,
223 * @ci: The locked swap cluster.253 swp_entry_t entry, void *shadow)
224 * @folio: The folio.
225 * @entry: The first swap entry that the folio corresponds to.
226 * @shadow: shadow value to be filled in the swap cache.
227 *
228 * Removes a folio from the swap cache and fills a shadow in place.
229 * This won't put the folio's refcount. The caller has to do that.
230 *
231 * Context: Caller must ensure the folio is locked and in the swap cache
232 * using the index of @entry, and lock the cluster that holds the entries.
233 */
234void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
235 swp_entry_t entry, void *shadow)
236{254{
237 int count;
238 unsigned long old_tb;255 unsigned long old_tb;
239 struct swap_info_struct *si;256 struct swap_info_struct *si;
240 unsigned int ci_start, ci_off, ci_end;257 unsigned int ci_start, ci_off, ci_end;
@@ -254,19 +271,17 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 13 unmodified lines
254 old_tb = __swap_table_get(ci, ci_off);271 old_tb = __swap_table_get(ci, ci_off);
255 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) ||272 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) ||
256 swp_tb_to_folio(old_tb) != folio);273 swp_tb_to_folio(old_tb) != folio);
257 count = __swp_tb_get_count(old_tb);274 if (__swp_tb_get_count(old_tb))
258 if (count)
259 folio_swapped = true;275 folio_swapped = true;
260 else276 else
261 need_free = true;277 need_free = true;
262 /* If shadow is NULL, we sets an empty shadow. */278 /* If shadow is NULL, we set an empty shadow. */
263 __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow, count));279 __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow,
280 __swp_tb_get_flags(old_tb)));
264 } while (++ci_off < ci_end);281 } while (++ci_off < ci_end);
265 282
266 folio->swap.val = 0;283 folio->swap.val = 0;
267 folio_clear_swapcache(folio);284 folio_clear_swapcache(folio);
268 node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
269 lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
270 285
271 if (!folio_swapped) {286 if (!folio_swapped) {
272 __swap_cluster_free_entries(si, ci, ci_start, nr_pages);287 __swap_cluster_free_entries(si, ci, ci_start, nr_pages);
@@ -279,6 +294,29 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 6 unmodified lines
279 }294 }
280}295}
281 296
297/**
298 * __swap_cache_del_folio - Removes a folio from the swap cache.
299 * @ci: The locked swap cluster.
300 * @folio: The folio.
301 * @entry: The first swap entry that the folio corresponds to.
302 * @shadow: shadow value to be filled in the swap cache.
303 *
304 * Removes a folio from the swap cache and fills a shadow in place.
305 * This won't put the folio's refcount. The caller has to do that.
306 *
307 * Context: Caller must ensure the folio is locked and in the swap cache
308 * using the index of @entry, and lock the cluster that holds the entries.
309 */
310void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
311 swp_entry_t entry, void *shadow)
312{
313 unsigned long nr_pages = folio_nr_pages(folio);
314
315 __swap_cache_do_del_folio(ci, folio, entry, shadow);
316 node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
317 lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
318}
319
282/**320/**
283 * swap_cache_del_folio - Removes a folio from the swap cache.321 * swap_cache_del_folio - Removes a folio from the swap cache.
284 * @folio: The folio.322 * @folio: The folio.
@@ -333,7 +371,7 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 48 unmodified lines
333 do {371 do {
334 old_tb = __swap_table_get(ci, ci_off);372 old_tb = __swap_table_get(ci, ci_off);
335 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) || swp_tb_to_folio(old_tb) != old);373 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) || swp_tb_to_folio(old_tb) != old);
336 __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));374 __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
337 } while (++ci_off < ci_end);375 } while (++ci_off < ci_end);
338 376
339 /*377 /*
@@ -350,6 +388,153 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 10 unmodified lines
350 }388 }
351}389}
352 390
391/*
392 * Try to allocate a folio of given order in the swap cache.
393 *
394 * This helper resolves the potential races of swap allocation
395 * and prepares a folio to be used for swap IO. May return following
396 * value:
397 *
398 * -ENOMEM / -EBUSY: Order is too large or in conflict with sub slot,
399 * caller should shrink the order and retry
400 * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
401 * should abort or try to use the cached folio instead
402 */
403static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
404 swp_entry_t targ_entry, gfp_t gfp,
405 unsigned int order, struct vm_fault *vmf,
406 struct mempolicy *mpol, pgoff_t ilx)
407{
408 int err;
409 swp_entry_t entry;
410 struct folio *folio;
411 void *shadow = NULL;
412 unsigned short memcg_id;
413 unsigned long address, nr_pages = 1UL << order;
414 struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
415
416 VM_WARN_ON_ONCE(nr_pages > SWAPFILE_CLUSTER);
417 entry.val = round_down(targ_entry.val, nr_pages);
418
419 /* Check if the slot and range are available, skip allocation if not */
420 spin_lock(&ci->lock);
421 err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
422 spin_unlock(&ci->lock);
423 if (unlikely(err))
424 return ERR_PTR(err);
425
426 /*
427 * Limit THP gfp. The limitation is a no-op for typical
428 * GFP_HIGHUSER_MOVABLE but matters for shmem.
429 */
430 if (order)
431 gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
432
433 if (mpol || !vmf) {
434 folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
435 } else {
436 address = round_down(vmf->address, PAGE_SIZE << order);
437 folio = vma_alloc_folio(gfp, order, vmf->vma, address);
438 }
439 if (unlikely(!folio))
440 return ERR_PTR(-ENOMEM);
441
442 /* Double check the range is still not in conflict */
443 spin_lock(&ci->lock);
444 err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
445 if (unlikely(err)) {
446 spin_unlock(&ci->lock);
447 folio_put(folio);
448 return ERR_PTR(err);
449 }
450
451 __folio_set_locked(folio);
452 __folio_set_swapbacked(folio);
453 __swap_cache_do_add_folio(ci, folio, entry);
454 spin_unlock(&ci->lock);
455
456 if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
457 vmf ? vmf->vma->vm_mm : NULL, gfp)) {
458 spin_lock(&ci->lock);
459 __swap_cache_do_del_folio(ci, folio, entry, shadow);
460 spin_unlock(&ci->lock);
461 folio_unlock(folio);
462 /* nr_pages refs from swap cache, 1 from allocation */
463 folio_put_refs(folio, nr_pages + 1);
464 count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
465 return ERR_PTR(-ENOMEM);
466 }
467
468 if (order > 1 && folio_memcg_alloc_deferred(folio)) {
469 spin_lock(&ci->lock);
470 __swap_cache_do_del_folio(ci, folio, entry, shadow);
471 spin_unlock(&ci->lock);
472 folio_unlock(folio);
473 /* nr_pages refs from swap cache, 1 from allocation */
474 folio_put_refs(folio, nr_pages + 1);
475 return ERR_PTR(-ENOMEM);
476 }
477
478 /* memsw uncharges swap when folio is added to swap cache */
479 memcg1_swapin(folio);
480 if (shadow)
481 workingset_refault(folio, shadow);
482
483 node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
484 lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
485
486 /* Caller will initiate read into locked new_folio */
487 folio_add_lru(folio);
488 return folio;
489}
490
491/**
492 * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
493 * @targ_entry: swap entry indicating the target slot
494 * @gfp: memory allocation flags
495 * @orders: allocation orders, must be non zero
496 * @vmf: fault information
497 * @mpol: NUMA memory allocation policy to be applied
498 * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
499 *
500 * Allocate a folio in the swap cache for one swap slot, typically before
501 * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
502 * @targ_entry must have a non-zero swap count (swapped out).
503 *
504 * Context: Caller must protect the swap device with reference count or locks.
505 * Return: Returns the folio if allocation succeeded and folio is in the swap
506 * cache. Returns error code if failed due to race, OOM or invalid arguments.
507 */
508struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
509 unsigned long orders, struct vm_fault *vmf,
510 struct mempolicy *mpol, pgoff_t ilx)
511{
512 int order, err;
513 struct folio *ret;
514 struct swap_cluster_info *ci;
515
516 ci = __swap_entry_to_cluster(targ_entry);
517 order = highest_order(orders);
518
519 /* orders must be non-zero, and must not exceed cluster size. */
520 if (WARN_ON_ONCE(!orders || (1UL << order) > SWAPFILE_CLUSTER))
521 return ERR_PTR(-EINVAL);
522
523 do {
524 ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
525 vmf, mpol, ilx);
526 if (!IS_ERR(ret))
527 break;
528 err = PTR_ERR(ret);
529 if (!order || (err && err != -EBUSY && err != -ENOMEM))
530 break;
531 count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
532 order = next_order(&orders, order);
533 } while (orders);
534
535 return ret;
536}
537
353/*538/*
354 * If we are the only user, then try to free up the swap cache.539 * If we are the only user, then try to free up the swap cache.
355 *540 *
@@ -448,140 +633,64 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, · 92 unmodified lines
448 }633 }
449}634}
450 635
451/**636static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
452 * __swap_cache_prepare_and_add - Prepare the folio and add it to swap cache.637 struct mempolicy *mpol, pgoff_t ilx,
453 * @entry: swap entry to be bound to the folio.638 struct swap_iocb **plug, bool readahead)
454 * @folio: folio to be added.
455 * @gfp: memory allocation flags for charge, can be 0 if @charged if true.
456 * @charged: if the folio is already charged.
457 *
458 * Update the swap_map and add folio as swap cache, typically before swapin.
459 * All swap slots covered by the folio must have a non-zero swap count.
460 *
461 * Context: Caller must protect the swap device with reference count or locks.
462 * Return: Returns the folio being added on success. Returns the existing folio
463 * if @entry is already cached. Returns NULL if raced with swapin or swapoff.
464 */
465static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
466 struct folio *folio,
467 gfp_t gfp, bool charged)
468{639{
469 struct folio *swapcache = NULL;640 struct folio *folio;
470 void *shadow;
471 int ret;
472
473 __folio_set_locked(folio);
474 __folio_set_swapbacked(folio);
475
476 if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry))
477 goto failed;
478 641
479 for (;;) {642 do {
480 ret = swap_cache_add_folio(folio, entry, &shadow);643 folio = swap_cache_get_folio(entry);
481 if (!ret)644 if (folio)
482 break;645 return folio;
646 folio = swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
647 } while (PTR_ERR(folio) == -EEXIST);
483 648
484 /*649 if (IS_ERR_OR_NULL(folio))
485 * Large order allocation needs special handling on650 return NULL;
486 * race: if a smaller folio exists in cache, swapin needs
487 * to fallback to order 0, and doing a swap cache lookup
488 * might return a folio that is irrelevant to the faulting
489 * entry because @entry is aligned down. Just return NULL.
490 */
491 if (ret != -EEXIST || folio_test_large(folio))
492 goto failed;
493 651
494 swapcache = swap_cache_get_folio(entry);652 swap_read_folio(folio, plug);
495 if (swapcache)653 if (readahead) {
496 goto failed;654 folio_set_readahead(folio);
655 count_vm_event(SWAP_RA);
497 }656 }
498 657
499 memcg1_swapin(entry, folio_nr_pages(folio));
500 if (shadow)
501 workingset_refault(folio, shadow);
502
503 /* Caller will initiate read into locked folio */
504 folio_add_lru(folio);
505 return folio;658 return folio;
506
507failed:
508 folio_unlock(folio);
509 return swapcache;
510}659}
511 660
512/**661/**
513 * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.662 * swapin_sync - swap-in one or multiple entries skipping readahead.
514 * @entry: the swapped out swap entry to be binded to the folio.663 * @entry: swap entry indicating the target slot
515 * @gfp_mask: memory allocation flags664 * @gfp: memory allocation flags
665 * @orders: allocation orders
666 * @vmf: fault information
516 * @mpol: NUMA memory allocation policy to be applied667 * @mpol: NUMA memory allocation policy to be applied
517 * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE668 * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
518 * @new_page_allocated: sets true if allocation happened, false otherwise
519 *669 *
520 * Allocate a folio in the swap cache for one swap slot, typically before670 * This allocates a folio suitable for given @orders, or returns the
521 * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by671 * existing folio in the swap cache for @entry. This initiates the IO, too,
522 * @entry must have a non-zero swap count (swapped out).672 * if needed. @entry is rounded down if @orders allow large allocation.
523 * Currently only supports order 0.
524 *673 *
525 * Context: Caller must protect the swap device with reference count or locks.674 * Context: Caller must ensure @entry is valid and pin the swap device with refcount.
526 * Return: Returns the existing folio if @entry is cached already. Returns675 * Return: Returns the folio on success, error code if failed.
527 * NULL if failed due to -ENOMEM or @entry have a swap count < 1.
528 */676 */
529struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,677struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
530 struct mempolicy *mpol, pgoff_t ilx,678 struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
531 bool *new_page_allocated)
532{679{
533 struct swap_info_struct *si = __swap_entry_to_info(entry);
534 struct folio *folio;680 struct folio *folio;
535 struct folio *result = NULL;
536 681
537 *new_page_allocated = false;682 do {
538 /* Check the swap cache again for readahead path. */683 folio = swap_cache_get_folio(entry);
539 folio = swap_cache_get_folio(entry);684 if (folio)
540 if (folio)685 return folio;
541 return folio;686 folio = swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
542 687 } while (PTR_ERR(folio) == -EEXIST);
543 /* Skip allocation for unused and bad swap slot for readahead. */
544 if (!swap_entry_swapped(si, entry))
545 return NULL;
546
547 /* Allocate a new folio to be added into the swap cache. */
548 folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
549 if (!folio)
550 return NULL;
551 /* Try add the new folio, returns existing folio or NULL on failure. */
552 result = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
553 if (result == folio)
554 *new_page_allocated = true;
555 else
556 folio_put(folio);
557 return result;
558}
559 688
560/**689 if (IS_ERR(folio))
561 * swapin_folio - swap-in one or multiple entries skipping readahead.690 return folio;
562 * @entry: starting swap entry to swap in
563 * @folio: a new allocated and charged folio
564 *
565 * Reads @entry into @folio, @folio will be added to the swap cache.
566 * If @folio is a large folio, the @entry will be rounded down to align
567 * with the folio size.
568 *
569 * Return: returns pointer to @folio on success. If folio is a large folio
570 * and this raced with another swapin, NULL will be returned to allow fallback
571 * to order 0. Else, if another folio was already added to the swap cache,
572 * return that swap cache folio instead.
573 */
574struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
575{
576 struct folio *swapcache;
577 pgoff_t offset = swp_offset(entry);
578 unsigned long nr_pages = folio_nr_pages(folio);
579 691
580 entry = swp_entry(swp_type(entry), round_down(offset, nr_pages));692 swap_read_folio(folio, NULL);
581 swapcache = __swap_cache_prepare_and_add(entry, folio, 0, true);693 return folio;
582 if (swapcache == folio)
583 swap_read_folio(folio, NULL);
584 return swapcache;
585}694}
586 695
587/*696/*
@@ -595,7 +704,6 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 7 unmodified lines
595 struct swap_iocb **plug)704 struct swap_iocb **plug)
596{705{
597 struct swap_info_struct *si;706 struct swap_info_struct *si;
598 bool page_allocated;
599 struct mempolicy *mpol;707 struct mempolicy *mpol;
600 pgoff_t ilx;708 pgoff_t ilx;
601 struct folio *folio;709 struct folio *folio;
@@ -605,13 +713,9 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 3 unmodified lines
605 return NULL;713 return NULL;
606 714
607 mpol = get_vma_policy(vma, addr, 0, &ilx);715 mpol = get_vma_policy(vma, addr, 0, &ilx);
608 folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,716 folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false);
609 &page_allocated);
610 mpol_cond_put(mpol);717 mpol_cond_put(mpol);
611 718
612 if (page_allocated)
613 swap_read_folio(folio, plug);
614
615 put_swap_device(si);719 put_swap_device(si);
616 return folio;720 return folio;
617}721}
@@ -696,7 +800,7 @@ static unsigned long swapin_nr_pages(unsigned long offset) · 78 unmodified lines
696 * are fairly likely to have been swapped out from the same node.800 * are fairly likely to have been swapped out from the same node.
697 */801 */
698struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,802struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
699 struct mempolicy *mpol, pgoff_t ilx)803 struct mempolicy *mpol, pgoff_t ilx)
700{804{
701 struct folio *folio;805 struct folio *folio;
702 unsigned long entry_offset = swp_offset(entry);806 unsigned long entry_offset = swp_offset(entry);
@@ -706,7 +810,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 3 unmodified lines
706 struct swap_info_struct *si = __swap_entry_to_info(entry);810 struct swap_info_struct *si = __swap_entry_to_info(entry);
707 struct blk_plug plug;811 struct blk_plug plug;
708 struct swap_iocb *splug = NULL;812 struct swap_iocb *splug = NULL;
709 bool page_allocated;813 swp_entry_t ra_entry;
710 814
711 mask = swapin_nr_pages(offset) - 1;815 mask = swapin_nr_pages(offset) - 1;
712 if (!mask)816 if (!mask)
@@ -723,18 +827,11 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 10 unmodified lines
723 blk_start_plug(&plug);827 blk_start_plug(&plug);
724 for (offset = start_offset; offset <= end_offset ; offset++) {828 for (offset = start_offset; offset <= end_offset ; offset++) {
725 /* Ok, do the async read-ahead now */829 /* Ok, do the async read-ahead now */
726 folio = swap_cache_alloc_folio(830 ra_entry = swp_entry(swp_type(entry), offset);
727 swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx,831 folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx,
728 &page_allocated);832 &splug, offset != entry_offset);
729 if (!folio)833 if (!folio)
730 continue;834 continue;
731 if (page_allocated) {
732 swap_read_folio(folio, &splug);
733 if (offset != entry_offset) {
734 folio_set_readahead(folio);
735 count_vm_event(SWAP_RA);
736 }
737 }
738 folio_put(folio);835 folio_put(folio);
739 }836 }
740 blk_finish_plug(&plug);837 blk_finish_plug(&plug);
@@ -742,11 +839,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 1 unmodified line
742 lru_add_drain(); /* Push any new pages onto the LRU now */839 lru_add_drain(); /* Push any new pages onto the LRU now */
743skip:840skip:
744 /* The page was likely read above, so no need for plugging here */841 /* The page was likely read above, so no need for plugging here */
745 folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,842 return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
746 &page_allocated);
747 if (unlikely(page_allocated))
748 swap_read_folio(folio, NULL);
749 return folio;
750}843}
751 844
752static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,845static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
@@ -812,8 +905,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 59 unmodified lines
812 pte_t *pte = NULL, pentry;905 pte_t *pte = NULL, pentry;
813 int win;906 int win;
814 unsigned long start, end, addr;907 unsigned long start, end, addr;
815 pgoff_t ilx;908 pgoff_t ilx = targ_ilx;
816 bool page_allocated;
817 909
818 win = swap_vma_ra_win(vmf, &start, &end);910 win = swap_vma_ra_win(vmf, &start, &end);
819 if (win == 1)911 if (win == 1)
@@ -847,19 +939,12 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 27 unmodified lines
847 if (!si)939 if (!si)
848 continue;940 continue;
849 }941 }
850 folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,942 folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx,
851 &page_allocated);943 &splug, addr != vmf->address);
852 if (si)944 if (si)
853 put_swap_device(si);945 put_swap_device(si);
854 if (!folio)946 if (!folio)
855 continue;947 continue;
856 if (page_allocated) {
857 swap_read_folio(folio, &splug);
858 if (addr != vmf->address) {
859 folio_set_readahead(folio);
860 count_vm_event(SWAP_RA);
861 }
862 }
863 folio_put(folio);948 folio_put(folio);
864 }949 }
865 if (pte)950 if (pte)
@@ -869,10 +954,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 3 unmodified lines
869 lru_add_drain();954 lru_add_drain();
870skip:955skip:
871 /* The folio was likely read above, so no need for plugging here */956 /* The folio was likely read above, so no need for plugging here */
872 folio = swap_cache_alloc_folio(targ_entry, gfp_mask, mpol, targ_ilx,957 folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
873 &page_allocated);958 NULL, false);
874 if (unlikely(page_allocated))
875 swap_read_folio(folio, NULL);
876 return folio;959 return folio;
877}960}
878 961
mm/swap_table.h+149−30
@@ -11,6 +11,11 @@ struct swap_table { · 10 unmodified lines
11 atomic_long_t entries[SWAPFILE_CLUSTER];11 atomic_long_t entries[SWAPFILE_CLUSTER];
12};12};
13 13
14/* For storing memcg private id */
15struct swap_memcg_table {
16 unsigned short id[SWAPFILE_CLUSTER];
17};
18
14#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)19#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)
15 20
16/*21/*
@@ -21,12 +26,14 @@ struct swap_table { · 4 unmodified lines
21 * Swap table entry type and bits layouts:26 * Swap table entry type and bits layouts:
22 *27 *
23 * NULL: |---------------- 0 ---------------| - Free slot28 * NULL: |---------------- 0 ---------------| - Free slot
24 * Shadow: | SWAP_COUNT |---- SHADOW_VAL ---|1| - Swapped out slot29 * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
25 * PFN: | SWAP_COUNT |------ PFN -------|10| - Cached slot30 * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
26 * Pointer: |----------- Pointer ----------|100| - (Unused)31 * Pointer: |----------- Pointer ----------|100| - (Unused)
27 * Bad: |------------- 1 -------------|1000| - Bad slot32 * Bad: |------------- 1 -------------|1000| - Bad slot
28 *33 *
29 * SWAP_COUNT is `SWP_TB_COUNT_BITS` long, each entry is an atomic long.34 * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
35 * and together they form the `SWP_TB_FLAGS_BITS` wide flags field.
36 * Each entry is an atomic long.
30 *37 *
31 * Usages:38 * Usages:
32 *39 *
@@ -49,14 +56,6 @@ struct swap_table { · 16 unmodified lines
49 * - Bad: Swap slot is reserved, protects swap header or holes on swap devices.56 * - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
50 */57 */
51 58
52#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
53#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
54#elif defined(MAX_PHYSMEM_BITS)
55#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
56#else
57#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
58#endif
59
60/* NULL Entry, all 0 */59/* NULL Entry, all 0 */
61#define SWP_TB_NULL 0UL60#define SWP_TB_NULL 0UL
62 61
@@ -64,22 +63,26 @@ struct swap_table { · 1 unmodified line
64#define SWP_TB_SHADOW_MARK 0b1UL63#define SWP_TB_SHADOW_MARK 0b1UL
65 64
66/* Cached: PFN */65/* Cached: PFN */
67#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWP_TB_PFN_MARK_BITS)66#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWAP_CACHE_PFN_MARK_BITS)
68#define SWP_TB_PFN_MARK 0b10UL67#define SWP_TB_PFN_MARK 0b10UL
69#define SWP_TB_PFN_MARK_BITS 268#define SWP_TB_PFN_MARK_MASK (BIT(SWAP_CACHE_PFN_MARK_BITS) - 1)
70#define SWP_TB_PFN_MARK_MASK (BIT(SWP_TB_PFN_MARK_BITS) - 1)
71 69
72/* SWAP_COUNT part for PFN or shadow, the width can be shrunk or extended */70/* Flags: For PFN or shadow, contains SWAP_COUNT, width changes */
73#define SWP_TB_COUNT_BITS min(4, BITS_PER_LONG - SWP_TB_PFN_BITS)71#define SWP_TB_FLAGS_BITS min(5, BITS_PER_LONG - SWP_TB_PFN_BITS)
72#define SWP_TB_COUNT_BITS (SWP_TB_FLAGS_BITS - SWAP_TABLE_HAS_ZEROFLAG)
73#define SWP_TB_FLAGS_MASK (~((~0UL) >> SWP_TB_FLAGS_BITS))
74#define SWP_TB_COUNT_MASK (~((~0UL) >> SWP_TB_COUNT_BITS))74#define SWP_TB_COUNT_MASK (~((~0UL) >> SWP_TB_COUNT_BITS))
75#define SWP_TB_FLAGS_SHIFT (BITS_PER_LONG - SWP_TB_FLAGS_BITS)
75#define SWP_TB_COUNT_SHIFT (BITS_PER_LONG - SWP_TB_COUNT_BITS)76#define SWP_TB_COUNT_SHIFT (BITS_PER_LONG - SWP_TB_COUNT_BITS)
76#define SWP_TB_COUNT_MAX ((1 << SWP_TB_COUNT_BITS) - 1)77#define SWP_TB_COUNT_MAX ((1 << SWP_TB_COUNT_BITS) - 1)
78/* The first flag is zero bit (SWAP_TABLE_HAS_ZEROFLAG) */
79#define SWP_TB_ZERO_FLAG BIT(BITS_PER_LONG - SWP_TB_FLAGS_BITS)
77 80
78/* Bad slot: ends with 0b1000 and rests of bits are all 1 */81/* Bad slot: ends with 0b1000 and rests of bits are all 1 */
79#define SWP_TB_BAD ((~0UL) << 3)82#define SWP_TB_BAD ((~0UL) << 3)
80 83
81/* Macro for shadow offset calculation */84/* Macro for shadow offset calculation */
82#define SWAP_COUNT_SHIFT SWP_TB_COUNT_BITS85#define SWAP_COUNT_SHIFT SWP_TB_FLAGS_BITS
83 86
84/*87/*
85 * Helpers for casting one type of info into a swap table entry.88 * Helpers for casting one type of info into a swap table entry.
@@ -97,40 +100,47 @@ static inline unsigned long __count_to_swp_tb(unsigned char count) · 11 unmodified lines
97 * used (count > 0 && count < SWP_TB_COUNT_MAX), and100 * used (count > 0 && count < SWP_TB_COUNT_MAX), and
98 * overflow (count == SWP_TB_COUNT_MAX).101 * overflow (count == SWP_TB_COUNT_MAX).
99 */102 */
100 BUILD_BUG_ON(SWP_TB_COUNT_MAX < 2 || SWP_TB_COUNT_BITS < 2);103 BUILD_BUG_ON(SWP_TB_COUNT_BITS < SWAP_COUNT_MIN_BITS);
101 VM_WARN_ON(count > SWP_TB_COUNT_MAX);104 VM_WARN_ON(count > SWP_TB_COUNT_MAX);
102 return ((unsigned long)count) << SWP_TB_COUNT_SHIFT;105 return ((unsigned long)count) << SWP_TB_COUNT_SHIFT;
103}106}
104 107
105static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned int count)108static inline unsigned long __flags_to_swp_tb(unsigned char flags)
109{
110 BUILD_BUG_ON(SWP_TB_FLAGS_BITS > BITS_PER_BYTE);
111 VM_WARN_ON(flags >> SWP_TB_FLAGS_BITS);
112 return ((unsigned long)flags) << SWP_TB_FLAGS_SHIFT;
113}
114
115static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned char flags)
106{116{
107 unsigned long swp_tb;117 unsigned long swp_tb;
108 118
109 BUILD_BUG_ON(sizeof(unsigned long) != sizeof(void *));119 BUILD_BUG_ON(sizeof(unsigned long) != sizeof(void *));
110 BUILD_BUG_ON(SWAP_CACHE_PFN_BITS >120 BUILD_BUG_ON(SWAP_CACHE_PFN_BITS >
111 (BITS_PER_LONG - SWP_TB_PFN_MARK_BITS - SWP_TB_COUNT_BITS));121 (BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - SWP_TB_FLAGS_BITS));
112 122
113 swp_tb = (pfn << SWP_TB_PFN_MARK_BITS) | SWP_TB_PFN_MARK;123 swp_tb = (pfn << SWAP_CACHE_PFN_MARK_BITS) | SWP_TB_PFN_MARK;
114 VM_WARN_ON_ONCE(swp_tb & SWP_TB_COUNT_MASK);124 VM_WARN_ON_ONCE(swp_tb & SWP_TB_FLAGS_MASK);
115 125
116 return swp_tb | __count_to_swp_tb(count);126 return swp_tb | __flags_to_swp_tb(flags);
117}127}
118 128
119static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned int count)129static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned char flags)
120{130{
121 return pfn_to_swp_tb(folio_pfn(folio), count);131 return pfn_to_swp_tb(folio_pfn(folio), flags);
122}132}
123 133
124static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned int count)134static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned char flags)
125{135{
126 BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) !=136 BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) !=
127 BITS_PER_BYTE * sizeof(unsigned long));137 BITS_PER_BYTE * sizeof(unsigned long));
128 BUILD_BUG_ON((unsigned long)xa_mk_value(0) != SWP_TB_SHADOW_MARK);138 BUILD_BUG_ON((unsigned long)xa_mk_value(0) != SWP_TB_SHADOW_MARK);
129 139
130 VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow));140 VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow));
131 VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_COUNT_MASK));141 VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_FLAGS_MASK));
132 142
133 return (unsigned long)shadow | __count_to_swp_tb(count) | SWP_TB_SHADOW_MARK;143 return (unsigned long)shadow | SWP_TB_SHADOW_MARK | __flags_to_swp_tb(flags);
134}144}
135 145
136/*146/*
@@ -168,14 +178,14 @@ static inline bool swp_tb_is_countable(unsigned long swp_tb) · 31 unmodified lines
168static inline struct folio *swp_tb_to_folio(unsigned long swp_tb)178static inline struct folio *swp_tb_to_folio(unsigned long swp_tb)
169{179{
170 VM_WARN_ON(!swp_tb_is_folio(swp_tb));180 VM_WARN_ON(!swp_tb_is_folio(swp_tb));
171 return pfn_folio((swp_tb & ~SWP_TB_COUNT_MASK) >> SWP_TB_PFN_MARK_BITS);181 return pfn_folio((swp_tb & ~SWP_TB_FLAGS_MASK) >> SWAP_CACHE_PFN_MARK_BITS);
172}182}
173 183
174static inline void *swp_tb_to_shadow(unsigned long swp_tb)184static inline void *swp_tb_to_shadow(unsigned long swp_tb)
175{185{
176 VM_WARN_ON(!swp_tb_is_shadow(swp_tb));186 VM_WARN_ON(!swp_tb_is_shadow(swp_tb));
177 /* No shift needed, xa_value is stored as it is in the lower bits. */187 /* No shift needed, xa_value is stored as it is in the lower bits. */
178 return (void *)(swp_tb & ~SWP_TB_COUNT_MASK);188 return (void *)(swp_tb & ~SWP_TB_FLAGS_MASK);
179}189}
180 190
181static inline unsigned char __swp_tb_get_count(unsigned long swp_tb)191static inline unsigned char __swp_tb_get_count(unsigned long swp_tb)
@@ -184,6 +194,12 @@ static inline unsigned char __swp_tb_get_count(unsigned long swp_tb) · 2 unmodified lines
184 return ((swp_tb & SWP_TB_COUNT_MASK) >> SWP_TB_COUNT_SHIFT);194 return ((swp_tb & SWP_TB_COUNT_MASK) >> SWP_TB_COUNT_SHIFT);
185}195}
186 196
197static inline unsigned char __swp_tb_get_flags(unsigned long swp_tb)
198{
199 VM_WARN_ON(!swp_tb_is_countable(swp_tb));
200 return ((swp_tb & SWP_TB_FLAGS_MASK) >> SWP_TB_FLAGS_SHIFT);
201}
202
187static inline int swp_tb_get_count(unsigned long swp_tb)203static inline int swp_tb_get_count(unsigned long swp_tb)
188{204{
189 if (swp_tb_is_countable(swp_tb))205 if (swp_tb_is_countable(swp_tb))
@@ -247,4 +263,107 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci, · 57 unmodified lines
247 263
248 return swp_tb;264 return swp_tb;
249}265}
266
267static inline void __swap_table_set_zero(struct swap_cluster_info *ci,
268 unsigned int ci_off)
269{
270#if SWAP_TABLE_HAS_ZEROFLAG
271 unsigned long swp_tb = __swap_table_get(ci, ci_off);
272
273 BUILD_BUG_ON(SWP_TB_ZERO_FLAG & ~SWP_TB_FLAGS_MASK);
274 VM_WARN_ON(!swp_tb_is_countable(swp_tb));
275 swp_tb |= SWP_TB_ZERO_FLAG;
276 __swap_table_set(ci, ci_off, swp_tb);
277#else
278 lockdep_assert_held(&ci->lock);
279 __set_bit(ci_off, ci->zero_bitmap);
280#endif
281}
282
283static inline bool __swap_table_test_zero(struct swap_cluster_info *ci,
284 unsigned int ci_off)
285{
286#if SWAP_TABLE_HAS_ZEROFLAG
287 unsigned long swp_tb = __swap_table_get(ci, ci_off);
288
289 VM_WARN_ON(!swp_tb_is_countable(swp_tb));
290 return !!(swp_tb & SWP_TB_ZERO_FLAG);
291#else
292 return test_bit(ci_off, ci->zero_bitmap);
293#endif
294}
295
296static inline void __swap_table_clear_zero(struct swap_cluster_info *ci,
297 unsigned int ci_off)
298{
299#if SWAP_TABLE_HAS_ZEROFLAG
300 unsigned long swp_tb = __swap_table_get(ci, ci_off);
301
302 VM_WARN_ON(!swp_tb_is_countable(swp_tb));
303 swp_tb &= ~SWP_TB_ZERO_FLAG;
304 __swap_table_set(ci, ci_off, swp_tb);
305#else
306 lockdep_assert_held(&ci->lock);
307 __clear_bit(ci_off, ci->zero_bitmap);
308#endif
309}
310
311#ifdef CONFIG_MEMCG
312static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
313 unsigned int ci_off, unsigned long nr, unsigned short id)
314{
315 lockdep_assert_held(&ci->lock);
316 VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
317 if (WARN_ON_ONCE(!ci->memcg_table))
318 return;
319 do {
320 ci->memcg_table->id[ci_off++] = id;
321 } while (--nr);
322}
323
324static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
325 unsigned int ci_off)
326{
327 lockdep_assert_held(&ci->lock);
328 VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
329 if (unlikely(!ci->memcg_table))
330 return 0;
331 return ci->memcg_table->id[ci_off];
332}
333
334static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
335 unsigned int ci_off,
336 unsigned long nr)
337{
338 unsigned short old = __swap_cgroup_get(ci, ci_off);
339
340 if (!old)
341 return 0;
342 do {
343 VM_WARN_ON_ONCE(ci->memcg_table->id[ci_off] != old);
344 ci->memcg_table->id[ci_off++] = 0;
345 } while (--nr);
346
347 return old;
348}
349#else
350static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
351 unsigned int ci_off, unsigned long nr, unsigned short id)
352{
353}
354
355static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
356 unsigned int ci_off)
357{
358 return 0;
359}
360
361static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
362 unsigned int ci_off,
363 unsigned long nr)
364{
365 return 0;
366}
367#endif
368
250#endif369#endif
mm/swapfile.c+302−158
@@ -45,7 +45,6 @@ · 44 unmodified lines
45 45
46#include <asm/tlbflush.h>46#include <asm/tlbflush.h>
47#include <linux/leafops.h>47#include <linux/leafops.h>
48#include <linux/swap_cgroup.h>
49#include "swap_table.h"48#include "swap_table.h"
50#include "internal.h"49#include "internal.h"
51#include "swap.h"50#include "swap.h"
@@ -133,7 +132,7 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = { · 81 unmodified lines
133/* May return NULL on invalid type, caller must check for NULL return */132/* May return NULL on invalid type, caller must check for NULL return */
134static struct swap_info_struct *swap_type_to_info(int type)133static struct swap_info_struct *swap_type_to_info(int type)
135{134{
136 if (type >= MAX_SWAPFILES)135 if (type < 0 || type >= MAX_SWAPFILES)
137 return NULL;136 return NULL;
138 return READ_ONCE(swap_info[type]); /* rcu_dereference() */137 return READ_ONCE(swap_info[type]); /* rcu_dereference() */
139}138}
@@ -411,20 +410,7 @@ static inline unsigned int cluster_offset(struct swap_info_struct *si, · 271 unmodified lines
411 return cluster_index(si, ci) * SWAPFILE_CLUSTER;410 return cluster_index(si, ci) * SWAPFILE_CLUSTER;
412}411}
413 412
414static struct swap_table *swap_table_alloc(gfp_t gfp)413static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
415{
416 struct folio *folio;
417
418 if (!SWP_TABLE_USE_PAGE)
419 return kmem_cache_zalloc(swap_table_cachep, gfp);
420
421 folio = folio_alloc(gfp | __GFP_ZERO, 0);
422 if (folio)
423 return folio_address(folio);
424 return NULL;
425}
426
427static void swap_table_free_folio_rcu_cb(struct rcu_head *head)
428{414{
429 struct folio *folio;415 struct folio *folio;
430 416
@@ -432,15 +418,76 @@ static void swap_table_free_folio_rcu_cb(struct rcu_head *head) · 1 unmodified line
432 folio_put(folio);418 folio_put(folio);
433}419}
434 420
435static void swap_table_free(struct swap_table *table)421static void swap_cluster_free_table(struct swap_cluster_info *ci)
436{422{
423 struct swap_table *table;
424
425#ifdef CONFIG_MEMCG
426 kfree(ci->memcg_table);
427 ci->memcg_table = NULL;
428#endif
429
430#if !SWAP_TABLE_HAS_ZEROFLAG
431 kfree(ci->zero_bitmap);
432 ci->zero_bitmap = NULL;
433#endif
434
435 table = (struct swap_table *)rcu_access_pointer(ci->table);
436 if (!table)
437 return;
438
439 rcu_assign_pointer(ci->table, NULL);
437 if (!SWP_TABLE_USE_PAGE) {440 if (!SWP_TABLE_USE_PAGE) {
438 kmem_cache_free(swap_table_cachep, table);441 kmem_cache_free(swap_table_cachep, table);
439 return;442 return;
440 }443 }
441 444
442 call_rcu(&(folio_page(virt_to_folio(table), 0)->rcu_head),445 call_rcu(&(folio_page(virt_to_folio(table), 0)->rcu_head),
443 swap_table_free_folio_rcu_cb);446 swap_cluster_free_table_folio_rcu_cb);
447}
448
449static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
450{
451 struct swap_table *table = NULL;
452 struct folio *folio;
453
454 /* The cluster must be empty and not on any list during allocation. */
455 VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
456 if (rcu_access_pointer(ci->table))
457 return 0;
458
459 if (SWP_TABLE_USE_PAGE) {
460 folio = folio_alloc(gfp | __GFP_ZERO, 0);
461 if (folio)
462 table = folio_address(folio);
463 } else {
464 table = kmem_cache_zalloc(swap_table_cachep, gfp);
465 }
466 if (!table)
467 return -ENOMEM;
468
469 rcu_assign_pointer(ci->table, table);
470
471#ifdef CONFIG_MEMCG
472 if (!mem_cgroup_disabled()) {
473 VM_WARN_ON_ONCE(ci->memcg_table);
474 ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
475 if (!ci->memcg_table) {
476 swap_cluster_free_table(ci);
477 return -ENOMEM;
478 }
479 }
480#endif
481
482#if !SWAP_TABLE_HAS_ZEROFLAG
483 VM_WARN_ON_ONCE(ci->zero_bitmap);
484 ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
485 if (!ci->zero_bitmap) {
486 swap_cluster_free_table(ci);
487 return -ENOMEM;
488 }
489#endif
490 return 0;
444}491}
445 492
446/*493/*
@@ -465,33 +512,22 @@ static void swap_cluster_assert_empty(struct swap_cluster_info *ci, · 18 unmodified lines
465 bad_slots++;512 bad_slots++;
466 else513 else
467 WARN_ON_ONCE(!swp_tb_is_null(swp_tb));514 WARN_ON_ONCE(!swp_tb_is_null(swp_tb));
515 WARN_ON_ONCE(__swap_cgroup_get(ci, ci_off));
468 } while (++ci_off < ci_end);516 } while (++ci_off < ci_end);
469 517
470 WARN_ON_ONCE(bad_slots != (swapoff ? ci->count : 0));518 WARN_ON_ONCE(bad_slots != (swapoff ? ci->count : 0));
471 WARN_ON_ONCE(nr == SWAPFILE_CLUSTER && ci->extend_table);519 WARN_ON_ONCE(nr == SWAPFILE_CLUSTER && ci->extend_table);
472}520}
473 521
474static void swap_cluster_free_table(struct swap_cluster_info *ci)
475{
476 struct swap_table *table;
477
478 /* Only empty cluster's table is allow to be freed */
479 lockdep_assert_held(&ci->lock);
480 table = (void *)rcu_dereference_protected(ci->table, true);
481 rcu_assign_pointer(ci->table, NULL);
482
483 swap_table_free(table);
484}
485
486/*522/*
487 * Allocate swap table for one cluster. Attempt an atomic allocation first,523 * Allocate swap table for one cluster. Attempt an atomic allocation first,
488 * then fallback to sleeping allocation.524 * then fallback to sleeping allocation.
489 */525 */
490static struct swap_cluster_info *526static struct swap_cluster_info *
491swap_cluster_alloc_table(struct swap_info_struct *si,527swap_cluster_populate(struct swap_info_struct *si,
492 struct swap_cluster_info *ci)528 struct swap_cluster_info *ci)
493{529{
494 struct swap_table *table;530 int ret;
495 531
496 /*532 /*
497 * Only cluster isolation from the allocator does table allocation.533 * Only cluster isolation from the allocator does table allocation.
@@ -502,14 +538,9 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 4 unmodified lines
502 lockdep_assert_held(&si->global_cluster_lock);538 lockdep_assert_held(&si->global_cluster_lock);
503 lockdep_assert_held(&ci->lock);539 lockdep_assert_held(&ci->lock);
504 540
505 /* The cluster must be free and was just isolated from the free list. */541 if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
506 VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));542 __GFP_NOWARN))
507
508 table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
509 if (table) {
510 rcu_assign_pointer(ci->table, table);
511 return ci;543 return ci;
512 }
513 544
514 /*545 /*
515 * Try a sleep allocation. Each isolated free cluster may cause546 * Try a sleep allocation. Each isolated free cluster may cause
@@ -521,7 +552,8 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 5 unmodified lines
521 spin_unlock(&si->global_cluster_lock);552 spin_unlock(&si->global_cluster_lock);
522 local_unlock(&percpu_swap_cluster.lock);553 local_unlock(&percpu_swap_cluster.lock);
523 554
524 table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL);555 ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
556 GFP_KERNEL);
525 557
526 /*558 /*
527 * Back to atomic context. We might have migrated to a new CPU with a559 * Back to atomic context. We might have migrated to a new CPU with a
@@ -536,20 +568,11 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 8 unmodified lines
536 spin_lock(&si->global_cluster_lock);568 spin_lock(&si->global_cluster_lock);
537 spin_lock(&ci->lock);569 spin_lock(&ci->lock);
538 570
539 /* Nothing except this helper should touch a dangling empty cluster. */571 if (ret) {
540 if (WARN_ON_ONCE(cluster_table_is_alloced(ci))) {
541 if (table)
542 swap_table_free(table);
543 return ci;
544 }
545
546 if (!table) {
547 move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);572 move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
548 spin_unlock(&ci->lock);573 spin_unlock(&ci->lock);
549 return NULL;574 return NULL;
550 }575 }
551
552 rcu_assign_pointer(ci->table, table);
553 return ci;576 return ci;
554}577}
555 578
@@ -621,12 +644,11 @@ static struct swap_cluster_info *isolate_lock_cluster( · 65 unmodified lines
621 }644 }
622 spin_unlock(&si->lock);645 spin_unlock(&si->lock);
623 646
624 if (found && !cluster_table_is_alloced(found)) {647 /* Cluster's table is freed when and only when it's on the free list. */
625 /* Only an empty free cluster's swap table can be freed. */648 if (found && flags == CLUSTER_FLAG_FREE) {
626 VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);
627 VM_WARN_ON_ONCE(list != &si->free_clusters);649 VM_WARN_ON_ONCE(list != &si->free_clusters);
628 VM_WARN_ON_ONCE(!cluster_is_empty(found));650 VM_WARN_ON_ONCE(cluster_table_is_alloced(found));
629 return swap_cluster_alloc_table(si, found);651 return swap_cluster_populate(si, found);
630 }652 }
631 653
632 return found;654 return found;
@@ -769,7 +791,6 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si, · 136 unmodified lines
769 unsigned int ci_off = offset % SWAPFILE_CLUSTER;791 unsigned int ci_off = offset % SWAPFILE_CLUSTER;
770 unsigned long idx = offset / SWAPFILE_CLUSTER;792 unsigned long idx = offset / SWAPFILE_CLUSTER;
771 struct swap_cluster_info *ci;793 struct swap_cluster_info *ci;
772 struct swap_table *table;
773 int ret = 0;794 int ret = 0;
774 795
775 /* si->max may got shrunk by swap swap_activate() */796 /* si->max may got shrunk by swap swap_activate() */
@@ -790,12 +811,9 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si, · 14 unmodified lines
790 }811 }
791 812
792 ci = cluster_info + idx;813 ci = cluster_info + idx;
793 if (!ci->table) {814 /* Need to allocate swap table first for initial bad slot marking. */
794 table = swap_table_alloc(GFP_KERNEL);815 if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
795 if (!table)816 return -ENOMEM;
796 return -ENOMEM;
797 rcu_assign_pointer(ci->table, table);
798 }
799 spin_lock(&ci->lock);817 spin_lock(&ci->lock);
800 /* Check for duplicated bad swap slots. */818 /* Check for duplicated bad swap slots. */
801 if (__swap_table_xchg(ci, ci_off, SWP_TB_BAD) != SWP_TB_NULL) {819 if (__swap_table_xchg(ci, ci_off, SWP_TB_BAD) != SWP_TB_NULL) {
@@ -922,8 +940,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si, · 120 unmodified lines
922 order = 0;940 order = 0;
923 nr_pages = 1;941 nr_pages = 1;
924 swap_cluster_assert_empty(ci, ci_off, 1, false);942 swap_cluster_assert_empty(ci, ci_off, 1, false);
925 /* Sets a fake shadow as placeholder */943 /* Fake shadow placeholder with no flag, hibernation does not use the zeromap */
926 __swap_table_set(ci, ci_off, shadow_to_swp_tb(NULL, 1));944 __swap_table_set(ci, ci_off, __swp_tb_mk_count(shadow_to_swp_tb(NULL, 0), 1));
927 } else {945 } else {
928 /* Allocation without folio is only possible with hibernation */946 /* Allocation without folio is only possible with hibernation */
929 WARN_ON_ONCE(1);947 WARN_ON_ONCE(1);
@@ -1054,6 +1072,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) · 124 unmodified lines
1054 swap_cluster_unlock(ci);1072 swap_cluster_unlock(ci);
1055 if (to_scan <= 0)1073 if (to_scan <= 0)
1056 break;1074 break;
1075 cond_resched();
1057 }1076 }
1058}1077}
1059 1078
@@ -1295,14 +1314,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, · 235 unmodified lines
1295 void (*swap_slot_free_notify)(struct block_device *, unsigned long);1314 void (*swap_slot_free_notify)(struct block_device *, unsigned long);
1296 unsigned int i;1315 unsigned int i;
1297 1316
1298 /*1317 for (i = 0; i < nr_entries; i++)
1299 * Use atomic clear_bit operations only on zeromap instead of non-atomic
1300 * bitmap_clear to prevent adjacent bits corruption due to simultaneous writes.
1301 */
1302 for (i = 0; i < nr_entries; i++) {
1303 clear_bit(offset + i, si->zeromap);
1304 zswap_invalidate(swp_entry(si->type, offset + i));1318 zswap_invalidate(swp_entry(si->type, offset + i));
1305 }
1306 1319
1307 if (si->flags & SWP_BLKDEV)1320 if (si->flags & SWP_BLKDEV)
1308 swap_slot_free_notify =1321 swap_slot_free_notify =
@@ -1442,8 +1455,10 @@ start_over: · 133 unmodified lines
1442}1455}
1443 1456
1444static int swap_extend_table_alloc(struct swap_info_struct *si,1457static int swap_extend_table_alloc(struct swap_info_struct *si,
1445 struct swap_cluster_info *ci, gfp_t gfp)1458 struct swap_cluster_info *ci,
1459 unsigned int ci_off, gfp_t gfp)
1446{1460{
1461 int count;
1447 void *table;1462 void *table;
1448 1463
1449 table = kzalloc(sizeof(ci->extend_table[0]) * SWAPFILE_CLUSTER, gfp);1464 table = kzalloc(sizeof(ci->extend_table[0]) * SWAPFILE_CLUSTER, gfp);
@@ -1451,11 +1466,27 @@ static int swap_extend_table_alloc(struct swap_info_struct *si, · 1 unmodified line
1451 return -ENOMEM;1466 return -ENOMEM;
1452 1467
1453 spin_lock(&ci->lock);1468 spin_lock(&ci->lock);
1454 if (!ci->extend_table)1469 /*
1455 ci->extend_table = table;1470 * Extend table allocation requires releasing ci lock first so it's
1456 else1471 * possible that the slot has been freed, no longer overflowed, or
1457 kfree(table);1472 * a concurrent extend table allocation has already succeeded, so
1473 * the allocation is no longer needed.
1474 */
1475 if (!cluster_table_is_alloced(ci))
1476 goto out_free;
1477 count = swp_tb_get_count(__swap_table_get(ci, ci_off));
1478 if (count < (SWP_TB_COUNT_MAX - 1))
1479 goto out_free;
1480 if (ci->extend_table)
1481 goto out_free;
1482
1483 ci->extend_table = table;
1484 spin_unlock(&ci->lock);
1485 return 0;
1486
1487out_free:
1458 spin_unlock(&ci->lock);1488 spin_unlock(&ci->lock);
1489 kfree(table);
1459 return 0;1490 return 0;
1460}1491}
1461 1492
@@ -1471,7 +1502,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp) · 9 unmodified lines
1471 return 0;1502 return 0;
1472 1503
1473 ci = __swap_offset_to_cluster(si, offset);1504 ci = __swap_offset_to_cluster(si, offset);
1474 ret = swap_extend_table_alloc(si, ci, gfp);1505 ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
1475 1506
1476 put_swap_device(si);1507 put_swap_device(si);
1477 return ret;1508 return ret;
@@ -1518,13 +1549,21 @@ static void __swap_cluster_put_entry(struct swap_cluster_info *ci, · 40 unmodified lines
1518 if (count == (SWP_TB_COUNT_MAX - 1)) {1549 if (count == (SWP_TB_COUNT_MAX - 1)) {
1519 ci->extend_table[ci_off] = 0;1550 ci->extend_table[ci_off] = 0;
1520 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, count));1551 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, count));
1521 swap_extend_table_try_free(ci);
1522 } else {1552 } else {
1523 ci->extend_table[ci_off] = count;1553 ci->extend_table[ci_off] = count;
1524 }1554 }
1525 } else {1555 } else {
1526 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, --count));1556 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, --count));
1527 }1557 }
1558
1559 /*
1560 * `SWP_TB_COUNT_MAX - 1` triggers extend table allocation. If the
1561 * count was above that, then the extend table is no longer needed,
1562 * so free it. And if we just put the count value from MAX - 1, it's
1563 * also possible that a pending dup just attached an extend table.
1564 */
1565 if (unlikely(count == SWP_TB_COUNT_MAX - 2 || count == SWP_TB_COUNT_MAX - 1))
1566 swap_extend_table_try_free(ci);
1528}1567}
1529 1568
1530/**1569/**
@@ -1664,7 +1703,7 @@ restart: · 133 unmodified lines
1664 if (unlikely(err)) {1703 if (unlikely(err)) {
1665 if (err == -ENOMEM) {1704 if (err == -ENOMEM) {
1666 spin_unlock(&ci->lock);1705 spin_unlock(&ci->lock);
1667 err = swap_extend_table_alloc(si, ci, GFP_ATOMIC);1706 err = swap_extend_table_alloc(si, ci, ci_off, GFP_ATOMIC);
1668 spin_lock(&ci->lock);1707 spin_lock(&ci->lock);
1669 if (!err)1708 if (!err)
1670 goto restart;1709 goto restart;
@@ -1730,7 +1769,7 @@ again: · 59 unmodified lines
1730 }1769 }
1731 1770
1732 /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */1771 /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
1733 if (unlikely(mem_cgroup_try_charge_swap(folio, folio->swap)))1772 if (unlikely(mem_cgroup_try_charge_swap(folio)))
1734 swap_cache_del_folio(folio);1773 swap_cache_del_folio(folio);
1735 1774
1736 if (unlikely(!folio_test_swapcache(folio)))1775 if (unlikely(!folio_test_swapcache(folio)))
@@ -1826,8 +1865,7 @@ void folio_put_swap(struct folio *folio, struct page *subpage) · 89 unmodified lines
1826 * do_swap_page()1865 * do_swap_page()
1827 * ... swapoff+swapon1866 * ... swapoff+swapon
1828 * swap_cache_alloc_folio()1867 * swap_cache_alloc_folio()
1829 * swap_cache_add_folio()1868 * // check swap_map
1830 * // check swap_map
1831 * // verify PTE not changed1869 * // verify PTE not changed
1832 *1870 *
1833 * In __swap_duplicate(), the swap_map need to be checked before1871 * In __swap_duplicate(), the swap_map need to be checked before
@@ -1873,21 +1911,44 @@ void __swap_cluster_free_entries(struct swap_info_struct *si, · 39 unmodified lines
1873 unsigned int ci_start, unsigned int nr_pages)1911 unsigned int ci_start, unsigned int nr_pages)
1874{1912{
1875 unsigned long old_tb;1913 unsigned long old_tb;
1914 unsigned short batch_id = 0, id_cur;
1876 unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;1915 unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
1877 unsigned long offset = cluster_offset(si, ci) + ci_start;1916 unsigned long ci_head = cluster_offset(si, ci);
1917 unsigned int batch_off = ci_off;
1878 1918
1879 VM_WARN_ON(ci->count < nr_pages);1919 VM_WARN_ON(ci->count < nr_pages);
1880 1920
1881 ci->count -= nr_pages;1921 ci->count -= nr_pages;
1882 do {1922 do {
1883 old_tb = __swap_table_get(ci, ci_off);1923 old_tb = __swap_table_get(ci, ci_off);
1884 /* Release the last ref, or after swap cache is dropped */1924 /*
1925 * Freeing is done after release of the last swap count
1926 * ref, or after swap cache is dropped
1927 */
1885 VM_WARN_ON(!swp_tb_is_shadow(old_tb) || __swp_tb_get_count(old_tb) > 1);1928 VM_WARN_ON(!swp_tb_is_shadow(old_tb) || __swp_tb_get_count(old_tb) > 1);
1929
1930 /* Resetting the slot to NULL also clears the inline flags. */
1886 __swap_table_set(ci, ci_off, null_to_swp_tb());1931 __swap_table_set(ci, ci_off, null_to_swp_tb());
1932 if (!SWAP_TABLE_HAS_ZEROFLAG)
1933 __swap_table_clear_zero(ci, ci_off);
1934
1935 /*
1936 * Uncharge swap slots by memcg in batches. Consecutive
1937 * slots with the same cgroup id are uncharged together.
1938 */
1939 id_cur = __swap_cgroup_clear(ci, ci_off, 1);
1940 if (batch_id != id_cur) {
1941 if (batch_id)
1942 mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
1943 batch_id = id_cur;
1944 batch_off = ci_off;
1945 }
1887 } while (++ci_off < ci_end);1946 } while (++ci_off < ci_end);
1888 1947
1889 mem_cgroup_uncharge_swap(swp_entry(si->type, offset), nr_pages);1948 if (batch_id)
1890 swap_range_free(si, offset, nr_pages);1949 mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
1950
1951 swap_range_free(si, ci_head + ci_start, nr_pages);
1891 swap_cluster_assert_empty(ci, ci_start, nr_pages, false);1952 swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
1892 1953
1893 if (!ci->count)1954 if (!ci->count)
@@ -2077,7 +2138,16 @@ out: · 183 unmodified lines
2077}2138}
2078 2139
2079#ifdef CONFIG_HIBERNATION2140#ifdef CONFIG_HIBERNATION
2080/* Allocate a slot for hibernation */2141/**
2142 * swap_alloc_hibernation_slot() - Allocate a swap slot for hibernation.
2143 * @type: swap device type index to allocate from.
2144 *
2145 * The caller must ensure the swap device is stable, either by pinning
2146 * it (SWP_HIBERNATION) or by freezing user-space.
2147 *
2148 * Return: a valid swp_entry_t on success, or an empty entry (val == 0)
2149 * on failure.
2150 */
2081swp_entry_t swap_alloc_hibernation_slot(int type)2151swp_entry_t swap_alloc_hibernation_slot(int type)
2082{2152{
2083 struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);2153 struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);
@@ -2088,46 +2158,42 @@ swp_entry_t swap_alloc_hibernation_slot(int type) · 4 unmodified lines
2088 if (!si)2158 if (!si)
2089 goto fail;2159 goto fail;
2090 2160
2091 /* This is called for allocating swap entry, not cache */2161 /*
2092 if (get_swap_device_info(si)) {2162 * Try the local cluster first if it matches the device. If
2093 if (si->flags & SWP_WRITEOK) {2163 * not, try grab a new cluster and override local cluster.
2094 /*2164 */
2095 * Try the local cluster first if it matches the device. If2165 local_lock(&percpu_swap_cluster.lock);
2096 * not, try grab a new cluster and override local cluster.2166 pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
2097 */2167 pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
2098 local_lock(&percpu_swap_cluster.lock);2168 if (pcp_si == si && pcp_offset) {
2099 pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);2169 ci = swap_cluster_lock(si, pcp_offset);
2100 pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);2170 if (cluster_is_usable(ci, 0))
2101 if (pcp_si == si && pcp_offset) {2171 offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
2102 ci = swap_cluster_lock(si, pcp_offset);2172 else
2103 if (cluster_is_usable(ci, 0))2173 swap_cluster_unlock(ci);
2104 offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
2105 else
2106 swap_cluster_unlock(ci);
2107 }
2108 if (!offset)
2109 offset = cluster_alloc_swap_entry(si, NULL);
2110 local_unlock(&percpu_swap_cluster.lock);
2111 if (offset)
2112 entry = swp_entry(si->type, offset);
2113 }
2114 put_swap_device(si);
2115 }2174 }
2175 if (!offset)
2176 offset = cluster_alloc_swap_entry(si, NULL);
2177 local_unlock(&percpu_swap_cluster.lock);
2178 if (offset)
2179 entry = swp_entry(si->type, offset);
2180
2116fail:2181fail:
2117 return entry;2182 return entry;
2118}2183}
2119 2184
2120/* Free a slot allocated by swap_alloc_hibernation_slot */2185/**
2186 * swap_free_hibernation_slot() - Free a swap slot allocated for hibernation.
2187 * @entry: swap entry to free.
2188 *
2189 * The caller must ensure the swap device is stable.
2190 */
2121void swap_free_hibernation_slot(swp_entry_t entry)2191void swap_free_hibernation_slot(swp_entry_t entry)
2122{2192{
2123 struct swap_info_struct *si;2193 struct swap_info_struct *si = __swap_entry_to_info(entry);
2124 struct swap_cluster_info *ci;2194 struct swap_cluster_info *ci;
2125 pgoff_t offset = swp_offset(entry);2195 pgoff_t offset = swp_offset(entry);
2126 2196
2127 si = get_swap_device(entry);
2128 if (WARN_ON(!si))
2129 return;
2130
2131 ci = swap_cluster_lock(si, offset);2197 ci = swap_cluster_lock(si, offset);
2132 __swap_cluster_put_entry(ci, offset % SWAPFILE_CLUSTER);2198 __swap_cluster_put_entry(ci, offset % SWAPFILE_CLUSTER);
2133 __swap_cluster_free_entries(si, ci, offset % SWAPFILE_CLUSTER, 1);2199 __swap_cluster_free_entries(si, ci, offset % SWAPFILE_CLUSTER, 1);
@@ -2135,25 +2201,17 @@ void swap_free_hibernation_slot(swp_entry_t entry) · 1 unmodified line
2135 2201
2136 /* In theory readahead might add it to the swap cache by accident */2202 /* In theory readahead might add it to the swap cache by accident */
2137 __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);2203 __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
2138 put_swap_device(si);
2139}2204}
2140 2205
2141/*2206static int __find_hibernation_swap_type(dev_t device, sector_t offset)
2142 * Find the swap type that corresponds to given device (if any).
2143 *
2144 * @offset - number of the PAGE_SIZE-sized block of the device, starting
2145 * from 0, in which the swap header is expected to be located.
2146 *
2147 * This is needed for the suspend to disk (aka swsusp).
2148 */
2149int swap_type_of(dev_t device, sector_t offset)
2150{2207{
2151 int type;2208 int type;
2152 2209
2210 lockdep_assert_held(&swap_lock);
2211
2153 if (!device)2212 if (!device)
2154 return -1;2213 return -EINVAL;
2155 2214
2156 spin_lock(&swap_lock);
2157 for (type = 0; type < nr_swapfiles; type++) {2215 for (type = 0; type < nr_swapfiles; type++) {
2158 struct swap_info_struct *sis = swap_info[type];2216 struct swap_info_struct *sis = swap_info[type];
2159 2217
@@ -2163,16 +2221,118 @@ int swap_type_of(dev_t device, sector_t offset) · 3 unmodified lines
2163 if (device == sis->bdev->bd_dev) {2221 if (device == sis->bdev->bd_dev) {
2164 struct swap_extent *se = first_se(sis);2222 struct swap_extent *se = first_se(sis);
2165 2223
2166 if (se->start_block == offset) {2224 if (se->start_block == offset)
2167 spin_unlock(&swap_lock);
2168 return type;2225 return type;
2169 }
2170 }2226 }
2171 }2227 }
2172 spin_unlock(&swap_lock);
2173 return -ENODEV;2228 return -ENODEV;
2174}2229}
2175 2230
2231/**
2232 * pin_hibernation_swap_type - Pin the swap device for hibernation
2233 * @device: Block device containing the resume image
2234 * @offset: Offset identifying the swap area
2235 *
2236 * Locate the swap device for @device/@offset and mark it as pinned
2237 * for hibernation. While pinned, swapoff() is prevented.
2238 *
2239 * Only one uswsusp context may pin a swap device at a time.
2240 * If already pinned, this function returns -EBUSY.
2241 *
2242 * Return:
2243 * >= 0 on success (swap type).
2244 * -EINVAL if @device is invalid.
2245 * -ENODEV if the swap device is not found.
2246 * -EBUSY if the device is already pinned for hibernation.
2247 */
2248int pin_hibernation_swap_type(dev_t device, sector_t offset)
2249{
2250 int type;
2251 struct swap_info_struct *si;
2252
2253 spin_lock(&swap_lock);
2254
2255 type = __find_hibernation_swap_type(device, offset);
2256 if (type < 0) {
2257 spin_unlock(&swap_lock);
2258 return type;
2259 }
2260
2261 si = swap_type_to_info(type);
2262 if (WARN_ON_ONCE(!si)) {
2263 spin_unlock(&swap_lock);
2264 return -ENODEV;
2265 }
2266
2267 /*
2268 * hibernate_acquire() prevents concurrent hibernation sessions.
2269 * This check additionally guards against double-pinning within
2270 * the same session.
2271 */
2272 if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) {
2273 spin_unlock(&swap_lock);
2274 return -EBUSY;
2275 }
2276
2277 si->flags |= SWP_HIBERNATION;
2278
2279 spin_unlock(&swap_lock);
2280 return type;
2281}
2282
2283/**
2284 * unpin_hibernation_swap_type - Unpin the swap device for hibernation
2285 * @type: Swap type previously returned by pin_hibernation_swap_type()
2286 *
2287 * Clear the hibernation pin on the given swap device, allowing
2288 * swapoff() to proceed normally.
2289 *
2290 * If @type does not refer to a valid swap device, this function
2291 * does nothing.
2292 */
2293void unpin_hibernation_swap_type(int type)
2294{
2295 struct swap_info_struct *si;
2296
2297 spin_lock(&swap_lock);
2298 si = swap_type_to_info(type);
2299 if (!si) {
2300 spin_unlock(&swap_lock);
2301 return;
2302 }
2303 si->flags &= ~SWP_HIBERNATION;
2304 spin_unlock(&swap_lock);
2305}
2306
2307/**
2308 * find_hibernation_swap_type - Find swap type for hibernation
2309 * @device: Block device containing the resume image
2310 * @offset: Offset within the device identifying the swap area
2311 *
2312 * Locate the swap device corresponding to @device and @offset.
2313 *
2314 * Unlike pin_hibernation_swap_type(), this function only performs a
2315 * lookup and does not mark the swap device as pinned for hibernation.
2316 *
2317 * This is safe in the sysfs-based hibernation path where user space
2318 * is already frozen and swapoff() cannot run concurrently.
2319 *
2320 * Return:
2321 * A non-negative swap type on success.
2322 * -EINVAL if @device is invalid.
2323 * -ENODEV if no matching swap device is found.
2324 */
2325int find_hibernation_swap_type(dev_t device, sector_t offset)
2326{
2327 int type;
2328
2329 spin_lock(&swap_lock);
2330 type = __find_hibernation_swap_type(device, offset);
2331 spin_unlock(&swap_lock);
2332
2333 return type;
2334}
2335
2176int find_first_swap(dev_t *device)2336int find_first_swap(dev_t *device)
2177{2337{
2178 int type;2338 int type;
@@ -2869,7 +3029,7 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info, · 690 unmodified lines
2869 ci = cluster_info + i;3029 ci = cluster_info + i;
2870 /* Cluster with bad marks count will have a remaining table */3030 /* Cluster with bad marks count will have a remaining table */
2871 spin_lock(&ci->lock);3031 spin_lock(&ci->lock);
2872 if (rcu_dereference_protected(ci->table, true)) {3032 if (cluster_table_is_alloced(ci)) {
2873 swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);3033 swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
2874 swap_cluster_free_table(ci);3034 swap_cluster_free_table(ci);
2875 }3035 }
@@ -2903,7 +3063,6 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si) · 27 unmodified lines
2903SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)3063SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
2904{3064{
2905 struct swap_info_struct *p = NULL;3065 struct swap_info_struct *p = NULL;
2906 unsigned long *zeromap;
2907 struct swap_cluster_info *cluster_info;3066 struct swap_cluster_info *cluster_info;
2908 struct file *swap_file, *victim;3067 struct file *swap_file, *victim;
2909 struct address_space *mapping;3068 struct address_space *mapping;
@@ -2936,6 +3095,14 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 26 unmodified lines
2936 spin_unlock(&swap_lock);3095 spin_unlock(&swap_lock);
2937 goto out_dput;3096 goto out_dput;
2938 }3097 }
3098
3099 /* Refuse swapoff while the device is pinned for hibernation */
3100 if (p->flags & SWP_HIBERNATION) {
3101 err = -EBUSY;
3102 spin_unlock(&swap_lock);
3103 goto out_dput;
3104 }
3105
2939 if (!security_vm_enough_memory_mm(current->mm, p->pages))3106 if (!security_vm_enough_memory_mm(current->mm, p->pages))
2940 vm_unacct_memory(p->pages);3107 vm_unacct_memory(p->pages);
2941 else {3108 else {
@@ -2991,8 +3158,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 49 unmodified lines
2991 3158
2992 swap_file = p->swap_file;3159 swap_file = p->swap_file;
2993 p->swap_file = NULL;3160 p->swap_file = NULL;
2994 zeromap = p->zeromap;
2995 p->zeromap = NULL;
2996 maxpages = p->max;3161 maxpages = p->max;
2997 cluster_info = p->cluster_info;3162 cluster_info = p->cluster_info;
2998 p->max = 0;3163 p->max = 0;
@@ -3004,10 +3169,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 5 unmodified lines
3004 mutex_unlock(&swapon_mutex);3169 mutex_unlock(&swapon_mutex);
3005 kfree(p->global_cluster);3170 kfree(p->global_cluster);
3006 p->global_cluster = NULL;3171 p->global_cluster = NULL;
3007 kvfree(zeromap);
3008 free_swap_cluster_info(cluster_info, maxpages);3172 free_swap_cluster_info(cluster_info, maxpages);
3009 /* Destroy swap account information */
3010 swap_cgroup_swapoff(p->type);
3011 3173
3012 inode = mapping->host;3174 inode = mapping->host;
3013 3175
@@ -3538,21 +3700,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · 524 unmodified lines
3538 if (error)3700 if (error)
3539 goto bad_swap_unlock_inode;3701 goto bad_swap_unlock_inode;
3540 3702
3541 error = swap_cgroup_swapon(si->type, maxpages);
3542 if (error)
3543 goto bad_swap_unlock_inode;
3544
3545 /*
3546 * Use kvmalloc_array instead of bitmap_zalloc as the allocation order might
3547 * be above MAX_PAGE_ORDER incase of a large swap file.
3548 */
3549 si->zeromap = kvmalloc_array(BITS_TO_LONGS(maxpages), sizeof(long),
3550 GFP_KERNEL | __GFP_ZERO);
3551 if (!si->zeromap) {
3552 error = -ENOMEM;
3553 goto bad_swap_unlock_inode;
3554 }
3555
3556 if (si->bdev && bdev_stable_writes(si->bdev))3703 if (si->bdev && bdev_stable_writes(si->bdev))
3557 si->flags |= SWP_STABLE_WRITES;3704 si->flags |= SWP_STABLE_WRITES;
3558 3705
@@ -3652,11 +3799,8 @@ bad_swap: · 93 unmodified lines
3652 si->global_cluster = NULL;3799 si->global_cluster = NULL;
3653 inode = NULL;3800 inode = NULL;
3654 destroy_swap_extents(si, swap_file);3801 destroy_swap_extents(si, swap_file);
3655 swap_cgroup_swapoff(si->type);
3656 free_swap_cluster_info(si->cluster_info, si->max);3802 free_swap_cluster_info(si->cluster_info, si->max);
3657 si->cluster_info = NULL;3803 si->cluster_info = NULL;
3658 kvfree(si->zeromap);
3659 si->zeromap = NULL;
3660 /*3804 /*
3661 * Clear the SWP_USED flag after all resources are freed so3805 * Clear the SWP_USED flag after all resources are freed so
3662 * alloc_swap_info can reuse this si safely.3806 * alloc_swap_info can reuse this si safely.
mm/userfaultfd.c+2247−12
@@ -2,7 +2,12 @@ · 1 unmodified line
2/*2/*
3 * mm/userfaultfd.c3 * mm/userfaultfd.c
4 *4 *
5 * Copyright (C) 2007 Davide Libenzi <davidel@xmailserver.org>
6 * Copyright (C) 2008-2009 Red Hat, Inc.
5 * Copyright (C) 2015 Red Hat, Inc.7 * Copyright (C) 2015 Red Hat, Inc.
8 *
9 * Some part derived from fs/eventfd.c (anon inode setup) and
10 * mm/ksm.c (mm hashing).
6 */11 */
7 12
8#include <linux/mm.h>13#include <linux/mm.h>
@@ -14,6 +19,17 @@ · 5 unmodified lines
14#include <linux/userfaultfd_k.h>19#include <linux/userfaultfd_k.h>
15#include <linux/mmu_notifier.h>20#include <linux/mmu_notifier.h>
16#include <linux/hugetlb.h>21#include <linux/hugetlb.h>
22#include <linux/list.h>
23#include <linux/sched/mm.h>
24#include <linux/mm_inline.h>
25#include <linux/poll.h>
26#include <linux/slab.h>
27#include <linux/seq_file.h>
28#include <linux/bug.h>
29#include <linux/anon_inodes.h>
30#include <linux/syscalls.h>
31#include <linux/miscdevice.h>
32#include <linux/uio.h>
17#include <linux/file.h>33#include <linux/file.h>
18#include <linux/cleanup.h>34#include <linux/cleanup.h>
19#include <asm/tlbflush.h>35#include <asm/tlbflush.h>
@@ -1017,7 +1033,7 @@ out: · 997 unmodified lines
1017 return copied ? copied : err;1033 return copied ? copied : err;
1018}1034}
1019 1035
1020ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,1036static ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
1021 unsigned long src_start, unsigned long len,1037 unsigned long src_start, unsigned long len,
1022 uffd_flags_t flags)1038 uffd_flags_t flags)
1023{1039{
@@ -1025,7 +1041,7 @@ ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start, · 1 unmodified line
1025 uffd_flags_set_mode(flags, MFILL_ATOMIC_COPY));1041 uffd_flags_set_mode(flags, MFILL_ATOMIC_COPY));
1026}1042}
1027 1043
1028ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,1044static ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
1029 unsigned long start,1045 unsigned long start,
1030 unsigned long len)1046 unsigned long len)
1031{1047{
@@ -1033,7 +1049,7 @@ ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx, · 1 unmodified line
1033 uffd_flags_set_mode(0, MFILL_ATOMIC_ZEROPAGE));1049 uffd_flags_set_mode(0, MFILL_ATOMIC_ZEROPAGE));
1034}1050}
1035 1051
1036ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,1052static ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
1037 unsigned long len, uffd_flags_t flags)1053 unsigned long len, uffd_flags_t flags)
1038{1054{
1039 1055
@@ -1049,7 +1065,7 @@ ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start, · 9 unmodified lines
1049 uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE));1065 uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE));
1050}1066}
1051 1067
1052ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,1068static ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
1053 unsigned long len, uffd_flags_t flags)1069 unsigned long len, uffd_flags_t flags)
1054{1070{
1055 return mfill_atomic(ctx, start, 0, len,1071 return mfill_atomic(ctx, start, 0, len,
@@ -1085,7 +1101,7 @@ long uffd_wp_range(struct vm_area_struct *dst_vma, · 29 unmodified lines
1085 return ret;1101 return ret;
1086}1102}
1087 1103
1088int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,1104static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
1089 unsigned long len, bool enable_wp)1105 unsigned long len, bool enable_wp)
1090{1106{
1091 struct mm_struct *dst_mm = ctx->mm;1107 struct mm_struct *dst_mm = ctx->mm;
@@ -1915,7 +1931,7 @@ static void uffd_move_unlock(struct vm_area_struct *dst_vma, · 823 unmodified lines
1915 * in the regions or not, but preventing the risk of having to split1931 * in the regions or not, but preventing the risk of having to split
1916 * the hugepmd during the remap.1932 * the hugepmd during the remap.
1917 */1933 */
1918ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,1934static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
1919 unsigned long src_start, unsigned long len, __u64 mode)1935 unsigned long src_start, unsigned long len, __u64 mode)
1920{1936{
1921 struct mm_struct *mm = ctx->mm;1937 struct mm_struct *mm = ctx->mm;
@@ -2090,7 +2106,7 @@ out: · 168 unmodified lines
2090 return moved ? moved : err;2106 return moved ? moved : err;
2091}2107}
2092 2108
2093bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,2109static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
2094 bool wp_async)2110 bool wp_async)
2095{2111{
2096 const struct vm_uffd_ops *ops = vma_uffd_ops(vma);2112 const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
@@ -2147,12 +2163,12 @@ static void userfaultfd_set_ctx(struct vm_area_struct *vma, · 50 unmodified lines
2147 (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags);2163 (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags);
2148}2164}
2149 2165
2150void userfaultfd_reset_ctx(struct vm_area_struct *vma)2166static void userfaultfd_reset_ctx(struct vm_area_struct *vma)
2151{2167{
2152 userfaultfd_set_ctx(vma, NULL, 0);2168 userfaultfd_set_ctx(vma, NULL, 0);
2153}2169}
2154 2170
2155struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,2171static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
2156 struct vm_area_struct *prev,2172 struct vm_area_struct *prev,
2157 struct vm_area_struct *vma,2173 struct vm_area_struct *vma,
2158 unsigned long start,2174 unsigned long start,
@@ -2191,7 +2207,7 @@ struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi, · 32 unmodified lines
2191}2207}
2192 2208
2193/* Assumes mmap write lock taken, and mm_struct pinned. */2209/* Assumes mmap write lock taken, and mm_struct pinned. */
2194int userfaultfd_register_range(struct userfaultfd_ctx *ctx,2210static int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
2195 struct vm_area_struct *vma,2211 struct vm_area_struct *vma,
2196 vm_flags_t vm_flags,2212 vm_flags_t vm_flags,
2197 unsigned long start, unsigned long end,2213 unsigned long start, unsigned long end,
@@ -2255,7 +2271,7 @@ skip: · 57 unmodified lines
2255 return 0;2271 return 0;
2256}2272}
2257 2273
2258void userfaultfd_release_new(struct userfaultfd_ctx *ctx)2274static void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
2259{2275{
2260 struct mm_struct *mm = ctx->mm;2276 struct mm_struct *mm = ctx->mm;
2261 struct vm_area_struct *vma;2277 struct vm_area_struct *vma;
@@ -2270,7 +2286,7 @@ void userfaultfd_release_new(struct userfaultfd_ctx *ctx) · 8 unmodified lines
2270 mmap_write_unlock(mm);2286 mmap_write_unlock(mm);
2271}2287}
2272 2288
2273void userfaultfd_release_all(struct mm_struct *mm,2289static void userfaultfd_release_all(struct mm_struct *mm,
2274 struct userfaultfd_ctx *ctx)2290 struct userfaultfd_ctx *ctx)
2275{2291{
2276 struct vm_area_struct *vma, *prev;2292 struct vm_area_struct *vma, *prev;
@@ -2305,3 +2321,2222 @@ void userfaultfd_release_all(struct mm_struct *mm, · 28 unmodified lines
2305 mmap_write_unlock(mm);2321 mmap_write_unlock(mm);
2306 mmput(mm);2322 mmput(mm);
2307}2323}
2324
2325static int sysctl_unprivileged_userfaultfd __read_mostly;
2326
2327#ifdef CONFIG_SYSCTL
2328static const struct ctl_table vm_userfaultfd_table[] = {
2329 {
2330 .procname = "unprivileged_userfaultfd",
2331 .data = &sysctl_unprivileged_userfaultfd,
2332 .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
2333 .mode = 0644,
2334 .proc_handler = proc_dointvec_minmax,
2335 .extra1 = SYSCTL_ZERO,
2336 .extra2 = SYSCTL_ONE,
2337 },
2338};
2339#endif
2340
2341static struct kmem_cache *userfaultfd_ctx_cachep __ro_after_init;
2342
2343struct userfaultfd_fork_ctx {
2344 struct userfaultfd_ctx *orig;
2345 struct userfaultfd_ctx *new;
2346 struct list_head list;
2347};
2348
2349struct userfaultfd_unmap_ctx {
2350 struct userfaultfd_ctx *ctx;
2351 unsigned long start;
2352 unsigned long end;
2353 struct list_head list;
2354};
2355
2356struct userfaultfd_wait_queue {
2357 struct uffd_msg msg;
2358 wait_queue_entry_t wq;
2359 struct userfaultfd_ctx *ctx;
2360 bool waken;
2361};
2362
2363struct userfaultfd_wake_range {
2364 unsigned long start;
2365 unsigned long len;
2366};
2367
2368/* internal indication that UFFD_API ioctl was successfully executed */
2369#define UFFD_FEATURE_INITIALIZED (1u << 31)
2370
2371static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
2372{
2373 return ctx->features & UFFD_FEATURE_INITIALIZED;
2374}
2375
2376static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
2377{
2378 return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
2379}
2380
2381/*
2382 * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
2383 * meaningful when userfaultfd_wp()==true on the vma and when it's
2384 * anonymous.
2385 */
2386bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
2387{
2388 struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
2389
2390 if (!ctx)
2391 return false;
2392
2393 return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
2394}
2395
2396static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
2397 int wake_flags, void *key)
2398{
2399 struct userfaultfd_wake_range *range = key;
2400 int ret;
2401 struct userfaultfd_wait_queue *uwq;
2402 unsigned long start, len;
2403
2404 uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
2405 ret = 0;
2406 /* len == 0 means wake all */
2407 start = range->start;
2408 len = range->len;
2409 if (len && (start > uwq->msg.arg.pagefault.address ||
2410 start + len <= uwq->msg.arg.pagefault.address))
2411 goto out;
2412 WRITE_ONCE(uwq->waken, true);
2413 /*
2414 * The Program-Order guarantees provided by the scheduler
2415 * ensure uwq->waken is visible before the task is woken.
2416 */
2417 ret = wake_up_state(wq->private, mode);
2418 if (ret) {
2419 /*
2420 * Wake only once, autoremove behavior.
2421 *
2422 * After the effect of list_del_init is visible to the other
2423 * CPUs, the waitqueue may disappear from under us, see the
2424 * !list_empty_careful() in handle_userfault().
2425 *
2426 * try_to_wake_up() has an implicit smp_mb(), and the
2427 * wq->private is read before calling the extern function
2428 * "wake_up_state" (which in turns calls try_to_wake_up).
2429 */
2430 list_del_init(&wq->entry);
2431 }
2432out:
2433 return ret;
2434}
2435
2436/**
2437 * userfaultfd_ctx_get - Acquires a reference to the internal userfaultfd
2438 * context.
2439 * @ctx: [in] Pointer to the userfaultfd context.
2440 */
2441static void userfaultfd_ctx_get(struct userfaultfd_ctx *ctx)
2442{
2443 refcount_inc(&ctx->refcount);
2444}
2445
2446/**
2447 * userfaultfd_ctx_put - Releases a reference to the internal userfaultfd
2448 * context.
2449 * @ctx: [in] Pointer to userfaultfd context.
2450 *
2451 * The userfaultfd context reference must have been previously acquired either
2452 * with userfaultfd_ctx_get() or userfaultfd_ctx_fdget().
2453 */
2454static void userfaultfd_ctx_put(struct userfaultfd_ctx *ctx)
2455{
2456 if (refcount_dec_and_test(&ctx->refcount)) {
2457 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_pending_wqh.lock));
2458 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_pending_wqh));
2459 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_wqh.lock));
2460 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_wqh));
2461 VM_WARN_ON_ONCE(spin_is_locked(&ctx->event_wqh.lock));
2462 VM_WARN_ON_ONCE(waitqueue_active(&ctx->event_wqh));
2463 VM_WARN_ON_ONCE(spin_is_locked(&ctx->fd_wqh.lock));
2464 VM_WARN_ON_ONCE(waitqueue_active(&ctx->fd_wqh));
2465 mmdrop(ctx->mm);
2466 kmem_cache_free(userfaultfd_ctx_cachep, ctx);
2467 }
2468}
2469
2470static inline void msg_init(struct uffd_msg *msg)
2471{
2472 BUILD_BUG_ON(sizeof(struct uffd_msg) != 32);
2473 /*
2474 * Must use memset to zero out the paddings or kernel data is
2475 * leaked to userland.
2476 */
2477 memset(msg, 0, sizeof(struct uffd_msg));
2478}
2479
2480static inline struct uffd_msg userfault_msg(unsigned long address,
2481 unsigned long real_address,
2482 unsigned int flags,
2483 unsigned long reason,
2484 unsigned int features)
2485{
2486 struct uffd_msg msg;
2487
2488 msg_init(&msg);
2489 msg.event = UFFD_EVENT_PAGEFAULT;
2490
2491 msg.arg.pagefault.address = (features & UFFD_FEATURE_EXACT_ADDRESS) ?
2492 real_address : address;
2493
2494 /*
2495 * These flags indicate why the userfault occurred:
2496 * - UFFD_PAGEFAULT_FLAG_WP indicates a write protect fault.
2497 * - UFFD_PAGEFAULT_FLAG_MINOR indicates a minor fault.
2498 * - Neither of these flags being set indicates a MISSING fault.
2499 *
2500 * Separately, UFFD_PAGEFAULT_FLAG_WRITE indicates it was a write
2501 * fault. Otherwise, it was a read fault.
2502 */
2503 if (flags & FAULT_FLAG_WRITE)
2504 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;
2505 if (reason & VM_UFFD_WP)
2506 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;
2507 if (reason & VM_UFFD_MINOR)
2508 msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;
2509 if (features & UFFD_FEATURE_THREAD_ID)
2510 msg.arg.pagefault.feat.ptid = task_pid_vnr(current);
2511 return msg;
2512}
2513
2514#ifdef CONFIG_HUGETLB_PAGE
2515/*
2516 * Same functionality as userfaultfd_must_wait below with modifications for
2517 * hugepmd ranges.
2518 */
2519static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
2520 struct vm_fault *vmf,
2521 unsigned long reason)
2522{
2523 struct vm_area_struct *vma = vmf->vma;
2524 pte_t *ptep, pte;
2525
2526 assert_fault_locked(vmf);
2527
2528 ptep = hugetlb_walk(vma, vmf->address, vma_mmu_pagesize(vma));
2529 if (!ptep)
2530 return true;
2531
2532 pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
2533
2534 /*
2535 * Lockless access: we're in a wait_event so it's ok if it
2536 * changes under us.
2537 */
2538
2539 /* Entry is still missing, wait for userspace to resolve the fault. */
2540 if (huge_pte_none(pte))
2541 return true;
2542 /* UFFD PTE markers require userspace to resolve the fault. */
2543 if (pte_is_uffd_marker(pte))
2544 return true;
2545 /*
2546 * Concurrent migration may have replaced the present PTE with a
2547 * non-marker swap entry between fault delivery and this lockless
2548 * re-check. huge_pte_write() on a swap entry decodes random offset
2549 * bits, so gate it on pte_present(). The migration completion path
2550 * will re-deliver the fault if it still needs userspace.
2551 */
2552 if (!pte_present(pte))
2553 return false;
2554 /*
2555 * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
2556 * resolve the fault.
2557 */
2558 if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
2559 return true;
2560
2561 return false;
2562}
2563#else
2564static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
2565 struct vm_fault *vmf,
2566 unsigned long reason)
2567{
2568 /* Should never get here. */
2569 VM_WARN_ON_ONCE(1);
2570 return false;
2571}
2572#endif /* CONFIG_HUGETLB_PAGE */
2573
2574/*
2575 * Verify the pagetables are still not ok after having registered into
2576 * the fault_pending_wqh to avoid userland having to UFFDIO_WAKE any
2577 * userfault that has already been resolved, if userfaultfd_read_iter and
2578 * UFFDIO_COPY|ZEROPAGE are being run simultaneously on two different
2579 * threads.
2580 */
2581static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
2582 struct vm_fault *vmf,
2583 unsigned long reason)
2584{
2585 struct mm_struct *mm = ctx->mm;
2586 unsigned long address = vmf->address;
2587 pgd_t *pgd;
2588 p4d_t *p4d;
2589 pud_t *pud;
2590 pmd_t *pmd, _pmd;
2591 pte_t *pte;
2592 pte_t ptent;
2593 bool ret;
2594
2595 assert_fault_locked(vmf);
2596
2597 pgd = pgd_offset(mm, address);
2598 if (!pgd_present(*pgd))
2599 return true;
2600 p4d = p4d_offset(pgd, address);
2601 if (!p4d_present(*p4d))
2602 return true;
2603 pud = pud_offset(p4d, address);
2604 if (!pud_present(*pud))
2605 return true;
2606 pmd = pmd_offset(pud, address);
2607again:
2608 _pmd = pmdp_get_lockless(pmd);
2609 if (pmd_none(_pmd))
2610 return true;
2611
2612 /*
2613 * A race could arise which would result in a softleaf entry such as
2614 * migration entry unexpectedly being present in the PMD, so explicitly
2615 * check for this and bail out if so.
2616 */
2617 if (!pmd_present(_pmd))
2618 return false;
2619
2620 if (pmd_trans_huge(_pmd))
2621 return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
2622
2623 pte = pte_offset_map(pmd, address);
2624 if (!pte)
2625 goto again;
2626
2627 /*
2628 * Lockless access: we're in a wait_event so it's ok if it
2629 * changes under us.
2630 */
2631 ptent = ptep_get(pte);
2632
2633 ret = true;
2634 /* Entry is still missing, wait for userspace to resolve the fault. */
2635 if (pte_none(ptent))
2636 goto out;
2637 /* UFFD PTE markers require userspace to resolve the fault. */
2638 if (pte_is_uffd_marker(ptent))
2639 goto out;
2640 /*
2641 * Concurrent swap-out / migration may have replaced the present PTE
2642 * with a non-marker swap entry between fault delivery and this
2643 * lockless re-check. pte_write() on a swap entry decodes random
2644 * offset bits, so gate it on pte_present(). The page-in path will
2645 * re-deliver the fault if it still needs userspace.
2646 */
2647 if (!pte_present(ptent)) {
2648 ret = false;
2649 goto out;
2650 }
2651 /*
2652 * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
2653 * resolve the fault.
2654 */
2655 if (!pte_write(ptent) && (reason & VM_UFFD_WP))
2656 goto out;
2657
2658 ret = false;
2659out:
2660 pte_unmap(pte);
2661 return ret;
2662}
2663
2664static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags)
2665{
2666 if (flags & FAULT_FLAG_INTERRUPTIBLE)
2667 return TASK_INTERRUPTIBLE;
2668
2669 if (flags & FAULT_FLAG_KILLABLE)
2670 return TASK_KILLABLE;
2671
2672 return TASK_UNINTERRUPTIBLE;
2673}
2674
2675/*
2676 * The locking rules involved in returning VM_FAULT_RETRY depending on
2677 * FAULT_FLAG_ALLOW_RETRY, FAULT_FLAG_RETRY_NOWAIT and
2678 * FAULT_FLAG_KILLABLE are not straightforward. The "Caution"
2679 * recommendation in __lock_page_or_retry is not an understatement.
2680 *
2681 * If FAULT_FLAG_ALLOW_RETRY is set, the mmap_lock must be released
2682 * before returning VM_FAULT_RETRY only if FAULT_FLAG_RETRY_NOWAIT is
2683 * not set.
2684 *
2685 * If FAULT_FLAG_ALLOW_RETRY is set but FAULT_FLAG_KILLABLE is not
2686 * set, VM_FAULT_RETRY can still be returned if and only if there are
2687 * fatal_signal_pending()s, and the mmap_lock must be released before
2688 * returning it.
2689 */
2690vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
2691{
2692 struct vm_area_struct *vma = vmf->vma;
2693 struct mm_struct *mm = vma->vm_mm;
2694 struct userfaultfd_ctx *ctx;
2695 struct userfaultfd_wait_queue uwq;
2696 vm_fault_t ret = VM_FAULT_SIGBUS;
2697 bool must_wait;
2698 unsigned int blocking_state;
2699
2700 /*
2701 * We don't do userfault handling for the final child pid update
2702 * and when coredumping (faults triggered by get_dump_page()).
2703 */
2704 if (current->flags & (PF_EXITING|PF_DUMPCORE))
2705 goto out;
2706
2707 assert_fault_locked(vmf);
2708
2709 ctx = vma->vm_userfaultfd_ctx.ctx;
2710 if (!ctx)
2711 goto out;
2712
2713 VM_WARN_ON_ONCE(ctx->mm != mm);
2714
2715 /* Any unrecognized flag is a bug. */
2716 VM_WARN_ON_ONCE(reason & ~__VM_UFFD_FLAGS);
2717 /* 0 or > 1 flags set is a bug; we expect exactly 1. */
2718 VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
2719
2720 if (ctx->features & UFFD_FEATURE_SIGBUS)
2721 goto out;
2722 if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
2723 goto out;
2724
2725 /*
2726 * Check that we can return VM_FAULT_RETRY.
2727 *
2728 * NOTE: it should become possible to return VM_FAULT_RETRY
2729 * even if FAULT_FLAG_TRIED is set without leading to gup()
2730 * -EBUSY failures, if the userfaultfd is to be extended for
2731 * VM_UFFD_WP tracking and we intend to arm the userfault
2732 * without first stopping userland access to the memory. For
2733 * VM_UFFD_MISSING userfaults this is enough for now.
2734 */
2735 if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
2736 /*
2737 * Validate the invariant that nowait must allow retry
2738 * to be sure not to return SIGBUS erroneously on
2739 * nowait invocations.
2740 */
2741 VM_WARN_ON_ONCE(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
2742#ifdef CONFIG_DEBUG_VM
2743 if (printk_ratelimit()) {
2744 pr_warn("FAULT_FLAG_ALLOW_RETRY missing %x\n",
2745 vmf->flags);
2746 dump_stack();
2747 }
2748#endif
2749 goto out;
2750 }
2751
2752 /*
2753 * Handle nowait, not much to do other than tell it to retry
2754 * and wait.
2755 */
2756 ret = VM_FAULT_RETRY;
2757 if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
2758 goto out;
2759
2760 if (unlikely(READ_ONCE(ctx->released))) {
2761 /*
2762 * If a concurrent release is detected, do not return
2763 * VM_FAULT_SIGBUS or VM_FAULT_NOPAGE, but instead always
2764 * return VM_FAULT_RETRY with lock released proactively.
2765 *
2766 * If we were to return VM_FAULT_SIGBUS here, the non
2767 * cooperative manager would be instead forced to
2768 * always call UFFDIO_UNREGISTER before it can safely
2769 * close the uffd, to avoid involuntary SIGBUS triggered.
2770 *
2771 * If we were to return VM_FAULT_NOPAGE, it would work for
2772 * the fault path, in which the lock will be released
2773 * later. However for GUP, faultin_page() does nothing
2774 * special on NOPAGE, so GUP would spin retrying without
2775 * releasing the mmap read lock, causing possible livelock.
2776 *
2777 * Here only VM_FAULT_RETRY would make sure the mmap lock
2778 * be released immediately, so that the thread concurrently
2779 * releasing the userfault would always make progress.
2780 */
2781 release_fault_lock(vmf);
2782 goto out;
2783 }
2784
2785 /* take the reference before dropping the mmap_lock */
2786 userfaultfd_ctx_get(ctx);
2787
2788 init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
2789 uwq.wq.private = current;
2790 uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
2791 reason, ctx->features);
2792 uwq.ctx = ctx;
2793 uwq.waken = false;
2794
2795 blocking_state = userfaultfd_get_blocking_state(vmf->flags);
2796
2797 /*
2798 * Take the vma lock now, in order to safely call
2799 * userfaultfd_huge_must_wait() later. Since acquiring the
2800 * (sleepable) vma lock can modify the current task state, that
2801 * must be before explicitly calling set_current_state().
2802 */
2803 if (is_vm_hugetlb_page(vma))
2804 hugetlb_vma_lock_read(vma);
2805
2806 spin_lock_irq(&ctx->fault_pending_wqh.lock);
2807 /*
2808 * After the __add_wait_queue the uwq is visible to userland
2809 * through poll/read().
2810 */
2811 __add_wait_queue(&ctx->fault_pending_wqh, &uwq.wq);
2812 /*
2813 * The smp_mb() after __set_current_state prevents the reads
2814 * following the spin_unlock to happen before the list_add in
2815 * __add_wait_queue.
2816 */
2817 set_current_state(blocking_state);
2818 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2819
2820 if (is_vm_hugetlb_page(vma)) {
2821 must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason);
2822 hugetlb_vma_unlock_read(vma);
2823 } else {
2824 must_wait = userfaultfd_must_wait(ctx, vmf, reason);
2825 }
2826
2827 release_fault_lock(vmf);
2828
2829 if (likely(must_wait && !READ_ONCE(ctx->released))) {
2830 wake_up_poll(&ctx->fd_wqh, EPOLLIN);
2831 schedule();
2832 }
2833
2834 __set_current_state(TASK_RUNNING);
2835
2836 /*
2837 * Here we race with the list_del; list_add in
2838 * userfaultfd_ctx_read(), however because we don't ever run
2839 * list_del_init() to refile across the two lists, the prev
2840 * and next pointers will never point to self. list_add also
2841 * would never let any of the two pointers to point to
2842 * self. So list_empty_careful won't risk to see both pointers
2843 * pointing to self at any time during the list refile. The
2844 * only case where list_del_init() is called is the full
2845 * removal in the wake function and there we don't re-list_add
2846 * and it's fine not to block on the spinlock. The uwq on this
2847 * kernel stack can be released after the list_del_init.
2848 */
2849 if (!list_empty_careful(&uwq.wq.entry)) {
2850 spin_lock_irq(&ctx->fault_pending_wqh.lock);
2851 /*
2852 * No need of list_del_init(), the uwq on the stack
2853 * will be freed shortly anyway.
2854 */
2855 list_del(&uwq.wq.entry);
2856 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2857 }
2858
2859 /*
2860 * ctx may go away after this if the userfault pseudo fd is
2861 * already released.
2862 */
2863 userfaultfd_ctx_put(ctx);
2864
2865out:
2866 return ret;
2867}
2868
2869static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
2870 struct userfaultfd_wait_queue *ewq)
2871{
2872 struct userfaultfd_ctx *release_new_ctx;
2873
2874 if (WARN_ON_ONCE(current->flags & PF_EXITING))
2875 goto out;
2876
2877 ewq->ctx = ctx;
2878 init_waitqueue_entry(&ewq->wq, current);
2879 release_new_ctx = NULL;
2880
2881 spin_lock_irq(&ctx->event_wqh.lock);
2882 /*
2883 * After the __add_wait_queue the uwq is visible to userland
2884 * through poll/read().
2885 */
2886 __add_wait_queue(&ctx->event_wqh, &ewq->wq);
2887 for (;;) {
2888 set_current_state(TASK_KILLABLE);
2889 if (ewq->msg.event == 0)
2890 break;
2891 if (READ_ONCE(ctx->released) ||
2892 fatal_signal_pending(current)) {
2893 /*
2894 * &ewq->wq may be queued in fork_event, but
2895 * __remove_wait_queue ignores the head
2896 * parameter. It would be a problem if it
2897 * didn't.
2898 */
2899 __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
2900 if (ewq->msg.event == UFFD_EVENT_FORK) {
2901 struct userfaultfd_ctx *new;
2902
2903 new = (struct userfaultfd_ctx *)
2904 (unsigned long)
2905 ewq->msg.arg.reserved.reserved1;
2906 release_new_ctx = new;
2907 }
2908 break;
2909 }
2910
2911 spin_unlock_irq(&ctx->event_wqh.lock);
2912
2913 wake_up_poll(&ctx->fd_wqh, EPOLLIN);
2914 schedule();
2915
2916 spin_lock_irq(&ctx->event_wqh.lock);
2917 }
2918 __set_current_state(TASK_RUNNING);
2919 spin_unlock_irq(&ctx->event_wqh.lock);
2920
2921 if (release_new_ctx) {
2922 userfaultfd_release_new(release_new_ctx);
2923 userfaultfd_ctx_put(release_new_ctx);
2924 }
2925
2926 /*
2927 * ctx may go away after this if the userfault pseudo fd is
2928 * already released.
2929 */
2930out:
2931 atomic_dec(&ctx->mmap_changing);
2932 VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
2933 userfaultfd_ctx_put(ctx);
2934}
2935
2936static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
2937 struct userfaultfd_wait_queue *ewq)
2938{
2939 ewq->msg.event = 0;
2940 wake_up_locked(&ctx->event_wqh);
2941 __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
2942}
2943
2944int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
2945{
2946 struct userfaultfd_ctx *ctx = NULL, *octx;
2947 struct userfaultfd_fork_ctx *fctx;
2948
2949 octx = vma->vm_userfaultfd_ctx.ctx;
2950 if (!octx)
2951 return 0;
2952
2953 if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
2954 userfaultfd_reset_ctx(vma);
2955 return 0;
2956 }
2957
2958 list_for_each_entry(fctx, fcs, list)
2959 if (fctx->orig == octx) {
2960 ctx = fctx->new;
2961 break;
2962 }
2963
2964 if (!ctx) {
2965 fctx = kmalloc_obj(*fctx);
2966 if (!fctx)
2967 return -ENOMEM;
2968
2969 ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
2970 if (!ctx) {
2971 kfree(fctx);
2972 return -ENOMEM;
2973 }
2974
2975 refcount_set(&ctx->refcount, 1);
2976 ctx->flags = octx->flags;
2977 ctx->features = octx->features;
2978 ctx->released = false;
2979 init_rwsem(&ctx->map_changing_lock);
2980 atomic_set(&ctx->mmap_changing, 0);
2981 ctx->mm = vma->vm_mm;
2982 mmgrab(ctx->mm);
2983
2984 userfaultfd_ctx_get(octx);
2985 down_write(&octx->map_changing_lock);
2986 atomic_inc(&octx->mmap_changing);
2987 up_write(&octx->map_changing_lock);
2988 fctx->orig = octx;
2989 fctx->new = ctx;
2990 list_add_tail(&fctx->list, fcs);
2991 }
2992
2993 vma->vm_userfaultfd_ctx.ctx = ctx;
2994 return 0;
2995}
2996
2997static void dup_fctx(struct userfaultfd_fork_ctx *fctx)
2998{
2999 struct userfaultfd_ctx *ctx = fctx->orig;
3000 struct userfaultfd_wait_queue ewq;
3001
3002 msg_init(&ewq.msg);
3003
3004 ewq.msg.event = UFFD_EVENT_FORK;
3005 ewq.msg.arg.reserved.reserved1 = (unsigned long)fctx->new;
3006
3007 userfaultfd_event_wait_completion(ctx, &ewq);
3008}
3009
3010void dup_userfaultfd_complete(struct list_head *fcs)
3011{
3012 struct userfaultfd_fork_ctx *fctx, *n;
3013
3014 list_for_each_entry_safe(fctx, n, fcs, list) {
3015 dup_fctx(fctx);
3016 list_del(&fctx->list);
3017 kfree(fctx);
3018 }
3019}
3020
3021void dup_userfaultfd_fail(struct list_head *fcs)
3022{
3023 struct userfaultfd_fork_ctx *fctx, *n;
3024
3025 /*
3026 * An error has occurred on fork, we will tear memory down, but have
3027 * allocated memory for fctx's and raised reference counts for both the
3028 * original and child contexts (and on the mm for each as a result).
3029 *
3030 * These would ordinarily be taken care of by a user handling the event,
3031 * but we are no longer doing so, so manually clean up here.
3032 *
3033 * mm tear down will take care of cleaning up VMA contexts.
3034 */
3035 list_for_each_entry_safe(fctx, n, fcs, list) {
3036 struct userfaultfd_ctx *octx = fctx->orig;
3037 struct userfaultfd_ctx *ctx = fctx->new;
3038
3039 atomic_dec(&octx->mmap_changing);
3040 VM_WARN_ON_ONCE(atomic_read(&octx->mmap_changing) < 0);
3041 userfaultfd_ctx_put(octx);
3042 userfaultfd_ctx_put(ctx);
3043
3044 list_del(&fctx->list);
3045 kfree(fctx);
3046 }
3047}
3048
3049void mremap_userfaultfd_prep(struct vm_area_struct *vma,
3050 struct vm_userfaultfd_ctx *vm_ctx)
3051{
3052 struct userfaultfd_ctx *ctx;
3053
3054 ctx = vma->vm_userfaultfd_ctx.ctx;
3055
3056 if (!ctx)
3057 return;
3058
3059 if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
3060 vm_ctx->ctx = ctx;
3061 userfaultfd_ctx_get(ctx);
3062 down_write(&ctx->map_changing_lock);
3063 atomic_inc(&ctx->mmap_changing);
3064 up_write(&ctx->map_changing_lock);
3065 } else {
3066 /* Drop uffd context if remap feature not enabled */
3067 userfaultfd_reset_ctx(vma);
3068 }
3069}
3070
3071void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *vm_ctx,
3072 unsigned long from, unsigned long to,
3073 unsigned long len)
3074{
3075 struct userfaultfd_ctx *ctx = vm_ctx->ctx;
3076 struct userfaultfd_wait_queue ewq;
3077
3078 if (!ctx)
3079 return;
3080
3081 msg_init(&ewq.msg);
3082
3083 ewq.msg.event = UFFD_EVENT_REMAP;
3084 ewq.msg.arg.remap.from = from;
3085 ewq.msg.arg.remap.to = to;
3086 ewq.msg.arg.remap.len = len;
3087
3088 userfaultfd_event_wait_completion(ctx, &ewq);
3089}
3090
3091void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
3092{
3093 struct userfaultfd_ctx *ctx = vm_ctx->ctx;
3094
3095 if (!ctx)
3096 return;
3097
3098 atomic_dec(&ctx->mmap_changing);
3099 VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
3100 userfaultfd_ctx_put(ctx);
3101}
3102
3103bool userfaultfd_remove(struct vm_area_struct *vma,
3104 unsigned long start, unsigned long end)
3105{
3106 struct mm_struct *mm = vma->vm_mm;
3107 struct userfaultfd_ctx *ctx;
3108 struct userfaultfd_wait_queue ewq;
3109
3110 ctx = vma->vm_userfaultfd_ctx.ctx;
3111 if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
3112 return true;
3113
3114 userfaultfd_ctx_get(ctx);
3115 down_write(&ctx->map_changing_lock);
3116 atomic_inc(&ctx->mmap_changing);
3117 up_write(&ctx->map_changing_lock);
3118 mmap_read_unlock(mm);
3119
3120 msg_init(&ewq.msg);
3121
3122 ewq.msg.event = UFFD_EVENT_REMOVE;
3123 ewq.msg.arg.remove.start = start;
3124 ewq.msg.arg.remove.end = end;
3125
3126 userfaultfd_event_wait_completion(ctx, &ewq);
3127
3128 return false;
3129}
3130
3131static bool has_unmap_ctx(struct userfaultfd_ctx *ctx, struct list_head *unmaps,
3132 unsigned long start, unsigned long end)
3133{
3134 struct userfaultfd_unmap_ctx *unmap_ctx;
3135
3136 list_for_each_entry(unmap_ctx, unmaps, list)
3137 if (unmap_ctx->ctx == ctx && unmap_ctx->start == start &&
3138 unmap_ctx->end == end)
3139 return true;
3140
3141 return false;
3142}
3143
3144int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
3145 unsigned long end, struct list_head *unmaps)
3146{
3147 struct userfaultfd_unmap_ctx *unmap_ctx;
3148 struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
3149
3150 if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
3151 has_unmap_ctx(ctx, unmaps, start, end))
3152 return 0;
3153
3154 unmap_ctx = kzalloc_obj(*unmap_ctx);
3155 if (!unmap_ctx)
3156 return -ENOMEM;
3157
3158 userfaultfd_ctx_get(ctx);
3159 down_write(&ctx->map_changing_lock);
3160 atomic_inc(&ctx->mmap_changing);
3161 up_write(&ctx->map_changing_lock);
3162 unmap_ctx->ctx = ctx;
3163 unmap_ctx->start = start;
3164 unmap_ctx->end = end;
3165 list_add_tail(&unmap_ctx->list, unmaps);
3166
3167 return 0;
3168}
3169
3170void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf)
3171{
3172 struct userfaultfd_unmap_ctx *ctx, *n;
3173 struct userfaultfd_wait_queue ewq;
3174
3175 list_for_each_entry_safe(ctx, n, uf, list) {
3176 msg_init(&ewq.msg);
3177
3178 ewq.msg.event = UFFD_EVENT_UNMAP;
3179 ewq.msg.arg.remove.start = ctx->start;
3180 ewq.msg.arg.remove.end = ctx->end;
3181
3182 userfaultfd_event_wait_completion(ctx->ctx, &ewq);
3183
3184 list_del(&ctx->list);
3185 kfree(ctx);
3186 }
3187}
3188
3189static int userfaultfd_release(struct inode *inode, struct file *file)
3190{
3191 struct userfaultfd_ctx *ctx = file->private_data;
3192 struct mm_struct *mm = ctx->mm;
3193 /* len == 0 means wake all */
3194 struct userfaultfd_wake_range range = { .len = 0, };
3195
3196 WRITE_ONCE(ctx->released, true);
3197
3198 userfaultfd_release_all(mm, ctx);
3199
3200 /*
3201 * After no new page faults can wait on this fault_*wqh, flush
3202 * the last page faults that may have been already waiting on
3203 * the fault_*wqh.
3204 */
3205 spin_lock_irq(&ctx->fault_pending_wqh.lock);
3206 __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &range);
3207 __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
3208 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
3209
3210 /* Flush pending events that may still wait on event_wqh */
3211 wake_up_all(&ctx->event_wqh);
3212
3213 wake_up_poll(&ctx->fd_wqh, EPOLLHUP);
3214 userfaultfd_ctx_put(ctx);
3215 return 0;
3216}
3217
3218/* fault_pending_wqh.lock must be hold by the caller */
3219static inline struct userfaultfd_wait_queue *find_userfault_in(
3220 wait_queue_head_t *wqh)
3221{
3222 wait_queue_entry_t *wq;
3223 struct userfaultfd_wait_queue *uwq;
3224
3225 lockdep_assert_held(&wqh->lock);
3226
3227 uwq = NULL;
3228 if (!waitqueue_active(wqh))
3229 goto out;
3230 /* walk in reverse to provide FIFO behavior to read userfaults */
3231 wq = list_last_entry(&wqh->head, typeof(*wq), entry);
3232 uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
3233out:
3234 return uwq;
3235}
3236
3237static inline struct userfaultfd_wait_queue *find_userfault(
3238 struct userfaultfd_ctx *ctx)
3239{
3240 return find_userfault_in(&ctx->fault_pending_wqh);
3241}
3242
3243static inline struct userfaultfd_wait_queue *find_userfault_evt(
3244 struct userfaultfd_ctx *ctx)
3245{
3246 return find_userfault_in(&ctx->event_wqh);
3247}
3248
3249static __poll_t userfaultfd_poll(struct file *file, poll_table *wait)
3250{
3251 struct userfaultfd_ctx *ctx = file->private_data;
3252 __poll_t ret;
3253
3254 poll_wait(file, &ctx->fd_wqh, wait);
3255
3256 if (!userfaultfd_is_initialized(ctx))
3257 return EPOLLERR;
3258
3259 /*
3260 * poll() never guarantees that read won't block.
3261 * userfaults can be waken before they're read().
3262 */
3263 if (unlikely(!(file->f_flags & O_NONBLOCK)))
3264 return EPOLLERR;
3265 /*
3266 * lockless access to see if there are pending faults
3267 * __pollwait last action is the add_wait_queue but
3268 * the spin_unlock would allow the waitqueue_active to
3269 * pass above the actual list_add inside
3270 * add_wait_queue critical section. So use a full
3271 * memory barrier to serialize the list_add write of
3272 * add_wait_queue() with the waitqueue_active read
3273 * below.
3274 */
3275 ret = 0;
3276 smp_mb();
3277 if (waitqueue_active(&ctx->fault_pending_wqh))
3278 ret = EPOLLIN;
3279 else if (waitqueue_active(&ctx->event_wqh))
3280 ret = EPOLLIN;
3281
3282 return ret;
3283}
3284
3285static const struct file_operations userfaultfd_fops;
3286
3287static int resolve_userfault_fork(struct userfaultfd_ctx *new,
3288 struct inode *inode,
3289 struct uffd_msg *msg)
3290{
3291 int fd;
3292
3293 fd = anon_inode_create_getfd("[userfaultfd]", &userfaultfd_fops, new,
3294 O_RDONLY | (new->flags & UFFD_SHARED_FCNTL_FLAGS), inode);
3295 if (fd < 0)
3296 return fd;
3297
3298 msg->arg.reserved.reserved1 = 0;
3299 msg->arg.fork.ufd = fd;
3300 return 0;
3301}
3302
3303static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
3304 struct uffd_msg *msg, struct inode *inode)
3305{
3306 ssize_t ret;
3307 DECLARE_WAITQUEUE(wait, current);
3308 struct userfaultfd_wait_queue *uwq;
3309 /*
3310 * Handling fork event requires sleeping operations, so
3311 * we drop the event_wqh lock, then do these ops, then
3312 * lock it back and wake up the waiter. While the lock is
3313 * dropped the ewq may go away so we keep track of it
3314 * carefully.
3315 */
3316 LIST_HEAD(fork_event);
3317 struct userfaultfd_ctx *fork_nctx = NULL;
3318
3319 /* always take the fd_wqh lock before the fault_pending_wqh lock */
3320 spin_lock_irq(&ctx->fd_wqh.lock);
3321 __add_wait_queue(&ctx->fd_wqh, &wait);
3322 for (;;) {
3323 set_current_state(TASK_INTERRUPTIBLE);
3324 spin_lock(&ctx->fault_pending_wqh.lock);
3325 uwq = find_userfault(ctx);
3326 if (uwq) {
3327 /*
3328 * Use a seqcount to repeat the lockless check
3329 * in wake_userfault() to avoid missing
3330 * wakeups because during the refile both
3331 * waitqueue could become empty if this is the
3332 * only userfault.
3333 */
3334 write_seqcount_begin(&ctx->refile_seq);
3335
3336 /*
3337 * The fault_pending_wqh.lock prevents the uwq
3338 * to disappear from under us.
3339 *
3340 * Refile this userfault from
3341 * fault_pending_wqh to fault_wqh, it's not
3342 * pending anymore after we read it.
3343 *
3344 * Use list_del() by hand (as
3345 * userfaultfd_wake_function also uses
3346 * list_del_init() by hand) to be sure nobody
3347 * changes __remove_wait_queue() to use
3348 * list_del_init() in turn breaking the
3349 * !list_empty_careful() check in
3350 * handle_userfault(). The uwq->wq.head list
3351 * must never be empty at any time during the
3352 * refile, or the waitqueue could disappear
3353 * from under us. The "wait_queue_head_t"
3354 * parameter of __remove_wait_queue() is unused
3355 * anyway.
3356 */
3357 list_del(&uwq->wq.entry);
3358 add_wait_queue(&ctx->fault_wqh, &uwq->wq);
3359
3360 write_seqcount_end(&ctx->refile_seq);
3361
3362 /* careful to always initialize msg if ret == 0 */
3363 *msg = uwq->msg;
3364 spin_unlock(&ctx->fault_pending_wqh.lock);
3365 ret = 0;
3366 break;
3367 }
3368 spin_unlock(&ctx->fault_pending_wqh.lock);
3369
3370 spin_lock(&ctx->event_wqh.lock);
3371 uwq = find_userfault_evt(ctx);
3372 if (uwq) {
3373 *msg = uwq->msg;
3374
3375 if (uwq->msg.event == UFFD_EVENT_FORK) {
3376 fork_nctx = (struct userfaultfd_ctx *)
3377 (unsigned long)
3378 uwq->msg.arg.reserved.reserved1;
3379 list_move(&uwq->wq.entry, &fork_event);
3380 /*
3381 * fork_nctx can be freed as soon as
3382 * we drop the lock, unless we take a
3383 * reference on it.
3384 */
3385 userfaultfd_ctx_get(fork_nctx);
3386 spin_unlock(&ctx->event_wqh.lock);
3387 ret = 0;
3388 break;
3389 }
3390
3391 userfaultfd_event_complete(ctx, uwq);
3392 spin_unlock(&ctx->event_wqh.lock);
3393 ret = 0;
3394 break;
3395 }
3396 spin_unlock(&ctx->event_wqh.lock);
3397
3398 if (signal_pending(current)) {
3399 ret = -ERESTARTSYS;
3400 break;
3401 }
3402 if (no_wait) {
3403 ret = -EAGAIN;
3404 break;
3405 }
3406 spin_unlock_irq(&ctx->fd_wqh.lock);
3407 schedule();
3408 spin_lock_irq(&ctx->fd_wqh.lock);
3409 }
3410 __remove_wait_queue(&ctx->fd_wqh, &wait);
3411 __set_current_state(TASK_RUNNING);
3412 spin_unlock_irq(&ctx->fd_wqh.lock);
3413
3414 if (!ret && msg->event == UFFD_EVENT_FORK) {
3415 ret = resolve_userfault_fork(fork_nctx, inode, msg);
3416 spin_lock_irq(&ctx->event_wqh.lock);
3417 if (!list_empty(&fork_event)) {
3418 /*
3419 * The fork thread didn't abort, so we can
3420 * drop the temporary refcount.
3421 */
3422 userfaultfd_ctx_put(fork_nctx);
3423
3424 uwq = list_first_entry(&fork_event,
3425 typeof(*uwq),
3426 wq.entry);
3427 /*
3428 * If fork_event list wasn't empty and in turn
3429 * the event wasn't already released by fork
3430 * (the event is allocated on fork kernel
3431 * stack), put the event back to its place in
3432 * the event_wq. fork_event head will be freed
3433 * as soon as we return so the event cannot
3434 * stay queued there no matter the current
3435 * "ret" value.
3436 */
3437 list_del(&uwq->wq.entry);
3438 __add_wait_queue(&ctx->event_wqh, &uwq->wq);
3439
3440 /*
3441 * Leave the event in the waitqueue and report
3442 * error to userland if we failed to resolve
3443 * the userfault fork.
3444 */
3445 if (likely(!ret))
3446 userfaultfd_event_complete(ctx, uwq);
3447 } else {
3448 /*
3449 * Here the fork thread aborted and the
3450 * refcount from the fork thread on fork_nctx
3451 * has already been released. We still hold
3452 * the reference we took before releasing the
3453 * lock above. If resolve_userfault_fork
3454 * failed we've to drop it because the
3455 * fork_nctx has to be freed in such case. If
3456 * it succeeded we'll hold it because the new
3457 * uffd references it.
3458 */
3459 if (ret)
3460 userfaultfd_ctx_put(fork_nctx);
3461 }
3462 spin_unlock_irq(&ctx->event_wqh.lock);
3463 }
3464
3465 return ret;
3466}
3467
3468static ssize_t userfaultfd_read_iter(struct kiocb *iocb, struct iov_iter *to)
3469{
3470 struct file *file = iocb->ki_filp;
3471 struct userfaultfd_ctx *ctx = file->private_data;
3472 ssize_t _ret, ret = 0;
3473 struct uffd_msg msg;
3474 struct inode *inode = file_inode(file);
3475 bool no_wait;
3476
3477 if (!userfaultfd_is_initialized(ctx))
3478 return -EINVAL;
3479
3480 no_wait = file->f_flags & O_NONBLOCK || iocb->ki_flags & IOCB_NOWAIT;
3481 for (;;) {
3482 if (iov_iter_count(to) < sizeof(msg))
3483 return ret ? ret : -EINVAL;
3484 _ret = userfaultfd_ctx_read(ctx, no_wait, &msg, inode);
3485 if (_ret < 0)
3486 return ret ? ret : _ret;
3487 _ret = !copy_to_iter_full(&msg, sizeof(msg), to);
3488 if (_ret)
3489 return ret ? ret : -EFAULT;
3490 ret += sizeof(msg);
3491 /*
3492 * Allow to read more than one fault at time but only
3493 * block if waiting for the very first one.
3494 */
3495 no_wait = true;
3496 }
3497}
3498
3499static void __wake_userfault(struct userfaultfd_ctx *ctx,
3500 struct userfaultfd_wake_range *range)
3501{
3502 spin_lock_irq(&ctx->fault_pending_wqh.lock);
3503 /* wake all in the range and autoremove */
3504 if (waitqueue_active(&ctx->fault_pending_wqh))
3505 __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
3506 range);
3507 if (waitqueue_active(&ctx->fault_wqh))
3508 __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, range);
3509 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
3510}
3511
3512static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
3513 struct userfaultfd_wake_range *range)
3514{
3515 unsigned seq;
3516 bool need_wakeup;
3517
3518 /*
3519 * To be sure waitqueue_active() is not reordered by the CPU
3520 * before the pagetable update, use an explicit SMP memory
3521 * barrier here. PT lock release or mmap_read_unlock(mm) still
3522 * have release semantics that can allow the
3523 * waitqueue_active() to be reordered before the pte update.
3524 */
3525 smp_mb();
3526
3527 /*
3528 * Use waitqueue_active because it's very frequent to
3529 * change the address space atomically even if there are no
3530 * userfaults yet. So we take the spinlock only when we're
3531 * sure we've userfaults to wake.
3532 */
3533 do {
3534 seq = read_seqcount_begin(&ctx->refile_seq);
3535 need_wakeup = waitqueue_active(&ctx->fault_pending_wqh) ||
3536 waitqueue_active(&ctx->fault_wqh);
3537 cond_resched();
3538 } while (read_seqcount_retry(&ctx->refile_seq, seq));
3539 if (need_wakeup)
3540 __wake_userfault(ctx, range);
3541}
3542
3543static __always_inline int validate_unaligned_range(
3544 struct mm_struct *mm, __u64 start, __u64 len)
3545{
3546 __u64 task_size = mm->task_size;
3547
3548 if (len & ~PAGE_MASK)
3549 return -EINVAL;
3550 if (!len)
3551 return -EINVAL;
3552 if (start >= task_size)
3553 return -EINVAL;
3554 if (len > task_size - start)
3555 return -EINVAL;
3556 if (start + len <= start)
3557 return -EINVAL;
3558 return 0;
3559}
3560
3561static __always_inline int validate_range(struct mm_struct *mm,
3562 __u64 start, __u64 len)
3563{
3564 if (start & ~PAGE_MASK)
3565 return -EINVAL;
3566
3567 return validate_unaligned_range(mm, start, len);
3568}
3569
3570static int userfaultfd_register(struct userfaultfd_ctx *ctx,
3571 unsigned long arg)
3572{
3573 struct mm_struct *mm = ctx->mm;
3574 struct vm_area_struct *vma, *cur;
3575 int ret;
3576 struct uffdio_register uffdio_register;
3577 struct uffdio_register __user *user_uffdio_register;
3578 vm_flags_t vm_flags;
3579 bool found;
3580 bool basic_ioctls;
3581 unsigned long start, end;
3582 struct vma_iterator vmi;
3583 bool wp_async = userfaultfd_wp_async_ctx(ctx);
3584
3585 user_uffdio_register = (struct uffdio_register __user *) arg;
3586
3587 ret = -EFAULT;
3588 if (copy_from_user(&uffdio_register, user_uffdio_register,
3589 sizeof(uffdio_register)-sizeof(__u64)))
3590 goto out;
3591
3592 ret = -EINVAL;
3593 if (!uffdio_register.mode)
3594 goto out;
3595 if (uffdio_register.mode & ~UFFD_API_REGISTER_MODES)
3596 goto out;
3597 vm_flags = 0;
3598 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
3599 vm_flags |= VM_UFFD_MISSING;
3600 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
3601 if (!pgtable_supports_uffd_wp())
3602 goto out;
3603
3604 vm_flags |= VM_UFFD_WP;
3605 }
3606 if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
3607#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
3608 goto out;
3609#endif
3610 vm_flags |= VM_UFFD_MINOR;
3611 }
3612
3613 ret = validate_range(mm, uffdio_register.range.start,
3614 uffdio_register.range.len);
3615 if (ret)
3616 goto out;
3617
3618 start = uffdio_register.range.start;
3619 end = start + uffdio_register.range.len;
3620
3621 ret = -ENOMEM;
3622 if (!mmget_not_zero(mm))
3623 goto out;
3624
3625 ret = -EINVAL;
3626 mmap_write_lock(mm);
3627 vma_iter_init(&vmi, mm, start);
3628 vma = vma_find(&vmi, end);
3629 if (!vma)
3630 goto out_unlock;
3631
3632 /*
3633 * If the first vma contains huge pages, make sure start address
3634 * is aligned to huge page size.
3635 */
3636 if (is_vm_hugetlb_page(vma)) {
3637 unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
3638
3639 if (start & (vma_hpagesize - 1))
3640 goto out_unlock;
3641 }
3642
3643 /*
3644 * Search for not compatible vmas.
3645 */
3646 found = false;
3647 basic_ioctls = false;
3648 cur = vma;
3649 do {
3650 cond_resched();
3651
3652 VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
3653 !!(cur->vm_flags & __VM_UFFD_FLAGS));
3654
3655 /* check not compatible vmas */
3656 ret = -EINVAL;
3657 if (!vma_can_userfault(cur, vm_flags, wp_async))
3658 goto out_unlock;
3659
3660 /*
3661 * UFFDIO_COPY will fill file holes even without
3662 * PROT_WRITE. This check enforces that if this is a
3663 * MAP_SHARED, the process has write permission to the backing
3664 * file. If VM_MAYWRITE is set it also enforces that on a
3665 * MAP_SHARED vma: there is no F_WRITE_SEAL and no further
3666 * F_WRITE_SEAL can be taken until the vma is destroyed.
3667 */
3668 ret = -EPERM;
3669 if (unlikely(!(cur->vm_flags & VM_MAYWRITE)))
3670 goto out_unlock;
3671
3672 /*
3673 * If this vma contains ending address, and huge pages
3674 * check alignment.
3675 */
3676 if (is_vm_hugetlb_page(cur) && end <= cur->vm_end &&
3677 end > cur->vm_start) {
3678 unsigned long vma_hpagesize = vma_kernel_pagesize(cur);
3679
3680 ret = -EINVAL;
3681
3682 if (end & (vma_hpagesize - 1))
3683 goto out_unlock;
3684 }
3685 if ((vm_flags & VM_UFFD_WP) && !(cur->vm_flags & VM_MAYWRITE))
3686 goto out_unlock;
3687
3688 /*
3689 * Check that this vma isn't already owned by a
3690 * different userfaultfd. We can't allow more than one
3691 * userfaultfd to own a single vma simultaneously or we
3692 * wouldn't know which one to deliver the userfaults to.
3693 */
3694 ret = -EBUSY;
3695 if (cur->vm_userfaultfd_ctx.ctx &&
3696 cur->vm_userfaultfd_ctx.ctx != ctx)
3697 goto out_unlock;
3698
3699 /*
3700 * Note vmas containing huge pages
3701 */
3702 if (is_vm_hugetlb_page(cur))
3703 basic_ioctls = true;
3704
3705 found = true;
3706 } for_each_vma_range(vmi, cur, end);
3707 VM_WARN_ON_ONCE(!found);
3708
3709 ret = userfaultfd_register_range(ctx, vma, vm_flags, start, end,
3710 wp_async);
3711
3712out_unlock:
3713 mmap_write_unlock(mm);
3714 mmput(mm);
3715 if (!ret) {
3716 __u64 ioctls_out;
3717
3718 ioctls_out = basic_ioctls ? UFFD_API_RANGE_IOCTLS_BASIC :
3719 UFFD_API_RANGE_IOCTLS;
3720
3721 /*
3722 * Declare the WP ioctl only if the WP mode is
3723 * specified and all checks passed with the range
3724 */
3725 if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_WP))
3726 ioctls_out &= ~((__u64)1 << _UFFDIO_WRITEPROTECT);
3727
3728 /* CONTINUE ioctl is only supported for MINOR ranges. */
3729 if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
3730 ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
3731
3732 /*
3733 * Now that we scanned all vmas we can already tell
3734 * userland which ioctls methods are guaranteed to
3735 * succeed on this range.
3736 */
3737 if (put_user(ioctls_out, &user_uffdio_register->ioctls))
3738 ret = -EFAULT;
3739 }
3740out:
3741 return ret;
3742}
3743
3744static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
3745 unsigned long arg)
3746{
3747 struct mm_struct *mm = ctx->mm;
3748 struct vm_area_struct *vma, *prev, *cur;
3749 int ret;
3750 struct uffdio_range uffdio_unregister;
3751 bool found;
3752 unsigned long start, end, vma_end;
3753 const void __user *buf = (void __user *)arg;
3754 struct vma_iterator vmi;
3755 bool wp_async = userfaultfd_wp_async_ctx(ctx);
3756
3757 ret = -EFAULT;
3758 if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
3759 goto out;
3760
3761 ret = validate_range(mm, uffdio_unregister.start,
3762 uffdio_unregister.len);
3763 if (ret)
3764 goto out;
3765
3766 start = uffdio_unregister.start;
3767 end = start + uffdio_unregister.len;
3768
3769 ret = -ENOMEM;
3770 if (!mmget_not_zero(mm))
3771 goto out;
3772
3773 mmap_write_lock(mm);
3774 ret = -EINVAL;
3775 vma_iter_init(&vmi, mm, start);
3776 vma = vma_find(&vmi, end);
3777 if (!vma)
3778 goto out_unlock;
3779
3780 /*
3781 * If the first vma contains huge pages, make sure start address
3782 * is aligned to huge page size.
3783 */
3784 if (is_vm_hugetlb_page(vma)) {
3785 unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
3786
3787 if (start & (vma_hpagesize - 1))
3788 goto out_unlock;
3789 }
3790
3791 /*
3792 * Search for not compatible vmas.
3793 */
3794 found = false;
3795 cur = vma;
3796 do {
3797 cond_resched();
3798
3799 VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
3800 !!(cur->vm_flags & __VM_UFFD_FLAGS));
3801
3802 /*
3803 * Prevent unregistering through a different userfaultfd than
3804 * the one used for registration.
3805 */
3806 if (cur->vm_userfaultfd_ctx.ctx &&
3807 cur->vm_userfaultfd_ctx.ctx != ctx)
3808 goto out_unlock;
3809
3810 /*
3811 * Check not compatible vmas, not strictly required
3812 * here as not compatible vmas cannot have an
3813 * userfaultfd_ctx registered on them, but this
3814 * provides for more strict behavior to notice
3815 * unregistration errors.
3816 */
3817 if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
3818 goto out_unlock;
3819
3820 found = true;
3821 } for_each_vma_range(vmi, cur, end);
3822 VM_WARN_ON_ONCE(!found);
3823
3824 vma_iter_set(&vmi, start);
3825 prev = vma_prev(&vmi);
3826 if (vma->vm_start < start)
3827 prev = vma;
3828
3829 ret = 0;
3830 for_each_vma_range(vmi, vma, end) {
3831 cond_resched();
3832
3833 /* VMA not registered with userfaultfd. */
3834 if (!vma->vm_userfaultfd_ctx.ctx)
3835 goto skip;
3836
3837 VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx != ctx);
3838 VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, wp_async));
3839 VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
3840
3841 if (vma->vm_start > start)
3842 start = vma->vm_start;
3843 vma_end = min(end, vma->vm_end);
3844
3845 if (userfaultfd_missing(vma)) {
3846 /*
3847 * Wake any concurrent pending userfault while
3848 * we unregister, so they will not hang
3849 * permanently and it avoids userland to call
3850 * UFFDIO_WAKE explicitly.
3851 */
3852 struct userfaultfd_wake_range range;
3853 range.start = start;
3854 range.len = vma_end - start;
3855 wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range);
3856 }
3857
3858 vma = userfaultfd_clear_vma(&vmi, prev, vma,
3859 start, vma_end);
3860 if (IS_ERR(vma)) {
3861 ret = PTR_ERR(vma);
3862 break;
3863 }
3864
3865skip:
3866 prev = vma;
3867 start = vma->vm_end;
3868 }
3869
3870out_unlock:
3871 mmap_write_unlock(mm);
3872 mmput(mm);
3873out:
3874 return ret;
3875}
3876
3877/*
3878 * userfaultfd_wake may be used in combination with the
3879 * UFFDIO_*_MODE_DONTWAKE to wakeup userfaults in batches.
3880 */
3881static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
3882 unsigned long arg)
3883{
3884 int ret;
3885 struct uffdio_range uffdio_wake;
3886 struct userfaultfd_wake_range range;
3887 const void __user *buf = (void __user *)arg;
3888
3889 ret = -EFAULT;
3890 if (copy_from_user(&uffdio_wake, buf, sizeof(uffdio_wake)))
3891 goto out;
3892
3893 ret = validate_range(ctx->mm, uffdio_wake.start, uffdio_wake.len);
3894 if (ret)
3895 goto out;
3896
3897 range.start = uffdio_wake.start;
3898 range.len = uffdio_wake.len;
3899
3900 /*
3901 * len == 0 means wake all and we don't want to wake all here,
3902 * so check it again to be sure.
3903 */
3904 VM_WARN_ON_ONCE(!range.len);
3905
3906 wake_userfault(ctx, &range);
3907 ret = 0;
3908
3909out:
3910 return ret;
3911}
3912
3913static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
3914 unsigned long arg)
3915{
3916 __s64 ret;
3917 struct uffdio_copy uffdio_copy;
3918 struct uffdio_copy __user *user_uffdio_copy;
3919 struct userfaultfd_wake_range range;
3920 uffd_flags_t flags = 0;
3921
3922 user_uffdio_copy = (struct uffdio_copy __user *) arg;
3923
3924 ret = -EAGAIN;
3925 if (unlikely(atomic_read(&ctx->mmap_changing))) {
3926 if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
3927 return -EFAULT;
3928 goto out;
3929 }
3930
3931 ret = -EFAULT;
3932 if (copy_from_user(&uffdio_copy, user_uffdio_copy,
3933 /* don't copy "copy" last field */
3934 sizeof(uffdio_copy)-sizeof(__s64)))
3935 goto out;
3936
3937 ret = validate_unaligned_range(ctx->mm, uffdio_copy.src,
3938 uffdio_copy.len);
3939 if (ret)
3940 goto out;
3941 ret = validate_range(ctx->mm, uffdio_copy.dst, uffdio_copy.len);
3942 if (ret)
3943 goto out;
3944
3945 ret = -EINVAL;
3946 if (uffdio_copy.mode & ~(UFFDIO_COPY_MODE_DONTWAKE|UFFDIO_COPY_MODE_WP))
3947 goto out;
3948 if (uffdio_copy.mode & UFFDIO_COPY_MODE_WP)
3949 flags |= MFILL_ATOMIC_WP;
3950 if (mmget_not_zero(ctx->mm)) {
3951 ret = mfill_atomic_copy(ctx, uffdio_copy.dst, uffdio_copy.src,
3952 uffdio_copy.len, flags);
3953 mmput(ctx->mm);
3954 } else {
3955 return -ESRCH;
3956 }
3957 if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
3958 return -EFAULT;
3959 if (ret < 0)
3960 goto out;
3961 VM_WARN_ON_ONCE(!ret);
3962 /* len == 0 would wake all */
3963 range.len = ret;
3964 if (!(uffdio_copy.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
3965 range.start = uffdio_copy.dst;
3966 wake_userfault(ctx, &range);
3967 }
3968 ret = range.len == uffdio_copy.len ? 0 : -EAGAIN;
3969out:
3970 return ret;
3971}
3972
3973static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
3974 unsigned long arg)
3975{
3976 __s64 ret;
3977 struct uffdio_zeropage uffdio_zeropage;
3978 struct uffdio_zeropage __user *user_uffdio_zeropage;
3979 struct userfaultfd_wake_range range;
3980
3981 user_uffdio_zeropage = (struct uffdio_zeropage __user *) arg;
3982
3983 ret = -EAGAIN;
3984 if (unlikely(atomic_read(&ctx->mmap_changing))) {
3985 if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
3986 return -EFAULT;
3987 goto out;
3988 }
3989
3990 ret = -EFAULT;
3991 if (copy_from_user(&uffdio_zeropage, user_uffdio_zeropage,
3992 /* don't copy "zeropage" last field */
3993 sizeof(uffdio_zeropage)-sizeof(__s64)))
3994 goto out;
3995
3996 ret = validate_range(ctx->mm, uffdio_zeropage.range.start,
3997 uffdio_zeropage.range.len);
3998 if (ret)
3999 goto out;
4000 ret = -EINVAL;
4001 if (uffdio_zeropage.mode & ~UFFDIO_ZEROPAGE_MODE_DONTWAKE)
4002 goto out;
4003
4004 if (mmget_not_zero(ctx->mm)) {
4005 ret = mfill_atomic_zeropage(ctx, uffdio_zeropage.range.start,
4006 uffdio_zeropage.range.len);
4007 mmput(ctx->mm);
4008 } else {
4009 return -ESRCH;
4010 }
4011 if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
4012 return -EFAULT;
4013 if (ret < 0)
4014 goto out;
4015 /* len == 0 would wake all */
4016 VM_WARN_ON_ONCE(!ret);
4017 range.len = ret;
4018 if (!(uffdio_zeropage.mode & UFFDIO_ZEROPAGE_MODE_DONTWAKE)) {
4019 range.start = uffdio_zeropage.range.start;
4020 wake_userfault(ctx, &range);
4021 }
4022 ret = range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
4023out:
4024 return ret;
4025}
4026
4027static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
4028 unsigned long arg)
4029{
4030 int ret;
4031 struct uffdio_writeprotect uffdio_wp;
4032 struct uffdio_writeprotect __user *user_uffdio_wp;
4033 struct userfaultfd_wake_range range;
4034 bool mode_wp, mode_dontwake;
4035
4036 if (atomic_read(&ctx->mmap_changing))
4037 return -EAGAIN;
4038
4039 user_uffdio_wp = (struct uffdio_writeprotect __user *) arg;
4040
4041 if (copy_from_user(&uffdio_wp, user_uffdio_wp,
4042 sizeof(struct uffdio_writeprotect)))
4043 return -EFAULT;
4044
4045 ret = validate_range(ctx->mm, uffdio_wp.range.start,
4046 uffdio_wp.range.len);
4047 if (ret)
4048 return ret;
4049
4050 if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE |
4051 UFFDIO_WRITEPROTECT_MODE_WP))
4052 return -EINVAL;
4053
4054 mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
4055 mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
4056
4057 if (mode_wp && mode_dontwake)
4058 return -EINVAL;
4059
4060 if (mmget_not_zero(ctx->mm)) {
4061 ret = mwriteprotect_range(ctx, uffdio_wp.range.start,
4062 uffdio_wp.range.len, mode_wp);
4063 mmput(ctx->mm);
4064 } else {
4065 return -ESRCH;
4066 }
4067
4068 if (ret)
4069 return ret;
4070
4071 if (!mode_wp && !mode_dontwake) {
4072 range.start = uffdio_wp.range.start;
4073 range.len = uffdio_wp.range.len;
4074 wake_userfault(ctx, &range);
4075 }
4076 return ret;
4077}
4078
4079static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
4080{
4081 __s64 ret;
4082 struct uffdio_continue uffdio_continue;
4083 struct uffdio_continue __user *user_uffdio_continue;
4084 struct userfaultfd_wake_range range;
4085 uffd_flags_t flags = 0;
4086
4087 user_uffdio_continue = (struct uffdio_continue __user *)arg;
4088
4089 ret = -EAGAIN;
4090 if (unlikely(atomic_read(&ctx->mmap_changing))) {
4091 if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
4092 return -EFAULT;
4093 goto out;
4094 }
4095
4096 ret = -EFAULT;
4097 if (copy_from_user(&uffdio_continue, user_uffdio_continue,
4098 /* don't copy the output fields */
4099 sizeof(uffdio_continue) - (sizeof(__s64))))
4100 goto out;
4101
4102 ret = validate_range(ctx->mm, uffdio_continue.range.start,
4103 uffdio_continue.range.len);
4104 if (ret)
4105 goto out;
4106
4107 ret = -EINVAL;
4108 if (uffdio_continue.mode & ~(UFFDIO_CONTINUE_MODE_DONTWAKE |
4109 UFFDIO_CONTINUE_MODE_WP))
4110 goto out;
4111 if (uffdio_continue.mode & UFFDIO_CONTINUE_MODE_WP)
4112 flags |= MFILL_ATOMIC_WP;
4113
4114 if (mmget_not_zero(ctx->mm)) {
4115 ret = mfill_atomic_continue(ctx, uffdio_continue.range.start,
4116 uffdio_continue.range.len, flags);
4117 mmput(ctx->mm);
4118 } else {
4119 return -ESRCH;
4120 }
4121
4122 if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
4123 return -EFAULT;
4124 if (ret < 0)
4125 goto out;
4126
4127 /* len == 0 would wake all */
4128 VM_WARN_ON_ONCE(!ret);
4129 range.len = ret;
4130 if (!(uffdio_continue.mode & UFFDIO_CONTINUE_MODE_DONTWAKE)) {
4131 range.start = uffdio_continue.range.start;
4132 wake_userfault(ctx, &range);
4133 }
4134 ret = range.len == uffdio_continue.range.len ? 0 : -EAGAIN;
4135
4136out:
4137 return ret;
4138}
4139
4140static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long arg)
4141{
4142 __s64 ret;
4143 struct uffdio_poison uffdio_poison;
4144 struct uffdio_poison __user *user_uffdio_poison;
4145 struct userfaultfd_wake_range range;
4146
4147 user_uffdio_poison = (struct uffdio_poison __user *)arg;
4148
4149 ret = -EAGAIN;
4150 if (unlikely(atomic_read(&ctx->mmap_changing))) {
4151 if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
4152 return -EFAULT;
4153 goto out;
4154 }
4155
4156 ret = -EFAULT;
4157 if (copy_from_user(&uffdio_poison, user_uffdio_poison,
4158 /* don't copy the output fields */
4159 sizeof(uffdio_poison) - (sizeof(__s64))))
4160 goto out;
4161
4162 ret = validate_range(ctx->mm, uffdio_poison.range.start,
4163 uffdio_poison.range.len);
4164 if (ret)
4165 goto out;
4166
4167 ret = -EINVAL;
4168 if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE)
4169 goto out;
4170
4171 if (mmget_not_zero(ctx->mm)) {
4172 ret = mfill_atomic_poison(ctx, uffdio_poison.range.start,
4173 uffdio_poison.range.len, 0);
4174 mmput(ctx->mm);
4175 } else {
4176 return -ESRCH;
4177 }
4178
4179 if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
4180 return -EFAULT;
4181 if (ret < 0)
4182 goto out;
4183
4184 /* len == 0 would wake all */
4185 VM_WARN_ON_ONCE(!ret);
4186 range.len = ret;
4187 if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) {
4188 range.start = uffdio_poison.range.start;
4189 wake_userfault(ctx, &range);
4190 }
4191 ret = range.len == uffdio_poison.range.len ? 0 : -EAGAIN;
4192
4193out:
4194 return ret;
4195}
4196
4197bool userfaultfd_wp_async(struct vm_area_struct *vma)
4198{
4199 return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
4200}
4201
4202static inline unsigned int uffd_ctx_features(__u64 user_features)
4203{
4204 /*
4205 * For the current set of features the bits just coincide. Set
4206 * UFFD_FEATURE_INITIALIZED to mark the features as enabled.
4207 */
4208 return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
4209}
4210
4211static int userfaultfd_move(struct userfaultfd_ctx *ctx,
4212 unsigned long arg)
4213{
4214 __s64 ret;
4215 struct uffdio_move uffdio_move;
4216 struct uffdio_move __user *user_uffdio_move;
4217 struct userfaultfd_wake_range range;
4218 struct mm_struct *mm = ctx->mm;
4219
4220 user_uffdio_move = (struct uffdio_move __user *) arg;
4221
4222 ret = -EAGAIN;
4223 if (unlikely(atomic_read(&ctx->mmap_changing))) {
4224 if (unlikely(put_user(ret, &user_uffdio_move->move)))
4225 return -EFAULT;
4226 goto out;
4227 }
4228
4229 if (copy_from_user(&uffdio_move, user_uffdio_move,
4230 /* don't copy "move" last field */
4231 sizeof(uffdio_move)-sizeof(__s64)))
4232 return -EFAULT;
4233
4234 /* Do not allow cross-mm moves. */
4235 if (mm != current->mm)
4236 return -EINVAL;
4237
4238 ret = validate_range(mm, uffdio_move.dst, uffdio_move.len);
4239 if (ret)
4240 return ret;
4241
4242 ret = validate_range(mm, uffdio_move.src, uffdio_move.len);
4243 if (ret)
4244 return ret;
4245
4246 if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES|
4247 UFFDIO_MOVE_MODE_DONTWAKE))
4248 return -EINVAL;
4249
4250 if (mmget_not_zero(mm)) {
4251 ret = move_pages(ctx, uffdio_move.dst, uffdio_move.src,
4252 uffdio_move.len, uffdio_move.mode);
4253 mmput(mm);
4254 } else {
4255 return -ESRCH;
4256 }
4257
4258 if (unlikely(put_user(ret, &user_uffdio_move->move)))
4259 return -EFAULT;
4260 if (ret < 0)
4261 goto out;
4262
4263 /* len == 0 would wake all */
4264 VM_WARN_ON(!ret);
4265 range.len = ret;
4266 if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) {
4267 range.start = uffdio_move.dst;
4268 wake_userfault(ctx, &range);
4269 }
4270 ret = range.len == uffdio_move.len ? 0 : -EAGAIN;
4271
4272out:
4273 return ret;
4274}
4275
4276/*
4277 * userland asks for a certain API version and we return which bits
4278 * and ioctl commands are implemented in this kernel for such API
4279 * version or -EINVAL if unknown.
4280 */
4281static int userfaultfd_api(struct userfaultfd_ctx *ctx,
4282 unsigned long arg)
4283{
4284 struct uffdio_api uffdio_api;
4285 void __user *buf = (void __user *)arg;
4286 unsigned int ctx_features;
4287 int ret;
4288 __u64 features;
4289
4290 ret = -EFAULT;
4291 if (copy_from_user(&uffdio_api, buf, sizeof(uffdio_api)))
4292 goto out;
4293 features = uffdio_api.features;
4294 ret = -EINVAL;
4295 if (uffdio_api.api != UFFD_API)
4296 goto err_out;
4297 ret = -EPERM;
4298 if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
4299 goto err_out;
4300
4301 /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
4302 if (features & UFFD_FEATURE_WP_ASYNC)
4303 features |= UFFD_FEATURE_WP_UNPOPULATED;
4304
4305 /* report all available features and ioctls to userland */
4306 uffdio_api.features = UFFD_API_FEATURES;
4307#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
4308 uffdio_api.features &=
4309 ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
4310#endif
4311 if (!pgtable_supports_uffd_wp())
4312 uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
4313
4314 if (!uffd_supports_wp_marker()) {
4315 uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
4316 uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
4317 uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
4318 }
4319
4320 ret = -EINVAL;
4321 if (features & ~uffdio_api.features)
4322 goto err_out;
4323
4324 uffdio_api.ioctls = UFFD_API_IOCTLS;
4325 ret = -EFAULT;
4326 if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
4327 goto out;
4328
4329 /* only enable the requested features for this uffd context */
4330 ctx_features = uffd_ctx_features(features);
4331 ret = -EINVAL;
4332 if (cmpxchg(&ctx->features, 0, ctx_features) != 0)
4333 goto err_out;
4334
4335 ret = 0;
4336out:
4337 return ret;
4338err_out:
4339 memset(&uffdio_api, 0, sizeof(uffdio_api));
4340 if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
4341 ret = -EFAULT;
4342 goto out;
4343}
4344
4345static long userfaultfd_ioctl(struct file *file, unsigned cmd,
4346 unsigned long arg)
4347{
4348 int ret = -EINVAL;
4349 struct userfaultfd_ctx *ctx = file->private_data;
4350
4351 if (cmd != UFFDIO_API && !userfaultfd_is_initialized(ctx))
4352 return -EINVAL;
4353
4354 switch (cmd) {
4355 case UFFDIO_API:
4356 ret = userfaultfd_api(ctx, arg);
4357 break;
4358 case UFFDIO_REGISTER:
4359 ret = userfaultfd_register(ctx, arg);
4360 break;
4361 case UFFDIO_UNREGISTER:
4362 ret = userfaultfd_unregister(ctx, arg);
4363 break;
4364 case UFFDIO_WAKE:
4365 ret = userfaultfd_wake(ctx, arg);
4366 break;
4367 case UFFDIO_COPY:
4368 ret = userfaultfd_copy(ctx, arg);
4369 break;
4370 case UFFDIO_ZEROPAGE:
4371 ret = userfaultfd_zeropage(ctx, arg);
4372 break;
4373 case UFFDIO_MOVE:
4374 ret = userfaultfd_move(ctx, arg);
4375 break;
4376 case UFFDIO_WRITEPROTECT:
4377 ret = userfaultfd_writeprotect(ctx, arg);
4378 break;
4379 case UFFDIO_CONTINUE:
4380 ret = userfaultfd_continue(ctx, arg);
4381 break;
4382 case UFFDIO_POISON:
4383 ret = userfaultfd_poison(ctx, arg);
4384 break;
4385 }
4386 return ret;
4387}
4388
4389#ifdef CONFIG_PROC_FS
4390static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
4391{
4392 struct userfaultfd_ctx *ctx = f->private_data;
4393 wait_queue_entry_t *wq;
4394 unsigned long pending = 0, total = 0;
4395
4396 spin_lock_irq(&ctx->fault_pending_wqh.lock);
4397 list_for_each_entry(wq, &ctx->fault_pending_wqh.head, entry) {
4398 pending++;
4399 total++;
4400 }
4401 list_for_each_entry(wq, &ctx->fault_wqh.head, entry) {
4402 total++;
4403 }
4404 spin_unlock_irq(&ctx->fault_pending_wqh.lock);
4405
4406 /*
4407 * If more protocols will be added, there will be all shown
4408 * separated by a space. Like this:
4409 * protocols: aa:... bb:...
4410 */
4411 seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
4412 pending, total, UFFD_API, ctx->features,
4413 UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
4414}
4415#endif
4416
4417static const struct file_operations userfaultfd_fops = {
4418#ifdef CONFIG_PROC_FS
4419 .show_fdinfo = userfaultfd_show_fdinfo,
4420#endif
4421 .release = userfaultfd_release,
4422 .poll = userfaultfd_poll,
4423 .read_iter = userfaultfd_read_iter,
4424 .unlocked_ioctl = userfaultfd_ioctl,
4425 .compat_ioctl = compat_ptr_ioctl,
4426 .llseek = noop_llseek,
4427};
4428
4429static void init_once_userfaultfd_ctx(void *mem)
4430{
4431 struct userfaultfd_ctx *ctx = (struct userfaultfd_ctx *) mem;
4432
4433 init_waitqueue_head(&ctx->fault_pending_wqh);
4434 init_waitqueue_head(&ctx->fault_wqh);
4435 init_waitqueue_head(&ctx->event_wqh);
4436 init_waitqueue_head(&ctx->fd_wqh);
4437 seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock);
4438}
4439
4440static int new_userfaultfd(int flags)
4441{
4442 struct userfaultfd_ctx *ctx __free(kfree) = NULL;
4443
4444 VM_WARN_ON_ONCE(!current->mm);
4445
4446 /* Check the UFFD_* constants for consistency. */
4447 BUILD_BUG_ON(UFFD_USER_MODE_ONLY & UFFD_SHARED_FCNTL_FLAGS);
4448
4449 if (flags & ~(UFFD_SHARED_FCNTL_FLAGS | UFFD_USER_MODE_ONLY))
4450 return -EINVAL;
4451
4452 ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
4453 if (!ctx)
4454 return -ENOMEM;
4455
4456 refcount_set(&ctx->refcount, 1);
4457 ctx->flags = flags;
4458 ctx->features = 0;
4459 ctx->released = false;
4460 init_rwsem(&ctx->map_changing_lock);
4461 atomic_set(&ctx->mmap_changing, 0);
4462 ctx->mm = current->mm;
4463
4464 FD_PREPARE(fdf, flags & UFFD_SHARED_FCNTL_FLAGS,
4465 anon_inode_create_getfile("[userfaultfd]", &userfaultfd_fops, ctx,
4466 O_RDONLY | (flags & UFFD_SHARED_FCNTL_FLAGS),
4467 NULL));
4468 if (fdf.err)
4469 return fdf.err;
4470
4471 /* prevent the mm struct to be freed */
4472 mmgrab(ctx->mm);
4473 fd_prepare_file(fdf)->f_mode |= FMODE_NOWAIT;
4474 retain_and_null_ptr(ctx);
4475 return fd_publish(fdf);
4476}
4477
4478static inline bool userfaultfd_syscall_allowed(int flags)
4479{
4480 /* Userspace-only page faults are always allowed */
4481 if (flags & UFFD_USER_MODE_ONLY)
4482 return true;
4483
4484 /*
4485 * The user is requesting a userfaultfd which can handle kernel faults.
4486 * Privileged users are always allowed to do this.
4487 */
4488 if (capable(CAP_SYS_PTRACE))
4489 return true;
4490
4491 /* Otherwise, access to kernel fault handling is sysctl controlled. */
4492 return sysctl_unprivileged_userfaultfd;
4493}
4494
4495SYSCALL_DEFINE1(userfaultfd, int, flags)
4496{
4497 if (!userfaultfd_syscall_allowed(flags))
4498 return -EPERM;
4499
4500 return new_userfaultfd(flags);
4501}
4502
4503static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags)
4504{
4505 if (cmd != USERFAULTFD_IOC_NEW)
4506 return -EINVAL;
4507
4508 return new_userfaultfd(flags);
4509}
4510
4511static const struct file_operations userfaultfd_dev_fops = {
4512 .unlocked_ioctl = userfaultfd_dev_ioctl,
4513 .compat_ioctl = userfaultfd_dev_ioctl,
4514 .owner = THIS_MODULE,
4515 .llseek = noop_llseek,
4516};
4517
4518static struct miscdevice userfaultfd_misc = {
4519 .minor = MISC_DYNAMIC_MINOR,
4520 .name = "userfaultfd",
4521 .fops = &userfaultfd_dev_fops
4522};
4523
4524static int __init userfaultfd_init(void)
4525{
4526 int ret;
4527
4528 ret = misc_register(&userfaultfd_misc);
4529 if (ret)
4530 return ret;
4531
4532 userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache",
4533 sizeof(struct userfaultfd_ctx),
4534 0,
4535 SLAB_HWCACHE_ALIGN|SLAB_PANIC,
4536 init_once_userfaultfd_ctx);
4537#ifdef CONFIG_SYSCTL
4538 register_sysctl_init("vm", vm_userfaultfd_table);
4539#endif
4540 return 0;
4541}
4542__initcall(userfaultfd_init);
mm/util.c+22−10
@@ -1192,6 +1192,7 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, · 1191 unmodified lines
1192 desc->vm_file = vma->vm_file;1192 desc->vm_file = vma->vm_file;
1193 desc->vma_flags = vma->flags;1193 desc->vma_flags = vma->flags;
1194 desc->page_prot = vma->vm_page_prot;1194 desc->page_prot = vma->vm_page_prot;
1195 desc->vm_ops = vma->vm_ops;
1195 1196
1196 /* Default. */1197 /* Default. */
1197 desc->action.type = MMAP_NOTHING;1198 desc->action.type = MMAP_NOTHING;
@@ -1396,8 +1397,6 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 198 unmodified lines
1396 1397
1397 if (!err)1398 if (!err)
1398 err = call_vma_mapped(vma);1399 err = call_vma_mapped(vma);
1399 if (!err && action->success_hook)
1400 err = action->success_hook(vma);
1401 1400
1402 /* do_munmap() might take rmap lock, so release if held. */1401 /* do_munmap() might take rmap lock, so release if held. */
1403 maybe_rmap_unlock_action(vma, action);1402 maybe_rmap_unlock_action(vma, action);
@@ -1415,16 +1414,22 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 11 unmodified lines
1415 */1414 */
1416 len = vma_pages(vma) << PAGE_SHIFT;1415 len = vma_pages(vma) << PAGE_SHIFT;
1417 do_munmap(current->mm, vma->vm_start, len, NULL);1416 do_munmap(current->mm, vma->vm_start, len, NULL);
1418 if (action->error_hook) {1417
1419 /* We may want to filter the error. */1418 return action->error_override ?: err;
1420 err = action->error_hook(err);
1421 /* The caller should not clear the error. */
1422 VM_WARN_ON_ONCE(!err);
1423 }
1424 return err;
1425}1419}
1426 1420
1427#ifdef CONFIG_MMU1421#ifdef CONFIG_MMU
1422
1423static int check_mmap_action(struct mmap_action *action)
1424{
1425 const unsigned long override = action->error_override;
1426
1427 if (WARN_ON_ONCE(override && !IS_ERR_VALUE(override)))
1428 return -EINVAL;
1429
1430 return 0;
1431}
1432
1428/**1433/**
1429 * mmap_action_prepare - Perform preparatory setup for an VMA descriptor1434 * mmap_action_prepare - Perform preparatory setup for an VMA descriptor
1430 * action which need to be performed.1435 * action which need to be performed.
@@ -1434,7 +1439,14 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 3 unmodified lines
1434 */1439 */
1435int mmap_action_prepare(struct vm_area_desc *desc)1440int mmap_action_prepare(struct vm_area_desc *desc)
1436{1441{
1437 switch (desc->action.type) {1442 struct mmap_action *action = &desc->action;
1443 int err;
1444
1445 err = check_mmap_action(action);
1446 if (err)
1447 return err;
1448
1449 switch (action->type) {
1438 case MMAP_NOTHING:1450 case MMAP_NOTHING:
1439 return 0;1451 return 0;
1440 case MMAP_REMAP_PFN:1452 case MMAP_REMAP_PFN:
mm/vma.c+3−0
@@ -2697,6 +2697,8 @@ static void set_vma_user_defined_fields(struct vm_area_struct *vma, · 2696 unmodified lines
2697{2697{
2698 if (map->vm_ops)2698 if (map->vm_ops)
2699 vma->vm_ops = map->vm_ops;2699 vma->vm_ops = map->vm_ops;
2700 else /* Only /dev/zero should do this. */
2701 vma_set_anonymous(vma);
2700 vma->vm_private_data = map->vm_private_data;2702 vma->vm_private_data = map->vm_private_data;
2701}2703}
2702 2704
@@ -2744,6 +2746,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, · 41 unmodified lines
2744 .action = {2746 .action = {
2745 .type = MMAP_NOTHING, /* Default to no further action. */2747 .type = MMAP_NOTHING, /* Default to no further action. */
2746 },2748 },
2749 .vm_ops = &vma_dummy_vm_ops,
2747 };2750 };
2748 bool allocated_new = false;2751 bool allocated_new = false;
2749 int error;2752 int error;
mm/vmalloc.c+107−23
@@ -3416,6 +3416,32 @@ void vfree_atomic(const void *addr) · 3415 unmodified lines
3416 schedule_work(&p->wq);3416 schedule_work(&p->wq);
3417}3417}
3418 3418
3419/*
3420 * vm_area_free_pages - free a range of pages from a vmalloc allocation
3421 * @vm: the vm_struct containing the pages
3422 * @start_idx: first page index to free (inclusive)
3423 * @end_idx: last page index to free (exclusive)
3424 *
3425 * Free pages [start_idx, end_idx) updating NR_VMALLOC stat accounting.
3426 * Freed vm->pages[] entries are set to NULL.
3427 * Caller is responsible for unmapping (vunmap_range) and KASAN
3428 * poisoning before calling this.
3429 */
3430static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx,
3431 unsigned int end_idx)
3432{
3433 unsigned int i;
3434
3435 if (!(vm->flags & VM_MAP_PUT_PAGES)) {
3436 for (i = start_idx; i < end_idx; i++)
3437 mod_lruvec_page_state(vm->pages[i], NR_VMALLOC, -1);
3438 }
3439 free_pages_bulk(vm->pages + start_idx, end_idx - start_idx);
3440
3441 for (i = start_idx; i < end_idx; i++)
3442 vm->pages[i] = NULL;
3443}
3444
3419/**3445/**
3420 * vfree - Release memory allocated by vmalloc()3446 * vfree - Release memory allocated by vmalloc()
3421 * @addr: Memory base address3447 * @addr: Memory base address
@@ -3436,7 +3462,6 @@ void vfree_atomic(const void *addr) · 14 unmodified lines
3436void vfree(const void *addr)3462void vfree(const void *addr)
3437{3463{
3438 struct vm_struct *vm;3464 struct vm_struct *vm;
3439 int i;
3440 3465
3441 if (unlikely(in_interrupt())) {3466 if (unlikely(in_interrupt())) {
3442 vfree_atomic(addr);3467 vfree_atomic(addr);
@@ -3459,19 +3484,8 @@ void vfree(const void *addr) · 16 unmodified lines
3459 3484
3460 if (unlikely(vm->flags & VM_FLUSH_RESET_PERMS))3485 if (unlikely(vm->flags & VM_FLUSH_RESET_PERMS))
3461 vm_reset_perms(vm);3486 vm_reset_perms(vm);
3462 for (i = 0; i < vm->nr_pages; i++) {
3463 struct page *page = vm->pages[i];
3464 3487
3465 BUG_ON(!page);3488 vm_area_free_pages(vm, 0, vm->nr_pages);
3466 /*
3467 * High-order allocs for huge vmallocs are split, so
3468 * can be freed as an array of order-0 allocations
3469 */
3470 if (!(vm->flags & VM_MAP_PUT_PAGES))
3471 mod_lruvec_page_state(page, NR_VMALLOC, -1);
3472 __free_page(page);
3473 cond_resched();
3474 }
3475 kvfree(vm->pages);3489 kvfree(vm->pages);
3476 kfree(vm);3490 kfree(vm);
3477}3491}
@@ -3939,7 +3953,7 @@ fail: · 461 unmodified lines
3939 __GFP_NOFAIL | __GFP_ZERO |\3953 __GFP_NOFAIL | __GFP_ZERO |\
3940 __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\3954 __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\
3941 GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\3955 GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
3942 GFP_USER | __GFP_NOLOCKDEP)3956 GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)
3943 3957
3944static gfp_t vmalloc_fix_flags(gfp_t flags)3958static gfp_t vmalloc_fix_flags(gfp_t flags)
3945{3959{
@@ -3980,6 +3994,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags) · 34 unmodified lines
3980 *3994 *
3981 * %__GFP_NOWARN can be used to suppress failure messages.3995 * %__GFP_NOWARN can be used to suppress failure messages.
3982 *3996 *
3997 * %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages
3998 * (when prot=%PAGE_KERNEL).
3999 *
3983 * Can not be called from interrupt nor NMI contexts.4000 * Can not be called from interrupt nor NMI contexts.
3984 * Return: the address of the area or %NULL on failure4001 * Return: the address of the area or %NULL on failure
3985 */4002 */
@@ -3993,6 +4010,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · 7 unmodified lines
3993 kasan_vmalloc_flags_t kasan_flags = KASAN_VMALLOC_NONE;4010 kasan_vmalloc_flags_t kasan_flags = KASAN_VMALLOC_NONE;
3994 unsigned long original_align = align;4011 unsigned long original_align = align;
3995 unsigned int shift = PAGE_SHIFT;4012 unsigned int shift = PAGE_SHIFT;
4013 bool skip_vmalloc_kasan = kasan_hw_tags_enabled() && (gfp_mask & __GFP_SKIP_KASAN);
3996 4014
3997 if (WARN_ON_ONCE(!size))4015 if (WARN_ON_ONCE(!size))
3998 return NULL;4016 return NULL;
@@ -4023,7 +4041,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · 24 unmodified lines
4023again:4041again:
4024 area = __get_vm_area_node(size, align, shift, VM_ALLOC |4042 area = __get_vm_area_node(size, align, shift, VM_ALLOC |
4025 VM_UNINITIALIZED | vm_flags, start, end, node,4043 VM_UNINITIALIZED | vm_flags, start, end, node,
4026 gfp_mask, caller);4044 gfp_mask & ~__GFP_SKIP_KASAN, caller);
4027 if (!area) {4045 if (!area) {
4028 bool nofail = gfp_mask & __GFP_NOFAIL;4046 bool nofail = gfp_mask & __GFP_NOFAIL;
4029 warn_alloc(gfp_mask, NULL,4047 warn_alloc(gfp_mask, NULL,
@@ -4041,7 +4059,7 @@ again: · 11 unmodified lines
4041 * kasan_unpoison_vmalloc().4059 * kasan_unpoison_vmalloc().
4042 */4060 */
4043 if (pgprot_val(prot) == pgprot_val(PAGE_KERNEL)) {4061 if (pgprot_val(prot) == pgprot_val(PAGE_KERNEL)) {
4044 if (kasan_hw_tags_enabled()) {4062 if (kasan_hw_tags_enabled() && !skip_vmalloc_kasan) {
4045 /*4063 /*
4046 * Modify protection bits to allow tagging.4064 * Modify protection bits to allow tagging.
4047 * This must be done before mapping.4065 * This must be done before mapping.
@@ -4078,7 +4096,8 @@ again: · 30 unmodified lines
4078 (gfp_mask & __GFP_SKIP_ZERO))4096 (gfp_mask & __GFP_SKIP_ZERO))
4079 kasan_flags |= KASAN_VMALLOC_INIT;4097 kasan_flags |= KASAN_VMALLOC_INIT;
4080 /* KASAN_VMALLOC_PROT_NORMAL already set if required. */4098 /* KASAN_VMALLOC_PROT_NORMAL already set if required. */
4081 area->addr = kasan_unpoison_vmalloc(area->addr, size, kasan_flags);4099 if (!skip_vmalloc_kasan)
4100 area->addr = kasan_unpoison_vmalloc(area->addr, size, kasan_flags);
4082 4101
4083 /*4102 /*
4084 * In this function, newly allocated vm_struct has VM_UNINITIALIZED4103 * In this function, newly allocated vm_struct has VM_UNINITIALIZED
@@ -4324,16 +4343,70 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 239 unmodified lines
4324 if (unlikely(flags & __GFP_THISNODE) && nid != NUMA_NO_NODE &&4343 if (unlikely(flags & __GFP_THISNODE) && nid != NUMA_NO_NODE &&
4325 nid != page_to_nid(vmalloc_to_page(p)))4344 nid != page_to_nid(vmalloc_to_page(p)))
4326 goto need_realloc;4345 goto need_realloc;
4346 } else {
4347 /*
4348 * If p is NULL, vrealloc behaves exactly like vmalloc.
4349 * Skip the shrink and in-place grow paths.
4350 */
4351 goto need_realloc;
4327 }4352 }
4328 4353
4329 /*
4330 * TODO: Shrink the vm_area, i.e. unmap and free unused pages. What
4331 * would be a good heuristic for when to shrink the vm_area?
4332 */
4333 if (size <= old_size) {4354 if (size <= old_size) {
4355 unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
4356
4334 /* Zero out "freed" memory, potentially for future realloc. */4357 /* Zero out "freed" memory, potentially for future realloc. */
4335 if (want_init_on_free() || want_init_on_alloc(flags))4358 if (want_init_on_free() || want_init_on_alloc(flags))
4336 memset((void *)p + size, 0, old_size - size);4359 memset((void *)p + size, 0, old_size - size);
4360
4361 /*
4362 * Free tail pages when shrink crosses a page boundary.
4363 *
4364 * Skip huge page allocations (page_order > 0) as partial
4365 * freeing would require splitting.
4366 *
4367 * Skip VM_FLUSH_RESET_PERMS, as direct-map permissions must
4368 * be reset before pages are returned to the allocator.
4369 *
4370 * Skip VM_USERMAP, as remap_vmalloc_range_partial() validates
4371 * mapping requests against the unchanged vm->size; freeing
4372 * tail pages would cause vmalloc_to_page() to return NULL for
4373 * the unmapped range.
4374 *
4375 * Skip if either GFP_NOFS or GFP_NOIO are used.
4376 * kmemleak_free_part() internally allocates with
4377 * GFP_KERNEL, which could trigger a recursive deadlock
4378 * if we are under filesystem or I/O reclaim.
4379 */
4380 if (new_nr_pages < vm->nr_pages && !vm_area_page_order(vm) &&
4381 !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) &&
4382 gfp_has_io_fs(flags)) {
4383 unsigned long addr = (unsigned long)kasan_reset_tag(p);
4384 unsigned int old_nr_pages = vm->nr_pages;
4385
4386 /*
4387 * Use the node lock to synchronize with concurrent
4388 * readers (vmalloc_info_show).
4389 */
4390 struct vmap_node *vn = addr_to_node(addr);
4391
4392 spin_lock(&vn->busy.lock);
4393 vm->nr_pages = new_nr_pages;
4394 spin_unlock(&vn->busy.lock);
4395
4396 /* Notify kmemleak of the reduced allocation size before unmapping. */
4397 kmemleak_free_part(
4398 (void *)addr + ((unsigned long)new_nr_pages
4399 << PAGE_SHIFT),
4400 (unsigned long)(old_nr_pages - new_nr_pages)
4401 << PAGE_SHIFT);
4402
4403 vunmap_range(addr + ((unsigned long)new_nr_pages
4404 << PAGE_SHIFT),
4405 addr + ((unsigned long)old_nr_pages
4406 << PAGE_SHIFT));
4407
4408 vm_area_free_pages(vm, new_nr_pages, old_nr_pages);
4409 }
4337 vm->requested_size = size;4410 vm->requested_size = size;
4338 kasan_vrealloc(p, old_size, size);4411 kasan_vrealloc(p, old_size, size);
4339 return (void *)p;4412 return (void *)p;
@@ -4342,7 +4415,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 2 unmodified lines
4342 /*4415 /*
4343 * We already have the bytes available in the allocation; use them.4416 * We already have the bytes available in the allocation; use them.
4344 */4417 */
4345 if (size <= alloced_size) {4418 if (size <= vm->nr_pages << PAGE_SHIFT) {
4346 /*4419 /*
4347 * No need to zero memory here, as unused memory will have4420 * No need to zero memory here, as unused memory will have
4348 * already been zeroed at initial allocation time or during4421 * already been zeroed at initial allocation time or during
@@ -4641,7 +4714,18 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · 292 unmodified lines
4641 smp_rmb();4714 smp_rmb();
4642 4715
4643 vaddr = (char *) va->va_start;4716 vaddr = (char *) va->va_start;
4644 size = vm ? get_vm_area_size(vm) : va_size(va);4717 if (vm)
4718 /*
4719 * For VM_ALLOC areas, use nr_pages rather than
4720 * get_vm_area_size() because vrealloc() may shrink
4721 * the mapping without updating area->size. Other
4722 * mapping types (vmap, ioremap) don't set nr_pages.
4723 */
4724 size = (vm->flags & VM_ALLOC && vm->nr_pages) ?
4725 (vm->nr_pages << PAGE_SHIFT) :
4726 get_vm_area_size(vm);
4727 else
4728 size = va_size(va);
4645 4729
4646 if (addr >= vaddr + size)4730 if (addr >= vaddr + size)
4647 goto next_va;4731 goto next_va;
mm/vmpressure.c+12−3
@@ -218,6 +218,7 @@ static void vmpressure_work_fn(struct work_struct *work) · 217 unmodified lines
218/**218/**
219 * vmpressure() - Account memory pressure through scanned/reclaimed ratio219 * vmpressure() - Account memory pressure through scanned/reclaimed ratio
220 * @gfp: reclaimer's gfp mask220 * @gfp: reclaimer's gfp mask
221 * @order: allocation order being reclaimed for
221 * @memcg: cgroup memory controller handle222 * @memcg: cgroup memory controller handle
222 * @tree: legacy subtree mode223 * @tree: legacy subtree mode
223 * @scanned: number of pages scanned224 * @scanned: number of pages scanned
@@ -236,7 +237,7 @@ static void vmpressure_work_fn(struct work_struct *work) · 12 unmodified lines
236 *237 *
237 * This function does not return any value.238 * This function does not return any value.
238 */239 */
239void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,240void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
240 unsigned long scanned, unsigned long reclaimed)241 unsigned long scanned, unsigned long reclaimed)
241{242{
242 struct vmpressure *vmpr;243 struct vmpressure *vmpr;
@@ -307,7 +308,15 @@ void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree, · 64 unmodified lines
307 308
308 level = vmpressure_calc_level(scanned, reclaimed);309 level = vmpressure_calc_level(scanned, reclaimed);
309 310
310 if (level > VMPRESSURE_LOW) {311 /*
312 * Once we go above COSTLY_ORDER, reclaim relies heavily on
313 * compaction to make progress. Reclaim efficiency was never a
314 * great proxy for pressure to begin with, but it's outright
315 * misleading with these high orders. Don't throttle sockets
316 * because somebody is attempting something crazy like an order-7
317 * and predictably struggling.
318 */
319 if (level > VMPRESSURE_LOW && order <= PAGE_ALLOC_COSTLY_ORDER) {
311 /*320 /*
312 * Let the socket buffer allocator know that321 * Let the socket buffer allocator know that
313 * we are having trouble reclaiming LRU pages.322 * we are having trouble reclaiming LRU pages.
@@ -348,7 +357,7 @@ void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio) · 34 unmodified lines
348 * to the vmpressure() basically means that we signal 'critical'357 * to the vmpressure() basically means that we signal 'critical'
349 * level.358 * level.
350 */359 */
351 vmpressure(gfp, memcg, true, vmpressure_win, 0);360 vmpressure(gfp, 0, memcg, true, vmpressure_win, 0);
352}361}
353 362
354#define MAX_VMPRESSURE_ARGS_LEN (strlen("critical") + strlen("hierarchy") + 2)363#define MAX_VMPRESSURE_ARGS_LEN (strlen("critical") + strlen("hierarchy") + 2)
mm/vmscan.c+168−202
@@ -109,7 +109,7 @@ struct scan_control { · 108 unmodified lines
109 /* zone_reclaim_mode */109 /* zone_reclaim_mode */
110 unsigned int may_unmap:1;110 unsigned int may_unmap:1;
111 111
112 /* zome_reclaim_mode, boost reclaim, cgroup restrictions */112 /* zone_reclaim_mode, boost reclaim, cgroup restrictions */
113 unsigned int may_swap:1;113 unsigned int may_swap:1;
114 114
115 /* Not allow cache_trim_mode to be turned on as part of reclaim? */115 /* Not allow cache_trim_mode to be turned on as part of reclaim? */
@@ -169,11 +169,9 @@ struct scan_control { · 53 unmodified lines
169 169
170 struct {170 struct {
171 unsigned int dirty;171 unsigned int dirty;
172 unsigned int unqueued_dirty;
173 unsigned int congested;172 unsigned int congested;
174 unsigned int writeback;173 unsigned int writeback;
175 unsigned int immediate;174 unsigned int immediate;
176 unsigned int file_taken;
177 unsigned int taken;175 unsigned int taken;
178 } nr;176 } nr;
179 177
@@ -739,7 +737,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, · 559 unmodified lines
739 737
740 if (reclaimed && !mapping_exiting(mapping))738 if (reclaimed && !mapping_exiting(mapping))
741 shadow = workingset_eviction(folio, target_memcg);739 shadow = workingset_eviction(folio, target_memcg);
742 memcg1_swapout(folio, swap);740 __memcg1_swapout(folio, ci);
743 __swap_cache_del_folio(ci, folio, swap, shadow);741 __swap_cache_del_folio(ci, folio, swap, shadow);
744 swap_cluster_unlock_irq(ci);742 swap_cluster_unlock_irq(ci);
745 } else {743 } else {
@@ -850,7 +848,11 @@ static bool lru_gen_set_refs(struct folio *folio) · 104 unmodified lines
850 return false;848 return false;
851 }849 }
852 850
853 set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));851 /* Promote on second access */
852 if (folio_lru_refs(folio) > 1)
853 set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
854 else
855 folio_mark_accessed(folio);
854 return true;856 return true;
855}857}
856#else858#else
@@ -1944,6 +1946,44 @@ static int current_may_throttle(void) · 1087 unmodified lines
1944 return !(current->flags & PF_LOCAL_THROTTLE);1946 return !(current->flags & PF_LOCAL_THROTTLE);
1945}1947}
1946 1948
1949static void handle_reclaim_writeback(unsigned long nr_taken,
1950 struct pglist_data *pgdat,
1951 struct scan_control *sc,
1952 struct reclaim_stat *stat)
1953{
1954 /*
1955 * If dirty folios are scanned that are not queued for IO, it
1956 * implies that flushers are not doing their job. This can
1957 * happen when memory pressure pushes dirty folios to the end of
1958 * the LRU before the dirty limits are breached and the dirty
1959 * data has expired. It can also happen when the proportion of
1960 * dirty folios grows not through writes but through memory
1961 * pressure reclaiming all the clean cache. And in some cases,
1962 * the flushers simply cannot keep up with the allocation
1963 * rate. Nudge the flusher threads in case they are asleep.
1964 */
1965 if (stat->nr_unqueued_dirty == nr_taken) {
1966 wakeup_flusher_threads(WB_REASON_VMSCAN);
1967 /*
1968 * For cgroupv1 dirty throttling is achieved by waking up
1969 * the kernel flusher here and later waiting on folios
1970 * which are in writeback to finish (see shrink_folio_list()).
1971 *
1972 * Flusher may not be able to issue writeback quickly
1973 * enough for cgroupv1 writeback throttling to work
1974 * on a large system.
1975 */
1976 if (!writeback_throttling_sane(sc))
1977 reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
1978 }
1979
1980 sc->nr.dirty += stat->nr_dirty;
1981 sc->nr.congested += stat->nr_congested;
1982 sc->nr.writeback += stat->nr_writeback;
1983 sc->nr.immediate += stat->nr_immediate;
1984 sc->nr.taken += nr_taken;
1985}
1986
1947/*1987/*
1948 * shrink_inactive_list() is a helper for shrink_node(). It returns the number1988 * shrink_inactive_list() is a helper for shrink_node(). It returns the number
1949 * of reclaimed pages1989 * of reclaimed pages
@@ -2007,42 +2047,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, · 57 unmodified lines
2007 lruvec_lock_irq(lruvec);2047 lruvec_lock_irq(lruvec);
2008 lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,2048 lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
2009 nr_scanned - nr_reclaimed);2049 nr_scanned - nr_reclaimed);
2010 2050 handle_reclaim_writeback(nr_taken, pgdat, sc, &stat);
2011 /*
2012 * If dirty folios are scanned that are not queued for IO, it
2013 * implies that flushers are not doing their job. This can
2014 * happen when memory pressure pushes dirty folios to the end of
2015 * the LRU before the dirty limits are breached and the dirty
2016 * data has expired. It can also happen when the proportion of
2017 * dirty folios grows not through writes but through memory
2018 * pressure reclaiming all the clean cache. And in some cases,
2019 * the flushers simply cannot keep up with the allocation
2020 * rate. Nudge the flusher threads in case they are asleep.
2021 */
2022 if (stat.nr_unqueued_dirty == nr_taken) {
2023 wakeup_flusher_threads(WB_REASON_VMSCAN);
2024 /*
2025 * For cgroupv1 dirty throttling is achieved by waking up
2026 * the kernel flusher here and later waiting on folios
2027 * which are in writeback to finish (see shrink_folio_list()).
2028 *
2029 * Flusher may not be able to issue writeback quickly
2030 * enough for cgroupv1 writeback throttling to work
2031 * on a large system.
2032 */
2033 if (!writeback_throttling_sane(sc))
2034 reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
2035 }
2036
2037 sc->nr.dirty += stat.nr_dirty;
2038 sc->nr.congested += stat.nr_congested;
2039 sc->nr.unqueued_dirty += stat.nr_unqueued_dirty;
2040 sc->nr.writeback += stat.nr_writeback;
2041 sc->nr.immediate += stat.nr_immediate;
2042 sc->nr.taken += nr_taken;
2043 if (file)
2044 sc->nr.file_taken += nr_taken;
2045
2046 trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,2051 trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
2047 nr_scanned, nr_reclaimed, &stat, sc->priority, file);2052 nr_scanned, nr_reclaimed, &stat, sc->priority, file);
2048 return nr_reclaimed;2053 return nr_reclaimed;
@@ -3220,7 +3225,7 @@ static int folio_update_gen(struct folio *folio, int gen) · 1171 unmodified lines
3220}3225}
3221 3226
3222/* protect pages accessed multiple times through file descriptors */3227/* protect pages accessed multiple times through file descriptors */
3223static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming)3228static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
3224{3229{
3225 int type = folio_is_file_lru(folio);3230 int type = folio_is_file_lru(folio);
3226 struct lru_gen_folio *lrugen = &lruvec->lrugen;3231 struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -3239,9 +3244,6 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai · 12 unmodified lines
3239 3244
3240 new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);3245 new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
3241 new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;3246 new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
3242 /* for folio_end_writeback() */
3243 if (reclaiming)
3244 new_flags |= BIT(PG_reclaim);
3245 } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));3247 } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
3246 3248
3247 lru_gen_update_size(lruvec, folio, old_gen, new_gen);3249 lru_gen_update_size(lruvec, folio, old_gen, new_gen);
@@ -3855,7 +3857,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · 607 unmodified lines
3855 VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);3857 VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
3856 VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);3858 VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
3857 3859
3858 new_gen = folio_inc_gen(lruvec, folio, false);3860 new_gen = folio_inc_gen(lruvec, folio);
3859 list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);3861 list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
3860 3862
3861 /* don't count the workingset being lazily promoted */3863 /* don't count the workingset being lazily promoted */
@@ -3878,10 +3880,9 @@ done: · 16 unmodified lines
3878 return true;3880 return true;
3879}3881}
3880 3882
3881static bool try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)3883static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
3882{3884{
3883 int gen, type, zone;3885 int gen, type, zone;
3884 bool success = false;
3885 bool seq_inc_flag = false;3886 bool seq_inc_flag = false;
3886 struct lru_gen_folio *lrugen = &lruvec->lrugen;3887 struct lru_gen_folio *lrugen = &lruvec->lrugen;
3887 DEFINE_MIN_SEQ(lruvec);3888 DEFINE_MIN_SEQ(lruvec);
@@ -3907,11 +3908,10 @@ next: · 19 unmodified lines
3907 3908
3908 /*3909 /*
3909 * If min_seq[type] of both anonymous and file is not increased,3910 * If min_seq[type] of both anonymous and file is not increased,
3910 * we can directly return false to avoid unnecessary checking3911 * return here to avoid unnecessary checking overhead later.
3911 * overhead later.
3912 */3912 */
3913 if (!seq_inc_flag)3913 if (!seq_inc_flag)
3914 return success;3914 return;
3915 3915
3916 /* see the comment on lru_gen_folio */3916 /* see the comment on lru_gen_folio */
3917 if (swappiness && swappiness <= MAX_SWAPPINESS) {3917 if (swappiness && swappiness <= MAX_SWAPPINESS) {
@@ -3929,10 +3929,7 @@ next: · 11 unmodified lines
3929 3929
3930 reset_ctrl_pos(lruvec, type, true);3930 reset_ctrl_pos(lruvec, type, true);
3931 WRITE_ONCE(lrugen->min_seq[type], min_seq[type]);3931 WRITE_ONCE(lrugen->min_seq[type], min_seq[type]);
3932 success = true;
3933 }3932 }
3934
3935 return success;
3936}3933}
3937 3934
3938static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness)3935static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness)
@@ -4084,27 +4081,33 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control · 145 unmodified lines
4084 sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);4081 sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
4085}4082}
4086 4083
4087static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)4084static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
4088{4085{
4089 int gen, type, zone;4086 int gen, type, zone;
4090 unsigned long total = 0;4087 unsigned long seq, total = 0;
4091 int swappiness = get_swappiness(lruvec, sc);
4092 struct lru_gen_folio *lrugen = &lruvec->lrugen;4088 struct lru_gen_folio *lrugen = &lruvec->lrugen;
4093 struct mem_cgroup *memcg = lruvec_memcg(lruvec);
4094 DEFINE_MAX_SEQ(lruvec);4089 DEFINE_MAX_SEQ(lruvec);
4095 DEFINE_MIN_SEQ(lruvec);4090 DEFINE_MIN_SEQ(lruvec);
4096 4091
4097 for_each_evictable_type(type, swappiness) {4092 for_each_evictable_type(type, swappiness) {
4098 unsigned long seq;
4099
4100 for (seq = min_seq[type]; seq <= max_seq; seq++) {4093 for (seq = min_seq[type]; seq <= max_seq; seq++) {
4101 gen = lru_gen_from_seq(seq);4094 gen = lru_gen_from_seq(seq);
4102
4103 for (zone = 0; zone < MAX_NR_ZONES; zone++)4095 for (zone = 0; zone < MAX_NR_ZONES; zone++)
4104 total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);4096 total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
4105 }4097 }
4106 }4098 }
4107 4099
4100 return total;
4101}
4102
4103static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
4104{
4105 unsigned long total;
4106 int swappiness = get_swappiness(lruvec, sc);
4107 struct mem_cgroup *memcg = lruvec_memcg(lruvec);
4108
4109 total = lruvec_evictable_size(lruvec, swappiness);
4110
4108 /* whether the size is big enough to be helpful */4111 /* whether the size is big enough to be helpful */
4109 return mem_cgroup_online(memcg) ? (total >> sc->priority) : total;4112 return mem_cgroup_online(memcg) ? (total >> sc->priority) : total;
4110}4113}
@@ -4577,7 +4580,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 466 unmodified lines
4577 int tier_idx)4580 int tier_idx)
4578{4581{
4579 bool success;4582 bool success;
4580 bool dirty, writeback;
4581 int gen = folio_lru_gen(folio);4583 int gen = folio_lru_gen(folio);
4582 int type = folio_is_file_lru(folio);4584 int type = folio_is_file_lru(folio);
4583 int zone = folio_zonenum(folio);4585 int zone = folio_zonenum(folio);
@@ -4607,7 +4609,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 23 unmodified lines
4607 4609
4608 /* protected */4610 /* protected */
4609 if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {4611 if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {
4610 gen = folio_inc_gen(lruvec, folio, false);4612 gen = folio_inc_gen(lruvec, folio);
4611 list_move(&folio->lru, &lrugen->folios[gen][type][zone]);4613 list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
4612 4614
4613 /* don't count the workingset being lazily promoted */4615 /* don't count the workingset being lazily promoted */
@@ -4622,26 +4624,11 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 8 unmodified lines
4622 4624
4623 /* ineligible */4625 /* ineligible */
4624 if (zone > sc->reclaim_idx) {4626 if (zone > sc->reclaim_idx) {
4625 gen = folio_inc_gen(lruvec, folio, false);4627 gen = folio_inc_gen(lruvec, folio);
4626 list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]);4628 list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
4627 return true;4629 return true;
4628 }4630 }
4629 4631
4630 dirty = folio_test_dirty(folio);
4631 writeback = folio_test_writeback(folio);
4632 if (type == LRU_GEN_FILE && dirty) {
4633 sc->nr.file_taken += delta;
4634 if (!writeback)
4635 sc->nr.unqueued_dirty += delta;
4636 }
4637
4638 /* waiting for writeback */
4639 if (writeback || (type == LRU_GEN_FILE && dirty)) {
4640 gen = folio_inc_gen(lruvec, folio, true);
4641 list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
4642 return true;
4643 }
4644
4645 return false;4632 return false;
4646}4633}
4647 4634
@@ -4649,12 +4636,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 1 unmodified line
4649{4636{
4650 bool success;4637 bool success;
4651 4638
4652 /* swap constrained */
4653 if (!(sc->gfp_mask & __GFP_IO) &&
4654 (folio_test_dirty(folio) ||
4655 (folio_test_anon(folio) && !folio_test_swapcache(folio))))
4656 return false;
4657
4658 /* raced with release_pages() */4639 /* raced with release_pages() */
4659 if (!folio_try_get(folio))4640 if (!folio_try_get(folio))
4660 return false;4641 return false;
@@ -4669,9 +4650,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 8 unmodified lines
4669 if (!folio_test_referenced(folio))4650 if (!folio_test_referenced(folio))
4670 set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);4651 set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
4671 4652
4672 /* for shrink_folio_list() */
4673 folio_clear_reclaim(folio);
4674
4675 success = lru_gen_del_folio(lruvec, folio, true);4653 success = lru_gen_del_folio(lruvec, folio, true);
4676 VM_WARN_ON_ONCE_FOLIO(!success, folio);4654 VM_WARN_ON_ONCE_FOLIO(!success, folio);
4677 4655
@@ -4680,7 +4658,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 2 unmodified lines
4680 4658
4681static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,4659static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
4682 struct scan_control *sc, int type, int tier,4660 struct scan_control *sc, int type, int tier,
4683 struct list_head *list)4661 struct list_head *list, int *isolatedp)
4684{4662{
4685 int i;4663 int i;
4686 int gen;4664 int gen;
@@ -4689,10 +4667,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 2 unmodified lines
4689 int scanned = 0;4667 int scanned = 0;
4690 int isolated = 0;4668 int isolated = 0;
4691 int skipped = 0;4669 int skipped = 0;
4692 int scan_batch = min(nr_to_scan, MAX_LRU_BATCH);4670 unsigned long remaining = nr_to_scan;
4693 int remaining = scan_batch;
4694 struct lru_gen_folio *lrugen = &lruvec->lrugen;4671 struct lru_gen_folio *lrugen = &lruvec->lrugen;
4695 4672
4673 VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
4696 VM_WARN_ON_ONCE(!list_empty(list));4674 VM_WARN_ON_ONCE(!list_empty(list));
4697 4675
4698 if (get_nr_gens(lruvec, type) == MIN_NR_GENS)4676 if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
@@ -4745,16 +4723,12 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 46 unmodified lines
4745 mod_lruvec_state(lruvec, item, isolated);4723 mod_lruvec_state(lruvec, item, isolated);
4746 mod_lruvec_state(lruvec, PGREFILL, sorted);4724 mod_lruvec_state(lruvec, PGREFILL, sorted);
4747 mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated);4725 mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated);
4748 trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, scan_batch,4726 trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan,
4749 scanned, skipped, isolated,4727 scanned, skipped, isolated,
4750 type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);4728 type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
4751 if (type == LRU_GEN_FILE)4729
4752 sc->nr.file_taken += isolated;4730 *isolatedp = isolated;
4753 /*4731 return scanned;
4754 * There might not be eligible folios due to reclaim_idx. Check the
4755 * remaining to prevent livelock if it's not making progress.
4756 */
4757 return isolated || !remaining ? scanned : 0;
4758}4732}
4759 4733
4760static int get_tier_idx(struct lruvec *lruvec, int type)4734static int get_tier_idx(struct lruvec *lruvec, int type)
@@ -4798,33 +4772,41 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness) · 37 unmodified lines
4798 4772
4799static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,4773static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
4800 struct scan_control *sc, int swappiness,4774 struct scan_control *sc, int swappiness,
4801 int *type_scanned, struct list_head *list)4775 struct list_head *list, int *isolated,
4776 int *isolate_type, int *isolate_scanned)
4802{4777{
4803 int i;4778 int i;
4779 int total_scanned = 0;
4804 int type = get_type_to_scan(lruvec, swappiness);4780 int type = get_type_to_scan(lruvec, swappiness);
4805 4781
4806 for_each_evictable_type(i, swappiness) {4782 for_each_evictable_type(i, swappiness) {
4807 int scanned;4783 int scanned;
4808 int tier = get_tier_idx(lruvec, type);4784 int tier = get_tier_idx(lruvec, type);
4809 4785
4810 *type_scanned = type;4786 scanned = scan_folios(nr_to_scan, lruvec, sc,
4811 4787 type, tier, list, isolated);
4812 scanned = scan_folios(nr_to_scan, lruvec, sc, type, tier, list);
4813 if (scanned)
4814 return scanned;
4815 4788
4816 type = !type;4789 total_scanned += scanned;
4790 if (*isolated) {
4791 *isolate_type = type;
4792 *isolate_scanned = scanned;
4793 break;
4794 }
4795 /*
4796 * If scanned > 0 and isolated == 0, avoid falling back to the
4797 * other type, as this type remains sufficient. Falling back
4798 * too readily can disrupt the positive_ctrl_err() bias.
4799 */
4800 if (!scanned)
4801 type = !type;
4817 }4802 }
4818 4803
4819 return 0;4804 return total_scanned;
4820}4805}
4821 4806
4822static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,4807static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
4823 struct scan_control *sc, int swappiness)4808 struct scan_control *sc, int swappiness)
4824{4809{
4825 int type;
4826 int scanned;
4827 int reclaimed;
4828 LIST_HEAD(list);4810 LIST_HEAD(list);
4829 LIST_HEAD(clean);4811 LIST_HEAD(clean);
4830 struct folio *folio;4812 struct folio *folio;

332 more lines aren’t shown here.

mm/workingset.c+6−4

This diff is large: its 18 lines aren’t shown here.

mm/zswap.c+10−15

This diff is large: its 50 lines aren’t shown here.

net/rds/tcp_recv.c+1−1

This diff is large: its 9 lines aren’t shown here.

rust/kernel/page.rs+1−0

This diff is large: its 8 lines aren’t shown here.

samples/damon/mtier.c+6−4

This diff is large: its 29 lines aren’t shown here.

tools/mm/page-types.c+4−4

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/cgroup/lib/cgroup_util.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/cgroup/lib/include/cgroup_util.h+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/cgroup/test_core.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/cgroup/test_freezer.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/cgroup/test_kmem.c+7−6

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/cgroup/test_memcontrol.c+9−4

This diff is large: its 55 lines aren’t shown here.

tools/testing/selftests/cgroup/test_zswap.c+127−54

This diff is large: its 371 lines aren’t shown here.

tools/testing/selftests/damon/_damon_sysfs.py+28−3

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/damon/drgn_dump_damon_status.py+3−0

This diff is large: its 17 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.py+55−5

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.sh+62−0

This diff is large: its 90 lines aren’t shown here.

tools/testing/selftests/memfd/fuse_test.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/memfd/memfd_test.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+7−3

This diff is large: its 38 lines aren’t shown here.

tools/testing/selftests/mm/check_config.sh+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/droppable.c+8−1

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/mm/hmm-tests.c+33−3

This diff is large: its 73 lines aren’t shown here.

tools/testing/selftests/mm/khugepaged.c+15−3

This diff is large: its 33 lines aren’t shown here.

tools/testing/selftests/mm/ksft_kmemleak_dedup.sh+222−0

This diff is large: its 223 lines aren’t shown here.

tools/testing/selftests/mm/ksm_functional_tests.c+11−8

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/mm/mlock2-tests.c+75−9

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/mm/mremap_test.c+10−99

This diff is large: its 138 lines aren’t shown here.

tools/testing/selftests/mm/process_madv.c+28−0

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/mm/run_vmtests.sh+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/split_huge_page_test.c+8−4

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/proc/proc-maps-race.c+231−62

This diff is large: its 421 lines aren’t shown here.

tools/testing/vma/include/dup.h+4−16

This diff is large: its 35 lines aren’t shown here.