History

Merge tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton: - "selftests/mm: clean up build output and verbosity" (Li Wang) Remove some noise from the MM selftests build - "mm: Free contiguous order-0 pages efficiently" (Ryan Roberts) Speed up the freeing of a batch of 0-order pages by first scanning them for coalescing opportunities. This is applicable to vfree() and to the releasing of frozen pages - "mm/damon: introduce DAMOS failed region quota charge ratio" (SeongJae Park) Address a DAMOS usability issue: The DAMOS quota often exhausts prematurely because it charges for all memory attempted, causing slow and inconsistent performance when actions fail on unreclaimable memory. To fix this, a new feature lets users set a smaller, flexible quota charge ratio (via a numerator and denominator) for failed regions. Since failed actions cause less overhead, reducing their quota cost ensures more predictable and efficient DAMOS processing - "selftests/cgroup: improve zswap tests robustness and support large page sizes" (Li Wang) Fix various spurious failures and improves the overall robustness of the cgroup zswap selftests - "fix MAP_DROPPABLE not supported errno" (Anthony Yznaga) Fix an issue in the mlock selftests on arm32 - "mm: huge_memory: clean up defrag sysfs with shared" (Breno Leitao) Some maintenance work in the huge_memory code - "treewide: fixup gfp_t printks" (Brendan Jackman) Use the special vprintf() gfp_t conversion in various places - "mm: Fix vmemmap optimization accounting and initialization" (Muchun Song) Fix several bugs in the vmemmap optimization, mainly around incorrect page accounting and memmap initialization in the DAX and memory hotplug paths. It also fixes pageblock migratetype initialization and struct page initialization for ZONE_DEVICE compound pages - "mm/damon: repost non-hotfix reviewed patches in damon/next tree" A sprinkle of unrelated minor bugfixes for DAMON - "mm: remove page_mapped()" (David Hildenbrand) Remove this function from the tree, replacing it with folio_mapped() - "mm/damon: let DAMON be paused and resumed" (SeongJae Park) Allow DAMON to be paused and resumed without losing its current state - "kasan: hw_tags: Disable tagging for stack and page-tables" (Muhammad Usama Anjum) Simplify and speed up kasan by removing its ineffective tagging of stacks and page tables - "mm/damon/reclaim,lru_sort: monitor all system rams by default" (SeongJae Park) Simplify deployment on diverse hardware like NUMA systems by updating DAMON_RECLAIM and DAMON_LRU_SORT to automatically monitor the physical address range covering all System RAM areas by default, replacing the overly restrictive behavior that only targeted the single largest memory block to save on negligible overhead - "mm/damon/sysfs: document filters/ directory as deprecated" (SeongJae Park) Update some DAMON docs - "mm: use spinlock guards for zone lock" (Dmitry Ilvokhin) Switch zone->lock handling over to using the guard() mechanisms - "mm/filemap: tighten mmap_miss hit accounting" (fujunjie) Fix a flaw where the mmap_miss counter over-credited page cache hits during fault-arounds and page-fault retries. This results in significant reduction of redundant synchronous mmap readahead I/O, drastically cutting down execution time and gigabytes read for sparse random or strided memory access workloads - "selftests/cgroup: Fix false positive failures in test_percpu_basic" (Li Wang) Fix a couple of false-positives in the cgroup kmem selftests - "mm/damon/reclaim: support monitoring intervals auto-tuning" (SeongJae Park) Add a new parameter to DAMON permitting DAMON_RECLAIM to automatically tune DAMON's sampling and aggregation intervals - "mm/damon/stat: add kdamond_pid parameter" (SeongJae Park) Change DAMON_STAT to provide the pid of its kdamond - "mm/kmemleak: dedupe verbose scan output" (Breno Leitao) Remove large amounts of duplicated backtraces from the verbose-mode kmemleak output - "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)" (David Hildenbrand) Reduce our use of CONFIG_HAVE_BOOTMEM_INFO_NODE, with a view to removing it entirely in a later series - "mm/damon: validate min_region_size to be power of 2" (Liew Rui Yan) Prevent users from passing a non-power-of-2 value of `addr_unit', as this later results in undesirable behavior - "mm: document read_pages and simplify usage" (Frederick Mayle) - "tools/mm/page-types: Fix misc bugs" (Ye Liu) Fix three issues in tools/mm/page-types.c - "mm: misc cleanups from __GFP_UNMAPPED series" (Brendan Jackman) Implement several cleanups in the page allocator and related code - "mm, swap: swap table phase IV: unify allocation" (Kairui Song) Unify the allocation and charging of anon and shmem swap in folios, provides better synchronization, consolidates the metadata management, hence dropping the static array and map, and improves performance - "mm/damon: introduce data attributes monitoring" (SeongJae Park( Extend DAMON to monitor general data attributes other than accesses - "mm/vmalloc: free unused pages on vrealloc() shrink" (Shivam Kalra) Implement the TODO in vrealloc() to unmap and free unused pages when shrinking across a page boundary - "mm/damon: documentation and comment fixes" (niecheng) - "remove mmap_action success, error hooks" (Lorenzo Stoakes) Eliminate custom hooks from mmap_action by removing the problematic success_hook which allowed drivers to improperly access uninitialized VMAs. It replaces the error_hook with a simple error-code field and updates the memory char driver accordingly - "mm/damon: minor improvements for code readability and tests" (SeongJae Park) - "mm/damon: fix macro arguments and clarify quota goals doc" (Maksym Shcherba) - "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c" (Mike Rapoport) - "mm/mglru: improve reclaim loop and dirty folio" (Kairui Song and others) Clean up and slightly improves MGLRU's reclaim loop and dirty writeback handling. Large performance improvements are measured - "use vma locks for proc/pid/{smaps|numa_maps} reads" (Suren Baghdasaryan) Use per-vma locks when reading /proc/pid/smaps and numa_maps similar to reduce contention on central mmap_lock - "refactors thpsize_shmem_enabled_store() and thpsize_shmem_enabled_show()" (Ran Xiaokai) Some cleanup work in the THP code - "selftests/memfd: fix compilation warnings" (Konstantin Khorenko) Fix a few build glitches in the memfd selftest code. - "memcg: shrink obj_stock_pcp and cache multiple objcgs" (Shakeel Butt) Resolve a 68% performance regression caused by NUMA-node cache thrashing around struct obj_stock_pcp by shrinking its existing fields and expanding it into a multi-slot array that caches up to five obj_cgroup pointers per CPU, allowing per-node variants of the same memcg to coexist within a single 64-byte cache line. - "zram: writeback fixes" (Sergey Senozhatsky) address a couple of unrelated zram writeback issues - "mm: switch THP shrinker to list_lru" (Johannes Weiner) Resolve NUMA-awareness issues and streamlines callsite interaction by refactoring and extending the list_lru API to completely replace the complex, open-coded deferred split queue for Transparent Huge Pages - "mm: improve large folio readahead for exec memory" (Usama Arif) Improve large-folio readahead on systems like 64K-page arm64 by preventing the mmap_miss check from permanently disabling target-oriented VM_EXEC readahead, and by generalizing the force_thp_readahead gate to support mappings with any usefully large maximum folio order under the cache cap. - "userfaultfd/pagemap: pre-existing fixes" (Kiryl Shutsemau) Fix a bunch of minor issues in the userfaultfd/pagemap, all of which were flagged by Sashiko review of proposed new material - "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and vmemmap_check_pmd()" (Muchun Song) Provide generic versions of these two functions so the four arch-specific implementations can be removed. - "mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device" (Youngjun Park) Address a uswsusp-vs-swapoff race and reduces the swap device reference taking/releasing frequency. - "mm/hmm: A fix and a selftest" (Dev Jain) * tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits) selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries fs/proc/task_mmu: do not warn on seeing non-migration pmd entry lib/test_hmm: check alloc_page_vma() return value and handle OOM mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX mm/swap: remove redundant swap device reference in alloc/free mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device mm/filemap: use folio_next_index() for start vmalloc: fix NULL pointer dereference in is_vm_area_hugepages() sparc/mm: drop vmemmap_check_pmd helper and use generic code loongarch/mm: drop vmemmap_check_pmd helper and use generic code riscv/mm: drop vmemmap_pmd helpers and use generic code arm64/mm: drop vmemmap_pmd helpers and use generic code mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd() rust: page: mark Page::nid as inline userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks userfaultfd: gate must_wait writability check on pte_present() mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole() fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry() fs/proc/task_mmu: fix make_uffd_wp_huge_pte() prot-update race ...

Linus Torvalds committed 4 months ago

Commit a552c812 parents c98d767 e3d8707169 files · +9215 −5618Browse files

Unified · Split

169 files
  1. Documentation/ABI/testing/sysfs-kernel-mm-damon +53 −28
  2. Documentation/admin-guide/kernel-parameters.txt +4 −0
  3. Documentation/admin-guide/mm/damon/lru_sort.rst +5 −3
  4. Documentation/admin-guide/mm/damon/reclaim.rst +16 −3
  5. Documentation/admin-guide/mm/damon/stat.rst +7 −0
  6. Documentation/admin-guide/mm/damon/usage.rst +82 −26
  7. Documentation/admin-guide/mm/transhuge.rst +2 −2
  8. Documentation/admin-guide/sysctl/vm.rst +2 −0
  9. Documentation/mm/damon/design.rst +75 −3
  10. Documentation/mm/damon/maintainer-profile.rst +21 −0
  11. Documentation/mm/process_addrs.rst +1 −1
  12. MAINTAINERS +14 −2
  13. arch/arm64/mm/mmu.c +3 −16
  14. arch/loongarch/mm/init.c +3 −13
  15. arch/powerpc/mm/init_64.c +0 −8
  16. arch/powerpc/mm/mem.c +3 −2
  17. arch/powerpc/platforms/pseries/hotplug-memory.c +4 −10
  18. arch/riscv/mm/init.c +3 −15
  19. arch/s390/mm/init.c +3 −2
  20. arch/s390/mm/vmem.c +1 −2
  21. arch/sh/mm/cache-sh4.c +1 −1
  22. arch/sparc/mm/init_64.c +0 −31
  23. arch/x86/mm/init_64.c +3 −2
  24. drivers/base/memory.c +16 −25
  25. drivers/base/node.c +4 −2
  26. drivers/block/zram/zram_drv.c +9 −4
  27. drivers/char/mem.c +6 −19
  28. drivers/dax/kmem.c +6 −0
  29. drivers/gpu/drm/drm_managed.c +2 −2
  30. drivers/s390/char/sclp_mem.c +8 −9
  31. fs/Makefile +0 −1
  32. fs/proc/task_mmu.c +223 −46
  33. fs/userfaultfd.c +0 −2231
  34. include/asm-generic/pgalloc.h +1 −1
  35. include/linux/bootmem_info.h +0 −1
  36. include/linux/compaction.h +5 −5
  37. include/linux/damon.h +111 −25
  38. include/linux/gfp.h +4 −0
  39. include/linux/gfp_types.h +3 −3
  40. include/linux/highmem-internal.h +1 −1
  41. include/linux/huge_mm.h +42 −5
  42. include/linux/list_lru.h +70 −0
  43. include/linux/memcontrol.h +17 −14
  44. include/linux/memory.h +5 −2
  45. include/linux/memory_hotplug.h +5 −3
  46. include/linux/mm.h +44 −18
  47. include/linux/mm_inline.h +1 −1
  48. include/linux/mm_types.h +3 −16
  49. include/linux/mmu_notifier.h +2 −2
  50. include/linux/mmzone.h +6 −17
  51. include/linux/nodemask.h +9 −9
  52. include/linux/page_ref.h +18 −0
  53. include/linux/pageblock-flags.h +3 −3
  54. include/linux/pagemap.h +2 −0
  55. include/linux/swap.h +12 −12
  56. include/linux/swap_cgroup.h +0 −47
  57. include/linux/thread_info.h +1 −1
  58. include/linux/userfaultfd_k.h +2 −38
  59. include/linux/vmalloc.h +3 −1
  60. include/linux/vmpressure.h +5 −4
  61. include/trace/events/damon.h +38 −0
  62. include/trace/events/vmscan.h +52 −0
  63. kernel/bpf/arena.c +1 −1
  64. kernel/fork.c +3 −2
  65. kernel/power/snapshot.c +4 −4
  66. kernel/power/swap.c +1 −1
  67. kernel/power/user.c +12 −3
  68. lib/maple_tree.c +5 −2
  69. lib/test_hmm.c +40 −9
  70. lib/test_meminit.c +1 −1
  71. lib/test_vmalloc.c +62 −0
  72. mm/Kconfig +2 −5
  73. mm/Makefile +0 −3
  74. mm/bootmem_info.c +2 −23
  75. mm/compaction.c +8 −3
  76. mm/damon/core.c +578 −77
  77. mm/damon/lru_sort.c +49 −10
  78. mm/damon/ops-common.c +6 −3
  79. mm/damon/paddr.c +80 −4
  80. mm/damon/reclaim.c +79 −19
  81. mm/damon/stat.c +42 −50
  82. mm/damon/sysfs-common.c +41 −0
  83. mm/damon/sysfs-common.h +2 −0
  84. mm/damon/sysfs-schemes.c +244 −45
  85. mm/damon/sysfs.c +588 −0
  86. mm/damon/tests/core-kunit.h +152 −28
  87. mm/damon/tests/vaddr-kunit.h +20 −7
  88. mm/damon/vaddr.c +50 −29
  89. mm/filemap.c +80 −53
  90. mm/gup.c +4 −4
  91. mm/huge_memory.c +190 −289
  92. mm/hugetlb.c +4 −3
  93. mm/hugetlb_cma.c +16 −19
  94. mm/internal.h +13 −14
  95. mm/kasan/kasan_test_c.c +10 −0
  96. mm/kfence/kfence_test.c +1 −1
  97. mm/khugepaged.c +15 −6
  98. mm/kmemleak.c +140 −8
  99. mm/list_lru.c +162 −76
  100. mm/madvise.c +25 −35
  101. mm/memcontrol-v1.c +44 −24
  102. mm/memcontrol.c +197 −97
  103. mm/memory-failure.c +18 −29
  104. mm/memory.c +79 −127
  105. mm/memory_hotplug.c +18 −16
  106. mm/mempolicy.c +1 −1
  107. mm/memremap.c +2 −2
  108. mm/migrate.c +33 −30
  109. mm/migrate_device.c +3 −6
  110. mm/mm_init.c +36 −37
  111. mm/mmap.c +1 −1
  112. mm/mseal.c +3 −2
  113. mm/page_alloc.c +277 −106
  114. mm/page_io.c +73 −30
  115. mm/page_isolation.c +28 −39
  116. mm/page_owner.c +1 −1
  117. mm/page_vma_mapped.c +7 −2
  118. mm/percpu-internal.h +3 −3
  119. mm/readahead.c +30 −6
  120. mm/rmap.c +4 −4
  121. mm/shmem.c +119 −159
  122. mm/shrinker.c +8 −10
  123. mm/sparse-vmemmap.c +56 −26
  124. mm/sparse.c +5 −63
  125. mm/swap.c +53 −4
  126. mm/swap.h +32 −58
  127. mm/swap_cgroup.c +0 −172
  128. mm/swap_state.c +308 −225
  129. mm/swap_table.h +149 −30
  130. mm/swapfile.c +302 −158
  131. mm/userfaultfd.c +2247 −12
  132. mm/util.c +22 −10
  133. mm/vma.c +3 −0
  134. mm/vmalloc.c +107 −23
  135. mm/vmpressure.c +12 −3
  136. mm/vmscan.c +168 −202
  137. mm/workingset.c +6 −4
  138. mm/zswap.c +10 −15
  139. net/rds/tcp_recv.c +1 −1
  140. rust/kernel/page.rs +1 −0
  141. samples/damon/mtier.c +6 −4
  142. tools/mm/page-types.c +4 −4
  143. tools/testing/selftests/cgroup/lib/cgroup_util.c +9 −9
  144. tools/testing/selftests/cgroup/lib/include/cgroup_util.h +2 −2
  145. tools/testing/selftests/cgroup/test_core.c +1 −1
  146. tools/testing/selftests/cgroup/test_freezer.c +1 −1
  147. tools/testing/selftests/cgroup/test_kmem.c +7 −6
  148. tools/testing/selftests/cgroup/test_memcontrol.c +9 −4
  149. tools/testing/selftests/cgroup/test_zswap.c +127 −54
  150. tools/testing/selftests/damon/_damon_sysfs.py +28 −3
  151. tools/testing/selftests/damon/drgn_dump_damon_status.py +3 −0
  152. tools/testing/selftests/damon/sysfs.py +55 −5
  153. tools/testing/selftests/damon/sysfs.sh +62 −0
  154. tools/testing/selftests/memfd/fuse_test.c +1 −1
  155. tools/testing/selftests/memfd/memfd_test.c +2 −2
  156. tools/testing/selftests/mm/Makefile +7 −3
  157. tools/testing/selftests/mm/check_config.sh +1 −1
  158. tools/testing/selftests/mm/droppable.c +8 −1
  159. tools/testing/selftests/mm/hmm-tests.c +33 −3
  160. tools/testing/selftests/mm/khugepaged.c +15 −3
  161. tools/testing/selftests/mm/ksft_kmemleak_dedup.sh +222 −0
  162. tools/testing/selftests/mm/ksm_functional_tests.c +11 −8
  163. tools/testing/selftests/mm/mlock2-tests.c +75 −9
  164. tools/testing/selftests/mm/mremap_test.c +10 −99
  165. tools/testing/selftests/mm/process_madv.c +28 −0
  166. tools/testing/selftests/mm/run_vmtests.sh +1 −0
  167. tools/testing/selftests/mm/split_huge_page_test.c +8 −4
  168. tools/testing/selftests/proc/proc-maps-race.c +231 −62
  169. tools/testing/vma/include/dup.h +4 −16
Documentation/ABI/testing/sysfs-kernel-mm-damon+53−28
@@ -84,6 +84,13 @@ Description: Writing an integer to this file sets the 'address unit' · 83 unmodified lines
8484 parameter of the given operations set of the context. Reading
8585 the file returns the last-written 'address unit' value.
8686
87+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/pause
88+Date: Mar 2026
89+Contact: SeongJae Park <sj@kernel.org>
90+Description: Writing a boolean keyword to this file sets the 'pause' request
91+ parameter for the context. Reading the file returns the
92+ last-written 'pause' value.
93+
8794 What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/sample_us
8895 Date: Mar 2022
8996 Contact: SeongJae Park <sj@kernel.org>
@@ -322,6 +329,18 @@ Contact: SeongJae Park <sj@kernel.org> · 232 unmodified lines
322329 Description: Writing to and reading from this file sets and gets the
323330 goal-based effective quota auto-tuning algorithm to use.
324331
332+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_num
333+Date: Mar 2026
334+Contact: SeongJae Park <sj@kernel.org>
335+Description: Writing to and reading from this file sets and gets the
336+ action-failed memory quota charging ratio numerator.
337+
338+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_denom
339+Date: Mar 2026
340+Contact: SeongJae Park <sj@kernel.org>
341+Description: Writing to and reading from this file sets and gets the
342+ action-failed memory quota charging ratio denominator.
343+
325344 What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil
326345 Date: Mar 2022
327346 Contact: SeongJae Park <sj@kernel.org>
@@ -377,15 +396,20 @@ Contact: SeongJae Park <sj@kernel.org> · 49 unmodified lines
377396 Description: Writing to and reading from this file sets and gets the low
378397 watermark of the scheme in permil.
379398
380−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/nr_filters
381−Date: Dec 2022
399+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
400+Date: Feb 2025
401+Contact: SeongJae Park <sj@kernel.org>
402+Description: Directory for DAMON core layer-handled DAMOS filters.
403+
404+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/nr_filters
405+Date: Feb 2025
382406 Contact: SeongJae Park <sj@kernel.org>
383407 Description: Writing a number 'N' to this file creates the number of
384408 directories for setting filters of the scheme named '0' to
385− 'N-1' under the filters/ directory.
409+ 'N-1' under the core_filters/ directory.
386410
387−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/type
388−Date: Dec 2022
411+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/type
412+Date: Feb 2025
389413 Contact: SeongJae Park <sj@kernel.org>
390414 Description: Writing to and reading from this file sets and gets the type of
391415 the memory of the interest. 'anon' for anonymous pages,
@@ -393,77 +417,78 @@ Description: Writing to and reading from this file sets and gets the type of · 1 unmodified line
393417 'addr' for address range (an open-ended interval), or 'target'
394418 for DAMON monitoring target can be written and read.
395419
396−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/memcg_path
397−Date: Dec 2022
420+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/memcg_path
421+Date: Feb 2025
398422 Contact: SeongJae Park <sj@kernel.org>
399423 Description: If 'memcg' is written to the 'type' file, writing to and
400424 reading from this file sets and gets the path to the memory
401425 cgroup of the interest.
402426
403−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_start
404−Date: Jul 2023
427+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_start
428+Date: Feb 2025
405429 Contact: SeongJae Park <sj@kernel.org>
406430 Description: If 'addr' is written to the 'type' file, writing to or reading
407431 from this file sets or gets the start address of the address
408432 range for the filter.
409433
410−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_end
411−Date: Jul 2023
434+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_end
435+Date: Feb 2025
412436 Contact: SeongJae Park <sj@kernel.org>
413437 Description: If 'addr' is written to the 'type' file, writing to or reading
414438 from this file sets or gets the end address of the address
415439 range for the filter.
416440
417−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/min
441+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/min
418442 Date: Feb 2025
419443 Contact: SeongJae Park <sj@kernel.org>
420444 Description: If 'hugepage_size' is written to the 'type' file, writing to
421445 or reading from this file sets or gets the minimum size of the
422446 hugepage for the filter.
423447
424−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/max
448+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/max
425449 Date: Feb 2025
426450 Contact: SeongJae Park <sj@kernel.org>
427451 Description: If 'hugepage_size' is written to the 'type' file, writing to
428452 or reading from this file sets or gets the maximum size of the
429453 hugepage for the filter.
430454
431−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/target_idx
432−Date: Dec 2022
455+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/damon_target_idx
456+Date: Feb 2025
433457 Contact: SeongJae Park <sj@kernel.org>
434458 Description: If 'target' is written to the 'type' file, writing to or
435459 reading from this file sets or gets the index of the DAMON
436460 monitoring target of the interest.
437461
438−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/matching
439−Date: Dec 2022
462+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/matching
463+Date: Feb 2025
440464 Contact: SeongJae Park <sj@kernel.org>
441465 Description: Writing 'Y' or 'N' to this file sets whether the filter is for
442466 the memory of the 'type', or all except the 'type'.
443467
444−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/allow
445−Date: Jan 2025
468+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/allow
469+Date: Feb 2025
446470 Contact: SeongJae Park <sj@kernel.org>
447471 Description: Writing 'Y' or 'N' to this file sets whether to allow or reject
448472 applying the scheme's action to the memory that satisfies the
449473 'type' and the 'matching' of the directory.
450474
451−What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
452−Date: Feb 2025
453−Contact: SeongJae Park <sj@kernel.org>
454−Description: Directory for DAMON core layer-handled DAMOS filters. Files
455− under this directory works same to those of
456− /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
457− directory.
458−
459475 What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/ops_filters
460476 Date: Feb 2025
461477 Contact: SeongJae Park <sj@kernel.org>
462478 Description: Directory for DAMON operations set layer-handled DAMOS filters.
463479 Files under this directory works same to those of
464− /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
480+ /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
465481 directory.
466482
483+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
484+Date: Dec 2022
485+Contact: SeongJae Park <sj@kernel.org>
486+Description: Directory for DAMOS filters. Files under this directory works
487+ same to those of
488+ /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/{core,ops}_filters
489+ directory. This is deprecated. Use the core_filters and
490+ ops_filters instead.
491+
467492 What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/nr_dests
468493 Date: Jul 2025
469494 Contact: SeongJae Park <sj@kernel.org>
Documentation/admin-guide/kernel-parameters.txt+4−0
@@ -2067,6 +2067,10 @@ Kernel parameters · 2066 unmodified lines
20672067 Format: nn[KMGTPE] or (node format)
20682068 <node>:nn[KMGTPE][,<node>:nn[KMGTPE]]
20692069
2070+ The size must be a multiple of the gigantic page size.
2071+ When using node format, this applies to each per-node size.
2072+ Missaligned values are dropped with a warning.
2073+
20702074 Reserve a CMA area of given size and allocate gigantic
20712075 hugepages using the CMA allocator. If enabled, the
20722076 boot-time allocation of gigantic hugepages is skipped.
Documentation/admin-guide/mm/damon/lru_sort.rst+5−3
@@ -75,7 +75,7 @@ Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``. · 74 unmodified lines
7575
7676 Input parameters that updated while DAMON_LRU_SORT is running are not applied
7777 by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT reads values
78−of parametrs except ``enabled`` again. Once the re-reading is done, this
78+of parameters except ``enabled`` again. Once the re-reading is done, this
7979 parameter is set as ``N``. If invalid parameters are found while the
8080 re-reading, DAMON_LRU_SORT will be disabled.
8181
@@ -246,7 +246,8 @@ monitor_region_start · 164 unmodified lines
246246 Start of target memory region in physical address.
247247
248248 The start physical address of memory region that DAMON_LRU_SORT will do work
249−against. By default, biggest System RAM is used as the region.
249+against. By default, the system's entire physical memory is used as the
250+region.
250251
251252 monitor_region_end
252253 ------------------
@@ -254,7 +255,8 @@ monitor_region_end · 1 unmodified line
254255 End of target memory region in physical address.
255256
256257 The end physical address of memory region that DAMON_LRU_SORT will do work
257−against. By default, biggest System RAM is used as the region.
258+against. By default, the system's entire physical memory is used as the
259+region.
258260
259261 addr_unit
260262 ---------
Documentation/admin-guide/mm/damon/reclaim.rst+16−3
@@ -67,7 +67,7 @@ Make DAMON_RECLAIM reads the input parameters again, except ``enabled``. · 66 unmodified lines
6767
6868 Input parameters that updated while DAMON_RECLAIM is running are not applied
6969 by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
70−of parametrs except ``enabled`` again. Once the re-reading is done, this
70+of parameters except ``enabled`` again. Once the re-reading is done, this
7171 parameter is set as ``N``. If invalid parameters are found while the
7272 re-reading, DAMON_RECLAIM will be disabled.
7373
@@ -85,6 +85,17 @@ identifies the region as cold, and reclaims it. · 11 unmodified lines
8585
8686 120 seconds by default.
8787
88+autotune_monitoring_intervals
89+-----------------------------
90+
91+If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
92+sampling and aggregation intervals. The auto-tuning aims to capture meaningful
93+amount of access events in each DAMON-snapshot, while keeping the sampling
94+interval 5 milliseconds in minimum, and 10 seconds in maximum. Setting this as
95+``N`` disables the auto-tuning.
96+
97+Disabled by default.
98+
8899 quota_ms
89100 --------
90101
@@ -229,7 +240,8 @@ Start of target memory region in physical address. · 138 unmodified lines
229240
230241 The start physical address of memory region that DAMON_RECLAIM will do work
231242 against. That is, DAMON_RECLAIM will find cold memory regions in this region
232−and reclaims. By default, biggest System RAM is used as the region.
243+and reclaims. By default, the system's entire physical memory is used as the
244+region.
233245
234246 monitor_region_end
235247 ------------------
@@ -238,7 +250,8 @@ End of target memory region in physical address. · 2 unmodified lines
238250
239251 The end physical address of memory region that DAMON_RECLAIM will do work
240252 against. That is, DAMON_RECLAIM will find cold memory regions in this region
241−and reclaims. By default, biggest System RAM is used as the region.
253+and reclaims. By default, the system's entire physical memory is used as the
254+region.
242255
243256 addr_unit
244257 ---------
Documentation/admin-guide/mm/damon/stat.rst+7−0
@@ -89,3 +89,10 @@ percentiles of the idle time values via this read-only parameter. Reading the · 88 unmodified lines
8989 parameter returns 101 idle time values in milliseconds, separated by comma.
9090 Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle
9191 times.
92+
93+kdamond_pid
94+-----------
95+
96+PID of the DAMON thread.
97+
98+If DAMON_STAT is enabled, this becomes the PID of the worker thread. Else, -1.
Documentation/admin-guide/mm/damon/usage.rst+82−26
@@ -66,11 +66,17 @@ comma (","). · 65 unmodified lines
6666 │ :ref:`kdamonds <sysfs_kdamonds>`/nr_kdamonds
6767 │ │ :ref:`0 <sysfs_kdamond>`/state,pid,refresh_ms
6868 │ │ │ :ref:`contexts <sysfs_contexts>`/nr_contexts
69− │ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit
69+ │ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit,
70+ │ │ │ │ pause
7071 │ │ │ │ │ :ref:`monitoring_attrs <sysfs_monitoring_attrs>`/
7172 │ │ │ │ │ │ intervals/sample_us,aggr_us,update_us
7273 │ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us
7374 │ │ │ │ │ │ nr_regions/min,max
75+ │ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes
76+ │ │ │ │ │ │ │ 0/filters/nr_filters
77+ │ │ │ │ │ │ │ │ 0/type,matching,allow,path
78+ │ │ │ │ │ │ │ │ ...
79+ │ │ │ │ │ │ │ ...
7480 │ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
7581 │ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target
7682 │ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions
@@ -83,18 +89,23 @@ comma (","). · 6 unmodified lines
8389 │ │ │ │ │ │ │ │ sz/min,max
8490 │ │ │ │ │ │ │ │ nr_accesses/min,max
8591 │ │ │ │ │ │ │ │ age/min,max
86− │ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,effective_bytes,goal_tuner
92+ │ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,
93+ │ │ │ │ │ │ │ effective_bytes,goal_tuner,
94+ │ │ │ │ │ │ │ fail_charge_num,fail_charge_denom
8795 │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
8896 │ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals
8997 │ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path
9098 │ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low
9199 │ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters
92− │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max
100+ │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,damon_target_idx,min,max
93101 │ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests
94102 │ │ │ │ │ │ │ │ 0/id,weight
95103 │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots
96104 │ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes
97105 │ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed
106+ │ │ │ │ │ │ │ │ │ probes
107+ │ │ │ │ │ │ │ │ │ │ 0/hits
108+ │ │ │ │ │ │ │ │ │ │ ...
98109 │ │ │ │ │ │ │ │ ...
99110 │ │ │ │ │ │ ...
100111 │ │ │ │ ...
@@ -194,9 +205,9 @@ details). At the moment, only one context per kdamond is supported, so only · 93 unmodified lines
194205 contexts/<N>/
195206 -------------
196207
197−In each context directory, three files (``avail_operations``, ``operations``
198−and ``addr_unit``) and three directories (``monitoring_attrs``, ``targets``,
199−and ``schemes``) exist.
208+In each context directory, four files (``avail_operations``, ``operations``,
209+``addr_unit`` and ``pause``) and three directories (``monitoring_attrs``,
210+``targets``, and ``schemes``) exist.
200211
201212 DAMON supports multiple types of :ref:`monitoring operations
202213 <damon_design_configurable_operations_set>`, including those for virtual address
@@ -214,6 +225,9 @@ reading from the ``operations`` file. · 11 unmodified lines
214225 ``addr_unit`` file is for setting and getting the :ref:`address unit
215226 <damon_design_addr_unit>` parameter of the operations set.
216227
228+``pause`` file is for setting and getting the :ref:`pause request
229+<damon_design_execution_model_and_data_structures>` parameter of the context.
230+
217231 .. _sysfs_monitoring_attrs:
218232
219233 contexts/<N>/monitoring_attrs/
@@ -221,8 +235,8 @@ contexts/<N>/monitoring_attrs/ · 1 unmodified line
221235
222236 Files for specifying attributes of the monitoring including required quality
223237 and efficiency of the monitoring are in ``monitoring_attrs`` directory.
224−Specifically, two directories, ``intervals`` and ``nr_regions`` exist in this
225−directory.
238+Specifically, three directories, ``intervals``, ``nr_regions`` and ``probes``
239+exist in this directory.
226240
227241 Under ``intervals`` directory, three files for DAMON's sampling interval
228242 (``sample_us``), aggregation interval (``aggr_us``), and update interval
@@ -256,6 +270,29 @@ tuning-applied current values of the two intervals can be read from the · 27 unmodified lines
256270 ``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to
257271 the ``state`` file.
258272
273+.. _damon_usage_sysfs_probes:
274+
275+contexts/<N>/monitoring_attrs/probes/
276+-------------------------------------
277+
278+A directory for registering :ref:`data attributes monitoring
279+<damon_design_data_attrs_monitoring>` probes.
280+
281+In the beginning, this directory has only one file, ``nr_probes``. Writing a
282+number (``N``) to the file creates the number of child directories named ``0``
283+to ``N-1``. Each directory represents each monitoring probe.
284+
285+In each probe directory, one directory, ``filters`` exists. The directory
286+contains files for installing filters for the probe, that is used to determine
287+the data attribute for the probe.
288+
289+In the beginning, ``filters`` directory has only one file, ``nr_filters``.
290+Writing a number (``N``) to the file creates the number of child directories
291+named ``0`` to ``N-1``. Each directory represents each filter and works in a
292+way similar to that for :ref:`DAMOS filter <sysfs_filters>`. When the filter
293+``type`` is ``memcg``, ``path`` file acts as ``memcg_path`` for :ref:`DAMOS
294+filter <sysfs_filters>`.
295+
259296 .. _sysfs_targets:
260297
261298 contexts/<N>/targets/
@@ -337,7 +374,7 @@ to ``N-1``. Each directory represents each DAMON-based operation scheme. · 75 unmodified lines
337374 schemes/<N>/
338375 ------------
339376
340−In each scheme directory, eight directories (``access_pattern``, ``quotas``,
377+In each scheme directory, nine directories (``access_pattern``, ``quotas``,
341378 ``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``,
342379 ``stats``, and ``tried_regions``) and three files (``action``, ``target_nid``
343380 and ``apply_interval``) exist.
@@ -377,9 +414,10 @@ schemes/<N>/quotas/ · 33 unmodified lines
377414 The directory for the :ref:`quotas <damon_design_damos_quotas>` of the given
378415 DAMON-based operation scheme.
379416
380−Under ``quotas`` directory, five files (``ms``, ``bytes``,
381−``reset_interval_ms``, ``effective_bytes`` and ``goal_tuner``) and two
382−directories (``weights`` and ``goals``) exist.
417+Under ``quotas`` directory, seven files (``ms``, ``bytes``,
418+``reset_interval_ms``, ``effective_bytes``, ``goal_tuner``, ``fail_charge_num``
419+and ``fail_charge_denom``) and two directories (``weights`` and ``goals``)
420+exist.
383421
384422 You can set the ``time quota`` in milliseconds, ``size quota`` in bytes, and
385423 ``reset interval`` in milliseconds by writing the values to the three files,
@@ -398,6 +436,13 @@ the background design of the feature and the name of the selectable algorithms. · 12 unmodified lines
398436 Refer to :ref:`goals directory <sysfs_schemes_quota_goals>` for the goals
399437 setup.
400438
439+You can set the action-failed memory quota charging ratio by writing the
440+numerator and the denominator for the ratio to ``fail_charge_num`` and
441+``fail_charge_denom`` files, respectively. Reading those files will return the
442+current set values. Refer to :ref:`design
443+<damon_design_damos_quotas_failed_memory_charging_ratio>` for more details of
444+the ratio feature.
445+
401446 The time quota is internally transformed to a size quota. Between the
402447 transformed size quota and user-specified size quota, smaller one is applied.
403448 Based on the user-specified :ref:`goal <sysfs_schemes_quota_goals>`, the
@@ -429,10 +474,12 @@ to ``N-1``. Each directory represents each goal and current achievement. · 25 unmodified lines
429474 Among the multiple feedback, the best one is used.
430475
431476 Each goal directory contains five files, namely ``target_metric``,
432−``target_value``, ``current_value`` ``nid`` and ``path``. Users can set and
477+``target_value``, ``current_value``, ``nid``, and ``path``. Users can set and
433478 get the five parameters for the quota auto-tuning goals that specified on the
434479 :ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and
435−reading from each of the files. Note that users should further write
480+reading from each of the files. Because the kernel does not update
481+``current_value``, reading it only makes sense when ``target_metric`` is
482+``user_input``. Note that users should further write
436483 ``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond
437484 directory <sysfs_kdamond>` to pass the feedback to DAMON.
438485
@@ -447,7 +494,7 @@ given DAMON-based operation scheme. · 8 unmodified lines
447494 Under the watermarks directory, five files (``metric``, ``interval_us``,
448495 ``high``, ``mid``, and ``low``) for setting the metric, the time interval
449496 between check of the metric, and the three watermarks exist. You can set and
450−get the five values by writing to the files, respectively.
497+get the five values by writing to and reading from the files, respectively.
451498
452499 Keywords and meanings of those that can be written to the ``metric`` file are
453500 as below.
@@ -455,7 +502,7 @@ as below. · 1 unmodified line
455502 - none: Ignore the watermarks
456503 - free_mem_rate: System's free memory rate (per thousand)
457504
458−The ``interval`` should written in microseconds unit.
505+The ``interval_us`` should be written in microseconds unit.
459506
460507 .. _sysfs_filters:
461508
@@ -471,10 +518,10 @@ directory can be used for installing filters regardless of their handled · 9 unmodified lines
471518 layers. Filters that requested by ``core_filters`` and ``ops_filters`` will be
472519 installed before those of ``filters``. All three directories have same files.
473520
474−Use of ``filters`` directory can make expecting evaluation orders of given
475−filters with the files under directory bit confusing. Users are hence
476−recommended to use ``core_filters`` and ``ops_filters`` directories. The
477−``filters`` directory could be deprecated in future.
521+Use of ``filters`` directory can make filters evaluation orders confusing to
522+expect. For this reason, ``filters`` directory is deprecated. It is still
523+functioning, but is scheduled for removal in the near future. Users should use
524+``core_filters`` and ``ops_filters`` directories instead.
478525
479526 In the beginning, the directory has only one file, ``nr_filters``. Writing a
480527 number (``N``) to the file creates the number of child directories named ``0``
@@ -483,9 +530,9 @@ in the numeric order. · 2 unmodified lines
483530
484531 Each filter directory contains nine files, namely ``type``, ``matching``,
485532 ``allow``, ``memcg_path``, ``addr_start``, ``addr_end``, ``min``, ``max``
486−and ``target_idx``. To ``type`` file, you can write the type of the filter.
487−Refer to :ref:`the design doc <damon_design_damos_filters>` for available type
488−names, their meaning and on what layer those are handled.
533+and ``damon_target_idx``. To ``type`` file, you can write the type of the
534+filter. Refer to :ref:`the design doc <damon_design_damos_filters>` for
535+available type names, their meaning and on what layer those are handled.
489536
490537 For ``memcg`` type, you can specify the memory cgroup of the interest by
491538 writing the path of the memory cgroup from the cgroups mount point to
@@ -495,7 +542,7 @@ files, respectively. For ``hugepage_size`` type, you can specify the minimum · 3 unmodified lines
495542 and maximum size of the range (closed interval) to ``min`` and ``max`` files,
496543 respectively. For ``target`` type, you can specify the index of the target
497544 between the list of the DAMON context's monitoring targets list to
498−``target_idx`` file.
545+``damon_target_idx`` file.
499546
500547 You can write ``Y`` or ``N`` to ``matching`` file to specify whether the filter
501548 is for memory that matches the ``type``. You can write ``Y`` or ``N`` to
@@ -601,10 +648,19 @@ tried_regions/<N>/ · 99 unmodified lines
601648 ------------------
602649
603650 In each region directory, you will find five files (``start``, ``end``,
604−``nr_accesses``, ``age``, and ``sz_filter_passed``). Reading the files will
651+``nr_accesses``, ``age`` and ``sz_filter_passed``). Reading the files will
605652 show the properties of the region that corresponding DAMON-based operation
606653 scheme ``action`` has tried to be applied.
607654
655+tried_regions/<N>/probes/
656+-------------------------
657+
658+In each region directory, one directory (``probes``) also exists. In the
659+directory, subdirectories named ``0`` to ``N-1`` exists. ``N`` is the number
660+of installed probes. In each number-named directory, a file (``hits``) exist.
661+Reading the file shows the number of data attributes monitoring probe-hit
662+positive samples of the region.
663+
608664 Example
609665 ~~~~~~~
610666
@@ -677,7 +733,7 @@ show results using tracepoint supporting tools like ``perf``. For example:: · 66 unmodified lines
677733
678734 Each line of the perf script output represents each monitoring region. The
679735 first five fields are as usual other tracepoint outputs. The sixth field
680−(``target_id=X``) shows the ide of the monitoring target of the region. The
736+(``target_id=X``) shows the id of the monitoring target of the region. The
681737 seventh field (``nr_regions=X``) shows the total number of monitoring regions
682738 for the target. The eighth field (``X-Y:``) shows the start (``X``) and end
683739 (``Y``) addresses of the region in bytes. The ninth field (``X``) shows the
Documentation/admin-guide/mm/transhuge.rst+2−2
@@ -57,7 +57,7 @@ prominent because the size of each page isn't as huge as the PMD-sized · 56 unmodified lines
5757 variant and there is less memory to clear in each page fault. Some
5858 architectures also employ TLB compression mechanisms to squeeze more
5959 entries in when a set of PTEs are virtually and physically contiguous
60−and approporiately aligned. In this case, TLB misses will occur less
60+and appropriately aligned. In this case, TLB misses will occur less
6161 often.
6262
6363 THP can be enabled system wide or restricted to certain tasks or even
@@ -210,7 +210,7 @@ PMD-mappable transparent hugepage:: · 146 unmodified lines
210210 cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
211211
212212 All THPs at fault and collapse time will be added to _deferred_list,
213−and will therefore be split under memory presure if they are considered
213+and will therefore be split under memory pressure if they are considered
214214 "underused". A THP is underused if the number of zero-filled pages in
215215 the THP is above max_ptes_none (see below). It is possible to disable
216216 this behaviour by writing 0 to shrink_underused, and enable it by writing
Documentation/admin-guide/sysctl/vm.rst+2−0
@@ -1034,6 +1034,8 @@ min(3% of current process size, user_reserve_kbytes) of free memory. · 1033 unmodified lines
10341034 This is intended to prevent a user from starting a single memory hogging
10351035 process, such that they cannot recover (kill the hog).
10361036
1037+This setting has no effect when overcommit_memory is set to 0 or 1.
1038+
10371039 user_reserve_kbytes defaults to min(3% of the current process size, 128MB).
10381040
10391041 If this is reduced to zero, then the user will be allowed to allocate
Documentation/mm/damon/design.rst+75−3
@@ -19,6 +19,13 @@ types of monitoring. · 18 unmodified lines
1919 To know how user-space can do the configurations and start/stop DAMON, refer to
2020 :ref:`DAMON sysfs interface <sysfs_interface>` documentation.
2121
22+Users can also request each context execution to be paused and resumed. When
23+it is paused, the kdamond does nothing other than applying online parameter
24+update.
25+
26+To know how user-space can pause/resume each context, refer to :ref:`DAMON
27+sysfs context <sysfs_context>` usage documentation.
28+
2229
2330 Overall Architecture
2431 ====================
@@ -140,7 +147,7 @@ as Idle page tracking does. · 115 unmodified lines
140147 Address Unit
141148 ------------
142149
143−DAMON core layer uses ``unsinged long`` type for monitoring target address
150+DAMON core layer uses ``unsigned long`` type for monitoring target address
144151 ranges. In some cases, the address space for a given operations set could be
145152 too large to be handled with the type. ARM (32-bit) with large physical
146153 address extension is an example. For such cases, a per-operations set
@@ -269,6 +276,45 @@ interval``, DAMON checks if the region's size and access frequency · 122 unmodified lines
269276 (``nr_accesses``) has significantly changed. If so, the counter is reset to
270277 zero. Otherwise, the counter is increased.
271278
279+.. _damon_design_data_attrs_monitoring:
280+
281+Data Attributes Monitoring
282+~~~~~~~~~~~~~~~~~~~~~~~~~~
283+
284+Data access pattern is only one type of data attributes. In some use cases,
285+users need to know more data attributes information. For example, users may
286+need to know how much of a given hot or cold memory region is backed by
287+anonymous pages, or belong to a specific cgroup. For such use case, data
288+attributes monitoring feature is provided.
289+
290+Using the feature, users can register data attributes of their interest to the
291+DAMON :ref:`context <damon_design_execution_model_and_data_structures>`. The
292+registration is made by specifying a probe per attribute. Each of the probe
293+specifies a rule to determine if a given memory region has the related
294+attribute. The rule is constructed with multiple filters. The filters work
295+same to :ref:`DAMOS filters <damon_design_damos_filters>` except the supported
296+filter types. Currently only ``anon`` and ``memcg`` filter types are supported
297+for data attributes monitoring.
298+
299+If such probes are registered, DAMON executes the probes for each region's
300+sampling memory when it does the access :ref:`sampling
301+<damon_design_region_based_sampling>`. The number of samples that identified
302+as having the data attribute (hitting the probe) per :ref:`aggregation interval
303+<damon_design_monitoring>` is accounted in a per-region per-probe counter.
304+Users can therefore know how much of a given DAMON region has a specific data
305+attribute by reading the per-region per-probe probe hits counter after each
306+aggregation interval.
307+
308+This is a sampling based mechanism. Hence, it is lightweight but the output
309+may include some measurement errors. The output should be used with good
310+understanding of statistics.
311+
312+Another way to do this for higher accuracy is using :ref:`DAMOS filter
313+<damon_design_damos_filters>` with ``stat`` :ref:`action
314+<damon_design_damos_action>` and ``sz_ops_filter_passed`` :ref:`stat
315+<damon_design_damos_stat>`. This approach provides the data attributes
316+information in page level. But, because it is operated in page level, the
317+overhead is proportional to the size of the memory.
272318
273319 Dynamic Target Space Updates Handling
274320 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -371,7 +417,7 @@ with theoretical maximum ``nr_accesses``, which can be calculated as · 96 unmodified lines
371417 ``aggregation interval / sampling interval``.
372418
373419 The mechanism calculates the ratio of access events for ``aggrs`` aggregations,
374−and increases or decrease the ``sampleing interval`` and ``aggregation
420+and increases or decrease the ``sampling interval`` and ``aggregation
375421 interval`` in same ratio, if the observed access ratio is lower or higher than
376422 the target, respectively. The ratio of the intervals change is decided in
377423 proportion to the distance between current samples ratio and the target ratio.
@@ -387,7 +433,7 @@ The tuning is turned off by default, and need to be set explicitly by the user. · 9 unmodified lines
387433 As a rule of thumbs and the Parreto principle, 4% access samples ratio target
388434 is recommended. Note that Parreto principle (80/20 rule) has applied twice.
389435 That is, assumes 4% (20% of 20%) DAMON-observed access events ratio (source)
390−to capture 64% (80% multipled by 80%) real access events (outcomes).
436+to capture 64% (80% multiplied by 80%) real access events (outcomes).
391437
392438 To know how user-space can use this feature via :ref:`DAMON sysfs interface
393439 <sysfs_interface>`, refer to :ref:`intervals_goal
@@ -474,6 +520,10 @@ that supports each action are as below. · 80 unmodified lines
474520 Supported by ``vaddr`` and ``fvaddr`` operations set. When
475521 TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
476522 fail.
523+ - ``collapse``: Call ``madvise()`` for the region with ``MADV_COLLAPSE``.
524+ Supported by ``vaddr`` and ``fvaddr`` operations set. When
525+ TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
526+ fail.
477527 - ``lru_prio``: Prioritize the region on its LRU lists.
478528 Supported by ``paddr`` operations set.
479529 - ``lru_deprio``: Deprioritize the region on its LRU lists.
@@ -565,6 +615,28 @@ interface <sysfs_interface>`, refer to :ref:`weights <sysfs_quotas>` part of · 85 unmodified lines
565615 the documentation.
566616
567617
618+.. _damon_design_damos_quotas_failed_memory_charging_ratio:
619+
620+Action-failed Memory Charging Ratio
621+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
622+
623+DAMOS action to a given region can fail for some subsets of the memory of the
624+region. For example, if the action is ``pageout`` and the region has some
625+unreclaimable pages, applying the action to the pages will fail. The amount of
626+system resource that is taken for such failed action applications is usually
627+different from that for successful action applications. For such cases, users
628+can set different charging ratio for such failed memory. The ratio can be
629+specified using ``fail_charge_num`` and ``fail_charge_denom`` parameters. The
630+two parameters represent the numerator and denominator of the ratio. The
631+feature is enabled only if ``fail_charge_denom`` is not zero.
632+
633+For example, let's suppose a DAMOS action is applied to a region of 1,000 MiB
634+size. The action is successfully applied to only 700 MiB of the region.
635+``fail_charge_num`` and ``fail_charge_denom`` are set to ``1`` and ``1024``,
636+respectively. Then only 700 MiB and 300 KiB of size (``700 MiB + 300 MiB * 1 /
637+1024``) will be charged.
638+
639+
568640 .. _damon_design_damos_quotas_auto_tuning:
569641
570642 Aim-oriented Feedback-driven Auto-tuning
Documentation/mm/damon/maintainer-profile.rst+21−0
@@ -100,3 +100,24 @@ There is also a public Google `calendar · 99 unmodified lines
100100 <https://calendar.google.com/calendar/u/0?cid=ZDIwOTA4YTMxNjc2MDQ3NTIyMmUzYTM5ZmQyM2U4NDA0ZGIwZjBiYmJlZGQxNDM0MmY4ZTRjOTE0NjdhZDRiY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t>`_
101101 that has the events. Anyone can subscribe to it. DAMON maintainer will also
102102 provide periodic reminders to the mailing list (damon@lists.linux.dev).
103+
104+AI Review
105+---------
106+
107+For patches that are publicly posted to DAMON mailing list
108+(damon@lists.linux.dev), AI reviews of the patches will be available at
109+sashiko.dev. The reviews could also be sent as mails to the author of the
110+patch.
111+
112+Patch authors are encouraged to check the AI reviews and share their opinions.
113+The sharing could be done as a reply to the mail thread. Consider reducing the
114+recipients list for such sharing, since some people are not really interested
115+in AI reviews. As a rule of thumb, drop stable@vger.kernel.org and individuals
116+except DAMON maintainer.
117+
118+`hkml` also provides a `feature
119+<https://github.com/sjp38/hackermail/blob/master/USAGE.md#forwarding-sashikodev-statuscomments-to-mailing-list>`_
120+for such sharing. Please feel free to use the feature.
121+
122+It is only an optional recommendation. DAMON maintainer could also ask any
123+question about the AI reviews, though.
Documentation/mm/process_addrs.rst+1−1
@@ -775,7 +775,7 @@ lock, releasing or downgrading the mmap write lock also releases the VMA write · 774 unmodified lines
775775 lock so there is no :c:func:`!vma_end_write` function.
776776
777777 Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily
778−modified so that readers can detect the presense of a writer. The reference counter is
778+modified so that readers can detect the presence of a writer. The reference counter is
779779 restored once the vma sequence number used for serialisation is updated.
780780
781781 This ensures the semantics we require - VMA write locks provide exclusive write
MAINTAINERS+14−2
@@ -6626,7 +6626,6 @@ F: mm/memcontrol.c · 6625 unmodified lines
66266626 F: mm/memcontrol-v1.c
66276627 F: mm/memcontrol-v1.h
66286628 F: mm/page_counter.c
6629−F: mm/swap_cgroup.c
66306629 F: samples/cgroup/*
66316630 F: tools/testing/selftests/cgroup/memcg_protection.m
66326631 F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
@@ -16920,6 +16919,7 @@ L: linux-mm@kvack.org · 10287 unmodified lines
1692016919 S: Maintained
1692116920 W: http://www.linux-mm.org
1692216921 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
16922+F: Documentation/admin-guide/sysctl/vm.rst
1692316923 F: include/linux/folio_batch.h
1692416924 F: include/linux/gfp.h
1692516925 F: include/linux/gfp_types.h
@@ -16992,6 +16992,7 @@ L: linux-mm@kvack.org · 66 unmodified lines
1699216992 S: Maintained
1699316993 W: http://www.linux-mm.org
1699416994 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
16995+F: Documentation/ABI/testing/sysfs-kernel-mm-ksm
1699516996 F: Documentation/admin-guide/mm/ksm.rst
1699616997 F: Documentation/mm/ksm.rst
1699716998 F: include/linux/ksm.h
@@ -17014,6 +17015,8 @@ L: linux-mm@kvack.org · 16 unmodified lines
1701417015 S: Maintained
1701517016 W: http://www.linux-mm.org
1701617017 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17018+F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
17019+F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
1701717020 F: include/linux/mempolicy.h
1701817021 F: include/uapi/linux/mempolicy.h
1701917022 F: include/linux/migrate.h
@@ -17056,6 +17059,10 @@ L: linux-mm@kvack.org · 36 unmodified lines
1705617059 S: Maintained
1705717060 W: http://www.linux-mm.org
1705817061 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17062+F: Documentation/ABI/testing/sysfs-kernel-mm
17063+F: Documentation/ABI/testing/sysfs-kernel-mm-cma
17064+F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
17065+F: Documentation/ABI/testing/sysfs-kernel-mm-numa
1705917066 F: Documentation/admin-guide/mm/
1706017067 F: Documentation/mm/
1706117068 F: include/linux/cma.h
@@ -17179,6 +17186,7 @@ R: Barry Song <baohua@kernel.org> · 117 unmodified lines
1717917186 R: Youngjun Park <youngjun.park@lge.com>
1718017187 L: linux-mm@kvack.org
1718117188 S: Maintained
17189+F: Documentation/ABI/testing/sysfs-kernel-mm-swap
1718217190 F: Documentation/mm/swap-table.rst
1718317191 F: include/linux/swap.h
1718417192 F: include/linux/swapfile.h
@@ -17206,6 +17214,7 @@ L: linux-mm@kvack.org · 21 unmodified lines
1720617214 S: Maintained
1720717215 W: http://www.linux-mm.org
1720817216 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17217+F: Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage
1720917218 F: Documentation/admin-guide/mm/transhuge.rst
1721017219 F: include/linux/huge_mm.h
1721117220 F: include/linux/khugepaged.h
@@ -17224,7 +17233,6 @@ R: Peter Xu <peterx@redhat.com> · 12 unmodified lines
1722417233 L: linux-mm@kvack.org
1722517234 S: Maintained
1722617235 F: Documentation/admin-guide/mm/userfaultfd.rst
17227−F: fs/userfaultfd.c
1722817236 F: include/asm-generic/pgtable_uffd.h
1722917237 F: include/linux/userfaultfd_k.h
1723017238 F: include/uapi/linux/userfaultfd.h
@@ -20352,6 +20360,10 @@ T: git git://git.infradead.org/users/willy/pagecache.git · 3121 unmodified lines
2035220360 F: Documentation/filesystems/locking.rst
2035320361 F: Documentation/filesystems/vfs.rst
2035420362 F: include/linux/pagemap.h
20363+F: include/linux/writeback.h
20364+F: include/trace/events/filemap.h
20365+F: include/trace/events/readahead.h
20366+F: include/trace/events/writeback.h
2035520367 F: mm/filemap.c
2035620368 F: mm/page-writeback.c
2035720369 F: mm/readahead.c
arch/arm64/mm/mmu.c+3−16
@@ -1781,20 +1781,6 @@ static void free_empty_tables(unsigned long addr, unsigned long end, · 1780 unmodified lines
17811781 }
17821782 #endif
17831783
1784−void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node,
1785− unsigned long addr, unsigned long next)
1786−{
1787− pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL));
1788−}
1789−
1790−int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
1791− unsigned long addr, unsigned long next)
1792−{
1793− vmemmap_verify((pte_t *)pmdp, node, addr, next);
1794−
1795− return pmd_leaf(READ_ONCE(*pmdp));
1796−}
1797−
17981784 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
17991785 struct vmem_altmap *altmap)
18001786 {
@@ -2030,12 +2016,13 @@ err: · 229 unmodified lines
20302016 return ret;
20312017 }
20322018
2033−void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
2019+void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
2020+ struct dev_pagemap *pgmap)
20342021 {
20352022 unsigned long start_pfn = start >> PAGE_SHIFT;
20362023 unsigned long nr_pages = size >> PAGE_SHIFT;
20372024
2038− __remove_pages(start_pfn, nr_pages, altmap);
2025+ __remove_pages(start_pfn, nr_pages, altmap, pgmap);
20392026 __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size);
20402027 }
20412028
arch/loongarch/mm/init.c+3−13
@@ -119,12 +119,13 @@ int arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) · 118 unmodified lines
119119 return ret;
120120 }
121121
122−void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
122+void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
123+ struct dev_pagemap *pgmap)
123124 {
124125 unsigned long start_pfn = start >> PAGE_SHIFT;
125126 unsigned long nr_pages = size >> PAGE_SHIFT;
126127
127− __remove_pages(start_pfn, nr_pages, altmap);
128+ __remove_pages(start_pfn, nr_pages, altmap, pgmap);
128129 }
129130 #endif
130131
@@ -139,17 +140,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, · 8 unmodified lines
139140 set_pmd_at(&init_mm, addr, pmd, entry);
140141 }
141142
142−int __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
143− unsigned long addr, unsigned long next)
144−{
145− int huge = pmd_val(pmdp_get(pmd)) & _PAGE_HUGE;
146−
147− if (huge)
148− vmemmap_verify((pte_t *)pmd, node, addr, next);
149−
150− return huge;
151−}
152−
153143 int __meminit vmemmap_populate(unsigned long start, unsigned long end,
154144 int node, struct vmem_altmap *altmap)
155145 {
arch/powerpc/mm/init_64.c+0−8
@@ -41,7 +41,6 @@ · 40 unmodified lines
4141 #include <linux/libfdt.h>
4242 #include <linux/memremap.h>
4343 #include <linux/memory.h>
44−#include <linux/bootmem_info.h>
4544
4645 #include <asm/pgalloc.h>
4746 #include <asm/page.h>
@@ -388,13 +387,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, · 340 unmodified lines
388387
389388 #endif
390389
391−#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE
392−void register_page_bootmem_memmap(unsigned long section_nr,
393− struct page *start_page, unsigned long size)
394−{
395−}
396−#endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
397−
398390 #endif /* CONFIG_SPARSEMEM_VMEMMAP */
399391
400392 #ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/mm/mem.c+3−2
@@ -158,12 +158,13 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, · 157 unmodified lines
158158 return rc;
159159 }
160160
161−void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
161+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
162+ struct dev_pagemap *pgmap)
162163 {
163164 unsigned long start_pfn = start >> PAGE_SHIFT;
164165 unsigned long nr_pages = size >> PAGE_SHIFT;
165166
166− __remove_pages(start_pfn, nr_pages, altmap);
167+ __remove_pages(start_pfn, nr_pages, altmap, pgmap);
167168 arch_remove_linear_mapping(start, size);
168169 }
169170 #endif
arch/powerpc/platforms/pseries/hotplug-memory.c+4−10
@@ -164,13 +164,7 @@ static int update_lmb_associativity_index(struct drmem_lmb *lmb) · 163 unmodified lines
164164
165165 static struct memory_block *lmb_to_memblock(struct drmem_lmb *lmb)
166166 {
167− unsigned long section_nr;
168− struct memory_block *mem_block;
169−
170− section_nr = pfn_to_section_nr(PFN_DOWN(lmb->base_addr));
171−
172− mem_block = find_memory_block(section_nr);
173− return mem_block;
167+ return memory_block_get(phys_to_block_id(lmb->base_addr));
174168 }
175169
176170 static int get_lmb_range(u32 drc_index, int n_lmbs,
@@ -220,7 +214,7 @@ static int dlpar_change_lmb_state(struct drmem_lmb *lmb, bool online) · 43 unmodified lines
220214 else
221215 rc = 0;
222216
223− put_device(&mem_block->dev);
217+ memory_block_put(mem_block);
224218
225219 return rc;
226220 }
@@ -319,12 +313,12 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb) · 92 unmodified lines
319313
320314 rc = dlpar_offline_lmb(lmb);
321315 if (rc) {
322− put_device(&mem_block->dev);
316+ memory_block_put(mem_block);
323317 return rc;
324318 }
325319
326320 __remove_memory(lmb->base_addr, memory_block_size);
327− put_device(&mem_block->dev);
321+ memory_block_put(mem_block);
328322
329323 /* Update memory regions for memory remove */
330324 memblock_remove(lmb->base_addr, memory_block_size);
arch/riscv/mm/init.c+3−15
@@ -1359,19 +1359,6 @@ void __init misc_mem_init(void) · 1358 unmodified lines
13591359 }
13601360
13611361 #ifdef CONFIG_SPARSEMEM_VMEMMAP
1362−void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
1363− unsigned long addr, unsigned long next)
1364−{
1365− pmd_set_huge(pmd, virt_to_phys(p), PAGE_KERNEL);
1366−}
1367−
1368−int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
1369− unsigned long addr, unsigned long next)
1370−{
1371− vmemmap_verify((pte_t *)pmdp, node, addr, next);
1372− return 1;
1373−}
1374−
13751362 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
13761363 struct vmem_altmap *altmap)
13771364 {
@@ -1742,9 +1729,10 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *param · 364 unmodified lines
17421729 return ret;
17431730 }
17441731
1745−void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
1732+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
1733+ struct dev_pagemap *pgmap)
17461734 {
1747− __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
1735+ __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, pgmap);
17481736 remove_linear_mapping(start, size);
17491737 flush_tlb_all();
17501738 }
arch/s390/mm/init.c+3−2
@@ -278,12 +278,13 @@ int arch_add_memory(int nid, u64 start, u64 size, · 277 unmodified lines
278278 return rc;
279279 }
280280
281−void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
281+void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
282+ struct dev_pagemap *pgmap)
282283 {
283284 unsigned long start_pfn = start >> PAGE_SHIFT;
284285 unsigned long nr_pages = size >> PAGE_SHIFT;
285286
286− __remove_pages(start_pfn, nr_pages, altmap);
287+ __remove_pages(start_pfn, nr_pages, altmap, pgmap);
287288 vmem_remove_mapping(start, size);
288289 }
289290 #endif /* CONFIG_MEMORY_HOTPLUG */
arch/s390/mm/vmem.c+1−2
@@ -4,7 +4,6 @@ · 3 unmodified lines
44 */
55
66 #include <linux/memory_hotplug.h>
7−#include <linux/bootmem_info.h>
87 #include <linux/cpufeature.h>
98 #include <linux/memblock.h>
109 #include <linux/pfn.h>
@@ -51,7 +50,7 @@ static void vmem_free_pages(unsigned long addr, int order, struct vmem_altmap *a · 40 unmodified lines
5150 if (PageReserved(page)) {
5251 /* allocated from memblock */
5352 while (nr_pages--)
54− free_bootmem_page(page++);
53+ free_reserved_page(page++);
5554 } else {
5655 free_pages(addr, order);
5756 }
arch/sh/mm/cache-sh4.c+1−1
@@ -248,7 +248,7 @@ static void sh4_flush_cache_page(void *args) · 247 unmodified lines
248248 */
249249 map_coherent = (current_cpu_data.dcache.n_aliases &&
250250 test_bit(PG_dcache_clean, folio_flags(folio, 0)) &&
251− page_mapped(page));
251+ folio_mapped(folio));
252252 if (map_coherent)
253253 vaddr = kmap_coherent(page, address);
254254 else
arch/sparc/mm/init_64.c+0−31
@@ -27,7 +27,6 @@ · 26 unmodified lines
2727 #include <linux/percpu.h>
2828 #include <linux/mmzone.h>
2929 #include <linux/gfp.h>
30−#include <linux/bootmem_info.h>
3130
3231 #include <asm/head.h>
3332 #include <asm/page.h>
@@ -2477,17 +2476,6 @@ int page_in_phys_avail(unsigned long paddr) · 2443 unmodified lines
24772476 return 0;
24782477 }
24792478
2480−static void __init register_page_bootmem_info(void)
2481−{
2482−#ifdef CONFIG_NUMA
2483− int i;
2484−
2485− for_each_online_node(i)
2486− if (NODE_DATA(i)->node_spanned_pages)
2487− register_page_bootmem_info_node(NODE_DATA(i));
2488−#endif
2489−}
2490−
24912479 void __init arch_setup_zero_pages(void)
24922480 {
24932481 phys_addr_t zero_page_pa = kern_base +
@@ -2498,14 +2486,6 @@ void __init arch_setup_zero_pages(void) · 4 unmodified lines
24982486
24992487 void __init mem_init(void)
25002488 {
2501− /*
2502− * Must be done after boot memory is put on freelist, because here we
2503− * might set fields in deferred struct pages that have not yet been
2504− * initialized, and memblock_free_all() initializes all the reserved
2505− * deferred pages for us.
2506− */
2507− register_page_bootmem_info();
2508−
25092489 if (tlb_type == cheetah || tlb_type == cheetah_plus)
25102490 cheetah_ecache_flush_init();
25112491 }
@@ -2579,17 +2559,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, · 67 unmodified lines
25792559 pmd_val(*pmd) = pte_base | __pa(p);
25802560 }
25812561
2582−int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
2583− unsigned long addr, unsigned long next)
2584−{
2585− int large = pmd_leaf(*pmdp);
2586−
2587− if (large)
2588− vmemmap_verify((pte_t *)pmdp, node, addr, next);
2589−
2590− return large;
2591−}
2592−
25932562 int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
25942563 int node, struct vmem_altmap *altmap)
25952564 {
arch/x86/mm/init_64.c+3−2
@@ -1300,12 +1300,13 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end) · 1299 unmodified lines
13001300 remove_pagetable(start, end, true, NULL);
13011301 }
13021302
1303−void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
1303+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
1304+ struct dev_pagemap *pgmap)
13041305 {
13051306 unsigned long start_pfn = start >> PAGE_SHIFT;
13061307 unsigned long nr_pages = size >> PAGE_SHIFT;
13071308
1308− __remove_pages(start_pfn, nr_pages, altmap);
1309+ __remove_pages(start_pfn, nr_pages, altmap, pgmap);
13091310 kernel_physical_mapping_remove(start, start + size);
13101311 }
13111312 #endif /* CONFIG_MEMORY_HOTPLUG */
drivers/base/memory.c+16−25
@@ -649,7 +649,7 @@ int __weak arch_get_memory_phys_device(unsigned long start_pfn) · 648 unmodified lines
649649 *
650650 * Called under device_hotplug_lock.
651651 */
652−struct memory_block *find_memory_block_by_id(unsigned long block_id)
652+struct memory_block *memory_block_get(unsigned long block_id)
653653 {
654654 struct memory_block *mem;
655655
@@ -659,16 +659,6 @@ struct memory_block *find_memory_block_by_id(unsigned long block_id) · 3 unmodified lines
659659 return mem;
660660 }
661661
662−/*
663− * Called under device_hotplug_lock.
664− */
665−struct memory_block *find_memory_block(unsigned long section_nr)
666−{
667− unsigned long block_id = memory_block_id(section_nr);
668−
669− return find_memory_block_by_id(block_id);
670−}
671−
672662 static struct attribute *memory_memblk_attrs[] = {
673663 &dev_attr_phys_index.attr,
674664 &dev_attr_state.attr,
@@ -701,7 +691,7 @@ static int __add_memory_block(struct memory_block *memory) · 26 unmodified lines
701691
702692 ret = device_register(&memory->dev);
703693 if (ret) {
704− put_device(&memory->dev);
694+ memory_block_put(memory);
705695 return ret;
706696 }
707697 ret = xa_err(xa_store(&memory_blocks, memory->dev.id, memory,
@@ -795,9 +785,9 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 87 unmodified lines
795785 struct memory_block *mem;
796786 int ret = 0;
797787
798− mem = find_memory_block_by_id(block_id);
788+ mem = memory_block_get(block_id);
799789 if (mem) {
800− put_device(&mem->dev);
790+ memory_block_put(mem);
801791 return -EEXIST;
802792 }
803793 mem = kzalloc_obj(*mem);
@@ -807,7 +797,6 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 3 unmodified lines
807797 mem->start_section_nr = block_id * sections_per_block;
808798 mem->state = state;
809799 mem->nid = nid;
810− mem->altmap = altmap;
811800 INIT_LIST_HEAD(&mem->group_next);
812801
813802 #ifndef CONFIG_NUMA
@@ -825,6 +814,8 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state · 11 unmodified lines
825814 if (ret)
826815 return ret;
827816
817+ mem->altmap = altmap;
818+
828819 if (group) {
829820 mem->group = group;
830821 list_add(&mem->group_next, &group->memory_blocks);
@@ -845,8 +836,8 @@ static void remove_memory_block(struct memory_block *memory) · 14 unmodified lines
845836 memory->group = NULL;
846837 }
847838
848− /* drop the ref. we got via find_memory_block() */
849− put_device(&memory->dev);
839+ /* drop the ref. we got via memory_block_get() */
840+ memory_block_put(memory);
850841 device_unregister(&memory->dev);
851842 }
852843
@@ -880,7 +871,7 @@ int create_memory_block_devices(unsigned long start, unsigned long size, · 27 unmodified lines
880871 end_block_id = block_id;
881872 for (block_id = start_block_id; block_id != end_block_id;
882873 block_id++) {
883− mem = find_memory_block_by_id(block_id);
874+ mem = memory_block_get(block_id);
884875 if (WARN_ON_ONCE(!mem))
885876 continue;
886877 remove_memory_block(mem);
@@ -908,7 +899,7 @@ void remove_memory_block_devices(unsigned long start, unsigned long size) · 21 unmodified lines
908899 return;
909900
910901 for (block_id = start_block_id; block_id != end_block_id; block_id++) {
911− mem = find_memory_block_by_id(block_id);
902+ mem = memory_block_get(block_id);
912903 if (WARN_ON_ONCE(!mem))
913904 continue;
914905 num_poisoned_pages_sub(-1UL, memblk_nr_poison(mem));
@@ -1015,12 +1006,12 @@ int walk_memory_blocks(unsigned long start, unsigned long size, · 100 unmodified lines
10151006 return 0;
10161007
10171008 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
1018− mem = find_memory_block_by_id(block_id);
1009+ mem = memory_block_get(block_id);
10191010 if (!mem)
10201011 continue;
10211012
10221013 ret = func(mem, arg);
1023− put_device(&mem->dev);
1014+ memory_block_put(mem);
10241015 if (ret)
10251016 break;
10261017 }
@@ -1228,22 +1219,22 @@ int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func, · 201 unmodified lines
12281219 void memblk_nr_poison_inc(unsigned long pfn)
12291220 {
12301221 const unsigned long block_id = pfn_to_block_id(pfn);
1231− struct memory_block *mem = find_memory_block_by_id(block_id);
1222+ struct memory_block *mem = memory_block_get(block_id);
12321223
12331224 if (mem) {
12341225 atomic_long_inc(&mem->nr_hwpoison);
1235− put_device(&mem->dev);
1226+ memory_block_put(mem);
12361227 }
12371228 }
12381229
12391230 void memblk_nr_poison_sub(unsigned long pfn, long i)
12401231 {
12411232 const unsigned long block_id = pfn_to_block_id(pfn);
1242− struct memory_block *mem = find_memory_block_by_id(block_id);
1233+ struct memory_block *mem = memory_block_get(block_id);
12431234
12441235 if (mem) {
12451236 atomic_long_sub(i, &mem->nr_hwpoison);
1246− put_device(&mem->dev);
1237+ memory_block_put(mem);
12471238 }
12481239 }
12491240
drivers/base/node.c+4−2
@@ -523,6 +523,7 @@ static ssize_t node_read_meminfo(struct device *dev, · 522 unmodified lines
523523 #ifdef CONFIG_UNACCEPTED_MEMORY
524524 "Node %d Unaccepted: %8lu kB\n"
525525 #endif
526+ "Node %d Balloon: %8lu kB\n"
526527 "Node %d GPUActive: %8lu kB\n"
527528 "Node %d GPUReclaim: %8lu kB\n"
528529 ,
@@ -559,6 +560,7 @@ static ssize_t node_read_meminfo(struct device *dev, · 30 unmodified lines
559560 nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED))
560561 #endif
561562 ,
563+ nid, K(node_page_state(pgdat, NR_BALLOON_PAGES)),
562564 nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)),
563565 nid, K(node_page_state(pgdat, NR_GPU_RECLAIM))
564566 );
@@ -847,13 +849,13 @@ static void register_memory_blocks_under_nodes(void) · 282 unmodified lines
847849 for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
848850 struct memory_block *mem;
849851
850− mem = find_memory_block_by_id(block_id);
852+ mem = memory_block_get(block_id);
851853 if (!mem)
852854 continue;
853855
854856 memory_block_add_nid_early(mem, nid);
855857 do_register_memory_block_under_node(nid, mem);
856− put_device(&mem->dev);
858+ memory_block_put(mem);
857859 }
858860
859861 }
drivers/block/zram/zram_drv.c+9−4
@@ -1127,6 +1127,9 @@ next: · 1126 unmodified lines
11271127 if (req)
11281128 release_wb_req(req);
11291129
1130+ if (blk_idx != INVALID_BDEV_BLOCK)
1131+ zram_release_bdev_block(zram, blk_idx);
1132+
11301133 while (atomic_read(&wb_ctl->num_inflight) > 0) {
11311134 wait_event(wb_ctl->done_wait, !list_empty(&wb_ctl->done_reqs));
11321135 err = zram_complete_done_reqs(zram, wb_ctl);
@@ -2131,6 +2134,8 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) · 998 unmodified lines
21312134 zs_obj_read_end(zram->mem_pool, handle, size, src);
21322135 zcomp_stream_put(zstrm);
21332136
2137+ memzero_page(page, size, PAGE_SIZE - size);
2138+
21342139 return 0;
21352140 }
21362141 #endif
@@ -2329,7 +2334,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) · 192 unmodified lines
23292334 * This is a partial IO. Read the full page before writing the changes.
23302335 */
23312336 static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,
2332− u32 index, int offset, struct bio *bio)
2337+ u32 index, int offset)
23332338 {
23342339 struct page *page = alloc_page(GFP_NOIO);
23352340 int ret;
@@ -2347,10 +2352,10 @@ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, · 11 unmodified lines
23472352 }
23482353
23492354 static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
2350− u32 index, int offset, struct bio *bio)
2355+ u32 index, int offset)
23512356 {
23522357 if (is_partial_io(bvec))
2353− return zram_bvec_write_partial(zram, bvec, index, offset, bio);
2358+ return zram_bvec_write_partial(zram, bvec, index, offset);
23542359 return zram_write_page(zram, bvec->bv_page, index);
23552360 }
23562361
@@ -2747,7 +2752,7 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) · 390 unmodified lines
27472752
27482753 bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset);
27492754
2750− if (zram_bvec_write(zram, &bv, index, offset, bio) < 0) {
2755+ if (zram_bvec_write(zram, &bv, index, offset) < 0) {
27512756 atomic64_inc(&zram->stats.failed_writes);
27522757 bio->bi_status = BLK_STS_IOERR;
27532758 break;
drivers/char/mem.c+6−19
@@ -322,11 +322,6 @@ static const struct vm_operations_struct mmap_mem_ops = { · 321 unmodified lines
322322 #endif
323323 };
324324
325−static int mmap_filter_error(int err)
326−{
327− return -EAGAIN;
328−}
329−
330325 static int mmap_mem_prepare(struct vm_area_desc *desc)
331326 {
332327 struct file *file = desc->file;
@@ -362,8 +357,7 @@ static int mmap_mem_prepare(struct vm_area_desc *desc) · 29 unmodified lines
362357
363358 /* Remap-pfn-range will mark the range with the I/O flag. */
364359 mmap_action_remap_full(desc, desc->pgoff);
365− /* We filter remap errors to -EAGAIN. */
366− desc->action.error_hook = mmap_filter_error;
360+ desc->action.error_override = -EAGAIN;
367361
368362 return 0;
369363 }
@@ -504,17 +498,6 @@ static ssize_t read_zero(struct file *file, char __user *buf, · 134 unmodified lines
504498 return cleared;
505499 }
506500
507−static int mmap_zero_private_success(const struct vm_area_struct *vma)
508−{
509− /*
510− * This is a highly unique situation where we mark a MAP_PRIVATE mapping
511− * of /dev/zero anonymous, despite it not being.
512− */
513− vma_set_anonymous((struct vm_area_struct *)vma);
514−
515− return 0;
516−}
517−
518501 static int mmap_zero_prepare(struct vm_area_desc *desc)
519502 {
520503 #ifndef CONFIG_MMU
@@ -523,7 +506,11 @@ static int mmap_zero_prepare(struct vm_area_desc *desc) · 2 unmodified lines
523506 if (vma_desc_test(desc, VMA_SHARED_BIT))
524507 return shmem_zero_setup_desc(desc);
525508
526− desc->action.success_hook = mmap_zero_private_success;
509+ /*
510+ * This is a highly unique situation where we mark a MAP_PRIVATE mapping
511+ * of /dev/zero anonymous, despite it not being.
512+ */
513+ vma_desc_set_anonymous(desc);
527514 return 0;
528515 }
529516
drivers/dax/kmem.c+6−0
@@ -227,6 +227,12 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 226 unmodified lines
227227 if (rc)
228228 continue;
229229
230+ /* range was never added during probe */
231+ if (!data->res[i]) {
232+ success++;
233+ continue;
234+ }
235+
230236 rc = remove_memory(range.start, range_len(&range));
231237 if (rc == 0) {
232238 remove_resource(data->res[i]);
drivers/gpu/drm/drm_managed.c+2−2
@@ -232,8 +232,8 @@ void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp) · 231 unmodified lines
232232
233233 dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
234234 if (!dr) {
235− drm_dbg_drmres(dev, "failed to allocate %zu bytes, %u flags\n",
236− size, gfp);
235+ drm_dbg_drmres(dev, "failed to allocate %zu bytes, %pGg\n",
236+ size, &gfp);
237237 return NULL;
238238 }
239239 dr->node.name = kstrdup_const("kmalloc", gfp);
drivers/s390/char/sclp_mem.c+8−9
@@ -204,7 +204,7 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute · 203 unmodified lines
204204 addr = sclp_mem->id * block_size;
205205 /*
206206 * Hold device_hotplug_lock when adding/removing memory blocks.
207− * Additionally, also protect calls to find_memory_block() and
207+ * Additionally, also protect calls to memory_block_get() and
208208 * sclp_attach_storage().
209209 */
210210 rc = lock_device_hotplug_sysfs();
@@ -231,20 +231,19 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute · 20 unmodified lines
231231 sclp_mem_change_state(addr, block_size, 0);
232232 goto out_unlock;
233233 }
234− mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));
235− put_device(&mem->dev);
234+ mem = memory_block_get(phys_to_block_id(addr));
235+ memory_block_put(mem);
236236 WRITE_ONCE(sclp_mem->config, 1);
237237 } else {
238238 if (!sclp_mem->config)
239239 goto out_unlock;
240− mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));
240+ mem = memory_block_get(phys_to_block_id(addr));
241241 if (mem->state != MEM_OFFLINE) {
242− put_device(&mem->dev);
242+ memory_block_put(mem);
243243 rc = -EBUSY;
244244 goto out_unlock;
245245 }
246− /* drop the ref just got via find_memory_block() */
247− put_device(&mem->dev);
246+ memory_block_put(mem);
248247 sclp_mem_change_state(addr, block_size, 0);
249248 __remove_memory(addr, block_size);
250249 #ifdef CONFIG_KASAN
@@ -294,11 +293,11 @@ static ssize_t sclp_memmap_on_memory_store(struct kobject *kobj, struct kobj_att · 43 unmodified lines
294293 return rc;
295294 block_size = memory_block_size_bytes();
296295 sclp_mem = container_of(kobj, struct sclp_mem, kobj);
297− mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(sclp_mem->id * block_size)));
296+ mem = memory_block_get(phys_to_block_id(sclp_mem->id * block_size));
298297 if (!mem) {
299298 WRITE_ONCE(sclp_mem->memmap_on_memory, value);
300299 } else {
301− put_device(&mem->dev);
300+ memory_block_put(mem);
302301 rc = -EBUSY;
303302 }
304303 unlock_device_hotplug();
fs/Makefile+0−1
@@ -27,7 +27,6 @@ obj-y += anon_inodes.o · 26 unmodified lines
2727 obj-$(CONFIG_SIGNALFD) += signalfd.o
2828 obj-$(CONFIG_TIMERFD) += timerfd.o
2929 obj-$(CONFIG_EVENTFD) += eventfd.o
30−obj-$(CONFIG_USERFAULTFD) += userfaultfd.o
3130 obj-$(CONFIG_AIO) += aio.o
3231 obj-$(CONFIG_FS_DAX) += dax.o
3332 obj-$(CONFIG_FS_ENCRYPTION) += crypto/
fs/proc/task_mmu.c+223−46
@@ -132,6 +132,22 @@ static void release_task_mempolicy(struct proc_maps_private *priv) · 131 unmodified lines
132132
133133 #ifdef CONFIG_PER_VMA_LOCK
134134
135+static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
136+{
137+ int ret = mmap_read_lock_killable(lock_ctx->mm);
138+
139+ if (!ret)
140+ lock_ctx->mmap_locked = true;
141+
142+ return ret;
143+}
144+
145+static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
146+{
147+ mmap_read_unlock(lock_ctx->mm);
148+ lock_ctx->mmap_locked = false;
149+}
150+
135151 static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx)
136152 {
137153 lock_ctx->locked_vma = NULL;
@@ -146,25 +162,11 @@ static void unlock_ctx_vma(struct proc_maps_locking_ctx *lock_ctx) · 8 unmodified lines
146162 }
147163 }
148164
149−static const struct seq_operations proc_pid_maps_op;
150−
151165 static inline bool lock_vma_range(struct seq_file *m,
152166 struct proc_maps_locking_ctx *lock_ctx)
153167 {
154− /*
155− * smaps and numa_maps perform page table walk, therefore require
156− * mmap_lock but maps can be read with locking just the vma and
157− * walking the vma tree under rcu read protection.
158− */
159− if (m->op != &proc_pid_maps_op) {
160− if (mmap_read_lock_killable(lock_ctx->mm))
161− return false;
162−
163− lock_ctx->mmap_locked = true;
164− } else {
165− rcu_read_lock();
166− reset_lock_ctx(lock_ctx);
167− }
168+ rcu_read_lock();
169+ reset_lock_ctx(lock_ctx);
168170
169171 return true;
170172 }
@@ -172,7 +174,7 @@ static inline bool lock_vma_range(struct seq_file *m, · 1 unmodified line
172174 static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
173175 {
174176 if (lock_ctx->mmap_locked) {
175− mmap_read_unlock(lock_ctx->mm);
177+ unlock_ctx_mm(lock_ctx);
176178 } else {
177179 unlock_ctx_vma(lock_ctx);
178180 rcu_read_unlock();
@@ -213,17 +215,45 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, · 34 unmodified lines
213215 return true;
214216 }
215217
218+static inline void drop_rcu(struct proc_maps_private *priv)
219+{
220+ if (priv->lock_ctx.mmap_locked)
221+ return;
222+
223+ rcu_read_unlock();
224+}
225+
226+static inline void reacquire_rcu(struct proc_maps_private *priv)
227+{
228+ if (priv->lock_ctx.mmap_locked)
229+ return;
230+
231+ rcu_read_lock();
232+ /* Reinitialize the iterator. */
233+ vma_iter_set(&priv->iter, priv->lock_ctx.locked_vma->vm_end);
234+}
235+
216236 #else /* CONFIG_PER_VMA_LOCK */
217237
238+static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
239+{
240+ return mmap_read_lock_killable(lock_ctx->mm);
241+}
242+
243+static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
244+{
245+ mmap_read_unlock(lock_ctx->mm);
246+}
247+
218248 static inline bool lock_vma_range(struct seq_file *m,
219249 struct proc_maps_locking_ctx *lock_ctx)
220250 {
221− return mmap_read_lock_killable(lock_ctx->mm) == 0;
251+ return lock_ctx_mm(lock_ctx) == 0;
222252 }
223253
224254 static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
225255 {
226− mmap_read_unlock(lock_ctx->mm);
256+ unlock_ctx_mm(lock_ctx);
227257 }
228258
229259 static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv,
@@ -238,6 +268,9 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, · 8 unmodified lines
238268 return false;
239269 }
240270
271+static inline void drop_rcu(struct proc_maps_private *priv) {}
272+static inline void reacquire_rcu(struct proc_maps_private *priv) {}
273+
241274 #endif /* CONFIG_PER_VMA_LOCK */
242275
243276 static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos)
@@ -538,12 +571,10 @@ static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx) · 294 unmodified lines
538571
539572 static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx)
540573 {
541− if (lock_ctx->mmap_locked) {
542− mmap_read_unlock(lock_ctx->mm);
543− lock_ctx->mmap_locked = false;
544− } else {
574+ if (lock_ctx->mmap_locked)
575+ unlock_ctx_mm(lock_ctx);
576+ else
545577 unlock_ctx_vma(lock_ctx);
546− }
547578 }
548579
549580 static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_locking_ctx *lock_ctx,
@@ -1280,21 +1311,75 @@ static const struct mm_walk_ops smaps_shmem_walk_ops = { · 730 unmodified lines
12801311 .walk_lock = PGWALK_RDLOCK,
12811312 };
12821313
1314+#ifdef CONFIG_PER_VMA_LOCK
1315+
1316+static const struct mm_walk_ops smaps_walk_vma_lock_ops = {
1317+ .pmd_entry = smaps_pte_range,
1318+ .hugetlb_entry = smaps_hugetlb_range,
1319+ .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
1320+};
1321+
1322+static const struct mm_walk_ops smaps_shmem_walk_vma_lock_ops = {
1323+ .pmd_entry = smaps_pte_range,
1324+ .hugetlb_entry = smaps_hugetlb_range,
1325+ .pte_hole = smaps_pte_hole,
1326+ .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
1327+};
1328+
1329+static inline const struct mm_walk_ops *
1330+get_smaps_walk_ops(struct proc_maps_private *priv)
1331+{
1332+ if (priv->lock_ctx.mmap_locked)
1333+ return &smaps_walk_ops;
1334+ return &smaps_walk_vma_lock_ops;
1335+}
1336+
1337+static inline const struct mm_walk_ops *
1338+get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
1339+{
1340+ if (priv->lock_ctx.mmap_locked)
1341+ return &smaps_shmem_walk_ops;
1342+ return &smaps_shmem_walk_vma_lock_ops;
1343+}
1344+
1345+#else /* CONFIG_PER_VMA_LOCK */
1346+
1347+static inline const struct mm_walk_ops *
1348+get_smaps_walk_ops(struct proc_maps_private *priv)
1349+{
1350+ return &smaps_walk_ops;
1351+}
1352+
1353+static inline const struct mm_walk_ops *
1354+get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
1355+{
1356+ return &smaps_shmem_walk_ops;
1357+}
1358+
1359+#endif /* CONFIG_PER_VMA_LOCK */
1360+
12831361 /*
12841362 * Gather mem stats from @vma with the indicated beginning
12851363 * address @start, and keep them in @mss.
12861364 *
12871365 * Use vm_start of @vma as the beginning address if @start is 0.
12881366 */
1289−static void smap_gather_stats(struct vm_area_struct *vma,
1290− struct mem_size_stats *mss, unsigned long start)
1367+static void smap_gather_stats(struct proc_maps_private *priv,
1368+ struct vm_area_struct *vma,
1369+ struct mem_size_stats *mss, unsigned long start)
12911370 {
1292− const struct mm_walk_ops *ops = &smaps_walk_ops;
1371+ const struct mm_walk_ops *ops = get_smaps_walk_ops(priv);
12931372
12941373 /* Invalid start */
12951374 if (start >= vma->vm_end)
12961375 return;
12971376
1377+ if (vma == get_gate_vma(priv->lock_ctx.mm))
1378+ return;
1379+
1380+ /* Might sleep. Drop RCU read lock but keep the VMA locked. */
1381+ drop_rcu(priv);
1382+
12981383 if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {
12991384 /*
13001385 * For shared or readonly shmem mappings we know that all
@@ -1312,15 +1397,16 @@ static void smap_gather_stats(struct vm_area_struct *vma, · 11 unmodified lines
13121397 !(vma->vm_flags & VM_WRITE))) {
13131398 mss->swap += shmem_swapped;
13141399 } else {
1315− ops = &smaps_shmem_walk_ops;
1400+ ops = get_smaps_shmem_walk_ops(priv);
13161401 }
13171402 }
13181403
1319− /* mmap_lock is held in m_start */
13201404 if (!start)
13211405 walk_page_vma(vma, ops, mss);
13221406 else
13231407 walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss);
1408+
1409+ reacquire_rcu(priv);
13241410 }
13251411
13261412 #define SEQ_PUT_DEC(str, val) \
@@ -1369,10 +1455,11 @@ static void __show_smap(struct seq_file *m, const struct mem_size_stats *mss, · 42 unmodified lines
13691455
13701456 static int show_smap(struct seq_file *m, void *v)
13711457 {
1458+ struct proc_maps_private *priv = m->private;
13721459 struct vm_area_struct *vma = v;
13731460 struct mem_size_stats mss = {};
13741461
1375− smap_gather_stats(vma, &mss, 0);
1462+ smap_gather_stats(priv, vma, &mss, 0);
13761463
13771464 show_map_vma(m, vma);
13781465
@@ -1413,7 +1500,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 34 unmodified lines
14131500 goto out_put_task;
14141501 }
14151502
1416− ret = mmap_read_lock_killable(mm);
1503+ ret = lock_ctx_mm(&priv->lock_ctx);
14171504 if (ret)
14181505 goto out_put_mm;
14191506
@@ -1425,7 +1512,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 5 unmodified lines
14251512
14261513 vma_start = vma->vm_start;
14271514 do {
1428− smap_gather_stats(vma, &mss, 0);
1515+ smap_gather_stats(priv, vma, &mss, 0);
14291516 last_vma_end = vma->vm_end;
14301517
14311518 /*
@@ -1434,8 +1521,8 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 2 unmodified lines
14341521 */
14351522 if (mmap_lock_is_contended(mm)) {
14361523 vma_iter_invalidate(&vmi);
1437− mmap_read_unlock(mm);
1438− ret = mmap_read_lock_killable(mm);
1524+ unlock_ctx_mm(&priv->lock_ctx);
1525+ ret = lock_ctx_mm(&priv->lock_ctx);
14391526 if (ret) {
14401527 release_task_mempolicy(priv);
14411528 goto out_put_mm;
@@ -1484,14 +1571,14 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 42 unmodified lines
14841571
14851572 /* Case 1 and 2 above */
14861573 if (vma->vm_start >= last_vma_end) {
1487− smap_gather_stats(vma, &mss, 0);
1574+ smap_gather_stats(priv, vma, &mss, 0);
14881575 last_vma_end = vma->vm_end;
14891576 continue;
14901577 }
14911578
14921579 /* Case 4 above */
14931580 if (vma->vm_end > last_vma_end) {
1494− smap_gather_stats(vma, &mss, last_vma_end);
1581+ smap_gather_stats(priv, vma, &mss, last_vma_end);
14951582 last_vma_end = vma->vm_end;
14961583 }
14971584 }
@@ -1505,7 +1592,7 @@ empty_set: · 7 unmodified lines
15051592 __show_smap(m, &mss, true);
15061593
15071594 release_task_mempolicy(priv);
1508− mmap_read_unlock(mm);
1595+ unlock_ctx_mm(&priv->lock_ctx);
15091596
15101597 out_put_mm:
15111598 mmput(mm);
@@ -2042,7 +2129,6 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr, · 530 unmodified lines
20422129 flags |= PM_SOFT_DIRTY;
20432130 if (pmd_swp_uffd_wp(pmd))
20442131 flags |= PM_UFFD_WP;
2045− VM_WARN_ON_ONCE(!pmd_is_migration_entry(pmd));
20462132 page = softleaf_to_page(entry);
20472133 }
20482134
@@ -2523,12 +2609,16 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma, · 474 unmodified lines
25232609 if (softleaf_is_hwpoison(entry) || softleaf_is_marker(entry))
25242610 return;
25252611
2526− if (softleaf_is_migration(entry))
2612+ if (softleaf_is_migration(entry)) {
25272613 set_huge_pte_at(vma->vm_mm, addr, ptep,
25282614 pte_swp_mkuffd_wp(ptent), psize);
2529− else
2530− huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
2531− huge_pte_mkuffd_wp(ptent));
2615+ } else {
2616+ pte_t old_pte, new_pte;
2617+
2618+ old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
2619+ new_pte = huge_pte_mkuffd_wp(old_pte);
2620+ huge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);
2621+ }
25322622 }
25332623 #endif /* CONFIG_HUGETLB_PAGE */
25342624
@@ -2869,7 +2959,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 334 unmodified lines
28692959 if (~categories & PAGE_IS_WRITTEN)
28702960 goto out_unlock;
28712961
2872− if (end != start + HPAGE_SIZE) {
2962+ if (end != start + huge_page_size(hstate_vma(vma))) {
28732963 /* Partial HugeTLB page WP isn't possible. */
28742964 pagemap_scan_backout_range(p, start, end);
28752965 p->arg.walk_end = start;
@@ -2886,8 +2976,62 @@ out_unlock: · 10 unmodified lines
28862976
28872977 return ret;
28882978 }
2979+
2980+/*
2981+ * Write-protect the unpopulated hugetlb entries covering [addr, end) by
2982+ * installing uffd-wp markers inline, exactly as pagemap_scan_hugetlb_entry()
2983+ * does for populated entries.
2984+ *
2985+ * walk_hugetlb_range() currently calls ->pte_hole() once per huge page, so the
2986+ * loop normally runs a single iteration; it is written to cover the full range
2987+ * in case the walker ever coalesces adjacent holes.
2988+ *
2989+ * The obvious route -- uffd_wp_range() -> hugetlb_change_protection() --
2990+ * cannot be used here: it takes hugetlb_vma_lock_write(), but the page-table
2991+ * walker (walk_hugetlb_range()) already holds hugetlb_vma_lock_read() on the
2992+ * same VMA, so the scanning thread would deadlock against itself. PMD sharing
2993+ * is disabled on uffd-wp VMAs (hugetlb_unshare_all_pmds() at registration), so
2994+ * the vma lock guards nothing that matters for these entries anyway.
2995+ */
2996+static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
2997+ unsigned long addr, unsigned long end)
2998+{
2999+ struct hstate *h = hstate_vma(vma);
3000+ unsigned long psize = huge_page_size(h);
3001+ struct mm_struct *mm = vma->vm_mm;
3002+ spinlock_t *ptl;
3003+ pte_t *ptep;
3004+ pte_t pte;
3005+
3006+ for (addr = ALIGN_DOWN(addr, psize); addr < end; addr += psize) {
3007+ ptep = huge_pte_alloc(mm, vma, addr, psize);
3008+ if (!ptep)
3009+ return -ENOMEM;
3010+
3011+ i_mmap_lock_write(vma->vm_file->f_mapping);
3012+ ptl = huge_pte_lock(h, mm, ptep);
3013+ pte = huge_ptep_get(mm, addr, ptep);
3014+ make_uffd_wp_huge_pte(vma, addr, ptep, pte);
3015+ /*
3016+ * A none entry has no cached translation, so installing the
3017+ * marker needs no TLB flush. Flush only if a fault populated
3018+ * the entry between huge_pte_alloc() and the page table lock.
3019+ */
3020+ if (!huge_pte_none(pte))
3021+ flush_hugetlb_tlb_range(vma, addr, addr + psize);
3022+ spin_unlock(ptl);
3023+ i_mmap_unlock_write(vma->vm_file->f_mapping);
3024+ }
3025+
3026+ return 0;
3027+}
28893028 #else
28903029 #define pagemap_scan_hugetlb_entry NULL
3030+static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
3031+ unsigned long addr, unsigned long end)
3032+{
3033+ return 0;
3034+}
28913035 #endif
28923036
28933037 static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
@@ -2907,7 +3051,10 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, · 13 unmodified lines
29073051 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
29083052 return ret;
29093053
2910− err = uffd_wp_range(vma, addr, end - addr, true);
3054+ if (is_vm_hugetlb_page(vma))
3055+ err = pagemap_scan_hugetlb_hole_wp(vma, addr, end);
3056+ else
3057+ err = uffd_wp_range(vma, addr, end - addr, true);
29113058 if (err < 0)
29123059 ret = err;
29133060
@@ -3291,6 +3438,31 @@ static const struct mm_walk_ops show_numa_ops = { · 377 unmodified lines
32913438 .walk_lock = PGWALK_RDLOCK,
32923439 };
32933440
3441+#ifdef CONFIG_PER_VMA_LOCK
3442+static const struct mm_walk_ops show_numa_vma_lock_ops = {
3443+ .hugetlb_entry = gather_hugetlb_stats,
3444+ .pmd_entry = gather_pte_stats,
3445+ .walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
3446+};
3447+
3448+static inline const struct mm_walk_ops *
3449+get_show_numa_ops(struct proc_maps_private *priv)
3450+{
3451+ if (priv->lock_ctx.mmap_locked)
3452+ return &show_numa_ops;
3453+ return &show_numa_vma_lock_ops;
3454+}
3455+
3456+#else /* CONFIG_PER_VMA_LOCK */
3457+
3458+static inline const struct mm_walk_ops *
3459+get_show_numa_ops(struct proc_maps_private *priv)
3460+{
3461+ return &show_numa_ops;
3462+}
3463+
3464+#endif /* CONFIG_PER_VMA_LOCK */
3465+
32943466 /*
32953467 * Display pages allocated per node and memory policy via /proc.
32963468 */
@@ -3335,8 +3507,13 @@ static int show_numa_map(struct seq_file *m, void *v) · 38 unmodified lines
33353507 if (is_vm_hugetlb_page(vma))
33363508 seq_puts(m, " huge");
33373509
3338− /* mmap_lock is held by m_start */
3339− walk_page_vma(vma, &show_numa_ops, md);
3510+ /* Skip walking pages if gate VMA */
3511+ if (vma != get_gate_vma(proc_priv->lock_ctx.mm)) {
3512+ /* Might sleep. Drop RCU read lock but keep the VMA locked. */
3513+ drop_rcu(proc_priv);
3514+ walk_page_vma(vma, get_show_numa_ops(proc_priv), md);
3515+ reacquire_rcu(proc_priv);
3516+ }
33403517
33413518 if (!md->pages)
33423519 goto out;
fs/userfaultfd.c+0−2231
@@ -1,2231 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0-only
2−/*
3− * fs/userfaultfd.c
4− *
5− * Copyright (C) 2007 Davide Libenzi <davidel@xmailserver.org>
6− * Copyright (C) 2008-2009 Red Hat, Inc.
7− * Copyright (C) 2015 Red Hat, Inc.
8− *
9− * Some part derived from fs/eventfd.c (anon inode setup) and
10− * mm/ksm.c (mm hashing).
11− */
12−
13−#include <linux/list.h>
14−#include <linux/hashtable.h>
15−#include <linux/sched/signal.h>
16−#include <linux/sched/mm.h>
17−#include <linux/mm.h>
18−#include <linux/mm_inline.h>
19−#include <linux/mmu_notifier.h>
20−#include <linux/poll.h>
21−#include <linux/slab.h>
22−#include <linux/seq_file.h>
23−#include <linux/file.h>
24−#include <linux/bug.h>
25−#include <linux/anon_inodes.h>
26−#include <linux/syscalls.h>
27−#include <linux/userfaultfd_k.h>
28−#include <linux/mempolicy.h>
29−#include <linux/ioctl.h>
30−#include <linux/security.h>
31−#include <linux/hugetlb.h>
32−#include <linux/leafops.h>
33−#include <linux/miscdevice.h>
34−#include <linux/uio.h>
35−
36−static int sysctl_unprivileged_userfaultfd __read_mostly;
37−
38−#ifdef CONFIG_SYSCTL
39−static const struct ctl_table vm_userfaultfd_table[] = {
40− {
41− .procname = "unprivileged_userfaultfd",
42− .data = &sysctl_unprivileged_userfaultfd,
43− .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
44− .mode = 0644,
45− .proc_handler = proc_dointvec_minmax,
46− .extra1 = SYSCTL_ZERO,
47− .extra2 = SYSCTL_ONE,
48− },
49−};
50−#endif
51−
52−static struct kmem_cache *userfaultfd_ctx_cachep __ro_after_init;
53−
54−struct userfaultfd_fork_ctx {
55− struct userfaultfd_ctx *orig;
56− struct userfaultfd_ctx *new;
57− struct list_head list;
58−};
59−
60−struct userfaultfd_unmap_ctx {
61− struct userfaultfd_ctx *ctx;
62− unsigned long start;
63− unsigned long end;
64− struct list_head list;
65−};
66−
67−struct userfaultfd_wait_queue {
68− struct uffd_msg msg;
69− wait_queue_entry_t wq;
70− struct userfaultfd_ctx *ctx;
71− bool waken;
72−};
73−
74−struct userfaultfd_wake_range {
75− unsigned long start;
76− unsigned long len;
77−};
78−
79−/* internal indication that UFFD_API ioctl was successfully executed */
80−#define UFFD_FEATURE_INITIALIZED (1u << 31)
81−
82−static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
83−{
84− return ctx->features & UFFD_FEATURE_INITIALIZED;
85−}
86−
87−static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
88−{
89− return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
90−}
91−
92−/*
93− * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
94− * meaningful when userfaultfd_wp()==true on the vma and when it's
95− * anonymous.
96− */
97−bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
98−{
99− struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
100−
101− if (!ctx)
102− return false;
103−
104− return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
105−}
106−
107−static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
108− int wake_flags, void *key)
109−{
110− struct userfaultfd_wake_range *range = key;
111− int ret;
112− struct userfaultfd_wait_queue *uwq;
113− unsigned long start, len;
114−
115− uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
116− ret = 0;
117− /* len == 0 means wake all */
118− start = range->start;
119− len = range->len;
120− if (len && (start > uwq->msg.arg.pagefault.address ||
121− start + len <= uwq->msg.arg.pagefault.address))
122− goto out;
123− WRITE_ONCE(uwq->waken, true);
124− /*
125− * The Program-Order guarantees provided by the scheduler
126− * ensure uwq->waken is visible before the task is woken.
127− */
128− ret = wake_up_state(wq->private, mode);
129− if (ret) {
130− /*
131− * Wake only once, autoremove behavior.
132− *
133− * After the effect of list_del_init is visible to the other
134− * CPUs, the waitqueue may disappear from under us, see the
135− * !list_empty_careful() in handle_userfault().
136− *
137− * try_to_wake_up() has an implicit smp_mb(), and the
138− * wq->private is read before calling the extern function
139− * "wake_up_state" (which in turns calls try_to_wake_up).
140− */
141− list_del_init(&wq->entry);
142− }
143−out:
144− return ret;
145−}
146−
147−/**
148− * userfaultfd_ctx_get - Acquires a reference to the internal userfaultfd
149− * context.
150− * @ctx: [in] Pointer to the userfaultfd context.
151− */
152−static void userfaultfd_ctx_get(struct userfaultfd_ctx *ctx)
153−{
154− refcount_inc(&ctx->refcount);
155−}
156−
157−/**
158− * userfaultfd_ctx_put - Releases a reference to the internal userfaultfd
159− * context.
160− * @ctx: [in] Pointer to userfaultfd context.
161− *
162− * The userfaultfd context reference must have been previously acquired either
163− * with userfaultfd_ctx_get() or userfaultfd_ctx_fdget().
164− */
165−static void userfaultfd_ctx_put(struct userfaultfd_ctx *ctx)
166−{
167− if (refcount_dec_and_test(&ctx->refcount)) {
168− VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_pending_wqh.lock));
169− VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_pending_wqh));
170− VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_wqh.lock));
171− VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_wqh));
172− VM_WARN_ON_ONCE(spin_is_locked(&ctx->event_wqh.lock));
173− VM_WARN_ON_ONCE(waitqueue_active(&ctx->event_wqh));
174− VM_WARN_ON_ONCE(spin_is_locked(&ctx->fd_wqh.lock));
175− VM_WARN_ON_ONCE(waitqueue_active(&ctx->fd_wqh));
176− mmdrop(ctx->mm);
177− kmem_cache_free(userfaultfd_ctx_cachep, ctx);
178− }
179−}
180−
181−static inline void msg_init(struct uffd_msg *msg)
182−{
183− BUILD_BUG_ON(sizeof(struct uffd_msg) != 32);
184− /*
185− * Must use memset to zero out the paddings or kernel data is
186− * leaked to userland.
187− */
188− memset(msg, 0, sizeof(struct uffd_msg));
189−}
190−
191−static inline struct uffd_msg userfault_msg(unsigned long address,
192− unsigned long real_address,
193− unsigned int flags,
194− unsigned long reason,
195− unsigned int features)
196−{
197− struct uffd_msg msg;
198−
199− msg_init(&msg);
200− msg.event = UFFD_EVENT_PAGEFAULT;
201−
202− msg.arg.pagefault.address = (features & UFFD_FEATURE_EXACT_ADDRESS) ?
203− real_address : address;
204−
205− /*
206− * These flags indicate why the userfault occurred:
207− * - UFFD_PAGEFAULT_FLAG_WP indicates a write protect fault.
208− * - UFFD_PAGEFAULT_FLAG_MINOR indicates a minor fault.
209− * - Neither of these flags being set indicates a MISSING fault.
210− *
211− * Separately, UFFD_PAGEFAULT_FLAG_WRITE indicates it was a write
212− * fault. Otherwise, it was a read fault.
213− */
214− if (flags & FAULT_FLAG_WRITE)
215− msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;
216− if (reason & VM_UFFD_WP)
217− msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;
218− if (reason & VM_UFFD_MINOR)
219− msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;
220− if (features & UFFD_FEATURE_THREAD_ID)
221− msg.arg.pagefault.feat.ptid = task_pid_vnr(current);
222− return msg;
223−}
224−
225−#ifdef CONFIG_HUGETLB_PAGE
226−/*
227− * Same functionality as userfaultfd_must_wait below with modifications for
228− * hugepmd ranges.
229− */
230−static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
231− struct vm_fault *vmf,
232− unsigned long reason)
233−{
234− struct vm_area_struct *vma = vmf->vma;
235− pte_t *ptep, pte;
236−
237− assert_fault_locked(vmf);
238−
239− ptep = hugetlb_walk(vma, vmf->address, vma_mmu_pagesize(vma));
240− if (!ptep)
241− return true;
242−
243− pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
244−
245− /*
246− * Lockless access: we're in a wait_event so it's ok if it
247− * changes under us.
248− */
249−
250− /* Entry is still missing, wait for userspace to resolve the fault. */
251− if (huge_pte_none(pte))
252− return true;
253− /* UFFD PTE markers require userspace to resolve the fault. */
254− if (pte_is_uffd_marker(pte))
255− return true;
256− /*
257− * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
258− * resolve the fault.
259− */
260− if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
261− return true;
262−
263− return false;
264−}
265−#else
266−static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
267− struct vm_fault *vmf,
268− unsigned long reason)
269−{
270− /* Should never get here. */
271− VM_WARN_ON_ONCE(1);
272− return false;
273−}
274−#endif /* CONFIG_HUGETLB_PAGE */
275−
276−/*
277− * Verify the pagetables are still not ok after having registered into
278− * the fault_pending_wqh to avoid userland having to UFFDIO_WAKE any
279− * userfault that has already been resolved, if userfaultfd_read_iter and
280− * UFFDIO_COPY|ZEROPAGE are being run simultaneously on two different
281− * threads.
282− */
283−static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
284− struct vm_fault *vmf,
285− unsigned long reason)
286−{
287− struct mm_struct *mm = ctx->mm;
288− unsigned long address = vmf->address;
289− pgd_t *pgd;
290− p4d_t *p4d;
291− pud_t *pud;
292− pmd_t *pmd, _pmd;
293− pte_t *pte;
294− pte_t ptent;
295− bool ret;
296−
297− assert_fault_locked(vmf);
298−
299− pgd = pgd_offset(mm, address);
300− if (!pgd_present(*pgd))
301− return true;
302− p4d = p4d_offset(pgd, address);
303− if (!p4d_present(*p4d))
304− return true;
305− pud = pud_offset(p4d, address);
306− if (!pud_present(*pud))
307− return true;
308− pmd = pmd_offset(pud, address);
309−again:
310− _pmd = pmdp_get_lockless(pmd);
311− if (pmd_none(_pmd))
312− return true;
313−
314− /*
315− * A race could arise which would result in a softleaf entry such as
316− * migration entry unexpectedly being present in the PMD, so explicitly
317− * check for this and bail out if so.
318− */
319− if (!pmd_present(_pmd))
320− return false;
321−
322− if (pmd_trans_huge(_pmd))
323− return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
324−
325− pte = pte_offset_map(pmd, address);
326− if (!pte)
327− goto again;
328−
329− /*
330− * Lockless access: we're in a wait_event so it's ok if it
331− * changes under us.
332− */
333− ptent = ptep_get(pte);
334−
335− ret = true;
336− /* Entry is still missing, wait for userspace to resolve the fault. */
337− if (pte_none(ptent))
338− goto out;
339− /* UFFD PTE markers require userspace to resolve the fault. */
340− if (pte_is_uffd_marker(ptent))
341− goto out;
342− /*
343− * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
344− * resolve the fault.
345− */
346− if (!pte_write(ptent) && (reason & VM_UFFD_WP))
347− goto out;
348−
349− ret = false;
350−out:
351− pte_unmap(pte);
352− return ret;
353−}
354−
355−static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags)
356−{
357− if (flags & FAULT_FLAG_INTERRUPTIBLE)
358− return TASK_INTERRUPTIBLE;
359−
360− if (flags & FAULT_FLAG_KILLABLE)
361− return TASK_KILLABLE;
362−
363− return TASK_UNINTERRUPTIBLE;
364−}
365−
366−/*
367− * The locking rules involved in returning VM_FAULT_RETRY depending on
368− * FAULT_FLAG_ALLOW_RETRY, FAULT_FLAG_RETRY_NOWAIT and
369− * FAULT_FLAG_KILLABLE are not straightforward. The "Caution"
370− * recommendation in __lock_page_or_retry is not an understatement.
371− *
372− * If FAULT_FLAG_ALLOW_RETRY is set, the mmap_lock must be released
373− * before returning VM_FAULT_RETRY only if FAULT_FLAG_RETRY_NOWAIT is
374− * not set.
375− *
376− * If FAULT_FLAG_ALLOW_RETRY is set but FAULT_FLAG_KILLABLE is not
377− * set, VM_FAULT_RETRY can still be returned if and only if there are
378− * fatal_signal_pending()s, and the mmap_lock must be released before
379− * returning it.
380− */
381−vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
382−{
383− struct vm_area_struct *vma = vmf->vma;
384− struct mm_struct *mm = vma->vm_mm;
385− struct userfaultfd_ctx *ctx;
386− struct userfaultfd_wait_queue uwq;
387− vm_fault_t ret = VM_FAULT_SIGBUS;
388− bool must_wait;
389− unsigned int blocking_state;
390−
391− /*
392− * We don't do userfault handling for the final child pid update
393− * and when coredumping (faults triggered by get_dump_page()).
394− */
395− if (current->flags & (PF_EXITING|PF_DUMPCORE))
396− goto out;
397−
398− assert_fault_locked(vmf);
399−
400− ctx = vma->vm_userfaultfd_ctx.ctx;
401− if (!ctx)
402− goto out;
403−
404− VM_WARN_ON_ONCE(ctx->mm != mm);
405−
406− /* Any unrecognized flag is a bug. */
407− VM_WARN_ON_ONCE(reason & ~__VM_UFFD_FLAGS);
408− /* 0 or > 1 flags set is a bug; we expect exactly 1. */
409− VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
410−
411− if (ctx->features & UFFD_FEATURE_SIGBUS)
412− goto out;
413− if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
414− goto out;
415−
416− /*
417− * Check that we can return VM_FAULT_RETRY.
418− *
419− * NOTE: it should become possible to return VM_FAULT_RETRY
420− * even if FAULT_FLAG_TRIED is set without leading to gup()
421− * -EBUSY failures, if the userfaultfd is to be extended for
422− * VM_UFFD_WP tracking and we intend to arm the userfault
423− * without first stopping userland access to the memory. For
424− * VM_UFFD_MISSING userfaults this is enough for now.
425− */
426− if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
427− /*
428− * Validate the invariant that nowait must allow retry
429− * to be sure not to return SIGBUS erroneously on
430− * nowait invocations.
431− */
432− VM_WARN_ON_ONCE(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
433−#ifdef CONFIG_DEBUG_VM
434− if (printk_ratelimit()) {
435− pr_warn("FAULT_FLAG_ALLOW_RETRY missing %x\n",
436− vmf->flags);
437− dump_stack();
438− }
439−#endif
440− goto out;
441− }
442−
443− /*
444− * Handle nowait, not much to do other than tell it to retry
445− * and wait.
446− */
447− ret = VM_FAULT_RETRY;
448− if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
449− goto out;
450−
451− if (unlikely(READ_ONCE(ctx->released))) {
452− /*
453− * If a concurrent release is detected, do not return
454− * VM_FAULT_SIGBUS or VM_FAULT_NOPAGE, but instead always
455− * return VM_FAULT_RETRY with lock released proactively.
456− *
457− * If we were to return VM_FAULT_SIGBUS here, the non
458− * cooperative manager would be instead forced to
459− * always call UFFDIO_UNREGISTER before it can safely
460− * close the uffd, to avoid involuntary SIGBUS triggered.
461− *
462− * If we were to return VM_FAULT_NOPAGE, it would work for
463− * the fault path, in which the lock will be released
464− * later. However for GUP, faultin_page() does nothing
465− * special on NOPAGE, so GUP would spin retrying without
466− * releasing the mmap read lock, causing possible livelock.
467− *
468− * Here only VM_FAULT_RETRY would make sure the mmap lock
469− * be released immediately, so that the thread concurrently
470− * releasing the userfault would always make progress.
471− */
472− release_fault_lock(vmf);
473− goto out;
474− }
475−
476− /* take the reference before dropping the mmap_lock */
477− userfaultfd_ctx_get(ctx);
478−
479− init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
480− uwq.wq.private = current;
481− uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
482− reason, ctx->features);
483− uwq.ctx = ctx;
484− uwq.waken = false;
485−
486− blocking_state = userfaultfd_get_blocking_state(vmf->flags);
487−
488− /*
489− * Take the vma lock now, in order to safely call
490− * userfaultfd_huge_must_wait() later. Since acquiring the
491− * (sleepable) vma lock can modify the current task state, that
492− * must be before explicitly calling set_current_state().
493− */
494− if (is_vm_hugetlb_page(vma))
495− hugetlb_vma_lock_read(vma);
496−
497− spin_lock_irq(&ctx->fault_pending_wqh.lock);
498− /*
499− * After the __add_wait_queue the uwq is visible to userland
500− * through poll/read().
501− */
502− __add_wait_queue(&ctx->fault_pending_wqh, &uwq.wq);
503− /*
504− * The smp_mb() after __set_current_state prevents the reads
505− * following the spin_unlock to happen before the list_add in
506− * __add_wait_queue.
507− */
508− set_current_state(blocking_state);
509− spin_unlock_irq(&ctx->fault_pending_wqh.lock);
510−
511− if (is_vm_hugetlb_page(vma)) {
512− must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason);
513− hugetlb_vma_unlock_read(vma);
514− } else {
515− must_wait = userfaultfd_must_wait(ctx, vmf, reason);
516− }
517−
518− release_fault_lock(vmf);
519−
520− if (likely(must_wait && !READ_ONCE(ctx->released))) {
521− wake_up_poll(&ctx->fd_wqh, EPOLLIN);
522− schedule();
523− }
524−
525− __set_current_state(TASK_RUNNING);
526−
527− /*
528− * Here we race with the list_del; list_add in
529− * userfaultfd_ctx_read(), however because we don't ever run
530− * list_del_init() to refile across the two lists, the prev
531− * and next pointers will never point to self. list_add also
532− * would never let any of the two pointers to point to
533− * self. So list_empty_careful won't risk to see both pointers
534− * pointing to self at any time during the list refile. The
535− * only case where list_del_init() is called is the full
536− * removal in the wake function and there we don't re-list_add
537− * and it's fine not to block on the spinlock. The uwq on this
538− * kernel stack can be released after the list_del_init.
539− */
540− if (!list_empty_careful(&uwq.wq.entry)) {
541− spin_lock_irq(&ctx->fault_pending_wqh.lock);
542− /*
543− * No need of list_del_init(), the uwq on the stack
544− * will be freed shortly anyway.
545− */
546− list_del(&uwq.wq.entry);
547− spin_unlock_irq(&ctx->fault_pending_wqh.lock);
548− }
549−
550− /*
551− * ctx may go away after this if the userfault pseudo fd is
552− * already released.
553− */
554− userfaultfd_ctx_put(ctx);
555−
556−out:
557− return ret;
558−}
559−
560−static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
561− struct userfaultfd_wait_queue *ewq)
562−{
563− struct userfaultfd_ctx *release_new_ctx;
564−
565− if (WARN_ON_ONCE(current->flags & PF_EXITING))
566− goto out;
567−
568− ewq->ctx = ctx;
569− init_waitqueue_entry(&ewq->wq, current);
570− release_new_ctx = NULL;
571−
572− spin_lock_irq(&ctx->event_wqh.lock);
573− /*
574− * After the __add_wait_queue the uwq is visible to userland
575− * through poll/read().
576− */
577− __add_wait_queue(&ctx->event_wqh, &ewq->wq);
578− for (;;) {
579− set_current_state(TASK_KILLABLE);
580− if (ewq->msg.event == 0)
581− break;
582− if (READ_ONCE(ctx->released) ||
583− fatal_signal_pending(current)) {
584− /*
585− * &ewq->wq may be queued in fork_event, but
586− * __remove_wait_queue ignores the head
587− * parameter. It would be a problem if it
588− * didn't.
589− */
590− __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
591− if (ewq->msg.event == UFFD_EVENT_FORK) {
592− struct userfaultfd_ctx *new;
593−
594− new = (struct userfaultfd_ctx *)
595− (unsigned long)
596− ewq->msg.arg.reserved.reserved1;
597− release_new_ctx = new;
598− }
599− break;
600− }
601−
602− spin_unlock_irq(&ctx->event_wqh.lock);
603−
604− wake_up_poll(&ctx->fd_wqh, EPOLLIN);
605− schedule();
606−
607− spin_lock_irq(&ctx->event_wqh.lock);
608− }
609− __set_current_state(TASK_RUNNING);
610− spin_unlock_irq(&ctx->event_wqh.lock);
611−
612− if (release_new_ctx) {
613− userfaultfd_release_new(release_new_ctx);
614− userfaultfd_ctx_put(release_new_ctx);
615− }
616−
617− /*
618− * ctx may go away after this if the userfault pseudo fd is
619− * already released.
620− */
621−out:
622− atomic_dec(&ctx->mmap_changing);
623− VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
624− userfaultfd_ctx_put(ctx);
625−}
626−
627−static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
628− struct userfaultfd_wait_queue *ewq)
629−{
630− ewq->msg.event = 0;
631− wake_up_locked(&ctx->event_wqh);
632− __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
633−}
634−
635−int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
636−{
637− struct userfaultfd_ctx *ctx = NULL, *octx;
638− struct userfaultfd_fork_ctx *fctx;
639−
640− octx = vma->vm_userfaultfd_ctx.ctx;
641− if (!octx)
642− return 0;
643−
644− if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
645− userfaultfd_reset_ctx(vma);
646− return 0;
647− }
648−
649− list_for_each_entry(fctx, fcs, list)
650− if (fctx->orig == octx) {
651− ctx = fctx->new;
652− break;
653− }
654−
655− if (!ctx) {
656− fctx = kmalloc_obj(*fctx);
657− if (!fctx)
658− return -ENOMEM;
659−
660− ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
661− if (!ctx) {
662− kfree(fctx);
663− return -ENOMEM;
664− }
665−
666− refcount_set(&ctx->refcount, 1);
667− ctx->flags = octx->flags;
668− ctx->features = octx->features;
669− ctx->released = false;
670− init_rwsem(&ctx->map_changing_lock);
671− atomic_set(&ctx->mmap_changing, 0);
672− ctx->mm = vma->vm_mm;
673− mmgrab(ctx->mm);
674−
675− userfaultfd_ctx_get(octx);
676− down_write(&octx->map_changing_lock);
677− atomic_inc(&octx->mmap_changing);
678− up_write(&octx->map_changing_lock);
679− fctx->orig = octx;
680− fctx->new = ctx;
681− list_add_tail(&fctx->list, fcs);
682− }
683−
684− vma->vm_userfaultfd_ctx.ctx = ctx;
685− return 0;
686−}
687−
688−static void dup_fctx(struct userfaultfd_fork_ctx *fctx)
689−{
690− struct userfaultfd_ctx *ctx = fctx->orig;
691− struct userfaultfd_wait_queue ewq;
692−
693− msg_init(&ewq.msg);
694−
695− ewq.msg.event = UFFD_EVENT_FORK;
696− ewq.msg.arg.reserved.reserved1 = (unsigned long)fctx->new;
697−
698− userfaultfd_event_wait_completion(ctx, &ewq);
699−}
700−
701−void dup_userfaultfd_complete(struct list_head *fcs)
702−{
703− struct userfaultfd_fork_ctx *fctx, *n;
704−
705− list_for_each_entry_safe(fctx, n, fcs, list) {
706− dup_fctx(fctx);
707− list_del(&fctx->list);
708− kfree(fctx);
709− }
710−}
711−
712−void dup_userfaultfd_fail(struct list_head *fcs)
713−{
714− struct userfaultfd_fork_ctx *fctx, *n;
715−
716− /*
717− * An error has occurred on fork, we will tear memory down, but have
718− * allocated memory for fctx's and raised reference counts for both the
719− * original and child contexts (and on the mm for each as a result).
720− *
721− * These would ordinarily be taken care of by a user handling the event,
722− * but we are no longer doing so, so manually clean up here.
723− *
724− * mm tear down will take care of cleaning up VMA contexts.
725− */
726− list_for_each_entry_safe(fctx, n, fcs, list) {
727− struct userfaultfd_ctx *octx = fctx->orig;
728− struct userfaultfd_ctx *ctx = fctx->new;
729−
730− atomic_dec(&octx->mmap_changing);
731− VM_WARN_ON_ONCE(atomic_read(&octx->mmap_changing) < 0);
732− userfaultfd_ctx_put(octx);
733− userfaultfd_ctx_put(ctx);
734−
735− list_del(&fctx->list);
736− kfree(fctx);
737− }
738−}
739−
740−void mremap_userfaultfd_prep(struct vm_area_struct *vma,
741− struct vm_userfaultfd_ctx *vm_ctx)
742−{
743− struct userfaultfd_ctx *ctx;
744−
745− ctx = vma->vm_userfaultfd_ctx.ctx;
746−
747− if (!ctx)
748− return;
749−
750− if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
751− vm_ctx->ctx = ctx;
752− userfaultfd_ctx_get(ctx);
753− down_write(&ctx->map_changing_lock);
754− atomic_inc(&ctx->mmap_changing);
755− up_write(&ctx->map_changing_lock);
756− } else {
757− /* Drop uffd context if remap feature not enabled */
758− userfaultfd_reset_ctx(vma);
759− }
760−}
761−
762−void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *vm_ctx,
763− unsigned long from, unsigned long to,
764− unsigned long len)
765−{
766− struct userfaultfd_ctx *ctx = vm_ctx->ctx;
767− struct userfaultfd_wait_queue ewq;
768−
769− if (!ctx)
770− return;
771−
772− msg_init(&ewq.msg);
773−
774− ewq.msg.event = UFFD_EVENT_REMAP;
775− ewq.msg.arg.remap.from = from;
776− ewq.msg.arg.remap.to = to;
777− ewq.msg.arg.remap.len = len;
778−
779− userfaultfd_event_wait_completion(ctx, &ewq);
780−}
781−
782−void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
783−{
784− struct userfaultfd_ctx *ctx = vm_ctx->ctx;
785−
786− if (!ctx)
787− return;
788−
789− userfaultfd_ctx_put(ctx);
790−}
791−
792−bool userfaultfd_remove(struct vm_area_struct *vma,
793− unsigned long start, unsigned long end)
794−{
795− struct mm_struct *mm = vma->vm_mm;
796− struct userfaultfd_ctx *ctx;
797− struct userfaultfd_wait_queue ewq;
798−
799− ctx = vma->vm_userfaultfd_ctx.ctx;
800− if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
801− return true;
802−
803− userfaultfd_ctx_get(ctx);
804− down_write(&ctx->map_changing_lock);
805− atomic_inc(&ctx->mmap_changing);
806− up_write(&ctx->map_changing_lock);
807− mmap_read_unlock(mm);
808−
809− msg_init(&ewq.msg);
810−
811− ewq.msg.event = UFFD_EVENT_REMOVE;
812− ewq.msg.arg.remove.start = start;
813− ewq.msg.arg.remove.end = end;
814−
815− userfaultfd_event_wait_completion(ctx, &ewq);
816−
817− return false;
818−}
819−
820−static bool has_unmap_ctx(struct userfaultfd_ctx *ctx, struct list_head *unmaps,
821− unsigned long start, unsigned long end)
822−{
823− struct userfaultfd_unmap_ctx *unmap_ctx;
824−
825− list_for_each_entry(unmap_ctx, unmaps, list)
826− if (unmap_ctx->ctx == ctx && unmap_ctx->start == start &&
827− unmap_ctx->end == end)
828− return true;
829−
830− return false;
831−}
832−
833−int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
834− unsigned long end, struct list_head *unmaps)
835−{
836− struct userfaultfd_unmap_ctx *unmap_ctx;
837− struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
838−
839− if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
840− has_unmap_ctx(ctx, unmaps, start, end))
841− return 0;
842−
843− unmap_ctx = kzalloc_obj(*unmap_ctx);
844− if (!unmap_ctx)
845− return -ENOMEM;
846−
847− userfaultfd_ctx_get(ctx);
848− down_write(&ctx->map_changing_lock);
849− atomic_inc(&ctx->mmap_changing);
850− up_write(&ctx->map_changing_lock);
851− unmap_ctx->ctx = ctx;
852− unmap_ctx->start = start;
853− unmap_ctx->end = end;
854− list_add_tail(&unmap_ctx->list, unmaps);
855−
856− return 0;
857−}
858−
859−void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf)
860−{
861− struct userfaultfd_unmap_ctx *ctx, *n;
862− struct userfaultfd_wait_queue ewq;
863−
864− list_for_each_entry_safe(ctx, n, uf, list) {
865− msg_init(&ewq.msg);
866−
867− ewq.msg.event = UFFD_EVENT_UNMAP;
868− ewq.msg.arg.remove.start = ctx->start;
869− ewq.msg.arg.remove.end = ctx->end;
870−
871− userfaultfd_event_wait_completion(ctx->ctx, &ewq);
872−
873− list_del(&ctx->list);
874− kfree(ctx);
875− }
876−}
877−
878−static int userfaultfd_release(struct inode *inode, struct file *file)
879−{
880− struct userfaultfd_ctx *ctx = file->private_data;
881− struct mm_struct *mm = ctx->mm;
882− /* len == 0 means wake all */
883− struct userfaultfd_wake_range range = { .len = 0, };
884−
885− WRITE_ONCE(ctx->released, true);
886−
887− userfaultfd_release_all(mm, ctx);
888−
889− /*
890− * After no new page faults can wait on this fault_*wqh, flush
891− * the last page faults that may have been already waiting on
892− * the fault_*wqh.
893− */
894− spin_lock_irq(&ctx->fault_pending_wqh.lock);
895− __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &range);
896− __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
897− spin_unlock_irq(&ctx->fault_pending_wqh.lock);
898−
899− /* Flush pending events that may still wait on event_wqh */
900− wake_up_all(&ctx->event_wqh);
901−
902− wake_up_poll(&ctx->fd_wqh, EPOLLHUP);
903− userfaultfd_ctx_put(ctx);
904− return 0;
905−}
906−
907−/* fault_pending_wqh.lock must be hold by the caller */
908−static inline struct userfaultfd_wait_queue *find_userfault_in(
909− wait_queue_head_t *wqh)
910−{
911− wait_queue_entry_t *wq;
912− struct userfaultfd_wait_queue *uwq;
913−
914− lockdep_assert_held(&wqh->lock);
915−
916− uwq = NULL;
917− if (!waitqueue_active(wqh))
918− goto out;
919− /* walk in reverse to provide FIFO behavior to read userfaults */
920− wq = list_last_entry(&wqh->head, typeof(*wq), entry);
921− uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
922−out:
923− return uwq;
924−}
925−
926−static inline struct userfaultfd_wait_queue *find_userfault(
927− struct userfaultfd_ctx *ctx)
928−{
929− return find_userfault_in(&ctx->fault_pending_wqh);
930−}
931−
932−static inline struct userfaultfd_wait_queue *find_userfault_evt(
933− struct userfaultfd_ctx *ctx)
934−{
935− return find_userfault_in(&ctx->event_wqh);
936−}
937−
938−static __poll_t userfaultfd_poll(struct file *file, poll_table *wait)
939−{
940− struct userfaultfd_ctx *ctx = file->private_data;
941− __poll_t ret;
942−
943− poll_wait(file, &ctx->fd_wqh, wait);
944−
945− if (!userfaultfd_is_initialized(ctx))
946− return EPOLLERR;
947−
948− /*
949− * poll() never guarantees that read won't block.
950− * userfaults can be waken before they're read().
951− */
952− if (unlikely(!(file->f_flags & O_NONBLOCK)))
953− return EPOLLERR;
954− /*
955− * lockless access to see if there are pending faults
956− * __pollwait last action is the add_wait_queue but
957− * the spin_unlock would allow the waitqueue_active to
958− * pass above the actual list_add inside
959− * add_wait_queue critical section. So use a full
960− * memory barrier to serialize the list_add write of
961− * add_wait_queue() with the waitqueue_active read
962− * below.
963− */
964− ret = 0;
965− smp_mb();
966− if (waitqueue_active(&ctx->fault_pending_wqh))
967− ret = EPOLLIN;
968− else if (waitqueue_active(&ctx->event_wqh))
969− ret = EPOLLIN;
970−
971− return ret;
972−}
973−
974−static const struct file_operations userfaultfd_fops;
975−
976−static int resolve_userfault_fork(struct userfaultfd_ctx *new,
977− struct inode *inode,
978− struct uffd_msg *msg)
979−{
980− int fd;
981−
982− fd = anon_inode_create_getfd("[userfaultfd]", &userfaultfd_fops, new,
983− O_RDONLY | (new->flags & UFFD_SHARED_FCNTL_FLAGS), inode);
984− if (fd < 0)
985− return fd;
986−
987− msg->arg.reserved.reserved1 = 0;
988− msg->arg.fork.ufd = fd;
989− return 0;
990−}
991−
992−static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
993− struct uffd_msg *msg, struct inode *inode)
994−{
995− ssize_t ret;
996− DECLARE_WAITQUEUE(wait, current);
997− struct userfaultfd_wait_queue *uwq;
998− /*
999− * Handling fork event requires sleeping operations, so
1000− * we drop the event_wqh lock, then do these ops, then
1001− * lock it back and wake up the waiter. While the lock is
1002− * dropped the ewq may go away so we keep track of it
1003− * carefully.
1004− */
1005− LIST_HEAD(fork_event);
1006− struct userfaultfd_ctx *fork_nctx = NULL;
1007−
1008− /* always take the fd_wqh lock before the fault_pending_wqh lock */
1009− spin_lock_irq(&ctx->fd_wqh.lock);
1010− __add_wait_queue(&ctx->fd_wqh, &wait);
1011− for (;;) {
1012− set_current_state(TASK_INTERRUPTIBLE);
1013− spin_lock(&ctx->fault_pending_wqh.lock);
1014− uwq = find_userfault(ctx);
1015− if (uwq) {
1016− /*
1017− * Use a seqcount to repeat the lockless check
1018− * in wake_userfault() to avoid missing
1019− * wakeups because during the refile both
1020− * waitqueue could become empty if this is the
1021− * only userfault.
1022− */
1023− write_seqcount_begin(&ctx->refile_seq);
1024−
1025− /*
1026− * The fault_pending_wqh.lock prevents the uwq
1027− * to disappear from under us.
1028− *
1029− * Refile this userfault from
1030− * fault_pending_wqh to fault_wqh, it's not
1031− * pending anymore after we read it.
1032− *
1033− * Use list_del() by hand (as
1034− * userfaultfd_wake_function also uses
1035− * list_del_init() by hand) to be sure nobody
1036− * changes __remove_wait_queue() to use
1037− * list_del_init() in turn breaking the
1038− * !list_empty_careful() check in
1039− * handle_userfault(). The uwq->wq.head list
1040− * must never be empty at any time during the
1041− * refile, or the waitqueue could disappear
1042− * from under us. The "wait_queue_head_t"
1043− * parameter of __remove_wait_queue() is unused
1044− * anyway.
1045− */
1046− list_del(&uwq->wq.entry);
1047− add_wait_queue(&ctx->fault_wqh, &uwq->wq);
1048−
1049− write_seqcount_end(&ctx->refile_seq);
1050−
1051− /* careful to always initialize msg if ret == 0 */
1052− *msg = uwq->msg;
1053− spin_unlock(&ctx->fault_pending_wqh.lock);
1054− ret = 0;
1055− break;
1056− }
1057− spin_unlock(&ctx->fault_pending_wqh.lock);
1058−
1059− spin_lock(&ctx->event_wqh.lock);
1060− uwq = find_userfault_evt(ctx);
1061− if (uwq) {
1062− *msg = uwq->msg;
1063−
1064− if (uwq->msg.event == UFFD_EVENT_FORK) {
1065− fork_nctx = (struct userfaultfd_ctx *)
1066− (unsigned long)
1067− uwq->msg.arg.reserved.reserved1;
1068− list_move(&uwq->wq.entry, &fork_event);
1069− /*
1070− * fork_nctx can be freed as soon as
1071− * we drop the lock, unless we take a
1072− * reference on it.
1073− */
1074− userfaultfd_ctx_get(fork_nctx);
1075− spin_unlock(&ctx->event_wqh.lock);
1076− ret = 0;
1077− break;
1078− }
1079−
1080− userfaultfd_event_complete(ctx, uwq);
1081− spin_unlock(&ctx->event_wqh.lock);
1082− ret = 0;
1083− break;
1084− }
1085− spin_unlock(&ctx->event_wqh.lock);
1086−
1087− if (signal_pending(current)) {
1088− ret = -ERESTARTSYS;
1089− break;
1090− }
1091− if (no_wait) {
1092− ret = -EAGAIN;
1093− break;
1094− }
1095− spin_unlock_irq(&ctx->fd_wqh.lock);
1096− schedule();
1097− spin_lock_irq(&ctx->fd_wqh.lock);
1098− }
1099− __remove_wait_queue(&ctx->fd_wqh, &wait);
1100− __set_current_state(TASK_RUNNING);
1101− spin_unlock_irq(&ctx->fd_wqh.lock);
1102−
1103− if (!ret && msg->event == UFFD_EVENT_FORK) {
1104− ret = resolve_userfault_fork(fork_nctx, inode, msg);
1105− spin_lock_irq(&ctx->event_wqh.lock);
1106− if (!list_empty(&fork_event)) {
1107− /*
1108− * The fork thread didn't abort, so we can
1109− * drop the temporary refcount.
1110− */
1111− userfaultfd_ctx_put(fork_nctx);
1112−
1113− uwq = list_first_entry(&fork_event,
1114− typeof(*uwq),
1115− wq.entry);
1116− /*
1117− * If fork_event list wasn't empty and in turn
1118− * the event wasn't already released by fork
1119− * (the event is allocated on fork kernel
1120− * stack), put the event back to its place in
1121− * the event_wq. fork_event head will be freed
1122− * as soon as we return so the event cannot
1123− * stay queued there no matter the current
1124− * "ret" value.
1125− */
1126− list_del(&uwq->wq.entry);
1127− __add_wait_queue(&ctx->event_wqh, &uwq->wq);
1128−
1129− /*
1130− * Leave the event in the waitqueue and report
1131− * error to userland if we failed to resolve
1132− * the userfault fork.
1133− */
1134− if (likely(!ret))
1135− userfaultfd_event_complete(ctx, uwq);
1136− } else {
1137− /*
1138− * Here the fork thread aborted and the
1139− * refcount from the fork thread on fork_nctx
1140− * has already been released. We still hold
1141− * the reference we took before releasing the
1142− * lock above. If resolve_userfault_fork
1143− * failed we've to drop it because the
1144− * fork_nctx has to be freed in such case. If
1145− * it succeeded we'll hold it because the new
1146− * uffd references it.
1147− */
1148− if (ret)
1149− userfaultfd_ctx_put(fork_nctx);
1150− }
1151− spin_unlock_irq(&ctx->event_wqh.lock);
1152− }
1153−
1154− return ret;
1155−}
1156−
1157−static ssize_t userfaultfd_read_iter(struct kiocb *iocb, struct iov_iter *to)
1158−{
1159− struct file *file = iocb->ki_filp;
1160− struct userfaultfd_ctx *ctx = file->private_data;
1161− ssize_t _ret, ret = 0;
1162− struct uffd_msg msg;
1163− struct inode *inode = file_inode(file);
1164− bool no_wait;
1165−
1166− if (!userfaultfd_is_initialized(ctx))
1167− return -EINVAL;
1168−
1169− no_wait = file->f_flags & O_NONBLOCK || iocb->ki_flags & IOCB_NOWAIT;
1170− for (;;) {
1171− if (iov_iter_count(to) < sizeof(msg))
1172− return ret ? ret : -EINVAL;
1173− _ret = userfaultfd_ctx_read(ctx, no_wait, &msg, inode);
1174− if (_ret < 0)
1175− return ret ? ret : _ret;
1176− _ret = !copy_to_iter_full(&msg, sizeof(msg), to);
1177− if (_ret)
1178− return ret ? ret : -EFAULT;
1179− ret += sizeof(msg);
1180− /*
1181− * Allow to read more than one fault at time but only
1182− * block if waiting for the very first one.
1183− */
1184− no_wait = true;
1185− }
1186−}
1187−
1188−static void __wake_userfault(struct userfaultfd_ctx *ctx,
1189− struct userfaultfd_wake_range *range)
1190−{
1191− spin_lock_irq(&ctx->fault_pending_wqh.lock);
1192− /* wake all in the range and autoremove */
1193− if (waitqueue_active(&ctx->fault_pending_wqh))
1194− __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
1195− range);
1196− if (waitqueue_active(&ctx->fault_wqh))
1197− __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, range);
1198− spin_unlock_irq(&ctx->fault_pending_wqh.lock);
1199−}
1200−
1201−static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
1202− struct userfaultfd_wake_range *range)
1203−{
1204− unsigned seq;
1205− bool need_wakeup;
1206−
1207− /*
1208− * To be sure waitqueue_active() is not reordered by the CPU
1209− * before the pagetable update, use an explicit SMP memory
1210− * barrier here. PT lock release or mmap_read_unlock(mm) still
1211− * have release semantics that can allow the
1212− * waitqueue_active() to be reordered before the pte update.
1213− */
1214− smp_mb();
1215−
1216− /*
1217− * Use waitqueue_active because it's very frequent to
1218− * change the address space atomically even if there are no
1219− * userfaults yet. So we take the spinlock only when we're
1220− * sure we've userfaults to wake.
1221− */
1222− do {
1223− seq = read_seqcount_begin(&ctx->refile_seq);
1224− need_wakeup = waitqueue_active(&ctx->fault_pending_wqh) ||
1225− waitqueue_active(&ctx->fault_wqh);
1226− cond_resched();
1227− } while (read_seqcount_retry(&ctx->refile_seq, seq));
1228− if (need_wakeup)
1229− __wake_userfault(ctx, range);
1230−}
1231−
1232−static __always_inline int validate_unaligned_range(
1233− struct mm_struct *mm, __u64 start, __u64 len)
1234−{
1235− __u64 task_size = mm->task_size;
1236−
1237− if (len & ~PAGE_MASK)
1238− return -EINVAL;
1239− if (!len)
1240− return -EINVAL;
1241− if (start >= task_size)
1242− return -EINVAL;
1243− if (len > task_size - start)
1244− return -EINVAL;
1245− if (start + len <= start)
1246− return -EINVAL;
1247− return 0;
1248−}
1249−
1250−static __always_inline int validate_range(struct mm_struct *mm,
1251− __u64 start, __u64 len)
1252−{
1253− if (start & ~PAGE_MASK)
1254− return -EINVAL;
1255−
1256− return validate_unaligned_range(mm, start, len);
1257−}
1258−
1259−static int userfaultfd_register(struct userfaultfd_ctx *ctx,
1260− unsigned long arg)
1261−{
1262− struct mm_struct *mm = ctx->mm;
1263− struct vm_area_struct *vma, *cur;
1264− int ret;
1265− struct uffdio_register uffdio_register;
1266− struct uffdio_register __user *user_uffdio_register;
1267− vm_flags_t vm_flags;
1268− bool found;
1269− bool basic_ioctls;
1270− unsigned long start, end;
1271− struct vma_iterator vmi;
1272− bool wp_async = userfaultfd_wp_async_ctx(ctx);
1273−
1274− user_uffdio_register = (struct uffdio_register __user *) arg;
1275−
1276− ret = -EFAULT;
1277− if (copy_from_user(&uffdio_register, user_uffdio_register,
1278− sizeof(uffdio_register)-sizeof(__u64)))
1279− goto out;
1280−
1281− ret = -EINVAL;
1282− if (!uffdio_register.mode)
1283− goto out;
1284− if (uffdio_register.mode & ~UFFD_API_REGISTER_MODES)
1285− goto out;
1286− vm_flags = 0;
1287− if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
1288− vm_flags |= VM_UFFD_MISSING;
1289− if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
1290− if (!pgtable_supports_uffd_wp())
1291− goto out;
1292−
1293− vm_flags |= VM_UFFD_WP;
1294− }
1295− if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
1296−#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
1297− goto out;
1298−#endif
1299− vm_flags |= VM_UFFD_MINOR;
1300− }
1301−
1302− ret = validate_range(mm, uffdio_register.range.start,
1303− uffdio_register.range.len);
1304− if (ret)
1305− goto out;
1306−
1307− start = uffdio_register.range.start;
1308− end = start + uffdio_register.range.len;
1309−
1310− ret = -ENOMEM;
1311− if (!mmget_not_zero(mm))
1312− goto out;
1313−
1314− ret = -EINVAL;
1315− mmap_write_lock(mm);
1316− vma_iter_init(&vmi, mm, start);
1317− vma = vma_find(&vmi, end);
1318− if (!vma)
1319− goto out_unlock;
1320−
1321− /*
1322− * If the first vma contains huge pages, make sure start address
1323− * is aligned to huge page size.
1324− */
1325− if (is_vm_hugetlb_page(vma)) {
1326− unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
1327−
1328− if (start & (vma_hpagesize - 1))
1329− goto out_unlock;
1330− }
1331−
1332− /*
1333− * Search for not compatible vmas.
1334− */
1335− found = false;
1336− basic_ioctls = false;
1337− cur = vma;
1338− do {
1339− cond_resched();
1340−
1341− VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
1342− !!(cur->vm_flags & __VM_UFFD_FLAGS));
1343−
1344− /* check not compatible vmas */
1345− ret = -EINVAL;
1346− if (!vma_can_userfault(cur, vm_flags, wp_async))
1347− goto out_unlock;
1348−
1349− /*
1350− * UFFDIO_COPY will fill file holes even without
1351− * PROT_WRITE. This check enforces that if this is a
1352− * MAP_SHARED, the process has write permission to the backing
1353− * file. If VM_MAYWRITE is set it also enforces that on a
1354− * MAP_SHARED vma: there is no F_WRITE_SEAL and no further
1355− * F_WRITE_SEAL can be taken until the vma is destroyed.
1356− */
1357− ret = -EPERM;
1358− if (unlikely(!(cur->vm_flags & VM_MAYWRITE)))
1359− goto out_unlock;
1360−
1361− /*
1362− * If this vma contains ending address, and huge pages
1363− * check alignment.
1364− */
1365− if (is_vm_hugetlb_page(cur) && end <= cur->vm_end &&
1366− end > cur->vm_start) {
1367− unsigned long vma_hpagesize = vma_kernel_pagesize(cur);
1368−
1369− ret = -EINVAL;
1370−
1371− if (end & (vma_hpagesize - 1))
1372− goto out_unlock;
1373− }
1374− if ((vm_flags & VM_UFFD_WP) && !(cur->vm_flags & VM_MAYWRITE))
1375− goto out_unlock;
1376−
1377− /*
1378− * Check that this vma isn't already owned by a
1379− * different userfaultfd. We can't allow more than one
1380− * userfaultfd to own a single vma simultaneously or we
1381− * wouldn't know which one to deliver the userfaults to.
1382− */
1383− ret = -EBUSY;
1384− if (cur->vm_userfaultfd_ctx.ctx &&
1385− cur->vm_userfaultfd_ctx.ctx != ctx)
1386− goto out_unlock;
1387−
1388− /*
1389− * Note vmas containing huge pages
1390− */
1391− if (is_vm_hugetlb_page(cur))
1392− basic_ioctls = true;
1393−
1394− found = true;
1395− } for_each_vma_range(vmi, cur, end);
1396− VM_WARN_ON_ONCE(!found);
1397−
1398− ret = userfaultfd_register_range(ctx, vma, vm_flags, start, end,
1399− wp_async);
1400−
1401−out_unlock:
1402− mmap_write_unlock(mm);
1403− mmput(mm);
1404− if (!ret) {
1405− __u64 ioctls_out;
1406−
1407− ioctls_out = basic_ioctls ? UFFD_API_RANGE_IOCTLS_BASIC :
1408− UFFD_API_RANGE_IOCTLS;
1409−
1410− /*
1411− * Declare the WP ioctl only if the WP mode is
1412− * specified and all checks passed with the range
1413− */
1414− if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_WP))
1415− ioctls_out &= ~((__u64)1 << _UFFDIO_WRITEPROTECT);
1416−
1417− /* CONTINUE ioctl is only supported for MINOR ranges. */
1418− if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
1419− ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
1420−
1421− /*
1422− * Now that we scanned all vmas we can already tell
1423− * userland which ioctls methods are guaranteed to
1424− * succeed on this range.
1425− */
1426− if (put_user(ioctls_out, &user_uffdio_register->ioctls))
1427− ret = -EFAULT;
1428− }
1429−out:
1430− return ret;
1431−}
1432−
1433−static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
1434− unsigned long arg)
1435−{
1436− struct mm_struct *mm = ctx->mm;
1437− struct vm_area_struct *vma, *prev, *cur;
1438− int ret;
1439− struct uffdio_range uffdio_unregister;
1440− bool found;
1441− unsigned long start, end, vma_end;
1442− const void __user *buf = (void __user *)arg;
1443− struct vma_iterator vmi;
1444− bool wp_async = userfaultfd_wp_async_ctx(ctx);
1445−
1446− ret = -EFAULT;
1447− if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
1448− goto out;
1449−
1450− ret = validate_range(mm, uffdio_unregister.start,
1451− uffdio_unregister.len);
1452− if (ret)
1453− goto out;
1454−
1455− start = uffdio_unregister.start;
1456− end = start + uffdio_unregister.len;
1457−
1458− ret = -ENOMEM;
1459− if (!mmget_not_zero(mm))
1460− goto out;
1461−
1462− mmap_write_lock(mm);
1463− ret = -EINVAL;
1464− vma_iter_init(&vmi, mm, start);
1465− vma = vma_find(&vmi, end);
1466− if (!vma)
1467− goto out_unlock;
1468−
1469− /*
1470− * If the first vma contains huge pages, make sure start address
1471− * is aligned to huge page size.
1472− */
1473− if (is_vm_hugetlb_page(vma)) {
1474− unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
1475−
1476− if (start & (vma_hpagesize - 1))
1477− goto out_unlock;
1478− }
1479−
1480− /*
1481− * Search for not compatible vmas.
1482− */
1483− found = false;
1484− cur = vma;
1485− do {
1486− cond_resched();
1487−
1488− VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
1489− !!(cur->vm_flags & __VM_UFFD_FLAGS));
1490−
1491− /*
1492− * Prevent unregistering through a different userfaultfd than
1493− * the one used for registration.
1494− */
1495− if (cur->vm_userfaultfd_ctx.ctx &&
1496− cur->vm_userfaultfd_ctx.ctx != ctx)
1497− goto out_unlock;
1498−
1499− /*
1500− * Check not compatible vmas, not strictly required
1501− * here as not compatible vmas cannot have an
1502− * userfaultfd_ctx registered on them, but this
1503− * provides for more strict behavior to notice
1504− * unregistration errors.
1505− */
1506− if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
1507− goto out_unlock;
1508−
1509− found = true;
1510− } for_each_vma_range(vmi, cur, end);
1511− VM_WARN_ON_ONCE(!found);
1512−
1513− vma_iter_set(&vmi, start);
1514− prev = vma_prev(&vmi);
1515− if (vma->vm_start < start)
1516− prev = vma;
1517−
1518− ret = 0;
1519− for_each_vma_range(vmi, vma, end) {
1520− cond_resched();
1521−
1522− /* VMA not registered with userfaultfd. */
1523− if (!vma->vm_userfaultfd_ctx.ctx)
1524− goto skip;
1525−
1526− VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx != ctx);
1527− VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, wp_async));
1528− VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
1529−
1530− if (vma->vm_start > start)
1531− start = vma->vm_start;
1532− vma_end = min(end, vma->vm_end);
1533−
1534− if (userfaultfd_missing(vma)) {
1535− /*
1536− * Wake any concurrent pending userfault while
1537− * we unregister, so they will not hang
1538− * permanently and it avoids userland to call
1539− * UFFDIO_WAKE explicitly.
1540− */
1541− struct userfaultfd_wake_range range;
1542− range.start = start;
1543− range.len = vma_end - start;
1544− wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range);
1545− }
1546−
1547− vma = userfaultfd_clear_vma(&vmi, prev, vma,
1548− start, vma_end);
1549− if (IS_ERR(vma)) {
1550− ret = PTR_ERR(vma);
1551− break;
1552− }
1553−
1554− skip:
1555− prev = vma;
1556− start = vma->vm_end;
1557− }
1558−
1559−out_unlock:
1560− mmap_write_unlock(mm);
1561− mmput(mm);
1562−out:
1563− return ret;
1564−}
1565−
1566−/*
1567− * userfaultfd_wake may be used in combination with the
1568− * UFFDIO_*_MODE_DONTWAKE to wakeup userfaults in batches.
1569− */
1570−static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
1571− unsigned long arg)
1572−{
1573− int ret;
1574− struct uffdio_range uffdio_wake;
1575− struct userfaultfd_wake_range range;
1576− const void __user *buf = (void __user *)arg;
1577−
1578− ret = -EFAULT;
1579− if (copy_from_user(&uffdio_wake, buf, sizeof(uffdio_wake)))
1580− goto out;
1581−
1582− ret = validate_range(ctx->mm, uffdio_wake.start, uffdio_wake.len);
1583− if (ret)
1584− goto out;
1585−
1586− range.start = uffdio_wake.start;
1587− range.len = uffdio_wake.len;
1588−
1589− /*
1590− * len == 0 means wake all and we don't want to wake all here,
1591− * so check it again to be sure.
1592− */
1593− VM_WARN_ON_ONCE(!range.len);
1594−
1595− wake_userfault(ctx, &range);
1596− ret = 0;
1597−
1598−out:
1599− return ret;
1600−}
1601−
1602−static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
1603− unsigned long arg)
1604−{
1605− __s64 ret;
1606− struct uffdio_copy uffdio_copy;
1607− struct uffdio_copy __user *user_uffdio_copy;
1608− struct userfaultfd_wake_range range;
1609− uffd_flags_t flags = 0;
1610−
1611− user_uffdio_copy = (struct uffdio_copy __user *) arg;
1612−
1613− ret = -EAGAIN;
1614− if (unlikely(atomic_read(&ctx->mmap_changing))) {
1615− if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
1616− return -EFAULT;
1617− goto out;
1618− }
1619−
1620− ret = -EFAULT;
1621− if (copy_from_user(&uffdio_copy, user_uffdio_copy,
1622− /* don't copy "copy" last field */
1623− sizeof(uffdio_copy)-sizeof(__s64)))
1624− goto out;
1625−
1626− ret = validate_unaligned_range(ctx->mm, uffdio_copy.src,
1627− uffdio_copy.len);
1628− if (ret)
1629− goto out;
1630− ret = validate_range(ctx->mm, uffdio_copy.dst, uffdio_copy.len);
1631− if (ret)
1632− goto out;
1633−
1634− ret = -EINVAL;
1635− if (uffdio_copy.mode & ~(UFFDIO_COPY_MODE_DONTWAKE|UFFDIO_COPY_MODE_WP))
1636− goto out;
1637− if (uffdio_copy.mode & UFFDIO_COPY_MODE_WP)
1638− flags |= MFILL_ATOMIC_WP;
1639− if (mmget_not_zero(ctx->mm)) {
1640− ret = mfill_atomic_copy(ctx, uffdio_copy.dst, uffdio_copy.src,
1641− uffdio_copy.len, flags);
1642− mmput(ctx->mm);
1643− } else {
1644− return -ESRCH;
1645− }
1646− if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
1647− return -EFAULT;
1648− if (ret < 0)
1649− goto out;
1650− VM_WARN_ON_ONCE(!ret);
1651− /* len == 0 would wake all */
1652− range.len = ret;
1653− if (!(uffdio_copy.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
1654− range.start = uffdio_copy.dst;
1655− wake_userfault(ctx, &range);
1656− }
1657− ret = range.len == uffdio_copy.len ? 0 : -EAGAIN;
1658−out:
1659− return ret;
1660−}
1661−
1662−static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
1663− unsigned long arg)
1664−{
1665− __s64 ret;
1666− struct uffdio_zeropage uffdio_zeropage;
1667− struct uffdio_zeropage __user *user_uffdio_zeropage;
1668− struct userfaultfd_wake_range range;
1669−
1670− user_uffdio_zeropage = (struct uffdio_zeropage __user *) arg;
1671−
1672− ret = -EAGAIN;
1673− if (unlikely(atomic_read(&ctx->mmap_changing))) {
1674− if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
1675− return -EFAULT;
1676− goto out;
1677− }
1678−
1679− ret = -EFAULT;
1680− if (copy_from_user(&uffdio_zeropage, user_uffdio_zeropage,
1681− /* don't copy "zeropage" last field */
1682− sizeof(uffdio_zeropage)-sizeof(__s64)))
1683− goto out;
1684−
1685− ret = validate_range(ctx->mm, uffdio_zeropage.range.start,
1686− uffdio_zeropage.range.len);
1687− if (ret)
1688− goto out;
1689− ret = -EINVAL;
1690− if (uffdio_zeropage.mode & ~UFFDIO_ZEROPAGE_MODE_DONTWAKE)
1691− goto out;
1692−
1693− if (mmget_not_zero(ctx->mm)) {
1694− ret = mfill_atomic_zeropage(ctx, uffdio_zeropage.range.start,
1695− uffdio_zeropage.range.len);
1696− mmput(ctx->mm);
1697− } else {
1698− return -ESRCH;
1699− }
1700− if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
1701− return -EFAULT;
1702− if (ret < 0)
1703− goto out;
1704− /* len == 0 would wake all */
1705− VM_WARN_ON_ONCE(!ret);
1706− range.len = ret;
1707− if (!(uffdio_zeropage.mode & UFFDIO_ZEROPAGE_MODE_DONTWAKE)) {
1708− range.start = uffdio_zeropage.range.start;
1709− wake_userfault(ctx, &range);
1710− }
1711− ret = range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
1712−out:
1713− return ret;
1714−}
1715−
1716−static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
1717− unsigned long arg)
1718−{
1719− int ret;
1720− struct uffdio_writeprotect uffdio_wp;
1721− struct uffdio_writeprotect __user *user_uffdio_wp;
1722− struct userfaultfd_wake_range range;
1723− bool mode_wp, mode_dontwake;
1724−
1725− if (atomic_read(&ctx->mmap_changing))
1726− return -EAGAIN;
1727−
1728− user_uffdio_wp = (struct uffdio_writeprotect __user *) arg;
1729−
1730− if (copy_from_user(&uffdio_wp, user_uffdio_wp,
1731− sizeof(struct uffdio_writeprotect)))
1732− return -EFAULT;
1733−
1734− ret = validate_range(ctx->mm, uffdio_wp.range.start,
1735− uffdio_wp.range.len);
1736− if (ret)
1737− return ret;
1738−
1739− if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE |
1740− UFFDIO_WRITEPROTECT_MODE_WP))
1741− return -EINVAL;
1742−
1743− mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
1744− mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
1745−
1746− if (mode_wp && mode_dontwake)
1747− return -EINVAL;
1748−
1749− if (mmget_not_zero(ctx->mm)) {
1750− ret = mwriteprotect_range(ctx, uffdio_wp.range.start,
1751− uffdio_wp.range.len, mode_wp);
1752− mmput(ctx->mm);
1753− } else {
1754− return -ESRCH;
1755− }
1756−
1757− if (ret)
1758− return ret;
1759−
1760− if (!mode_wp && !mode_dontwake) {
1761− range.start = uffdio_wp.range.start;
1762− range.len = uffdio_wp.range.len;
1763− wake_userfault(ctx, &range);
1764− }
1765− return ret;
1766−}
1767−
1768−static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
1769−{
1770− __s64 ret;
1771− struct uffdio_continue uffdio_continue;
1772− struct uffdio_continue __user *user_uffdio_continue;
1773− struct userfaultfd_wake_range range;
1774− uffd_flags_t flags = 0;
1775−
1776− user_uffdio_continue = (struct uffdio_continue __user *)arg;
1777−
1778− ret = -EAGAIN;
1779− if (unlikely(atomic_read(&ctx->mmap_changing))) {
1780− if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
1781− return -EFAULT;
1782− goto out;
1783− }
1784−
1785− ret = -EFAULT;
1786− if (copy_from_user(&uffdio_continue, user_uffdio_continue,
1787− /* don't copy the output fields */
1788− sizeof(uffdio_continue) - (sizeof(__s64))))
1789− goto out;
1790−
1791− ret = validate_range(ctx->mm, uffdio_continue.range.start,
1792− uffdio_continue.range.len);
1793− if (ret)
1794− goto out;
1795−
1796− ret = -EINVAL;
1797− if (uffdio_continue.mode & ~(UFFDIO_CONTINUE_MODE_DONTWAKE |
1798− UFFDIO_CONTINUE_MODE_WP))
1799− goto out;
1800− if (uffdio_continue.mode & UFFDIO_CONTINUE_MODE_WP)
1801− flags |= MFILL_ATOMIC_WP;
1802−
1803− if (mmget_not_zero(ctx->mm)) {
1804− ret = mfill_atomic_continue(ctx, uffdio_continue.range.start,
1805− uffdio_continue.range.len, flags);
1806− mmput(ctx->mm);
1807− } else {
1808− return -ESRCH;
1809− }
1810−
1811− if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
1812− return -EFAULT;
1813− if (ret < 0)
1814− goto out;
1815−
1816− /* len == 0 would wake all */
1817− VM_WARN_ON_ONCE(!ret);
1818− range.len = ret;
1819− if (!(uffdio_continue.mode & UFFDIO_CONTINUE_MODE_DONTWAKE)) {
1820− range.start = uffdio_continue.range.start;
1821− wake_userfault(ctx, &range);
1822− }
1823− ret = range.len == uffdio_continue.range.len ? 0 : -EAGAIN;
1824−
1825−out:
1826− return ret;
1827−}
1828−
1829−static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long arg)
1830−{
1831− __s64 ret;
1832− struct uffdio_poison uffdio_poison;
1833− struct uffdio_poison __user *user_uffdio_poison;
1834− struct userfaultfd_wake_range range;
1835−
1836− user_uffdio_poison = (struct uffdio_poison __user *)arg;
1837−
1838− ret = -EAGAIN;
1839− if (unlikely(atomic_read(&ctx->mmap_changing))) {
1840− if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
1841− return -EFAULT;
1842− goto out;
1843− }
1844−
1845− ret = -EFAULT;
1846− if (copy_from_user(&uffdio_poison, user_uffdio_poison,
1847− /* don't copy the output fields */
1848− sizeof(uffdio_poison) - (sizeof(__s64))))
1849− goto out;
1850−
1851− ret = validate_range(ctx->mm, uffdio_poison.range.start,
1852− uffdio_poison.range.len);
1853− if (ret)
1854− goto out;
1855−
1856− ret = -EINVAL;
1857− if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE)
1858− goto out;
1859−
1860− if (mmget_not_zero(ctx->mm)) {
1861− ret = mfill_atomic_poison(ctx, uffdio_poison.range.start,
1862− uffdio_poison.range.len, 0);
1863− mmput(ctx->mm);
1864− } else {
1865− return -ESRCH;
1866− }
1867−
1868− if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
1869− return -EFAULT;
1870− if (ret < 0)
1871− goto out;
1872−
1873− /* len == 0 would wake all */
1874− VM_WARN_ON_ONCE(!ret);
1875− range.len = ret;
1876− if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) {
1877− range.start = uffdio_poison.range.start;
1878− wake_userfault(ctx, &range);
1879− }
1880− ret = range.len == uffdio_poison.range.len ? 0 : -EAGAIN;
1881−
1882−out:
1883− return ret;
1884−}
1885−
1886−bool userfaultfd_wp_async(struct vm_area_struct *vma)
1887−{
1888− return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
1889−}
1890−
1891−static inline unsigned int uffd_ctx_features(__u64 user_features)
1892−{
1893− /*
1894− * For the current set of features the bits just coincide. Set
1895− * UFFD_FEATURE_INITIALIZED to mark the features as enabled.
1896− */
1897− return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
1898−}
1899−
1900−static int userfaultfd_move(struct userfaultfd_ctx *ctx,
1901− unsigned long arg)
1902−{
1903− __s64 ret;
1904− struct uffdio_move uffdio_move;
1905− struct uffdio_move __user *user_uffdio_move;
1906− struct userfaultfd_wake_range range;
1907− struct mm_struct *mm = ctx->mm;
1908−
1909− user_uffdio_move = (struct uffdio_move __user *) arg;
1910−
1911− ret = -EAGAIN;
1912− if (unlikely(atomic_read(&ctx->mmap_changing))) {
1913− if (unlikely(put_user(ret, &user_uffdio_move->move)))
1914− return -EFAULT;
1915− goto out;
1916− }
1917−
1918− if (copy_from_user(&uffdio_move, user_uffdio_move,
1919− /* don't copy "move" last field */
1920− sizeof(uffdio_move)-sizeof(__s64)))
1921− return -EFAULT;
1922−
1923− /* Do not allow cross-mm moves. */
1924− if (mm != current->mm)
1925− return -EINVAL;
1926−
1927− ret = validate_range(mm, uffdio_move.dst, uffdio_move.len);
1928− if (ret)
1929− return ret;
1930−
1931− ret = validate_range(mm, uffdio_move.src, uffdio_move.len);
1932− if (ret)
1933− return ret;
1934−
1935− if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES|
1936− UFFDIO_MOVE_MODE_DONTWAKE))
1937− return -EINVAL;
1938−
1939− if (mmget_not_zero(mm)) {
1940− ret = move_pages(ctx, uffdio_move.dst, uffdio_move.src,
1941− uffdio_move.len, uffdio_move.mode);
1942− mmput(mm);
1943− } else {
1944− return -ESRCH;
1945− }
1946−
1947− if (unlikely(put_user(ret, &user_uffdio_move->move)))
1948− return -EFAULT;
1949− if (ret < 0)
1950− goto out;
1951−
1952− /* len == 0 would wake all */
1953− VM_WARN_ON(!ret);
1954− range.len = ret;
1955− if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) {
1956− range.start = uffdio_move.dst;
1957− wake_userfault(ctx, &range);
1958− }
1959− ret = range.len == uffdio_move.len ? 0 : -EAGAIN;
1960−
1961−out:
1962− return ret;
1963−}
1964−
1965−/*
1966− * userland asks for a certain API version and we return which bits
1967− * and ioctl commands are implemented in this kernel for such API
1968− * version or -EINVAL if unknown.
1969− */
1970−static int userfaultfd_api(struct userfaultfd_ctx *ctx,
1971− unsigned long arg)
1972−{
1973− struct uffdio_api uffdio_api;
1974− void __user *buf = (void __user *)arg;
1975− unsigned int ctx_features;
1976− int ret;
1977− __u64 features;
1978−
1979− ret = -EFAULT;
1980− if (copy_from_user(&uffdio_api, buf, sizeof(uffdio_api)))
1981− goto out;
1982− features = uffdio_api.features;
1983− ret = -EINVAL;
1984− if (uffdio_api.api != UFFD_API)
1985− goto err_out;
1986− ret = -EPERM;
1987− if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
1988− goto err_out;
1989−
1990− /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
1991− if (features & UFFD_FEATURE_WP_ASYNC)
1992− features |= UFFD_FEATURE_WP_UNPOPULATED;
1993−
1994− /* report all available features and ioctls to userland */
1995− uffdio_api.features = UFFD_API_FEATURES;
1996−#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
1997− uffdio_api.features &=
1998− ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
1999−#endif
2000− if (!pgtable_supports_uffd_wp())
2001− uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
2002−
2003− if (!uffd_supports_wp_marker()) {
2004− uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
2005− uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
2006− uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
2007− }
2008−
2009− ret = -EINVAL;
2010− if (features & ~uffdio_api.features)
2011− goto err_out;
2012−
2013− uffdio_api.ioctls = UFFD_API_IOCTLS;
2014− ret = -EFAULT;
2015− if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
2016− goto out;
2017−
2018− /* only enable the requested features for this uffd context */
2019− ctx_features = uffd_ctx_features(features);
2020− ret = -EINVAL;
2021− if (cmpxchg(&ctx->features, 0, ctx_features) != 0)
2022− goto err_out;
2023−
2024− ret = 0;
2025−out:
2026− return ret;
2027−err_out:
2028− memset(&uffdio_api, 0, sizeof(uffdio_api));
2029− if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
2030− ret = -EFAULT;
2031− goto out;
2032−}
2033−
2034−static long userfaultfd_ioctl(struct file *file, unsigned cmd,
2035− unsigned long arg)
2036−{
2037− int ret = -EINVAL;
2038− struct userfaultfd_ctx *ctx = file->private_data;
2039−
2040− if (cmd != UFFDIO_API && !userfaultfd_is_initialized(ctx))
2041− return -EINVAL;
2042−
2043− switch(cmd) {
2044− case UFFDIO_API:
2045− ret = userfaultfd_api(ctx, arg);
2046− break;
2047− case UFFDIO_REGISTER:
2048− ret = userfaultfd_register(ctx, arg);
2049− break;
2050− case UFFDIO_UNREGISTER:
2051− ret = userfaultfd_unregister(ctx, arg);
2052− break;
2053− case UFFDIO_WAKE:
2054− ret = userfaultfd_wake(ctx, arg);
2055− break;
2056− case UFFDIO_COPY:
2057− ret = userfaultfd_copy(ctx, arg);
2058− break;
2059− case UFFDIO_ZEROPAGE:
2060− ret = userfaultfd_zeropage(ctx, arg);
2061− break;
2062− case UFFDIO_MOVE:
2063− ret = userfaultfd_move(ctx, arg);
2064− break;
2065− case UFFDIO_WRITEPROTECT:
2066− ret = userfaultfd_writeprotect(ctx, arg);
2067− break;
2068− case UFFDIO_CONTINUE:
2069− ret = userfaultfd_continue(ctx, arg);
2070− break;
2071− case UFFDIO_POISON:
2072− ret = userfaultfd_poison(ctx, arg);
2073− break;
2074− }
2075− return ret;
2076−}
2077−
2078−#ifdef CONFIG_PROC_FS
2079−static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
2080−{
2081− struct userfaultfd_ctx *ctx = f->private_data;
2082− wait_queue_entry_t *wq;
2083− unsigned long pending = 0, total = 0;
2084−
2085− spin_lock_irq(&ctx->fault_pending_wqh.lock);
2086− list_for_each_entry(wq, &ctx->fault_pending_wqh.head, entry) {
2087− pending++;
2088− total++;
2089− }
2090− list_for_each_entry(wq, &ctx->fault_wqh.head, entry) {
2091− total++;
2092− }
2093− spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2094−
2095− /*
2096− * If more protocols will be added, there will be all shown
2097− * separated by a space. Like this:
2098− * protocols: aa:... bb:...
2099− */
2100− seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
2101− pending, total, UFFD_API, ctx->features,
2102− UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
2103−}
2104−#endif
2105−
2106−static const struct file_operations userfaultfd_fops = {
2107−#ifdef CONFIG_PROC_FS
2108− .show_fdinfo = userfaultfd_show_fdinfo,
2109−#endif
2110− .release = userfaultfd_release,
2111− .poll = userfaultfd_poll,
2112− .read_iter = userfaultfd_read_iter,
2113− .unlocked_ioctl = userfaultfd_ioctl,
2114− .compat_ioctl = compat_ptr_ioctl,
2115− .llseek = noop_llseek,
2116−};
2117−
2118−static void init_once_userfaultfd_ctx(void *mem)
2119−{
2120− struct userfaultfd_ctx *ctx = (struct userfaultfd_ctx *) mem;
2121−
2122− init_waitqueue_head(&ctx->fault_pending_wqh);
2123− init_waitqueue_head(&ctx->fault_wqh);
2124− init_waitqueue_head(&ctx->event_wqh);
2125− init_waitqueue_head(&ctx->fd_wqh);
2126− seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock);
2127−}
2128−
2129−static int new_userfaultfd(int flags)
2130−{
2131− struct userfaultfd_ctx *ctx __free(kfree) = NULL;
2132−
2133− VM_WARN_ON_ONCE(!current->mm);
2134−
2135− /* Check the UFFD_* constants for consistency. */
2136− BUILD_BUG_ON(UFFD_USER_MODE_ONLY & UFFD_SHARED_FCNTL_FLAGS);
2137−
2138− if (flags & ~(UFFD_SHARED_FCNTL_FLAGS | UFFD_USER_MODE_ONLY))
2139− return -EINVAL;
2140−
2141− ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
2142− if (!ctx)
2143− return -ENOMEM;
2144−
2145− refcount_set(&ctx->refcount, 1);
2146− ctx->flags = flags;
2147− ctx->features = 0;
2148− ctx->released = false;
2149− init_rwsem(&ctx->map_changing_lock);
2150− atomic_set(&ctx->mmap_changing, 0);
2151− ctx->mm = current->mm;
2152−
2153− FD_PREPARE(fdf, flags & UFFD_SHARED_FCNTL_FLAGS,
2154− anon_inode_create_getfile("[userfaultfd]", &userfaultfd_fops, ctx,
2155− O_RDONLY | (flags & UFFD_SHARED_FCNTL_FLAGS),
2156− NULL));
2157− if (fdf.err)
2158− return fdf.err;
2159−
2160− /* prevent the mm struct to be freed */
2161− mmgrab(ctx->mm);
2162− fd_prepare_file(fdf)->f_mode |= FMODE_NOWAIT;
2163− retain_and_null_ptr(ctx);
2164− return fd_publish(fdf);
2165−}
2166−
2167−static inline bool userfaultfd_syscall_allowed(int flags)
2168−{
2169− /* Userspace-only page faults are always allowed */
2170− if (flags & UFFD_USER_MODE_ONLY)
2171− return true;
2172−
2173− /*
2174− * The user is requesting a userfaultfd which can handle kernel faults.
2175− * Privileged users are always allowed to do this.
2176− */
2177− if (capable(CAP_SYS_PTRACE))
2178− return true;
2179−
2180− /* Otherwise, access to kernel fault handling is sysctl controlled. */
2181− return sysctl_unprivileged_userfaultfd;
2182−}
2183−
2184−SYSCALL_DEFINE1(userfaultfd, int, flags)
2185−{
2186− if (!userfaultfd_syscall_allowed(flags))
2187− return -EPERM;
2188−
2189− return new_userfaultfd(flags);
2190−}
2191−
2192−static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags)
2193−{
2194− if (cmd != USERFAULTFD_IOC_NEW)
2195− return -EINVAL;
2196−
2197− return new_userfaultfd(flags);
2198−}
2199−
2200−static const struct file_operations userfaultfd_dev_fops = {
2201− .unlocked_ioctl = userfaultfd_dev_ioctl,
2202− .compat_ioctl = userfaultfd_dev_ioctl,
2203− .owner = THIS_MODULE,
2204− .llseek = noop_llseek,
2205−};
2206−
2207−static struct miscdevice userfaultfd_misc = {
2208− .minor = MISC_DYNAMIC_MINOR,
2209− .name = "userfaultfd",
2210− .fops = &userfaultfd_dev_fops
2211−};
2212−
2213−static int __init userfaultfd_init(void)
2214−{
2215− int ret;
2216−
2217− ret = misc_register(&userfaultfd_misc);
2218− if (ret)
2219− return ret;
2220−
2221− userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache",
2222− sizeof(struct userfaultfd_ctx),
2223− 0,
2224− SLAB_HWCACHE_ALIGN|SLAB_PANIC,
2225− init_once_userfaultfd_ctx);
2226−#ifdef CONFIG_SYSCTL
2227− register_sysctl_init("vm", vm_userfaultfd_table);
2228−#endif
2229− return 0;
2230−}
2231−__initcall(userfaultfd_init);
include/asm-generic/pgalloc.h+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44
55 #ifdef CONFIG_MMU
66
7−#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO)
7+#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
88 #define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)
99
1010 /**
include/linux/bootmem_info.h+0−1
@@ -82,7 +82,6 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 81 unmodified lines
8282
8383 static inline void free_bootmem_page(struct page *page)
8484 {
85− kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
8685 free_reserved_page(page);
8786 }
8887 #endif
include/linux/compaction.h+5−5
@@ -2,6 +2,8 @@ · 1 unmodified line
22 #ifndef _LINUX_COMPACTION_H
33 #define _LINUX_COMPACTION_H
44
5+#include <linux/swap.h>
6+
57 /*
68 * Determines how hard direct compaction should try to succeed.
79 * Lower value means higher priority, analogically to reclaim priority.
@@ -73,11 +75,9 @@ static inline unsigned long compact_gap(unsigned int order) · 65 unmodified lines
7375 * effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum
7476 * that the migrate scanner can have isolated on migrate list, and free
7577 * scanner is only invoked when the number of isolated free pages is
76− * lower than that. But it's not worth to complicate the formula here
77− * as a bigger gap for higher orders than strictly necessary can also
78− * improve chances of compaction success.
78+ * lower than that.
7979 */
80− return 2UL << order;
80+ return min(2UL << order, COMPACT_CLUSTER_MAX);
8181 }
8282
8383 static inline int current_is_kcompactd(void)
@@ -101,7 +101,7 @@ extern void compaction_defer_reset(struct zone *zone, int order, · 17 unmodified lines
101101 bool alloc_success);
102102
103103 bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
104− int alloc_flags);
104+ int alloc_flags, gfp_t gfp_mask);
105105
106106 extern void __meminit kcompactd_run(int nid);
107107 extern void __meminit kcompactd_stop(int nid);
include/linux/damon.h+111−25
@@ -8,23 +8,20 @@ · 7 unmodified lines
88 #ifndef _DAMON_H_
99 #define _DAMON_H_
1010
11+#include <linux/math64.h>
1112 #include <linux/memcontrol.h>
1213 #include <linux/mutex.h>
14+#include <linux/prandom.h>
1315 #include <linux/time64.h>
1416 #include <linux/types.h>
15−#include <linux/random.h>
1617
1718 /* Minimal region size. Every damon_region is aligned by this. */
1819 #define DAMON_MIN_REGION_SZ PAGE_SIZE
20+/* Maximum number of monitoring probes. */
21+#define DAMON_MAX_PROBES (4)
1922 /* Max priority score for DAMON-based operation schemes */
2023 #define DAMOS_MAX_SCORE (99)
2124
22−/* Get a random number in [l, r) */
23−static inline unsigned long damon_rand(unsigned long l, unsigned long r)
24−{
25− return l + get_random_u32_below(r - l);
26−}
27−
2825 /**
2926 * struct damon_addr_range - Represents an address region of [@start, @end).
3027 * @start: Start address of the region (inclusive).
@@ -52,6 +49,7 @@ struct damon_size_range { · 21 unmodified lines
5249 * @nr_accesses: Access frequency of this region.
5350 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
5451 * each sampling interval.
52+ * @probe_hits: Number of probe-positive region samples.
5553 * @list: List head for siblings.
5654 * @age: Age of this region.
5755 *
@@ -80,6 +78,7 @@ struct damon_region { · 22 unmodified lines
8078 unsigned long sampling_addr;
8179 unsigned int nr_accesses;
8280 unsigned int nr_accesses_bp;
81+ unsigned char probe_hits[DAMON_MAX_PROBES];
8382 struct list_head list;
8483
8584 unsigned int age;
@@ -121,6 +120,7 @@ struct damon_target { · 35 unmodified lines
121120 * @DAMOS_PAGEOUT: Reclaim the region.
122121 * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
123122 * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
123+ * @DAMOS_COLLAPSE: Call ``madvise()`` for the region with MADV_COLLAPSE.
124124 * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
125125 * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.
126126 * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.
@@ -140,6 +140,7 @@ enum damos_action { · 13 unmodified lines
140140 DAMOS_PAGEOUT,
141141 DAMOS_HUGEPAGE,
142142 DAMOS_NOHUGEPAGE,
143+ DAMOS_COLLAPSE,
143144 DAMOS_LRU_PRIO,
144145 DAMOS_LRU_DEPRIO,
145146 DAMOS_MIGRATE_HOT,
@@ -159,6 +160,8 @@ enum damos_action { · 13 unmodified lines
159160 * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.
160161 * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.
161162 * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.
163+ * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a
164+ * node in basis points (0-10000).
162165 * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.
163166 *
164167 * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@@ -172,6 +175,7 @@ enum damos_quota_goal_metric { · 7 unmodified lines
172175 DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
173176 DAMOS_QUOTA_ACTIVE_MEM_BP,
174177 DAMOS_QUOTA_INACTIVE_MEM_BP,
178+ DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
175179 NR_DAMOS_QUOTA_GOAL_METRICS,
176180 };
177181
@@ -233,6 +237,8 @@ enum damos_quota_goal_tuner { · 55 unmodified lines
233237 * @goals: Head of quota tuning goals (&damos_quota_goal) list.
234238 * @goal_tuner: Goal-based @esz tuning algorithm to use.
235239 * @esz: Effective size quota in bytes.
240+ * @fail_charge_num: Failed regions charge rate numerator.
241+ * @fail_charge_denom: Failed regions charge rate denominator.
236242 *
237243 * @weight_sz: Weight of the region's size for prioritization.
238244 * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.
@@ -262,6 +268,10 @@ enum damos_quota_goal_tuner { · 23 unmodified lines
262268 *
263269 * The resulting effective size quota in bytes is set to @esz.
264270 *
271+ * For DAMOS action applying failed amount of regions, charging those same to
272+ * those that the action has successfully applied may be unfair. For the
273+ * reason, 'the size * @fail_charge_num / @fail_charge_denom' is charged.
274+ *
265275 * For selecting regions within the quota, DAMON prioritizes current scheme's
266276 * target memory regions using the &struct damon_operations->get_scheme_score.
267277 * You could customize the prioritization logic by setting &weight_sz,
@@ -276,6 +286,9 @@ struct damos_quota { · 8 unmodified lines
276286 enum damos_quota_goal_tuner goal_tuner;
277287 unsigned long esz;
278288
289+ unsigned int fail_charge_num;
290+ unsigned int fail_charge_denom;
291+
279292 unsigned int weight_sz;
280293 unsigned int weight_nr_accesses;
281294 unsigned int weight_age;
@@ -617,6 +630,7 @@ enum damon_ops_id { · 335 unmodified lines
617630 * @update: Update operations-related data structures.
618631 * @prepare_access_checks: Prepare next access check of target regions.
619632 * @check_accesses: Check the accesses to target regions.
633+ * @apply_probes: Apply probes for each region.
620634 * @get_scheme_score: Get the score of a region for a scheme.
621635 * @apply_scheme: Apply a DAMON-based operation scheme.
622636 * @target_valid: Determine if the target is valid.
@@ -643,6 +657,8 @@ enum damon_ops_id { · 20 unmodified lines
643657 * last preparation and update the number of observed accesses of each region.
644658 * It should also return max number of observed accesses that made as a result
645659 * of its update. The value will be used for regions adjustment threshold.
660+ * @apply_probes should apply the data attribute probes to each region and
661+ * accordingly update the probe hits counter of the region.
646662 * @get_scheme_score should return the priority score of a region for a scheme
647663 * as an integer in [0, &DAMOS_MAX_SCORE].
648664 * @apply_scheme is called from @kdamond when a region for user provided
@@ -660,6 +676,7 @@ struct damon_operations { · 11 unmodified lines
660676 void (*update)(struct damon_ctx *context);
661677 void (*prepare_access_checks)(struct damon_ctx *context);
662678 unsigned int (*check_accesses)(struct damon_ctx *context);
679+ void (*apply_probes)(struct damon_ctx *context);
663680 int (*get_scheme_score)(struct damon_ctx *context,
664681 struct damon_region *r, struct damos *scheme);
665682 unsigned long (*apply_scheme)(struct damon_ctx *context,
@@ -721,6 +738,47 @@ struct damon_intervals_goal { · 55 unmodified lines
721738 unsigned long max_sample_us;
722739 };
723740
741+/**
742+ * enum damon_filter_type - Type of &struct damon_filter
743+ *
744+ * @DAMON_FILTER_TYPE_ANON: Anonymous pages.
745+ * @DAMON_FILTER_TYPE_MEMCG: Specific memcg's pages.
746+ */
747+enum damon_filter_type {
748+ DAMON_FILTER_TYPE_ANON,
749+ DAMON_FILTER_TYPE_MEMCG,
750+};
751+
752+/**
753+ * struct damon_filter - DAMON region filter for &struct damon_probe.
754+ *
755+ * @type: Type of the region.
756+ * @matching: Whether this filter is for the type-matching ones.
757+ * @allow: Whether the @type-@matching ones should pass this filter.
758+ * @memcg_id: Memcg id of the question if @type is DAMON_FILTER_MEMCG.
759+ * @list: Siblings list.
760+ */
761+struct damon_filter {
762+ enum damon_filter_type type;
763+ bool matching;
764+ bool allow;
765+ union {
766+ u64 memcg_id;
767+ };
768+ struct list_head list;
769+};
770+
771+/**
772+ * struct damon_probe - Data region attribute probe.
773+ *
774+ * @filters: Filters for assessing if a given region is for this probe.
775+ * @list: Siblings list.
776+ */
777+struct damon_probe {
778+ struct list_head filters;
779+ struct list_head list;
780+};
781+
724782 /**
725783 * struct damon_attrs - Monitoring attributes for accuracy/overhead control.
726784 *
@@ -787,6 +845,7 @@ struct damon_attrs { · 60 unmodified lines
787845 * @ops: Set of monitoring operations for given use cases.
788846 * @addr_unit: Scale factor for core to ops address conversion.
789847 * @min_region_sz: Minimum region size.
848+ * @pause: Pause kdamond main loop.
790849 * @adaptive_targets: Head of monitoring targets (&damon_target) list.
791850 * @schemes: Head of schemes (&damos) list.
792851 */
@@ -838,13 +897,34 @@ struct damon_ctx { · 45 unmodified lines
838897
839898 /* public: */
840899 struct damon_operations ops;
900+ struct list_head probes;
841901 unsigned long addr_unit;
842902 unsigned long min_region_sz;
903+ bool pause;
843904
844905 struct list_head adaptive_targets;
845906 struct list_head schemes;
907+
908+ /* Per-ctx PRNG state for damon_rand(); kdamond is the sole consumer. */
909+ struct rnd_state rnd_state;
846910 };
847911
912+/* Get a random number in [@l, @r) using @ctx's lockless PRNG. */
913+static inline unsigned long damon_rand(struct damon_ctx *ctx,
914+ unsigned long l, unsigned long r)
915+{
916+ unsigned long span = r - l;
917+ u64 rnd;
918+
919+ if (span <= U32_MAX) {
920+ rnd = prandom_u32_state(&ctx->rnd_state);
921+ return l + (unsigned long)((rnd * span) >> 32);
922+ }
923+ rnd = ((u64)prandom_u32_state(&ctx->rnd_state) << 32) |
924+ prandom_u32_state(&ctx->rnd_state);
925+ return l + mul_u64_u64_shr(rnd, span, 64);
926+}
927+
848928 static inline struct damon_region *damon_next_region(struct damon_region *r)
849929 {
850930 return container_of(r->list.next, struct damon_region, list);
@@ -870,15 +950,26 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 19 unmodified lines
870950 return r->ar.end - r->ar.start;
871951 }
872952
953+#define damon_for_each_filter(f, p) \
954+ list_for_each_entry(f, &(p)->filters, list)
955+
956+#define damon_for_each_filter_safe(f, next, p) \
957+ list_for_each_entry_safe(f, next, &(p)->filters, list)
958+
959+#define damon_for_each_probe(p, ctx) \
960+ list_for_each_entry(p, &(ctx)->probes, list)
961+
962+#define damon_for_each_probe_safe(p, next, ctx) \
963+ list_for_each_entry_safe(p, next, &(ctx)->probes, list)
873964
874965 #define damon_for_each_region(r, t) \
875− list_for_each_entry(r, &t->regions_list, list)
966+ list_for_each_entry(r, &(t)->regions_list, list)
876967
877968 #define damon_for_each_region_from(r, t) \
878− list_for_each_entry_from(r, &t->regions_list, list)
969+ list_for_each_entry_from(r, &(t)->regions_list, list)
879970
880971 #define damon_for_each_region_safe(r, next, t) \
881− list_for_each_entry_safe(r, next, &t->regions_list, list)
972+ list_for_each_entry_safe(r, next, &(t)->regions_list, list)
882973
883974 #define damon_for_each_target(t, ctx) \
884975 list_for_each_entry(t, &(ctx)->adaptive_targets, list)
@@ -893,7 +984,7 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 8 unmodified lines
893984 list_for_each_entry_safe(s, next, &(ctx)->schemes, list)
894985
895986 #define damos_for_each_quota_goal(goal, quota) \
896− list_for_each_entry(goal, &quota->goals, list)
987+ list_for_each_entry(goal, &(quota)->goals, list)
897988
898989 #define damos_for_each_quota_goal_safe(goal, next, quota) \
899990 list_for_each_entry_safe(goal, next, &(quota)->goals, list)
@@ -912,21 +1003,16 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 12 unmodified lines
9121003
9131004 #ifdef CONFIG_DAMON
9141005
915−struct damon_region *damon_new_region(unsigned long start, unsigned long end);
1006+struct damon_filter *damon_new_filter(enum damon_filter_type type,
1007+ bool matching, bool allow);
1008+void damon_add_filter(struct damon_probe *probe, struct damon_filter *f);
1009+void damon_destroy_filter(struct damon_filter *f);
9161010
917−/*
918− * Add a region between two other regions
919− */
920−static inline void damon_insert_region(struct damon_region *r,
921− struct damon_region *prev, struct damon_region *next,
922− struct damon_target *t)
923−{
924− __list_add(&r->list, &prev->list, &next->list);
925− t->nr_regions++;
926−}
1011+struct damon_probe *damon_new_probe(void);
1012+void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe);
1013+
1014+struct damon_region *damon_new_region(unsigned long start, unsigned long end);
9271015
928−void damon_add_region(struct damon_region *r, struct damon_target *t);
929−void damon_destroy_region(struct damon_region *r, struct damon_target *t);
9301016 int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
9311017 unsigned int nr_ranges, unsigned long min_region_sz);
9321018 void damon_update_region_access_rate(struct damon_region *r, bool accessed,
@@ -994,7 +1080,7 @@ int damon_kdamond_pid(struct damon_ctx *ctx); · 61 unmodified lines
9941080 int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
9951081 int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
9961082
997−int damon_set_region_biggest_system_ram_default(struct damon_target *t,
1083+int damon_set_region_system_rams_default(struct damon_target *t,
9981084 unsigned long *start, unsigned long *end,
9991085 unsigned long addr_unit,
10001086 unsigned long min_region_sz);
include/linux/gfp.h+4−0
@@ -239,6 +239,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 238 unmodified lines
239239 struct page **page_array);
240240 #define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))
241241
242+void free_pages_bulk(struct page **page_array, unsigned long nr_pages);
243+
242244 unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
243245 unsigned long nr_pages,
244246 struct page **page_array);
@@ -467,6 +469,8 @@ void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages); · 222 unmodified lines
467469 void free_contig_range(unsigned long pfn, unsigned long nr_pages);
468470 #endif
469471
472+void __free_contig_range(unsigned long pfn, unsigned long nr_pages);
473+
470474 DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))
471475
472476 #endif /* __LINUX_GFP_H */
include/linux/gfp_types.h+3−3
@@ -281,9 +281,9 @@ enum { · 280 unmodified lines
281281 *
282282 * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.
283283 * Used for userspace and vmalloc pages; the latter are unpoisoned by
284− * kasan_unpoison_vmalloc instead. For userspace pages, results in
285− * poisoning being skipped as well, see should_skip_kasan_poison for
286− * details. Only effective in HW_TAGS mode.
284+ * kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc
285+ * is skipped too. For userspace pages, results in poisoning being skipped as
286+ * well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode.
287287 */
288288 #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
289289 #define __GFP_COMP ((__force gfp_t)___GFP_COMP)
include/linux/highmem-internal.h+1−1
@@ -262,7 +262,7 @@ static inline bool is_kmap_addr(const void *x) · 261 unmodified lines
262262 * @__addr: Virtual address to be unmapped
263263 *
264264 * Unmaps an address previously mapped by kmap_atomic() and re-enables
265− * pagefaults. Depending on PREEMP_RT configuration, re-enables also
265+ * pagefaults. Depending on PREEMPT_RT configuration, re-enables also
266266 * migration and preemption. Users should not count on these side effects.
267267 *
268268 * Mappings should be unmapped in the reverse order that they were mapped.
include/linux/huge_mm.h+42−5
@@ -237,6 +237,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 236 unmodified lines
237237 return true;
238238 }
239239
240+/*
241+ * Make sure huge_gfp is always more limited than limit_gfp.
242+ * Some shmem users want THP allocation to be done less aggressively
243+ * and only in certain zone.
244+ */
245+static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
246+{
247+ gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
248+ gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
249+ gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
250+ gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
251+
252+ /* Allow allocations only from the originally specified zones. */
253+ result |= zoneflags;
254+
255+ /*
256+ * Minimize the result gfp by taking the union with the deny flags,
257+ * and the intersection of the allow flags.
258+ */
259+ result |= (limit_gfp & denyflags);
260+ result |= (huge_gfp & limit_gfp) & allowflags;
261+
262+ return result;
263+}
264+
240265 /*
241266 * Filter the bitfield of input orders to the ones suitable for use in the vma.
242267 * See thp_vma_suitable_order().
@@ -414,10 +439,10 @@ static inline int split_huge_page(struct page *page) · 171 unmodified lines
414439 {
415440 return split_huge_page_to_list_to_order(page, NULL, 0);
416441 }
442+
443+int folio_memcg_alloc_deferred(struct folio *folio);
444+
417445 void deferred_split_folio(struct folio *folio, bool partially_mapped);
418−#ifdef CONFIG_MEMCG
419−void reparent_deferred_split_queue(struct mem_cgroup *memcg);
420−#endif
421446
422447 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
423448 unsigned long address, bool freeze);
@@ -581,6 +606,11 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 157 unmodified lines
581606 return false;
582607 }
583608
609+static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
610+{
611+ return huge_gfp;
612+}
613+
584614 static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
585615 unsigned long addr, unsigned long orders)
586616 {
@@ -649,8 +679,15 @@ static inline int try_folio_split_to_order(struct folio *folio, · 62 unmodified lines
649679 return -EINVAL;
650680 }
651681
652−static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
653−static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}
682+static inline int folio_memcg_alloc_deferred(struct folio *folio)
683+{
684+ return 0;
685+}
686+
687+static inline void deferred_split_folio(struct folio *folio, bool partially_mapped)
688+{
689+}
690+
654691 #define split_huge_pmd(__vma, __pmd, __address) \
655692 do { } while (0)
656693
include/linux/list_lru.h+70−0
@@ -81,8 +81,75 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker · 80 unmodified lines
8181
8282 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
8383 gfp_t gfp);
84+
85+#ifdef CONFIG_MEMCG
86+/**
87+ * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
88+ * @folio: the newly allocated & charged folio
89+ * @lru: the list_lru this might be queued on
90+ * @gfp: gfp mask
91+ *
92+ * Allocate list_lru heads (per-memcg, per-node) needed to queue this
93+ * particular folio down the line.
94+ *
95+ * This does memcg_list_lru_alloc(), but on the memcg that @folio is
96+ * associated with. Handles folio_memcg() access rules in the fast
97+ * path (list_lru heads allocated) and the allocation slowpath.
98+ *
99+ * Returns 0 on success, a negative error value otherwise.
100+ */
101+int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
102+ gfp_t gfp);
103+#else
104+static inline int folio_memcg_list_lru_alloc(struct folio *folio,
105+ struct list_lru *lru, gfp_t gfp)
106+{
107+ return 0;
108+}
109+#endif
110+
84111 void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
85112
113+/**
114+ * list_lru_lock: lock the sublist for the given node and memcg
115+ * @lru: the lru pointer
116+ * @nid: the node id of the sublist to lock.
117+ * @memcg: pointer to the cgroup of the sublist to lock. On return,
118+ * updated to the cgroup whose sublist was actually locked,
119+ * which may be an ancestor if the original memcg was dying.
120+ *
121+ * Returns the locked list_lru_one sublist. The caller must call
122+ * list_lru_unlock() when done.
123+ *
124+ * You must ensure that the memcg is not freed during this call (e.g., with
125+ * rcu or by taking a css refcnt).
126+ *
127+ * Return: the locked list_lru_one, or NULL on failure
128+ */
129+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
130+ struct mem_cgroup **memcg);
131+
132+/**
133+ * list_lru_unlock: unlock a sublist locked by list_lru_lock()
134+ * @l: the list_lru_one to unlock
135+ */
136+void list_lru_unlock(struct list_lru_one *l);
137+
138+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
139+ struct mem_cgroup **memcg);
140+void list_lru_unlock_irq(struct list_lru_one *l);
141+
142+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
143+ struct mem_cgroup **memcg, unsigned long *irq_flags);
144+void list_lru_unlock_irqrestore(struct list_lru_one *l,
145+ unsigned long *irq_flags);
146+
147+/* Caller-locked variants, see list_lru_add() etc for documentation */
148+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
149+ struct list_head *item, int nid, struct mem_cgroup *memcg);
150+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
151+ struct list_head *item, int nid);
152+
86153 /**
87154 * list_lru_add: add an element to the lru list's tail
88155 * @lru: the lru pointer
@@ -115,6 +182,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren · 26 unmodified lines
115182 bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
116183 struct mem_cgroup *memcg);
117184
185+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
186+ struct mem_cgroup *memcg);
187+
118188 /**
119189 * list_lru_add_obj: add an element to the lru list's tail
120190 * @lru: the lru pointer
include/linux/memcontrol.h+17−14
@@ -29,6 +29,7 @@ struct obj_cgroup; · 28 unmodified lines
2929 struct page;
3030 struct mm_struct;
3131 struct kmem_cache;
32+struct swap_cluster_info;
3233
3334 /* Cgroup-specific page state, on top of universal node page state */
3435 enum memcg_stat_item {
@@ -277,10 +278,6 @@ struct mem_cgroup { · 242 unmodified lines
277278 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
278279 #endif
279280
280−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
281− struct deferred_split deferred_split_queue;
282−#endif
283−
284281 #ifdef CONFIG_LRU_GEN_WALKS_MMU
285282 /* per-memcg mm_struct list */
286283 struct lru_gen_mm_list mm_list;
@@ -646,8 +643,8 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 359 unmodified lines
646643
647644 int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
648645
649−int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
650− gfp_t gfp, swp_entry_t entry);
646+int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
647+ struct mm_struct *mm, gfp_t gfp);
651648
652649 void __mem_cgroup_uncharge(struct folio *folio);
653650
@@ -1137,7 +1134,7 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp) · 483 unmodified lines
11371134 }
11381135
11391136 static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
1140− struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
1137+ unsigned short id, struct mm_struct *mm, gfp_t gfp)
11411138 {
11421139 return 0;
11431140 }
@@ -1899,9 +1896,6 @@ static inline void mem_cgroup_exit_user_fault(void) · 755 unmodified lines
18991896 current->in_user_fault = 0;
19001897 }
19011898
1902−void memcg1_swapout(struct folio *folio, swp_entry_t entry);
1903−void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages);
1904−
19051899 #else /* CONFIG_MEMCG_V1 */
19061900 static inline
19071901 unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
@@ -1929,14 +1923,23 @@ static inline void mem_cgroup_exit_user_fault(void) · 21 unmodified lines
19291923 {
19301924 }
19311925
1932−static inline void memcg1_swapout(struct folio *folio, swp_entry_t entry)
1926+#endif /* CONFIG_MEMCG_V1 */
1927+
1928+#if defined(CONFIG_MEMCG_V1) && defined(CONFIG_SWAP)
1929+
1930+void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci);
1931+void memcg1_swapin(struct folio *folio);
1932+
1933+#else
1934+
1935+static inline void __memcg1_swapout(struct folio *folio,
1936+ struct swap_cluster_info *ci)
19331937 {
19341938 }
19351939
1936−static inline void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
1940+static inline void memcg1_swapin(struct folio *folio)
19371941 {
19381942 }
1939−
1940−#endif /* CONFIG_MEMCG_V1 */
1943+#endif
19411944
19421945 #endif /* _LINUX_MEMCONTROL_H */
include/linux/memory.h+5−2
@@ -158,7 +158,11 @@ int create_memory_block_devices(unsigned long start, unsigned long size, · 157 unmodified lines
158158 void remove_memory_block_devices(unsigned long start, unsigned long size);
159159 extern void memory_dev_init(void);
160160 extern int memory_notify(enum memory_block_state state, void *v);
161−extern struct memory_block *find_memory_block(unsigned long section_nr);
161+struct memory_block *memory_block_get(unsigned long block_id);
162+static inline void memory_block_put(struct memory_block *mem)
163+{
164+ put_device(&mem->dev);
165+}
162166 typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);
163167 extern int walk_memory_blocks(unsigned long start, unsigned long size,
164168 void *arg, walk_memory_blocks_func_t func);
@@ -171,7 +175,6 @@ struct memory_group *memory_group_find_by_id(int mgid); · 6 unmodified lines
171175 typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);
172176 int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
173177 struct memory_group *excluded, void *arg);
174−struct memory_block *find_memory_block_by_id(unsigned long block_id);
175178 #define hotplug_memory_notifier(fn, pri) ({ \
176179 static __meminitdata struct notifier_block fn##_mem_nb =\
177180 { .notifier_call = fn, .priority = pri };\
include/linux/memory_hotplug.h+5−3
@@ -135,9 +135,10 @@ static inline bool movable_node_is_enabled(void) · 134 unmodified lines
135135 return movable_node_enabled;
136136 }
137137
138−extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap);
138+extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
139+ struct dev_pagemap *pgmap);
139140 extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
140− struct vmem_altmap *altmap);
141+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap);
141142
142143 /* reasonably generic interface to expand the physical pages */
143144 extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -307,7 +308,8 @@ extern int sparse_add_section(int nid, unsigned long pfn, · 163 unmodified lines
307308 unsigned long nr_pages, struct vmem_altmap *altmap,
308309 struct dev_pagemap *pgmap);
309310 extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
310− struct vmem_altmap *altmap);
311+ struct vmem_altmap *altmap,
312+ struct dev_pagemap *pgmap);
311313 extern struct zone *zone_for_pfn_range(enum mmop online_type,
312314 int nid, struct memory_group *group, unsigned long start_pfn,
313315 unsigned long nr_pages);
include/linux/mm.h+44−18
@@ -496,6 +496,21 @@ enum { · 495 unmodified lines
496496 #else
497497 #define VM_UFFD_MINOR VM_NONE
498498 #endif
499+
500+/*
501+ * vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on
502+ * the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits
503+ * -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose
504+ * bitmap cannot hold it.
505+ */
506+#define VMA_UFFD_MISSING mk_vma_flags(VMA_UFFD_MISSING_BIT)
507+#define VMA_UFFD_WP mk_vma_flags(VMA_UFFD_WP_BIT)
508+#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
509+#define VMA_UFFD_MINOR mk_vma_flags(VMA_UFFD_MINOR_BIT)
510+#else
511+#define VMA_UFFD_MINOR EMPTY_VMA_FLAGS
512+#endif
513+
499514 #ifdef CONFIG_64BIT
500515 #define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
501516 #define VM_SEALED INIT_VM_FLAG(SEALED)
@@ -1238,6 +1253,30 @@ static __always_inline void vma_flags_set_mask(vma_flags_t *flags, · 736 unmodified lines
12381253 #define vma_flags_set(flags, ...) \
12391254 vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))
12401255
1256+static __always_inline vma_flags_t __mk_vma_flags_from_masks(size_t count,
1257+ const vma_flags_t *masks)
1258+{
1259+ vma_flags_t flags = EMPTY_VMA_FLAGS;
1260+ size_t i;
1261+
1262+ for (i = 0; i < count; i++)
1263+ vma_flags_set_mask(&flags, masks[i]);
1264+ return flags;
1265+}
1266+
1267+/*
1268+ * Combine pre-computed vma_flags_t masks into one value, e.g.:
1269+ *
1270+ * vma_flags_t flags = mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR);
1271+ *
1272+ * Unlike mk_vma_flags(), which takes bit numbers, this takes whole masks --
1273+ * each of which may be EMPTY_VMA_FLAGS when its feature is unavailable -- so a
1274+ * bit that does not exist on the current build is never materialised.
1275+ */
1276+#define mk_vma_flags_from_masks(...) \
1277+ __mk_vma_flags_from_masks(COUNT_ARGS(__VA_ARGS__), \
1278+ (const vma_flags_t []){__VA_ARGS__})
1279+
12411280 /* Clear all of the to-clear flags in flags, non-atomically. */
12421281 static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,
12431282 vma_flags_t to_clear)
@@ -1489,6 +1528,11 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma) · 245 unmodified lines
14891528 vma->vm_ops = NULL;
14901529 }
14911530
1531+static inline void vma_desc_set_anonymous(struct vm_area_desc *desc)
1532+{
1533+ desc->vm_ops = NULL;
1534+}
1535+
14921536 static inline bool vma_is_anonymous(struct vm_area_struct *vma)
14931537 {
14941538 return !vma->vm_ops;
@@ -1888,16 +1932,6 @@ static inline bool folio_mapped(const struct folio *folio) · 393 unmodified lines
18881932 return folio_mapcount(folio) >= 1;
18891933 }
18901934
1891−/*
1892− * Return true if this page is mapped into pagetables.
1893− * For compound page it returns true if any sub-page of compound page is mapped,
1894− * even if this particular sub-page is not itself mapped by any PTE or PMD.
1895− */
1896−static inline bool page_mapped(const struct page *page)
1897−{
1898− return folio_mapped(page_folio(page));
1899−}
1900−
19011935 static inline struct page *virt_to_head_page(const void *x)
19021936 {
19031937 struct page *page = virt_to_page(x);
@@ -4855,18 +4889,10 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) · 2951 unmodified lines
48554889 }
48564890 #endif
48574891
4858−void *sparse_buffer_alloc(unsigned long size);
48594892 unsigned long section_map_size(void);
48604893 struct page * __populate_section_memmap(unsigned long pfn,
48614894 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
48624895 struct dev_pagemap *pgmap);
4863−pgd_t *vmemmap_pgd_populate(unsigned long addr, int node);
4864−p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node);
4865−pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node);
4866−pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node);
4867−pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
4868− struct vmem_altmap *altmap, unsigned long ptpfn,
4869− unsigned long flags);
48704896 void *vmemmap_alloc_block(unsigned long size, int node);
48714897 struct vmem_altmap;
48724898 void *vmemmap_alloc_block_buf(unsigned long size, int node,
include/linux/mm_inline.h+1−1
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · 246 unmodified lines
247247 (folio_test_dirty(folio) || folio_test_writeback(folio))))
248248 gen = MIN_NR_GENS;
249249 else
250− gen = MAX_NR_GENS - folio_test_workingset(folio);
250+ gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
251251
252252 return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
253253 }
include/linux/mm_types.h+3−16
@@ -845,23 +845,10 @@ struct mmap_action { · 844 unmodified lines
845845 enum mmap_action_type type;
846846
847847 /*
848− * If specified, this hook is invoked after the selected action has been
849− * successfully completed. Note that the VMA write lock still held.
850− *
851− * The absolute minimum ought to be done here.
852− *
853− * Returns 0 on success, or an error code.
854− */
855− int (*success_hook)(const struct vm_area_struct *vma);
856−
857− /*
858− * If specified, this hook is invoked when an error occurred when
859− * attempting the selected action.
860− *
861− * The hook can return an error code in order to filter the error, but
862− * it is not valid to clear the error here.
848+ * If non-zero, replace errors that arise from mmap actions with this
849+ * value instead. Only valid error codes may be specified.
863850 */
864− int (*error_hook)(int err);
851+ int error_override;
865852
866853 /*
867854 * This should be set in rare instances where the operation required
include/linux/mmu_notifier.h+2−2
@@ -134,8 +134,8 @@ struct mmu_notifier_ops { · 133 unmodified lines
134134 * Invalidation of multiple concurrent ranges may be
135135 * optionally permitted by the driver. Either way the
136136 * establishment of sptes is forbidden in the range passed to
137− * invalidate_range_begin/end for the whole duration of the
138− * invalidate_range_begin/end critical section.
137+ * invalidate_range_start/end for the whole duration of the
138+ * invalidate_range_start/end critical section.
139139 *
140140 * invalidate_range_start() is called when all pages in the
141141 * range are still mapped and have at least a refcount of one.
include/linux/mmzone.h+6−17
@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt) · 176 unmodified lines
177177 return mt < MIGRATE_PCPTYPES;
178178 }
179179
180−#define for_each_migratetype_order(order, type) \
181− for (order = 0; order < NR_PAGE_ORDERS; order++) \
182− for (type = 0; type < MIGRATE_TYPES; type++)
180+#define for_each_free_list(list, zone, order) \
181+ for (order = 0; order < NR_PAGE_ORDERS; order++) \
182+ for (unsigned int __type = 0; \
183+ __type < MIGRATE_TYPES && \
184+ (list = &(zone)->free_area[order].free_list[__type], 1); \
185+ __type++)
183186
184187 extern int page_group_by_mobility_disabled;
185188
@@ -211,7 +214,6 @@ enum numa_stat_item { · 25 unmodified lines
211214 #endif
212215
213216 enum zone_stat_item {
214− /* First 128 byte cacheline (assuming 64 bit words) */
215217 NR_FREE_PAGES,
216218 NR_FREE_PAGES_BLOCKS,
217219 NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */
@@ -222,7 +224,6 @@ enum zone_stat_item { · 4 unmodified lines
222224 NR_ZONE_UNEVICTABLE,
223225 NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */
224226 NR_MLOCK, /* mlock()ed pages found and moved off LRU */
225− /* Second 128 byte cacheline */
226227 #if IS_ENABLED(CONFIG_ZSMALLOC)
227228 NR_ZSPAGES, /* allocated in zsmalloc */
228229 #endif
@@ -1428,14 +1429,6 @@ struct zonelist { · 1199 unmodified lines
14281429 */
14291430 extern struct page *mem_map;
14301431
1431−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1432−struct deferred_split {
1433− spinlock_t split_queue_lock;
1434− struct list_head split_queue;
1435− unsigned long split_queue_len;
1436−};
1437−#endif
1438−
14391432 #ifdef CONFIG_MEMORY_FAILURE
14401433 /*
14411434 * Per NUMA node memory failure handling statistics.
@@ -1561,10 +1554,6 @@ typedef struct pglist_data { · 119 unmodified lines
15611554 unsigned long first_deferred_pfn;
15621555 #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
15631556
1564−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1565− struct deferred_split deferred_split_queue;
1566−#endif
1567−
15681557 #ifdef CONFIG_NUMA_BALANCING
15691558 /* start time in ms of current promote rate limit period */
15701559 unsigned int nbp_rl_start;
include/linux/nodemask.h+9−9
@@ -24,23 +24,23 @@ · 23 unmodified lines
2424 * void nodes_setall(mask) set all bits
2525 * void nodes_clear(mask) clear all bits
2626 * int node_isset(node, mask) true iff bit 'node' set in mask
27− * int node_test_and_set(node, mask) test and set bit 'node' in mask
27+ * bool node_test_and_set(node, mask) test and set bit 'node' in mask
2828 *
29− * void nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
29+ * bool nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
3030 * void nodes_or(dst, src1, src2) dst = src1 | src2 [union]
3131 * void nodes_xor(dst, src1, src2) dst = src1 ^ src2
32− * void nodes_andnot(dst, src1, src2) dst = src1 & ~src2
32+ * bool nodes_andnot(dst, src1, src2) dst = src1 & ~src2
3333 * void nodes_complement(dst, src) dst = ~src
3434 *
35− * int nodes_equal(mask1, mask2) Does mask1 == mask2?
36− * int nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
37− * int nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
38− * int nodes_empty(mask) Is mask empty (no bits sets)?
39− * int nodes_full(mask) Is mask full (all bits sets)?
35+ * bool nodes_equal(mask1, mask2) Does mask1 == mask2?
36+ * bool nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
37+ * bool nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
38+ * bool nodes_empty(mask) Is mask empty (no bits sets)?
39+ * bool nodes_full(mask) Is mask full (all bits sets)?
4040 * int nodes_weight(mask) Hamming weight - number of set bits
4141 *
4242 * unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES
43− * unsigend int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
43+ * unsigned int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
4444 * unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,
4545 * or MAX_NUMNODES
4646 * unsigned int first_unset_node(mask) First node not set in mask, or
include/linux/page_ref.h+18−0
@@ -71,6 +71,12 @@ static inline int page_ref_count(const struct page *page) · 70 unmodified lines
7171 * folio_ref_count - The reference count on this folio.
7272 * @folio: The folio.
7373 *
74+ * Folios contain a reference count. When that reference count reaches
75+ * zero, the folio is referred to as frozen. At this point, it will
76+ * usually be returned to the memory allocator, but some parts of the
77+ * kernel freeze folios in order to perform unusual operations on them
78+ * such as splitting or migration.
79+ *
7480 * The refcount is usually incremented by calls to folio_get() and
7581 * decremented by calls to folio_put(). Some typical users of the
7682 * folio refcount:
@@ -82,6 +88,18 @@ static inline int page_ref_count(const struct page *page) · 5 unmodified lines
8288 * - Pipes
8389 * - Direct IO which references this page in the process address space
8490 *
91+ * The reference count has three components: expected, temporary and
92+ * spurious. The expected reference count of a folio is that which
93+ * we would logically expect it to be from just reading the code.
94+ * Temporary refcounts are gained by threads which need a temporary
95+ * reference to make sure the folio isn't reallocated while they use it.
96+ * Spurious refcounts are gained by threads which, thanks to RCU walks
97+ * of the page tables or file cache, find a stale pointer to a folio.
98+ * These threads will drop the refcount after discoveering the pointer
99+ * is stale, but it can surprise other users to see the spurious refcount
100+ * on a freshly allocated folio (eg they may see a refcount of 2 instead
101+ * of 1).
102+ *
85103 * Return: The number of references to this folio.
86104 */
87105 static inline int folio_ref_count(const struct folio *folio)
include/linux/pageblock-flags.h+3−3
@@ -36,12 +36,12 @@ enum pageblock_bits { · 35 unmodified lines
3636
3737 #define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))
3838
39−#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
39+#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
4040
4141 #ifdef CONFIG_MEMORY_ISOLATION
42−#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate))
42+#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
4343 #else
44−#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK
44+#define PAGEBLOCK_ISO_MASK 0
4545 #endif
4646
4747 #if defined(CONFIG_HUGETLB_PAGE)
include/linux/pagemap.h+2−0
@@ -1350,6 +1350,7 @@ struct readahead_control { · 1349 unmodified lines
13501350 struct file_ra_state *ra;
13511351 /* private: use the readahead_* accessors instead */
13521352 pgoff_t _index;
1353+ pgoff_t _max_index; /* limit readahead to _max_index, inclusive */
13531354 unsigned int _nr_pages;
13541355 unsigned int _batch_count;
13551356 bool dropbehind;
@@ -1363,6 +1364,7 @@ struct readahead_control { · 7 unmodified lines
13631364 .mapping = m, \
13641365 .ra = r, \
13651366 ._index = i, \
1367+ ._max_index = ULONG_MAX, \
13661368 }
13671369
13681370 #define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)
include/linux/swap.h+12−12
@@ -213,6 +213,7 @@ enum { · 212 unmodified lines
213213 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
214214 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
215215 SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
216+ SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
216217 /* add others here before... */
217218 };
218219
@@ -252,7 +253,6 @@ struct swap_info_struct { · 33 unmodified lines
252253 struct plist_node list; /* entry in swap_active_head */
253254 signed char type; /* strange name for an index */
254255 unsigned int max; /* size of this swap device */
255− unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
256256 struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
257257 struct list_head free_clusters; /* free clusters list */
258258 struct list_head full_clusters; /* full clusters list */
@@ -433,7 +433,9 @@ static inline long get_nr_swap_pages(void) · 174 unmodified lines
433433 }
434434
435435 extern void si_swapinfo(struct sysinfo *);
436−int swap_type_of(dev_t device, sector_t offset);
436+extern int pin_hibernation_swap_type(dev_t device, sector_t offset);
437+extern void unpin_hibernation_swap_type(int type);
438+extern int find_hibernation_swap_type(dev_t device, sector_t offset);
437439 int find_first_swap(dev_t *device);
438440 extern unsigned int count_swap_pages(int, int);
439441 extern sector_t swapdev_block(int, pgoff_t);
@@ -571,33 +573,31 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp) · 131 unmodified lines
571573 #endif
572574
573575 #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
574−int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);
575−static inline int mem_cgroup_try_charge_swap(struct folio *folio,
576− swp_entry_t entry)
576+int __mem_cgroup_try_charge_swap(struct folio *folio);
577+static inline int mem_cgroup_try_charge_swap(struct folio *folio)
577578 {
578579 if (mem_cgroup_disabled())
579580 return 0;
580− return __mem_cgroup_try_charge_swap(folio, entry);
581+ return __mem_cgroup_try_charge_swap(folio);
581582 }
582583
583−extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
584−static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
584+extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
585+static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
585586 {
586587 if (mem_cgroup_disabled())
587588 return;
588− __mem_cgroup_uncharge_swap(entry, nr_pages);
589+ __mem_cgroup_uncharge_swap(id, nr_pages);
589590 }
590591
591592 extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
592593 extern bool mem_cgroup_swap_full(struct folio *folio);
593594 #else
594−static inline int mem_cgroup_try_charge_swap(struct folio *folio,
595− swp_entry_t entry)
595+static inline int mem_cgroup_try_charge_swap(struct folio *folio)
596596 {
597597 return 0;
598598 }
599599
600−static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,
600+static inline void mem_cgroup_uncharge_swap(unsigned short id,
601601 unsigned int nr_pages)
602602 {
603603 }
include/linux/swap_cgroup.h+0−47
@@ -1,47 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0 */
2−#ifndef __LINUX_SWAP_CGROUP_H
3−#define __LINUX_SWAP_CGROUP_H
4−
5−#include <linux/swap.h>
6−
7−#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
8−
9−extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
10−extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
11−extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
12−extern int swap_cgroup_swapon(int type, unsigned long max_pages);
13−extern void swap_cgroup_swapoff(int type);
14−
15−#else
16−
17−static inline
18−void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
19−{
20−}
21−
22−static inline
23−unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
24−{
25− return 0;
26−}
27−
28−static inline
29−unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
30−{
31− return 0;
32−}
33−
34−static inline int
35−swap_cgroup_swapon(int type, unsigned long max_pages)
36−{
37− return 0;
38−}
39−
40−static inline void swap_cgroup_swapoff(int type)
41−{
42− return;
43−}
44−
45−#endif
46−
47−#endif /* __LINUX_SWAP_CGROUP_H */
include/linux/thread_info.h+1−1
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart, · 91 unmodified lines
9292 #define THREAD_ALIGN THREAD_SIZE
9393 #endif
9494
95−#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO)
95+#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)
9696
9797 /*
9898 * flag set/clear/test wrappers
include/linux/userfaultfd_k.h+2−38
@@ -23,8 +23,8 @@ · 22 unmodified lines
2323 /* The set of all possible UFFD-related VM flags. */
2424 #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
2525
26−#define __VMA_UFFD_FLAGS mk_vma_flags(VMA_UFFD_MISSING_BIT, VMA_UFFD_WP_BIT, \
27− VMA_UFFD_MINOR_BIT)
26+#define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \
27+ VMA_UFFD_MINOR)
2828
2929 /*
3030 * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -147,26 +147,12 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at · 116 unmodified lines
147147 /* Flags controlling behavior. These behavior changes are mode-independent. */
148148 #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
149149
150−extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
151− unsigned long src_start, unsigned long len,
152− uffd_flags_t flags);
153−extern ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
154− unsigned long dst_start,
155− unsigned long len);
156−extern ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long dst_start,
157− unsigned long len, uffd_flags_t flags);
158−extern ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
159− unsigned long len, uffd_flags_t flags);
160−extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
161− unsigned long len, bool enable_wp);
162150 extern long uffd_wp_range(struct vm_area_struct *vma,
163151 unsigned long start, unsigned long len, bool enable_wp);
164152
165153 /* move_pages */
166154 void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
167155 void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
168−ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
169− unsigned long src_start, unsigned long len, __u64 flags);
170156 int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,
171157 struct vm_area_struct *dst_vma,
172158 struct vm_area_struct *src_vma,
@@ -239,9 +225,6 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 66 unmodified lines
239225 return vma->vm_flags & __VM_UFFD_FLAGS;
240226 }
241227
242−bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
243− bool wp_async);
244−
245228 static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
246229 {
247230 struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
@@ -271,25 +254,6 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm, · 23 unmodified lines
271254 extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
272255 extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
273256
274−void userfaultfd_reset_ctx(struct vm_area_struct *vma);
275−
276−struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
277− struct vm_area_struct *prev,
278− struct vm_area_struct *vma,
279− unsigned long start,
280− unsigned long end);
281−
282−int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
283− struct vm_area_struct *vma,
284− vm_flags_t vm_flags,
285− unsigned long start, unsigned long end,
286− bool wp_async);
287−
288−void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
289−
290−void userfaultfd_release_all(struct mm_struct *mm,
291− struct userfaultfd_ctx *ctx);
292−
293257 static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
294258 {
295259 /* Only wr-protect mode uses pte markers */
include/linux/vmalloc.h+3−1
@@ -265,7 +265,9 @@ static inline bool is_vm_area_hugepages(const void *addr) · 264 unmodified lines
265265 * allocated in the vmalloc layer.
266266 */
267267 #ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
268− return find_vm_area(addr)->page_order > 0;
268+ struct vm_struct *area = find_vm_area(addr);
269+
270+ return area && area->page_order > 0;
269271 #else
270272 return false;
271273 #endif
include/linux/vmpressure.h+5−4
@@ -30,8 +30,8 @@ struct vmpressure { · 29 unmodified lines
3030 struct mem_cgroup;
3131
3232 #ifdef CONFIG_MEMCG
33−extern void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,
34− unsigned long scanned, unsigned long reclaimed);
33+void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
34+ unsigned long scanned, unsigned long reclaimed);
3535 extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);
3636
3737 extern void vmpressure_init(struct vmpressure *vmpr);
@@ -44,8 +44,9 @@ extern int vmpressure_register_event(struct mem_cgroup *memcg, · 6 unmodified lines
4444 extern void vmpressure_unregister_event(struct mem_cgroup *memcg,
4545 struct eventfd_ctx *eventfd);
4646 #else
47−static inline void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,
48− unsigned long scanned, unsigned long reclaimed) {}
47+static inline void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg,
48+ bool tree, unsigned long scanned,
49+ unsigned long reclaimed) {}
4950 static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,
5051 int prio) {}
5152 #endif /* CONFIG_MEMCG */
include/trace/events/damon.h+38−0
@@ -130,6 +130,44 @@ TRACE_EVENT(damon_monitor_intervals_tune, · 129 unmodified lines
130130 TP_printk("sample_us=%lu", __entry->sample_us)
131131 );
132132
133+TRACE_EVENT_CONDITION(damon_region_aggregated,
134+
135+ TP_PROTO(unsigned int target_id, struct damon_region *r,
136+ unsigned int nr_regions, unsigned int nr_probes),
137+
138+ TP_ARGS(target_id, r, nr_regions, nr_probes),
139+
140+ TP_CONDITION(nr_probes > 0),
141+
142+ TP_STRUCT__entry(
143+ __field(unsigned long, target_id)
144+ __field(unsigned long, start)
145+ __field(unsigned long, end)
146+ __field(unsigned int, nr_regions)
147+ __field(unsigned int, nr_accesses)
148+ __field(unsigned int, age)
149+ __dynamic_array(unsigned char, probe_hits, nr_probes)
150+ ),
151+
152+ TP_fast_assign(
153+ __entry->target_id = target_id;
154+ __entry->start = r->ar.start;
155+ __entry->end = r->ar.end;
156+ __entry->nr_regions = nr_regions;
157+ __entry->nr_accesses = r->nr_accesses;
158+ __entry->age = r->age;
159+ memcpy(__get_dynamic_array(probe_hits), r->probe_hits,
160+ sizeof(*r->probe_hits) * nr_probes);
161+ ),
162+
163+ TP_printk("target_id=%lu nr_regions=%u %lu-%lu: %u %u probe_hits=%s",
164+ __entry->target_id, __entry->nr_regions,
165+ __entry->start, __entry->end,
166+ __entry->nr_accesses, __entry->age,
167+ __print_hex(__get_dynamic_array(probe_hits),
168+ __get_dynamic_array_len(probe_hits)))
169+);
170+
133171 TRACE_EVENT(damon_aggregated,
134172
135173 TP_PROTO(unsigned int target_id, struct damon_region *r,
include/trace/events/vmscan.h+52−0
@@ -96,6 +96,58 @@ TRACE_EVENT(mm_vmscan_kswapd_wake, · 95 unmodified lines
9696 __entry->order)
9797 );
9898
99+TRACE_EVENT(mm_vmscan_balance_pgdat_begin,
100+
101+ TP_PROTO(int nid, int order, int highest_zoneidx),
102+
103+ TP_ARGS(nid, order, highest_zoneidx),
104+
105+ TP_STRUCT__entry(
106+ __field(int, nid)
107+ __field(int, order)
108+ __field(int, highest_zoneidx)
109+ ),
110+
111+ TP_fast_assign(
112+ __entry->nid = nid;
113+ __entry->order = order;
114+ __entry->highest_zoneidx = highest_zoneidx;
115+ ),
116+
117+ TP_printk("nid=%d order=%d highest_zoneidx=%-8s",
118+ __entry->nid,
119+ __entry->order,
120+ __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE))
121+);
122+
123+TRACE_EVENT(mm_vmscan_balance_pgdat_end,
124+
125+ TP_PROTO(int nid, int order, int highest_zoneidx,
126+ unsigned long nr_reclaimed),
127+
128+ TP_ARGS(nid, order, highest_zoneidx, nr_reclaimed),
129+
130+ TP_STRUCT__entry(
131+ __field(int, nid)
132+ __field(int, order)
133+ __field(int, highest_zoneidx)
134+ __field(unsigned long, nr_reclaimed)
135+ ),
136+
137+ TP_fast_assign(
138+ __entry->nid = nid;
139+ __entry->order = order;
140+ __entry->highest_zoneidx = highest_zoneidx;
141+ __entry->nr_reclaimed = nr_reclaimed;
142+ ),
143+
144+ TP_printk("nid=%d order=%d highest_zoneidx=%-8s nr_reclaimed=%lu",
145+ __entry->nid,
146+ __entry->order,
147+ __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE),
148+ __entry->nr_reclaimed)
149+);
150+
99151 TRACE_EVENT(mm_vmscan_wakeup_kswapd,
100152
101153 TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),
kernel/bpf/arena.c+1−1
@@ -891,7 +891,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 890 unmodified lines
891891
892892 llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {
893893 page = llist_entry(pos, struct page, pcp_llist);
894− if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */
894+ if (page_cnt == 1 && page_ref_count(page) > 1) /* maybe mapped by user space */
895895 /* Optimization for the common case of page_cnt==1:
896896 * If page wasn't mapped into some user vma there
897897 * is no need to call zap_pages which is slow. When
kernel/fork.c+3−2
@@ -205,7 +205,7 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]); · 204 unmodified lines
205205 * accounting is performed by the code assigning/releasing stacks to tasks.
206206 * We need a zeroed memory without __GFP_ACCOUNT.
207207 */
208−#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO)
208+#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
209209
210210 struct vm_stack {
211211 struct rcu_head rcu;
@@ -343,7 +343,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) · 131 unmodified lines
343343 }
344344
345345 /* Reset stack metadata. */
346− kasan_unpoison_range(vm_area->addr, THREAD_SIZE);
346+ if (!kasan_hw_tags_enabled())
347+ kasan_unpoison_range(vm_area->addr, THREAD_SIZE);
347348
348349 stack = kasan_reset_tag(vm_area->addr);
349350
kernel/power/snapshot.c+4−4
@@ -1244,8 +1244,9 @@ unsigned int snapshot_additional_pages(struct zone *zone) · 1243 unmodified lines
12441244 static void mark_free_pages(struct zone *zone)
12451245 {
12461246 unsigned long pfn, max_zone_pfn, page_count = WD_PAGE_COUNT;
1247+ struct list_head *free_list;
12471248 unsigned long flags;
1248− unsigned int order, t;
1249+ unsigned int order;
12491250 struct page *page;
12501251
12511252 if (zone_is_empty(zone))
@@ -1269,9 +1270,8 @@ static void mark_free_pages(struct zone *zone) · 17 unmodified lines
12691270 swsusp_unset_page_free(page);
12701271 }
12711272
1272− for_each_migratetype_order(order, t) {
1273− list_for_each_entry(page,
1274− &zone->free_area[order].free_list[t], buddy_list) {
1273+ for_each_free_list(free_list, zone, order) {
1274+ list_for_each_entry(page, free_list, buddy_list) {
12751275 unsigned long i;
12761276
12771277 pfn = page_to_pfn(page);
kernel/power/swap.c+1−1
@@ -341,7 +341,7 @@ static int swsusp_swap_check(void) · 340 unmodified lines
341341 * This is called before saving the image.
342342 */
343343 if (swsusp_resume_device)
344− res = swap_type_of(swsusp_resume_device, swsusp_resume_block);
344+ res = find_hibernation_swap_type(swsusp_resume_device, swsusp_resume_block);
345345 else
346346 res = find_first_swap(&swsusp_resume_device);
347347 if (res < 0)
kernel/power/user.c+12−3
@@ -71,7 +71,7 @@ static int snapshot_open(struct inode *inode, struct file *filp) · 70 unmodified lines
7171 memset(&data->handle, 0, sizeof(struct snapshot_handle));
7272 if ((filp->f_flags & O_ACCMODE) == O_RDONLY) {
7373 /* Hibernating. The image device should be accessible. */
74− data->swap = swap_type_of(swsusp_resume_device, 0);
74+ data->swap = pin_hibernation_swap_type(swsusp_resume_device, 0);
7575 data->mode = O_RDONLY;
7676 data->free_bitmaps = false;
7777 error = pm_notifier_call_chain_robust(PM_HIBERNATION_PREPARE, PM_POST_HIBERNATION);
@@ -90,8 +90,10 @@ static int snapshot_open(struct inode *inode, struct file *filp) · 12 unmodified lines
9090 data->free_bitmaps = !error;
9191 }
9292 }
93− if (error)
93+ if (error) {
94+ unpin_hibernation_swap_type(data->swap);
9495 hibernate_release();
96+ }
9597
9698 data->frozen = false;
9799 data->ready = false;
@@ -115,6 +117,7 @@ static int snapshot_release(struct inode *inode, struct file *filp) · 17 unmodified lines
115117 data = filp->private_data;
116118 data->dev = 0;
117119 free_all_swap_pages(data->swap);
120+ unpin_hibernation_swap_type(data->swap);
118121 if (data->frozen) {
119122 pm_restore_gfp_mask();
120123 free_basic_memory_bitmaps();
@@ -235,11 +238,17 @@ static int snapshot_set_swap_area(struct snapshot_data *data, · 114 unmodified lines
235238 offset = swap_area.offset;
236239 }
237240
241+ /*
242+ * Unpin the swap device if a swap area was already
243+ * set by SNAPSHOT_SET_SWAP_AREA.
244+ */
245+ unpin_hibernation_swap_type(data->swap);
246+
238247 /*
239248 * User space encodes device types as two-byte values,
240249 * so we need to recode them
241250 */
242− data->swap = swap_type_of(swdev, offset);
251+ data->swap = pin_hibernation_swap_type(swdev, offset);
243252 if (data->swap < 0)
244253 return swdev ? -ENODEV : -EINVAL;
245254 data->dev = swdev;
lib/maple_tree.c+5−2
@@ -5727,13 +5727,16 @@ int mtree_store(struct maple_tree *mt, unsigned long index, void *entry, · 5726 unmodified lines
57275727 EXPORT_SYMBOL(mtree_store);
57285728
57295729 /**
5730− * mtree_insert_range() - Insert an entry at a given range if there is no value.
5730+ * mtree_insert_range() - Insert an entry from [first, last] at a given range
5731+ * if there is no value.
57315732 * @mt: The maple tree
57325733 * @first: The start of the range
5733− * @last: The end of the range
5734+ * @last: The end of the range (inclusive)
57345735 * @entry: The entry to store
57355736 * @gfp: The GFP_FLAGS to use for allocations.
57365737 *
5738+ * Note that @last is inclusive. That is, @last = @first + length - 1;
5739+ *
57375740 * Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid
57385741 * request, -ENOMEM if memory could not be allocated.
57395742 */
lib/test_hmm.c+40−9
@@ -1063,6 +1063,25 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, · 1062 unmodified lines
10631063 /* Try with smaller pages if large allocation fails */
10641064 if (!dpage && order) {
10651065 dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr);
1066+ if (!dpage) {
1067+ /* Unlock and free pages already allocated. */
1068+ while (i > 0) {
1069+ struct page *fpage;
1070+
1071+ fpage = migrate_pfn_to_page(dst[--i]);
1072+ unlock_page(fpage);
1073+ __free_page(fpage);
1074+ }
1075+ /* Clear remaining dst entries to avoid
1076+ * migrate_vma_pages/finalize() using
1077+ * uninitialized values.
1078+ */
1079+ while (i < (1 << order)) {
1080+ dst[i] = 0;
1081+ i++;
1082+ }
1083+ return VM_FAULT_OOM;
1084+ }
10661085 lock_page(dpage);
10671086 dst[i] = migrate_pfn(page_to_pfn(dpage));
10681087 dst_page = pfn_to_page(page_to_pfn(dpage));
@@ -1111,9 +1130,6 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 42 unmodified lines
11111130 unsigned long *src_pfns;
11121131 unsigned long *dst_pfns;
11131132
1114− src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
1115− dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
1116−
11171133 start = cmd->addr;
11181134 end = start + size;
11191135 if (end < start)
@@ -1123,6 +1139,9 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 3 unmodified lines
11231139 if (!mmget_not_zero(mm))
11241140 return -EINVAL;
11251141
1142+ src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
1143+ dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
1144+
11261145 cmd->cpages = 0;
11271146 mmap_read_lock(mm);
11281147 for (addr = start; addr < end; addr = next) {
@@ -1148,7 +1167,11 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror, · 19 unmodified lines
11481167 goto out;
11491168
11501169 pr_debug("Migrating from device mem to sys mem\n");
1151− dmirror_devmem_fault_alloc_and_copy(&args, dmirror);
1170+ if (dmirror_devmem_fault_alloc_and_copy(&args, dmirror)) {
1171+ migrate_vma_finalize(&args);
1172+ ret = -ENOMEM;
1173+ goto out;
1174+ }
11521175
11531176 migrate_vma_pages(&args);
11541177 cmd->cpages += dmirror_successful_migrated_pages(&args);
@@ -1253,8 +1276,8 @@ out: · 98 unmodified lines
12531276 mmap_read_unlock(mm);
12541277 mmput(mm);
12551278 free_mem:
1256− kfree(src_pfns);
1257− kfree(dst_pfns);
1279+ kvfree(src_pfns);
1280+ kvfree(dst_pfns);
12581281 return ret;
12591282 }
12601283
@@ -1679,12 +1702,20 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) · 418 unmodified lines
16791702 if (order)
16801703 args.flags |= MIGRATE_VMA_SELECT_COMPOUND;
16811704
1682− if (migrate_vma_setup(&args))
1683− return VM_FAULT_SIGBUS;
1705+ /*
1706+ * In practice migrate_vma_setup() should never fail unless the
1707+ * test is wrong as it just tests some static VMA properties.
1708+ */
1709+ if (migrate_vma_setup(&args)) {
1710+ ret = VM_FAULT_SIGBUS;
1711+ goto err;
1712+ }
16841713
16851714 ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror);
1686− if (ret)
1715+ if (ret) {
1716+ migrate_vma_finalize(&args);
16871717 goto err;
1718+ }
16881719 migrate_vma_pages(&args);
16891720 /*
16901721 * No device finalize step is needed since
lib/test_meminit.c+1−1
@@ -386,7 +386,7 @@ static int __init test_kmemcache(int *total_failures) · 385 unmodified lines
386386 ctor = flags & 1;
387387 rcu = flags & 2;
388388 zero = flags & 4;
389− if (ctor & zero)
389+ if (ctor && zero)
390390 continue;
391391 num_tests += do_kmem_cache_size(size, ctor, rcu, zero,
392392 &failures);
lib/test_vmalloc.c+62−0
@@ -55,6 +55,7 @@ __param(int, run_test_mask, 7, · 54 unmodified lines
5555 "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n"
5656 "\t\tid: 1024, name: vm_map_ram_test\n"
5757 "\t\tid: 2048, name: no_block_alloc_test\n"
58+ "\t\tid: 4096, name: vrealloc_test\n"
5859 /* Add a new test case description here. */
5960 );
6061
@@ -421,6 +422,66 @@ cleanup: · 360 unmodified lines
421422 return nr_allocated != map_nr_pages;
422423 }
423424
425+static int vrealloc_test(void)
426+{
427+ void *ptr, *tmp;
428+ int i;
429+
430+ for (i = 0; i < test_loop_count; i++) {
431+ int err = -1;
432+
433+ ptr = vrealloc(NULL, PAGE_SIZE, GFP_KERNEL);
434+ if (!ptr)
435+ return -1;
436+
437+ *((__u8 *)ptr) = 'a';
438+
439+ /* Grow: beyond allocated pages, triggers full realloc. */
440+ tmp = vrealloc(ptr, 4 * PAGE_SIZE, GFP_KERNEL);
441+ if (!tmp)
442+ goto error;
443+ ptr = tmp;
444+
445+ if (*((__u8 *)ptr) != 'a')
446+ goto error;
447+
448+ /* Shrink: crosses page boundary, frees tail pages. */
449+ tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
450+ if (!tmp)
451+ goto error;
452+ ptr = tmp;
453+
454+ if (*((__u8 *)ptr) != 'a')
455+ goto error;
456+
457+ /* Shrink: within same page, no page freeing. */
458+ tmp = vrealloc(ptr, PAGE_SIZE / 2, GFP_KERNEL);
459+ if (!tmp)
460+ goto error;
461+ ptr = tmp;
462+
463+ if (*((__u8 *)ptr) != 'a')
464+ goto error;
465+
466+ /* Grow: within allocated page, in-place, no realloc. */
467+ tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
468+ if (!tmp)
469+ goto error;
470+ ptr = tmp;
471+
472+ if (*((__u8 *)ptr) != 'a')
473+ goto error;
474+
475+ err = 0;
476+error:
477+ vfree(ptr);
478+ if (err)
479+ return err;
480+ }
481+
482+ return 0;
483+}
484+
424485 struct test_case_desc {
425486 const char *test_name;
426487 int (*test_func)(void);
@@ -440,6 +501,7 @@ static struct test_case_desc test_case_array[] = { · 13 unmodified lines
440501 { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test, },
441502 { "vm_map_ram_test", vm_map_ram_test, },
442503 { "no_block_alloc_test", no_block_alloc_test, true },
504+ { "vrealloc_test", vrealloc_test, },
443505 /* Add a new test case here. */
444506 };
445507
mm/Kconfig+2−5
@@ -590,7 +590,7 @@ endchoice · 589 unmodified lines
590590
591591 config MEMORY_HOTREMOVE
592592 bool "Allow for memory hot remove"
593− select HAVE_BOOTMEM_INFO_NODE if (X86_64 || PPC64)
593+ select HAVE_BOOTMEM_INFO_NODE if X86_64
594594 depends on MEMORY_HOTPLUG
595595 select MIGRATION
596596
@@ -863,7 +863,6 @@ if TRANSPARENT_HUGEPAGE · 266 unmodified lines
863863
864864 choice
865865 prompt "Transparent Hugepage Support sysfs defaults"
866− depends on TRANSPARENT_HUGEPAGE
867866 default TRANSPARENT_HUGEPAGE_ALWAYS
868867 help
869868 Selects the sysfs defaults for Transparent Hugepage Support.
@@ -893,7 +892,6 @@ endchoice · 23 unmodified lines
893892
894893 choice
895894 prompt "Shmem hugepage allocation defaults"
896− depends on TRANSPARENT_HUGEPAGE
897895 default TRANSPARENT_HUGEPAGE_SHMEM_HUGE_NEVER
898896 help
899897 Selects the hugepage allocation policy defaults for
@@ -939,7 +937,6 @@ endchoice · 39 unmodified lines
939937
940938 choice
941939 prompt "Tmpfs hugepage allocation defaults"
942− depends on TRANSPARENT_HUGEPAGE
943940 default TRANSPARENT_HUGEPAGE_TMPFS_HUGE_NEVER
944941 help
945942 Selects the hugepage allocation policy defaults for
@@ -984,7 +981,7 @@ endchoice · 38 unmodified lines
984981
985982 config THP_SWAP
986983 def_bool y
987− depends on TRANSPARENT_HUGEPAGE && ARCH_WANTS_THP_SWAP && SWAP && 64BIT
984+ depends on ARCH_WANTS_THP_SWAP && SWAP && 64BIT
988985 help
989986 Swap transparent huge pages in one piece, without splitting.
990987 XXX: For now, swap cluster backing transparent huge page
mm/Makefile+0−3
@@ -103,9 +103,6 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o · 102 unmodified lines
103103 obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
104104 obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
105105 obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
106−ifdef CONFIG_SWAP
107−obj-$(CONFIG_MEMCG) += swap_cgroup.o
108−endif
109106 ifdef CONFIG_BPF_SYSCALL
110107 obj-$(CONFIG_MEMCG) += bpf_memcontrol.o
111108 endif
mm/bootmem_info.c+2−23
@@ -19,7 +19,6 @@ void get_page_bootmem(unsigned long info, struct page *page, · 18 unmodified lines
1919 {
2020 BUG_ON(type > 0xf);
2121 BUG_ON(info > (ULONG_MAX >> 4));
22− SetPagePrivate(page);
2322 set_page_private(page, info << 4 | type);
2423 page_ref_inc(page);
2524 }
@@ -32,20 +31,15 @@ void put_page_bootmem(struct page *page) · 6 unmodified lines
3231 type > MEMORY_HOTPLUG_MAX_BOOTMEM_TYPE);
3332
3433 if (page_ref_dec_return(page) == 1) {
35− ClearPagePrivate(page);
3634 set_page_private(page, 0);
37− INIT_LIST_HEAD(&page->lru);
38− kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
3935 free_reserved_page(page);
4036 }
4137 }
4238
4339 static void __init register_page_bootmem_info_section(unsigned long start_pfn)
4440 {
45− unsigned long mapsize, section_nr, i;
41+ unsigned long section_nr;
4642 struct mem_section *ms;
47− struct mem_section_usage *usage;
48− struct page *page;
4943
5044 start_pfn = SECTION_ALIGN_DOWN(start_pfn);
5145 section_nr = pfn_to_section_nr(start_pfn);
@@ -54,27 +48,12 @@ static void __init register_page_bootmem_info_section(unsigned long start_pfn) · 2 unmodified lines
5448 if (!preinited_vmemmap_section(ms))
5549 register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn),
5650 PAGES_PER_SECTION);
57−
58− usage = ms->usage;
59− page = virt_to_page(usage);
60−
61− mapsize = PAGE_ALIGN(mem_section_usage_size()) >> PAGE_SHIFT;
62−
63− for (i = 0; i < mapsize; i++, page++)
64− get_page_bootmem(section_nr, page, MIX_SECTION_INFO);
6551 }
6652
6753 void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
6854 {
69− unsigned long i, pfn, end_pfn, nr_pages;
55+ unsigned long pfn, end_pfn;
7056 int node = pgdat->node_id;
71− struct page *page;
72−
73− nr_pages = PAGE_ALIGN(sizeof(struct pglist_data)) >> PAGE_SHIFT;
74− page = virt_to_page(pgdat);
75−
76− for (i = 0; i < nr_pages; i++, page++)
77− get_page_bootmem(node, page, NODE_INFO);
7857
7958 pfn = pgdat->node_start_pfn;
8059 end_pfn = pgdat_end_pfn(pgdat);
mm/compaction.c+8−3
@@ -1123,7 +1123,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · 1122 unmodified lines
11231123 * To minimise LRU disruption, the caller can indicate with
11241124 * ISOLATE_ASYNC_MIGRATE that it only wants to isolate pages
11251125 * it will be able to migrate without blocking - clean pages
1126− * for the most part. PageWriteback would require blocking.
1126+ * for the most part. Writeback would require blocking.
11271127 */
11281128 if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio))
11291129 goto isolate_fail_put;
@@ -2340,7 +2340,8 @@ static enum compact_result __compact_finished(struct compact_control *cc) · 1210 unmodified lines
23402340 * Job done if allocation would steal freepages from
23412341 * other migratetype buddy lists.
23422342 */
2343− if (find_suitable_fallback(area, order, migratetype, true) >= 0)
2343+ if (find_suitable_fallback(area, order, migratetype, true, NULL)
2344+ == FALLBACK_FOUND)
23442345 /*
23452346 * Movable pages are OK in any pageblock. If we are
23462347 * stealing for a non-movable allocation, make sure
@@ -2447,7 +2448,7 @@ bool compaction_suitable(struct zone *zone, int order, unsigned long watermark, · 100 unmodified lines
24472448
24482449 /* Used by direct reclaimers */
24492450 bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
2450− int alloc_flags)
2451+ int alloc_flags, gfp_t gfp_mask)
24512452 {
24522453 struct zone *zone;
24532454 struct zoneref *z;
@@ -2460,6 +2461,10 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, · 6 unmodified lines
24602461 ac->highest_zoneidx, ac->nodemask) {
24612462 unsigned long available;
24622463
2464+ if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
2465+ !__cpuset_zone_allowed(zone, gfp_mask))
2466+ continue;
2467+
24632468 /*
24642469 * Do not consider all the reclaimable memory because we do not
24652470 * want to trash just for a single high order allocation which
mm/damon/core.c+578−77
@@ -13,10 +13,14 @@ · 12 unmodified lines
1313 #include <linux/memcontrol.h>
1414 #include <linux/mm.h>
1515 #include <linux/psi.h>
16+#include <linux/sched.h>
1617 #include <linux/slab.h>
1718 #include <linux/string.h>
1819 #include <linux/string_choices.h>
1920
21+/* for damon_get_folio() used by node eligible memory metrics */
22+#include "ops-common.h"
23+
2024 #define CREATE_TRACE_POINTS
2125 #include <trace/events/damon.h>
2226
@@ -109,6 +113,103 @@ int damon_select_ops(struct damon_ctx *ctx, enum damon_ops_id id) · 86 unmodified lines
109113 return err;
110114 }
111115
116+struct damon_filter *damon_new_filter(enum damon_filter_type type,
117+ bool matching, bool allow)
118+{
119+ struct damon_filter *filter;
120+
121+ filter = kmalloc_obj(*filter);
122+ if (!filter)
123+ return NULL;
124+ filter->type = type;
125+ filter->matching = matching;
126+ filter->allow = allow;
127+ INIT_LIST_HEAD(&filter->list);
128+ return filter;
129+}
130+
131+void damon_add_filter(struct damon_probe *p, struct damon_filter *f)
132+{
133+ list_add_tail(&f->list, &p->filters);
134+}
135+
136+static void damon_del_filter(struct damon_filter *f)
137+{
138+ list_del(&f->list);
139+}
140+
141+static void damon_free_filter(struct damon_filter *f)
142+{
143+ kfree(f);
144+}
145+
146+void damon_destroy_filter(struct damon_filter *f)
147+{
148+ damon_del_filter(f);
149+ damon_free_filter(f);
150+}
151+
152+static struct damon_filter *damon_nth_filter(int n, struct damon_probe *p)
153+{
154+ struct damon_filter *f;
155+ int i = 0;
156+
157+ damon_for_each_filter(f, p) {
158+ if (i++ == n)
159+ return f;
160+ }
161+ return NULL;
162+}
163+
164+struct damon_probe *damon_new_probe(void)
165+{
166+ struct damon_probe *p;
167+
168+ p = kmalloc_obj(*p);
169+ if (!p)
170+ return NULL;
171+ INIT_LIST_HEAD(&p->filters);
172+ INIT_LIST_HEAD(&p->list);
173+ return p;
174+}
175+
176+void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe)
177+{
178+ list_add_tail(&probe->list, &ctx->probes);
179+}
180+
181+static void damon_del_probe(struct damon_probe *p)
182+{
183+ list_del(&p->list);
184+}
185+
186+static void damon_free_probe(struct damon_probe *p)
187+{
188+ struct damon_filter *f, *next;
189+
190+ damon_for_each_filter_safe(f, next, p)
191+ damon_free_filter(f);
192+ kfree(p);
193+}
194+
195+static void damon_destroy_probe(struct damon_probe *p)
196+{
197+ damon_del_probe(p);
198+ damon_free_probe(p);
199+}
200+
201+static struct damon_probe *damon_nth_probe(int n, struct damon_ctx *ctx)
202+{
203+ struct damon_probe *p;
204+ int i = 0;
205+
206+ damon_for_each_probe(p, ctx) {
207+ if (i++ == n)
208+ return p;
209+ }
210+ return NULL;
211+}
212+
112213 #ifdef CONFIG_DAMON_DEBUG_SANITY
113214 static void damon_verify_new_region(unsigned long start, unsigned long end)
114215 {
@@ -128,6 +229,7 @@ static void damon_verify_new_region(unsigned long start, unsigned long end) · 13 unmodified lines
128229 struct damon_region *damon_new_region(unsigned long start, unsigned long end)
129230 {
130231 struct damon_region *region;
232+ int i;
131233
132234 damon_verify_new_region(start, end);
133235 region = kmem_cache_alloc(damon_region_cache, GFP_KERNEL);
@@ -138,6 +240,8 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 4 unmodified lines
138240 region->ar.end = end;
139241 region->nr_accesses = 0;
140242 region->nr_accesses_bp = 0;
243+ for (i = 0; i < DAMON_MAX_PROBES; i++)
244+ region->probe_hits[i] = 0;
141245 INIT_LIST_HEAD(&region->list);
142246
143247 region->age = 0;
@@ -146,12 +250,23 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 2 unmodified lines
146250 return region;
147251 }
148252
149−void damon_add_region(struct damon_region *r, struct damon_target *t)
253+static void damon_add_region(struct damon_region *r, struct damon_target *t)
150254 {
151255 list_add_tail(&r->list, &t->regions_list);
152256 t->nr_regions++;
153257 }
154258
259+/*
260+ * Add a region between two other regions
261+ */
262+static inline void damon_insert_region(struct damon_region *r,
263+ struct damon_region *prev, struct damon_region *next,
264+ struct damon_target *t)
265+{
266+ __list_add(&r->list, &prev->list, &next->list);
267+ t->nr_regions++;
268+}
269+
155270 #ifdef CONFIG_DAMON_DEBUG_SANITY
156271 static void damon_verify_del_region(struct damon_target *t)
157272 {
@@ -176,7 +291,8 @@ static void damon_free_region(struct damon_region *r) · 18 unmodified lines
176291 kmem_cache_free(damon_region_cache, r);
177292 }
178293
179−void damon_destroy_region(struct damon_region *r, struct damon_target *t)
294+static void damon_destroy_region(struct damon_region *r,
295+ struct damon_target *t)
180296 {
181297 damon_del_region(r, t);
182298 damon_free_region(r);
@@ -252,11 +368,25 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 69 unmodified lines
252368 damon_destroy_region(r, t);
253369 }
254370
371+ if (!damon_nr_regions(t)) {
372+ for (i = 0; i < nr_ranges; i++) {
373+ r = damon_new_region(
374+ ALIGN_DOWN(ranges[i].start,
375+ min_region_sz),
376+ ALIGN(ranges[i].end, min_region_sz));
377+ if (!r)
378+ return -ENOMEM;
379+ damon_add_region(r, t);
380+ }
381+ return 0;
382+ }
383+
255384 r = damon_first_region(t);
256385 /* Add new regions or resize existing regions to fit in the ranges */
257386 for (i = 0; i < nr_ranges; i++) {
258387 struct damon_region *first = NULL, *last, *newr;
259388 struct damon_addr_range *range;
389+ bool insert_before_r = false;
260390
261391 range = &ranges[i];
262392 /* Get the first/last regions intersecting with the range */
@@ -266,8 +396,10 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 3 unmodified lines
266396 first = r;
267397 last = r;
268398 }
269− if (r->ar.start >= range->end)
399+ if (r->ar.start >= range->end) {
400+ insert_before_r = true;
270401 break;
402+ }
271403 }
272404 if (!first) {
273405 /* no region intersects with this range */
@@ -277,7 +409,11 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 3 unmodified lines
277409 ALIGN(range->end, min_region_sz));
278410 if (!newr)
279411 return -ENOMEM;
280− damon_insert_region(newr, damon_prev_region(r), r, t);
412+ if (insert_before_r)
413+ damon_insert_region(newr, damon_prev_region(r),
414+ r, t);
415+ else
416+ damon_add_region(newr, t);
281417 } else {
282418 /* resize intersecting regions to fit in this range */
283419 first->ar.start = ALIGN_DOWN(range->start,
@@ -550,27 +686,8 @@ void damon_destroy_target(struct damon_target *t, struct damon_ctx *ctx) · 266 unmodified lines
550686 damon_free_target(t);
551687 }
552688
553−#ifdef CONFIG_DAMON_DEBUG_SANITY
554−static void damon_verify_nr_regions(struct damon_target *t)
555−{
556− struct damon_region *r;
557− unsigned int count = 0;
558−
559− damon_for_each_region(r, t)
560− count++;
561− WARN_ONCE(count != t->nr_regions, "t->nr_regions (%u) != count (%u)\n",
562− t->nr_regions, count);
563−}
564−#else
565−static void damon_verify_nr_regions(struct damon_target *t)
566−{
567−}
568−#endif
569−
570689 unsigned int damon_nr_regions(struct damon_target *t)
571690 {
572− damon_verify_nr_regions(t);
573−
574691 return t->nr_regions;
575692 }
576693
@@ -601,12 +718,16 @@ struct damon_ctx *damon_new_ctx(void) · 24 unmodified lines
601718 ctx->attrs.min_nr_regions = 10;
602719 ctx->attrs.max_nr_regions = 1000;
603720
721+ INIT_LIST_HEAD(&ctx->probes);
722+
604723 ctx->addr_unit = 1;
605724 ctx->min_region_sz = DAMON_MIN_REGION_SZ;
606725
607726 INIT_LIST_HEAD(&ctx->adaptive_targets);
608727 INIT_LIST_HEAD(&ctx->schemes);
609728
729+ prandom_seed_state(&ctx->rnd_state, get_random_u64());
730+
610731 return ctx;
611732 }
612733
@@ -621,12 +742,16 @@ static void damon_destroy_targets(struct damon_ctx *ctx) · 8 unmodified lines
621742 void damon_destroy_ctx(struct damon_ctx *ctx)
622743 {
623744 struct damos *s, *next_s;
745+ struct damon_probe *p, *next_p;
624746
625747 damon_destroy_targets(ctx);
626748
627749 damon_for_each_scheme_safe(s, next_s, ctx)
628750 damon_destroy_scheme(s);
629751
752+ damon_for_each_probe_safe(p, next_p, ctx)
753+ damon_destroy_probe(p);
754+
630755 kfree(ctx);
631756 }
632757
@@ -797,6 +922,9 @@ int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) · 164 unmodified lines
797922 attrs->aggr_interval / sample_interval;
798923 ctx->next_ops_update_sis = ctx->passed_sample_intervals +
799924 attrs->ops_update_interval / sample_interval;
925+ /*
926+ * next_intervals_tune_sis will be updated inside kdamond_fn().
927+ */
800928
801929 damon_update_monitoring_results(ctx, attrs, aggregating);
802930 ctx->attrs = *attrs;
@@ -918,6 +1046,8 @@ static int damos_commit_quota(struct damos_quota *dst, struct damos_quota *src) · 115 unmodified lines
9181046 if (err)
9191047 return err;
9201048 dst->goal_tuner = src->goal_tuner;
1049+ dst->fail_charge_num = src->fail_charge_num;
1050+ dst->fail_charge_denom = src->fail_charge_denom;
9211051 dst->weight_sz = src->weight_sz;
9221052 dst->weight_nr_accesses = src->weight_nr_accesses;
9231053 dst->weight_age = src->weight_age;
@@ -1310,6 +1440,86 @@ static int damon_commit_targets( · 386 unmodified lines
13101440 return 0;
13111441 }
13121442
1443+static void damon_commit_filter(struct damon_filter *dst,
1444+ struct damon_filter *src)
1445+{
1446+ dst->type = src->type;
1447+ dst->matching = src->matching;
1448+ dst->allow = src->allow;
1449+ switch (dst->type) {
1450+ case DAMON_FILTER_TYPE_MEMCG:
1451+ dst->memcg_id = src->memcg_id;
1452+ break;
1453+ default:
1454+ break;
1455+ }
1456+}
1457+
1458+static int damon_commit_filters(struct damon_probe *dst,
1459+ struct damon_probe *src)
1460+{
1461+ struct damon_filter *dst_filter, *next, *src_filter, *new_filter;
1462+ int i = 0, j = 0;
1463+
1464+ damon_for_each_filter_safe(dst_filter, next, dst) {
1465+ src_filter = damon_nth_filter(i++, src);
1466+ if (src_filter)
1467+ damon_commit_filter(dst_filter, src_filter);
1468+ else
1469+ damon_destroy_filter(dst_filter);
1470+ }
1471+
1472+ damon_for_each_filter_safe(src_filter, next, src) {
1473+ if (j++ < i)
1474+ continue;
1475+
1476+ new_filter = damon_new_filter(src_filter->type,
1477+ src_filter->matching, src_filter->allow);
1478+ if (!new_filter)
1479+ return -ENOMEM;
1480+ switch (src_filter->type) {
1481+ case DAMON_FILTER_TYPE_MEMCG:
1482+ new_filter->memcg_id = src_filter->memcg_id;
1483+ break;
1484+ default:
1485+ break;
1486+ }
1487+ damon_add_filter(dst, new_filter);
1488+ }
1489+ return 0;
1490+}
1491+
1492+static int damon_commit_probes(struct damon_ctx *dst, struct damon_ctx *src)
1493+{
1494+ struct damon_probe *dst_probe, *next, *src_probe, *new_probe;
1495+ int i = 0, j = 0, err;
1496+
1497+ damon_for_each_probe_safe(dst_probe, next, dst) {
1498+ src_probe = damon_nth_probe(i++, src);
1499+ if (src_probe) {
1500+ err = damon_commit_filters(dst_probe, src_probe);
1501+ if (err)
1502+ return err;
1503+ } else {
1504+ damon_destroy_probe(dst_probe);
1505+ }
1506+ }
1507+
1508+ damon_for_each_probe_safe(src_probe, next, src) {
1509+ if (j++ < i)
1510+ continue;
1511+
1512+ new_probe = damon_new_probe();
1513+ if (!new_probe)
1514+ return -ENOMEM;
1515+ damon_add_probe(dst, new_probe);
1516+ err = damon_commit_filters(new_probe, src_probe);
1517+ if (err)
1518+ return err;
1519+ }
1520+ return 0;
1521+}
1522+
13131523 /**
13141524 * damon_commit_ctx() - Commit parameters of a DAMON context to another.
13151525 * @dst: The commit destination DAMON context.
@@ -1326,11 +1536,26 @@ static int damon_commit_targets( · 10 unmodified lines
13261536 int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src)
13271537 {
13281538 int err;
1539+ struct damos *scheme;
1540+ struct damos_quota_goal *goal;
13291541
13301542 dst->maybe_corrupted = true;
13311543 if (!is_power_of_2(src->min_region_sz))
13321544 return -EINVAL;
13331545
1546+ /* node_eligible_mem_bp metric requires PADDR ops */
1547+ if (src->ops.id != DAMON_OPS_PADDR) {
1548+ damon_for_each_scheme(scheme, src) {
1549+ struct damos_quota *quota = &scheme->quota;
1550+
1551+ damos_for_each_quota_goal(goal, quota) {
1552+ if (goal->metric ==
1553+ DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP)
1554+ return -EINVAL;
1555+ }
1556+ }
1557+ }
1558+
13341559 err = damon_commit_schemes(dst, src);
13351560 if (err)
13361561 return err;
@@ -1349,7 +1574,11 @@ int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src) · 12 unmodified lines
13491574 if (err)
13501575 return err;
13511576 }
1577+ dst->pause = src->pause;
13521578 dst->ops = src->ops;
1579+ err = damon_commit_probes(dst, src);
1580+ if (err)
1581+ return err;
13531582 dst->addr_unit = src->addr_unit;
13541583 dst->min_region_sz = src->min_region_sz;
13551584
@@ -1706,15 +1935,28 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) · 350 unmodified lines
17061935 {
17071936 struct damon_target *t;
17081937 unsigned int ti = 0; /* target's index */
1938+ unsigned int nr_probes = 0;
1939+ struct damon_probe *probe;
1940+
1941+ if (trace_damon_region_aggregated_enabled()) {
1942+ damon_for_each_probe(probe, c)
1943+ nr_probes++;
1944+ }
17091945
17101946 damon_for_each_target(t, c) {
17111947 struct damon_region *r;
17121948
17131949 damon_for_each_region(r, t) {
1950+ int i;
1951+
17141952 trace_damon_aggregated(ti, r, damon_nr_regions(t));
1953+ trace_damon_region_aggregated(ti, r,
1954+ damon_nr_regions(t), nr_probes);
17151955 damon_warn_fix_nr_accesses_corruption(r);
17161956 r->last_nr_accesses = r->nr_accesses;
17171957 r->nr_accesses = 0;
1958+ for (i = 0; i < DAMON_MAX_PROBES; i++)
1959+ r->probe_hits[i] = 0;
17181960 damon_verify_reset_aggregated(r, c);
17191961 }
17201962 ti++;
@@ -2046,6 +2288,37 @@ static void damos_walk_cancel(struct damon_ctx *ctx) · 325 unmodified lines
20462288 mutex_unlock(&ctx->walk_control_lock);
20472289 }
20482290
2291+static void damos_charge_quota(struct damos_quota *quota,
2292+ unsigned long sz_region, unsigned long sz_applied)
2293+{
2294+ /*
2295+ * sz_applied could be bigger than sz_region, depending on ops
2296+ * implementation of the action, e.g., damos_pa_pageout(). Charge only
2297+ * the region size in the case.
2298+ */
2299+ if (!quota->fail_charge_denom || sz_applied > sz_region)
2300+ quota->charged_sz += sz_region;
2301+ else
2302+ quota->charged_sz += sz_applied + mult_frac(
2303+ (sz_region - sz_applied),
2304+ quota->fail_charge_num,
2305+ quota->fail_charge_denom);
2306+}
2307+
2308+static bool damos_quota_is_full(struct damos_quota *quota,
2309+ unsigned long min_region_sz)
2310+{
2311+ if (!damos_quota_is_set(quota))
2312+ return false;
2313+ if (quota->charged_sz >= quota->esz)
2314+ return true;
2315+ /*
2316+ * DAMOS action is applied per region, so <min_region_sz remaining
2317+ * quota means the quota is effectively full.
2318+ */
2319+ return quota->esz - quota->charged_sz < min_region_sz;
2320+}
2321+
20492322 static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t,
20502323 struct damon_region *r, struct damos *s)
20512324 {
@@ -2102,11 +2375,10 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 50 unmodified lines
21022375 ktime_get_coarse_ts64(&end);
21032376 quota->total_charged_ns += timespec64_to_ns(&end) -
21042377 timespec64_to_ns(&begin);
2105− quota->charged_sz += sz;
2106− if (damos_quota_is_set(quota) &&
2107− quota->charged_sz >= quota->esz) {
2378+ damos_charge_quota(quota, sz, sz_applied);
2379+ if (damos_quota_is_full(quota, c->min_region_sz)) {
21082380 quota->charge_target_from = t;
2109− quota->charge_addr_from = r->ar.end + 1;
2381+ quota->charge_addr_from = r->ar.end;
21102382 }
21112383 }
21122384 if (s->action != DAMOS_STAT)
@@ -2132,8 +2404,7 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 19 unmodified lines
21322404 continue;
21332405
21342406 /* Check the quota */
2135− if (damos_quota_is_set(quota) &&
2136− quota->charged_sz >= quota->esz)
2407+ if (damos_quota_is_full(quota, c->min_region_sz))
21372408 continue;
21382409
21392410 if (damos_skip_charged_region(t, r, s, c->min_region_sz))
@@ -2151,6 +2422,58 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 11 unmodified lines
21512422 }
21522423 }
21532424
2425+/*
2426+ * damos_apply_target() - Apply DAMOS schemes to a given target.
2427+ * @c: monitoring context to apply its DAMOS schemes to..
2428+ * @t: monitoring target to apply the schemes to.
2429+ * @max_region_sz: maximum region size for @c.
2430+ *
2431+ * This function could split regions for keeping the quota. To minimize
2432+ * overhead from the split operations increased number of regions, this
2433+ * function will also merge regions after the schemes applying attempt is done,
2434+ * for each region. The merge operation is made only when it doesn't lose the
2435+ * monitoring information and not violating @max_region_sz.
2436+ *
2437+ * Hence, after this function is called, the total number of regions could
2438+ * be increased or reduced. The increase could make max_nr_regions temporarily
2439+ * be violated, until the next per-aggregation interval regions merge operation
2440+ * is executed. The decrease will not violate min_nr_regions though, since it
2441+ * keeps @max_region_sz.
2442+ */
2443+static void damos_apply_target(struct damon_ctx *c, struct damon_target *t,
2444+ unsigned long max_region_sz)
2445+{
2446+ struct damon_region *r;
2447+
2448+ damon_for_each_region(r, t) {
2449+ struct damon_region *prev_r;
2450+
2451+ damon_do_apply_schemes(c, t, r);
2452+ /*
2453+ * damon_do_apply_scheems() could split the region for the
2454+ * quota. Keeping the new slices is an overhead. Merge back
2455+ * the slices into the previous region if it doesn't lose any
2456+ * information and not violating the max_region_sz.
2457+ */
2458+ if (damon_first_region(t) == r)
2459+ continue;
2460+ prev_r = damon_prev_region(r);
2461+ if (prev_r->ar.end != r->ar.start)
2462+ continue;
2463+ if (prev_r->age != r->age)
2464+ continue;
2465+ if (prev_r->last_nr_accesses != r->last_nr_accesses)
2466+ continue;
2467+ if (prev_r->nr_accesses != r->nr_accesses)
2468+ continue;
2469+ if (r->ar.end - prev_r->ar.start > max_region_sz)
2470+ continue;
2471+ prev_r->ar.end = r->ar.end;
2472+ damon_destroy_region(r, t);
2473+ r = prev_r;
2474+ }
2475+}
2476+
21542477 /*
21552478 * damon_feed_loop_next_input() - get next input to achieve a target score.
21562479 * @last_input The last input.
@@ -2287,7 +2610,115 @@ static unsigned long damos_get_node_memcg_used_bp( · 130 unmodified lines
22872610 numerator = i.totalram - used_pages;
22882611 return mult_frac(numerator, 10000, i.totalram);
22892612 }
2290−#else
2613+
2614+#ifdef CONFIG_DAMON_PADDR
2615+/*
2616+ * damos_calc_eligible_bytes() - Calculate raw eligible bytes per node.
2617+ * @c: The DAMON context.
2618+ * @s: The scheme.
2619+ * @nid: The target NUMA node id.
2620+ * @total: Output for total eligible bytes across all nodes.
2621+ *
2622+ * Iterates through each folio in eligible regions to accurately determine
2623+ * which node the memory resides on. Returns eligible bytes on the specified
2624+ * node and sets *total to the sum across all nodes.
2625+ *
2626+ * Note: This function requires damon_get_folio() from ops-common.c, which is
2627+ * only available when CONFIG_DAMON_PADDR is enabled. It also requires the
2628+ * context to be using PADDR operations for meaningful results.
2629+ */
2630+static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c,
2631+ struct damos *s, int nid, phys_addr_t *total)
2632+{
2633+ struct damon_target *t;
2634+ struct damon_region *r;
2635+ phys_addr_t total_eligible = 0;
2636+ phys_addr_t node_eligible = 0;
2637+
2638+ damon_for_each_target(t, c) {
2639+ damon_for_each_region(r, t) {
2640+ phys_addr_t addr, end_addr;
2641+
2642+ if (!__damos_valid_target(r, s))
2643+ continue;
2644+
2645+ /* Convert from core address units to physical bytes */
2646+ addr = (phys_addr_t)r->ar.start * c->addr_unit;
2647+ end_addr = (phys_addr_t)r->ar.end * c->addr_unit;
2648+ while (addr < end_addr) {
2649+ struct folio *folio;
2650+ phys_addr_t folio_start, folio_end;
2651+ phys_addr_t overlap_start, overlap_end;
2652+ phys_addr_t counted;
2653+
2654+ folio = damon_get_folio(PHYS_PFN(addr));
2655+ if (!folio) {
2656+ addr = PAGE_ALIGN_DOWN(addr +
2657+ PAGE_SIZE);
2658+ if (!addr)
2659+ break;
2660+ continue;
2661+ }
2662+
2663+ /*
2664+ * Calculate exact overlap between the region
2665+ * [addr, end_addr) and the folio range.
2666+ * The folio may start before addr if addr is
2667+ * in the middle of a large folio.
2668+ */
2669+ folio_start = PFN_PHYS(folio_pfn(folio));
2670+ folio_end = folio_start + folio_size(folio);
2671+
2672+ overlap_start = max(addr, folio_start);
2673+ overlap_end = min(end_addr, folio_end);
2674+
2675+ if (overlap_end > overlap_start) {
2676+ counted = overlap_end - overlap_start;
2677+ total_eligible += counted;
2678+ if (folio_nid(folio) == nid)
2679+ node_eligible += counted;
2680+ }
2681+
2682+ /* Advance past the entire folio */
2683+ addr = folio_end;
2684+ folio_put(folio);
2685+ }
2686+ cond_resched();
2687+ }
2688+ }
2689+
2690+ *total = total_eligible;
2691+ return node_eligible;
2692+}
2693+
2694+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2695+ struct damos *s, int nid)
2696+{
2697+ phys_addr_t total_eligible = 0;
2698+ phys_addr_t node_eligible;
2699+
2700+ if (c->ops.id != DAMON_OPS_PADDR)
2701+ return 0;
2702+
2703+ if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
2704+ return 0;
2705+
2706+ node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
2707+
2708+ if (!(unsigned long)total_eligible)
2709+ return 0;
2710+
2711+ return mult_frac((unsigned long)node_eligible, 10000,
2712+ (unsigned long)total_eligible);
2713+}
2714+#else /* CONFIG_DAMON_PADDR */
2715+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2716+ struct damos *s, int nid)
2717+{
2718+ return 0;
2719+}
2720+#endif /* CONFIG_DAMON_PADDR */
2721+#else /* CONFIG_NUMA */
22912722 static __kernel_ulong_t damos_get_node_mem_bp(
22922723 struct damos_quota_goal *goal)
22932724 {
@@ -2299,7 +2730,13 @@ static unsigned long damos_get_node_memcg_used_bp( · 5 unmodified lines
22992730 {
23002731 return 0;
23012732 }
2302−#endif
2733+
2734+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
2735+ struct damos *s, int nid)
2736+{
2737+ return 0;
2738+}
2739+#endif /* CONFIG_NUMA */
23032740
23042741 /*
23052742 * Returns LRU-active or inactive memory to total LRU memory size ratio.
@@ -2319,7 +2756,8 @@ static unsigned int damos_get_in_active_mem_bp(bool active_ratio) · 13 unmodified lines
23192756 return mult_frac(inactive, 10000, total);
23202757 }
23212758
2322−static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal)
2759+static void damos_set_quota_goal_current_value(struct damon_ctx *c,
2760+ struct damos *s, struct damos_quota_goal *goal)
23232761 {
23242762 u64 now_psi_total;
23252763
@@ -2345,19 +2783,24 @@ static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal) · 19 unmodified lines
23452783 goal->current_value = damos_get_in_active_mem_bp(
23462784 goal->metric == DAMOS_QUOTA_ACTIVE_MEM_BP);
23472785 break;
2786+ case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
2787+ goal->current_value = damos_get_node_eligible_mem_bp(c, s,
2788+ goal->nid);
2789+ break;
23482790 default:
23492791 break;
23502792 }
23512793 }
23522794
23532795 /* Return the highest score since it makes schemes least aggressive */
2354−static unsigned long damos_quota_score(struct damos_quota *quota)
2796+static unsigned long damos_quota_score(struct damon_ctx *c, struct damos *s)
23552797 {
23562798 struct damos_quota_goal *goal;
2799+ struct damos_quota *quota = &s->quota;
23572800 unsigned long highest_score = 0;
23582801
23592802 damos_for_each_quota_goal(goal, quota) {
2360− damos_set_quota_goal_current_value(goal);
2803+ damos_set_quota_goal_current_value(c, s, goal);
23612804 highest_score = max(highest_score,
23622805 mult_frac(goal->current_value, 10000,
23632806 goal->target_value));
@@ -2366,17 +2809,20 @@ static unsigned long damos_quota_score(struct damos_quota *quota) · 2 unmodified lines
23662809 return highest_score;
23672810 }
23682811
2369−static void damos_goal_tune_esz_bp_consist(struct damos_quota *quota)
2812+static void damos_goal_tune_esz_bp_consist(struct damon_ctx *c, struct damos *s)
23702813 {
2371− unsigned long score = damos_quota_score(quota);
2814+ struct damos_quota *quota = &s->quota;
2815+ unsigned long score = damos_quota_score(c, s);
23722816
23732817 quota->esz_bp = damon_feed_loop_next_input(
23742818 max(quota->esz_bp, 10000UL), score);
23752819 }
23762820
2377−static void damos_goal_tune_esz_bp_temporal(struct damos_quota *quota)
2821+static void damos_goal_tune_esz_bp_temporal(struct damon_ctx *c,
2822+ struct damos *s)
23782823 {
2379− unsigned long score = damos_quota_score(quota);
2824+ struct damos_quota *quota = &s->quota;
2825+ unsigned long score = damos_quota_score(c, s);
23802826
23812827 if (score >= 10000)
23822828 quota->esz_bp = 0;
@@ -2389,9 +2835,9 @@ static void damos_goal_tune_esz_bp_temporal(struct damos_quota *quota) · 6 unmodified lines
23892835 /*
23902836 * Called only if quota->ms, or quota->sz are set, or quota->goals is not empty
23912837 */
2392−static void damos_set_effective_quota(struct damos_quota *quota,
2393− struct damon_ctx *ctx)
2838+static void damos_set_effective_quota(struct damon_ctx *ctx, struct damos *s)
23942839 {
2840+ struct damos_quota *quota = &s->quota;
23952841 unsigned long throughput;
23962842 unsigned long esz = ULONG_MAX;
23972843
@@ -2402,9 +2848,9 @@ static void damos_set_effective_quota(struct damos_quota *quota, · 4 unmodified lines
24022848
24032849 if (!list_empty(&quota->goals)) {
24042850 if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_CONSIST)
2405− damos_goal_tune_esz_bp_consist(quota);
2851+ damos_goal_tune_esz_bp_consist(ctx, s);
24062852 else if (quota->goal_tuner == DAMOS_QUOTA_GOAL_TUNER_TEMPORAL)
2407− damos_goal_tune_esz_bp_temporal(quota);
2853+ damos_goal_tune_esz_bp_temporal(ctx, s);
24082854 esz = quota->esz_bp / 10000;
24092855 }
24102856
@@ -2452,22 +2898,23 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) · 41 unmodified lines
24522898 /* First charge window */
24532899 if (!quota->total_charged_sz && !quota->charged_from) {
24542900 quota->charged_from = jiffies;
2455− damos_set_effective_quota(quota, c);
2901+ damos_set_effective_quota(c, s);
2902+ if (trace_damos_esz_enabled())
2903+ damos_trace_esz(c, s, quota);
24562904 }
24572905
24582906 /* New charge window starts */
24592907 if (!time_in_range_open(jiffies, quota->charged_from,
24602908 quota->charged_from +
24612909 msecs_to_jiffies(quota->reset_interval))) {
2462− if (damos_quota_is_set(quota) &&
2463− quota->charged_sz >= quota->esz)
2910+ if (damos_quota_is_full(quota, c->min_region_sz))
24642911 s->stat.qt_exceeds++;
24652912 quota->total_charged_sz += quota->charged_sz;
24662913 quota->charged_from = jiffies;
24672914 quota->charged_sz = 0;
24682915 if (trace_damos_esz_enabled())
24692916 cached_esz = quota->esz;
2470− damos_set_effective_quota(quota, c);
2917+ damos_set_effective_quota(c, s);
24712918 if (trace_damos_esz_enabled() && quota->esz != cached_esz)
24722919 damos_trace_esz(c, s, quota);
24732920 }
@@ -2521,9 +2968,9 @@ static void damos_trace_stat(struct damon_ctx *c, struct damos *s) · 47 unmodified lines
25212968 static void kdamond_apply_schemes(struct damon_ctx *c)
25222969 {
25232970 struct damon_target *t;
2524− struct damon_region *r;
25252971 struct damos *s;
25262972 bool has_schemes_to_apply = false;
2973+ unsigned long max_region_sz;
25272974
25282975 damon_for_each_scheme(s, c) {
25292976 if (time_before(c->passed_sample_intervals, s->next_apply_sis))
@@ -2540,13 +2987,12 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 10 unmodified lines
25402987 if (!has_schemes_to_apply)
25412988 return;
25422989
2990+ max_region_sz = damon_region_sz_limit(c);
25432991 mutex_lock(&c->walk_control_lock);
25442992 damon_for_each_target(t, c) {
25452993 if (c->ops.target_valid && c->ops.target_valid(t) == false)
25462994 continue;
2547−
2548− damon_for_each_region(r, t)
2549− damon_do_apply_schemes(c, t, r);
2995+ damos_apply_target(c, t, max_region_sz);
25502996 }
25512997
25522998 damon_for_each_scheme(s, c) {
@@ -2582,12 +3028,17 @@ static void damon_merge_two_regions(struct damon_target *t, · 29 unmodified lines
25823028 struct damon_region *l, struct damon_region *r)
25833029 {
25843030 unsigned long sz_l = damon_sz_region(l), sz_r = damon_sz_region(r);
3031+ int i;
25853032
25863033 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
25873034 (sz_l + sz_r);
25883035 l->nr_accesses_bp = l->nr_accesses * 10000;
25893036 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
25903037 l->ar.end = r->ar.end;
3038+ /* todo: do this for only installed probes */
3039+ for (i = 0; i < DAMON_MAX_PROBES; i++)
3040+ l->probe_hits[i] = (l->probe_hits[i] * sz_l + r->probe_hits[i]
3041+ * sz_r) / (sz_l + sz_r);
25913042 damon_verify_merge_two_regions(l, r);
25923043 damon_destroy_region(r, t);
25933044 }
@@ -2710,13 +3161,16 @@ static void damon_split_region_at(struct damon_target *t, · 116 unmodified lines
27103161 new->last_nr_accesses = r->last_nr_accesses;
27113162 new->nr_accesses_bp = r->nr_accesses_bp;
27123163 new->nr_accesses = r->nr_accesses;
3164+ /* todo: do this for only installed probes */
3165+ memcpy(new->probe_hits, r->probe_hits, sizeof(r->probe_hits));
27133166
27143167 damon_insert_region(new, r, damon_next_region(r), t);
27153168 }
27163169
27173170 /* Split every region in the given target into 'nr_subs' regions */
2718−static void damon_split_regions_of(struct damon_target *t, int nr_subs,
2719− unsigned long min_region_sz)
3171+static void damon_split_regions_of(struct damon_ctx *ctx,
3172+ struct damon_target *t, int nr_subs,
3173+ unsigned long min_region_sz)
27203174 {
27213175 struct damon_region *r, *next;
27223176 unsigned long sz_region, sz_sub = 0;
@@ -2731,7 +3185,7 @@ static void damon_split_regions_of(struct damon_target *t, int nr_subs, · 8 unmodified lines
27313185 * Randomly select size of left sub-region to be at
27323186 * least 10 percent and at most 90% of original region
27333187 */
2734− sz_sub = ALIGN_DOWN(damon_rand(1, 10) *
3188+ sz_sub = ALIGN_DOWN(damon_rand(ctx, 1, 10) *
27353189 sz_region / 10, min_region_sz);
27363190 /* Do not allow blank region */
27373191 if (sz_sub == 0 || sz_sub >= sz_region)
@@ -2772,7 +3226,8 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 34 unmodified lines
27723226 nr_subregions = 3;
27733227
27743228 damon_for_each_target(t, ctx)
2775− damon_split_regions_of(t, nr_subregions, ctx->min_region_sz);
3229+ damon_split_regions_of(ctx, t, nr_subregions,
3230+ ctx->min_region_sz);
27763231
27773232 last_nr_regions = nr_regions;
27783233 }
@@ -2857,6 +3312,37 @@ static void kdamond_usleep(unsigned long usecs) · 78 unmodified lines
28573312 usleep_range_idle(usecs, usecs + 1);
28583313 }
28593314
3315+#ifdef CONFIG_DAMON_DEBUG_SANITY
3316+static void damon_verify_ctx(struct damon_ctx *c)
3317+{
3318+ struct damon_target *t;
3319+ struct damon_region *r;
3320+
3321+ damon_for_each_target(t, c) {
3322+ struct damon_region *prev_r = NULL;
3323+ unsigned int nr_regions = 0;
3324+
3325+ damon_for_each_region(r, t) {
3326+ WARN_ONCE(r->ar.start >= r->ar.end,
3327+ "region start (%lu) >= end (%lu)\n",
3328+ r->ar.start, r->ar.end);
3329+ WARN_ONCE(prev_r && prev_r->ar.end > r->ar.start,
3330+ "region overlap (%lu > %lu)\n",
3331+ prev_r->ar.end, r->ar.start);
3332+ prev_r = r;
3333+ nr_regions++;
3334+ }
3335+ WARN_ONCE(damon_nr_regions(t) != nr_regions,
3336+ "nr_regions mismatch: %u != %u\n",
3337+ damon_nr_regions(t), nr_regions);
3338+ }
3339+}
3340+#else
3341+static void damon_verify_ctx(struct damon_ctx *c)
3342+{
3343+}
3344+#endif
3345+
28603346 /*
28613347 * kdamond_call() - handle damon_call_control objects.
28623348 * @ctx: The &struct damon_ctx of the kdamond.
@@ -2872,6 +3358,8 @@ static void kdamond_call(struct damon_ctx *ctx, bool cancel) · 9 unmodified lines
28723358 struct damon_call_control *control, *next;
28733359 LIST_HEAD(controls);
28743360
3361+ damon_verify_ctx(ctx);
3362+
28753363 mutex_lock(&ctx->call_controls_lock);
28763364 list_splice_tail_init(&ctx->call_controls, &controls);
28773365 mutex_unlock(&ctx->call_controls_lock);
@@ -2997,6 +3485,8 @@ static int kdamond_fn(void *data) · 119 unmodified lines
29973485
29983486 if (ctx->ops.check_accesses)
29993487 max_nr_accesses = ctx->ops.check_accesses(ctx);
3488+ if (ctx->ops.apply_probes)
3489+ ctx->ops.apply_probes(ctx);
30003490
30013491 if (time_after_eq(ctx->passed_sample_intervals,
30023492 next_aggregation_sis)) {
@@ -3014,6 +3504,14 @@ static int kdamond_fn(void *data) · 11 unmodified lines
30143504 kdamond_call(ctx, false);
30153505 if (ctx->maybe_corrupted)
30163506 break;
3507+ while (ctx->pause) {
3508+ damos_walk_cancel(ctx);
3509+ kdamond_usleep(ctx->attrs.sample_interval);
3510+ /* allow caller unset pause via damon_call() */
3511+ kdamond_call(ctx, false);
3512+ if (kdamond_need_stop(ctx) || ctx->maybe_corrupted)
3513+ goto done;
3514+ }
30173515 if (!list_empty(&ctx->schemes))
30183516 kdamond_apply_schemes(ctx);
30193517 else
@@ -3096,14 +3594,20 @@ done: · 76 unmodified lines
30963594 return 0;
30973595 }
30983596
3099−static int walk_system_ram(struct resource *res, void *arg)
3597+struct damon_system_ram_range_walk_arg {
3598+ bool walked;
3599+ struct resource res;
3600+};
3601+
3602+static int damon_system_ram_walk_fn(struct resource *res, void *arg)
31003603 {
3101− struct resource *a = arg;
3604+ struct damon_system_ram_range_walk_arg *a = arg;
31023605
3103− if (resource_size(a) < resource_size(res)) {
3104− a->start = res->start;
3105− a->end = res->end;
3606+ if (!a->walked) {
3607+ a->walked = true;
3608+ a->res.start = res->start;
31063609 }
3610+ a->res.end = res->end;
31073611 return 0;
31083612 }
31093613
@@ -3120,27 +3624,24 @@ static unsigned long damon_res_to_core_addr(resource_size_t ra, · 10 unmodified lines
31203624 return ra / addr_unit;
31213625 }
31223626
3123−/*
3124− * Find biggest 'System RAM' resource and store its start and end address in
3125− * @start and @end, respectively. If no System RAM is found, returns false.
3126− */
3127−static bool damon_find_biggest_system_ram(unsigned long *start,
3627+static bool damon_find_system_rams_range(unsigned long *start,
31283628 unsigned long *end, unsigned long addr_unit)
3129−
31303629 {
3131− struct resource res = {};
3630+ struct damon_system_ram_range_walk_arg arg = {};
31323631
3133− walk_system_ram_res(0, -1, &res, walk_system_ram);
3134− *start = damon_res_to_core_addr(res.start, addr_unit);
3135− *end = damon_res_to_core_addr(res.end + 1, addr_unit);
3632+ walk_system_ram_res(0, -1, &arg, damon_system_ram_walk_fn);
3633+ if (!arg.walked)
3634+ return false;
3635+ *start = damon_res_to_core_addr(arg.res.start, addr_unit);
3636+ *end = damon_res_to_core_addr(arg.res.end + 1, addr_unit);
31363637 if (*end <= *start)
31373638 return false;
31383639 return true;
31393640 }
31403641
31413642 /**
3142− * damon_set_region_biggest_system_ram_default() - Set the region of the given
3143− * monitoring target as requested, or biggest 'System RAM'.
3643+ * damon_set_region_system_rams_default() - Set the region of the given
3644+ * monitoring target as requested, or to cover all 'System RAM' resources.
31443645 * @t: The monitoring target to set the region.
31453646 * @start: The pointer to the start address of the region.
31463647 * @end: The pointer to the end address of the region.
@@ -3148,14 +3649,14 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 1 unmodified line
31483649 * @min_region_sz: Minimum region size.
31493650 *
31503651 * This function sets the region of @t as requested by @start and @end. If the
3151− * values of @start and @end are zero, however, this function finds the biggest
3152− * 'System RAM' resource and sets the region to cover the resource. In the
3153− * latter case, this function saves the start and end addresses of the resource
3154− * in @start and @end, respectively.
3652+ * values of @start and @end are zero, however, this function finds 'System
3653+ * RAM' resources and sets the region to cover all the resource. In the latter
3654+ * case, this function saves the start and the end addresseses of the first and
3655+ * the last resources in @start and @end, respectively.
31553656 *
31563657 * Return: 0 on success, negative error code otherwise.
31573658 */
3158−int damon_set_region_biggest_system_ram_default(struct damon_target *t,
3659+int damon_set_region_system_rams_default(struct damon_target *t,
31593660 unsigned long *start, unsigned long *end,
31603661 unsigned long addr_unit, unsigned long min_region_sz)
31613662 {
@@ -3165,7 +3666,7 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 3 unmodified lines
31653666 return -EINVAL;
31663667
31673668 if (!*start && !*end &&
3168− !damon_find_biggest_system_ram(start, end, addr_unit))
3669+ !damon_find_system_rams_range(start, end, addr_unit))
31693670 return -EINVAL;
31703671
31713672 addr_range.start = *start;
mm/damon/lru_sort.c+49−10
@@ -39,7 +39,6 @@ static bool enabled __read_mostly; · 38 unmodified lines
3939 * the re-reading, DAMON_LRU_SORT will be disabled.
4040 */
4141 static bool commit_inputs __read_mostly;
42−module_param(commit_inputs, bool, 0600);
4342
4443 /*
4544 * Desired active to [in]active memory ratio in bp (1/10,000).
@@ -140,7 +139,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_lru_sort_mon_attrs); · 94 unmodified lines
140139 * Start of the target memory region in physical address.
141140 *
142141 * The start physical address of memory region that DAMON_LRU_SORT will do work
143− * against. By default, biggest System RAM is used as the region.
142+ * against. By default, the system's entire physical memory is used as the
143+ * region.
144144 */
145145 static unsigned long monitor_region_start __read_mostly;
146146 module_param(monitor_region_start, ulong, 0600);
@@ -149,7 +149,8 @@ module_param(monitor_region_start, ulong, 0600); · 2 unmodified lines
149149 * End of the target memory region in physical address.
150150 *
151151 * The end physical address of memory region that DAMON_LRU_SORT will do work
152− * against. By default, biggest System RAM is used as the region.
152+ * against. By default, the system's entire physical memory is used as the
153+ * region.
153154 */
154155 static unsigned long monitor_region_end __read_mostly;
155156 module_param(monitor_region_end, ulong, 0600);
@@ -285,6 +286,11 @@ static int damon_lru_sort_apply_parameters(void) · 129 unmodified lines
285286 param_ctx->addr_unit = addr_unit;
286287 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
287288
289+ if (!is_power_of_2(param_ctx->min_region_sz)) {
290+ err = -EINVAL;
291+ goto out;
292+ }
293+
288294 if (!damon_lru_sort_mon_attrs.sample_interval) {
289295 err = -EINVAL;
290296 goto out;
@@ -327,7 +333,7 @@ static int damon_lru_sort_apply_parameters(void) · 36 unmodified lines
327333 if (err)
328334 goto out;
329335
330− err = damon_set_region_biggest_system_ram_default(param_target,
336+ err = damon_set_region_system_rams_default(param_target,
331337 &monitor_region_start,
332338 &monitor_region_end,
333339 param_ctx->addr_unit,
@@ -340,18 +346,51 @@ out: · 6 unmodified lines
340346 return err;
341347 }
342348
343−static int damon_lru_sort_handle_commit_inputs(void)
349+static int damon_lru_sort_commit_inputs_fn(void *arg)
350+{
351+ return damon_lru_sort_apply_parameters();
352+}
353+
354+static int damon_lru_sort_commit_inputs_store(const char *val,
355+ const struct kernel_param *kp)
344356 {
357+ bool commit_inputs_request;
345358 int err;
359+ struct damon_call_control control = {
360+ .fn = damon_lru_sort_commit_inputs_fn,
361+ };
362+
363+ if (!val) {
364+ commit_inputs_request = true;
365+ } else {
366+ err = kstrtobool(val, &commit_inputs_request);
367+ if (err)
368+ return err;
369+ }
346370
347− if (!commit_inputs)
371+ if (!commit_inputs_request)
348372 return 0;
349373
350− err = damon_lru_sort_apply_parameters();
351− commit_inputs = false;
352− return err;
374+ /*
375+ * Skip damon_call() if ctx is not initialized to avoid
376+ * NULL pointer dereference.
377+ */
378+ if (!ctx)
379+ return -EINVAL;
380+
381+ err = damon_call(ctx, &control);
382+
383+ return err ? err : control.return_code;
353384 }
354385
386+static const struct kernel_param_ops commit_inputs_param_ops = {
387+ .flags = KERNEL_PARAM_OPS_FL_NOARG,
388+ .set = damon_lru_sort_commit_inputs_store,
389+ .get = param_get_bool,
390+};
391+
392+module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
393+
355394 static int damon_lru_sort_damon_call_fn(void *arg)
356395 {
357396 struct damon_ctx *c = arg;
@@ -365,7 +404,7 @@ static int damon_lru_sort_damon_call_fn(void *arg) · 7 unmodified lines
365404 damon_lru_sort_cold_stat = s->stat;
366405 }
367406
368− return damon_lru_sort_handle_commit_inputs();
407+ return 0;
369408 }
370409
371410 static struct damon_call_control call_control = {
mm/damon/ops-common.c+6−3
@@ -117,9 +117,12 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, · 116 unmodified lines
117117 damon_max_nr_accesses(&c->attrs);
118118
119119 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
120− for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
121− age_in_log++, age_in_sec >>= 1)
122− ;
120+ if (age_in_sec)
121+ age_in_log = min_t(int, ilog2(age_in_sec) + 1,
122+ DAMON_MAX_AGE_IN_LOG);
123+ else
124+ age_in_log = 0;
125+
123126
124127 /* If frequency is 0, higher age means it's colder */
125128 if (freq_subscore == 0)
mm/damon/paddr.c+80−4
@@ -49,11 +49,11 @@ static void damon_pa_mkold(phys_addr_t paddr) · 48 unmodified lines
4949 }
5050
5151 static void __damon_pa_prepare_access_check(struct damon_region *r,
52− unsigned long addr_unit)
52+ struct damon_ctx *ctx)
5353 {
54− r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
54+ r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
5555
56− damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, addr_unit));
56+ damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit));
5757 }
5858
5959 static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
@@ -63,7 +63,7 @@ static void damon_pa_prepare_access_checks(struct damon_ctx *ctx) · 3 unmodified lines
6363
6464 damon_for_each_target(t, ctx) {
6565 damon_for_each_region(r, t)
66− __damon_pa_prepare_access_check(r, ctx->addr_unit);
66+ __damon_pa_prepare_access_check(r, ctx);
6767 }
6868 }
6969
@@ -120,6 +120,81 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx) · 50 unmodified lines
120120 return max_nr_accesses;
121121 }
122122
123+static bool damon_pa_filter_match(struct damon_filter *filter,
124+ struct folio *folio)
125+{
126+ bool matched = false;
127+ struct mem_cgroup *memcg;
128+
129+ switch (filter->type) {
130+ case DAMON_FILTER_TYPE_ANON:
131+ if (!folio) {
132+ matched = false;
133+ break;
134+ }
135+ matched = folio_test_anon(folio);
136+ break;
137+ case DAMON_FILTER_TYPE_MEMCG:
138+ if (!folio) {
139+ matched = false;
140+ break;
141+ }
142+ rcu_read_lock();
143+ memcg = folio_memcg_check(folio);
144+ if (!memcg)
145+ matched = false;
146+ else
147+ matched = filter->memcg_id == mem_cgroup_id(memcg);
148+ rcu_read_unlock();
149+ break;
150+ default:
151+ break;
152+ }
153+ return matched == filter->matching;
154+}
155+
156+static bool damon_pa_filter_pass(phys_addr_t pa, struct folio *folio,
157+ struct damon_probe *p)
158+{
159+ struct damon_filter *f;
160+ bool pass = true;
161+
162+ damon_for_each_filter(f, p) {
163+ if (damon_pa_filter_match(f, folio)) {
164+ pass = f->allow;
165+ break;
166+ }
167+ pass = !f->allow;
168+ }
169+ return pass;
170+}
171+
172+static void damon_pa_apply_probes(struct damon_ctx *ctx)
173+{
174+ struct damon_target *t;
175+ struct damon_region *r;
176+ struct damon_probe *p;
177+
178+ damon_for_each_target(t, ctx) {
179+ damon_for_each_region(r, t) {
180+ int i = 0;
181+ phys_addr_t pa;
182+ struct folio *folio;
183+
184+ pa = damon_pa_phys_addr(r->sampling_addr,
185+ ctx->addr_unit);
186+ folio = damon_get_folio(PHYS_PFN(pa));
187+ damon_for_each_probe(p, ctx) {
188+ if (damon_pa_filter_pass(pa, folio, p))
189+ r->probe_hits[i]++;
190+ i++;
191+ }
192+ if (folio)
193+ folio_put(folio);
194+ }
195+ }
196+}
197+
123198 /*
124199 * damos_pa_filter_out - Return true if the page should be filtered out.
125200 */
@@ -371,6 +446,7 @@ static int __init damon_pa_initcall(void) · 245 unmodified lines
371446 .update = NULL,
372447 .prepare_access_checks = damon_pa_prepare_access_checks,
373448 .check_accesses = damon_pa_check_accesses,
449+ .apply_probes = damon_pa_apply_probes,
374450 .target_valid = NULL,
375451 .apply_scheme = damon_pa_apply_scheme,
376452 .get_scheme_score = damon_pa_scheme_score,
mm/damon/reclaim.c+79−19
@@ -39,7 +39,6 @@ static bool enabled __read_mostly; · 38 unmodified lines
3939 * re-reading, DAMON_RECLAIM will be disabled.
4040 */
4141 static bool commit_inputs __read_mostly;
42−module_param(commit_inputs, bool, 0600);
4342
4443 /*
4544 * Time threshold for cold memory regions identification in microseconds.
@@ -92,6 +91,20 @@ module_param(quota_mem_pressure_us, ulong, 0600); · 46 unmodified lines
9291 static unsigned long quota_autotune_feedback __read_mostly;
9392 module_param(quota_autotune_feedback, ulong, 0600);
9493
94+/*
95+ * Auto-tune monitoring intervals.
96+ *
97+ * If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
98+ * sampling and aggregation intervals. The auto-tuning aims to capture
99+ * meaningful amount of access events in each DAMON-snapshot, while keeping the
100+ * sampling intervals 5 milliseconds in minimum, and 10 seconds in maximum.
101+ * Setting this as ``N`` disables the auto-tuning.
102+ *
103+ * Disabled by default.
104+ */
105+static bool autotune_monitoring_intervals __read_mostly;
106+module_param(autotune_monitoring_intervals, bool, 0600);
107+
95108 static struct damos_watermarks damon_reclaim_wmarks = {
96109 .metric = DAMOS_WMARK_FREE_MEM_RATE,
97110 .interval = 5000000, /* 5 seconds */
@@ -114,7 +127,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_reclaim_mon_attrs); · 16 unmodified lines
114127 * Start of the target memory region in physical address.
115128 *
116129 * The start physical address of memory region that DAMON_RECLAIM will do work
117− * against. By default, biggest System RAM is used as the region.
130+ * against. By default, the system's entire physical memory is used as the
131+ * region.
118132 */
119133 static unsigned long monitor_region_start __read_mostly;
120134 module_param(monitor_region_start, ulong, 0600);
@@ -123,7 +137,8 @@ module_param(monitor_region_start, ulong, 0600); · 2 unmodified lines
123137 * End of the target memory region in physical address.
124138 *
125139 * The end physical address of memory region that DAMON_RECLAIM will do work
126− * against. By default, biggest System RAM is used as the region.
140+ * against. By default, the system's entire physical memory is used as the
141+ * region.
127142 */
128143 static unsigned long monitor_region_end __read_mostly;
129144 module_param(monitor_region_end, ulong, 0600);
@@ -151,7 +166,7 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_reclaim_stat, · 21 unmodified lines
151166 static struct damon_ctx *ctx;
152167 static struct damon_target *target;
153168
154−static struct damos *damon_reclaim_new_scheme(void)
169+static struct damos *damon_reclaim_new_scheme(unsigned long aggr_interval)
155170 {
156171 struct damos_access_pattern pattern = {
157172 /* Find regions having PAGE_SIZE or larger size */
@@ -161,8 +176,7 @@ static struct damos *damon_reclaim_new_scheme(void) · 3 unmodified lines
161176 .min_nr_accesses = 0,
162177 .max_nr_accesses = 0,
163178 /* for min_age or more micro-seconds */
164− .min_age_region = min_age /
165− damon_reclaim_mon_attrs.aggr_interval,
179+ .min_age_region = min_age / aggr_interval,
166180 .max_age_region = UINT_MAX,
167181 };
168182
@@ -183,6 +197,7 @@ static int damon_reclaim_apply_parameters(void) · 14 unmodified lines
183197 {
184198 struct damon_ctx *param_ctx;
185199 struct damon_target *param_target;
200+ struct damon_attrs attrs;
186201 struct damos *scheme;
187202 struct damos_quota_goal *goal;
188203 struct damos_filter *filter;
@@ -195,17 +210,31 @@ static int damon_reclaim_apply_parameters(void) · 6 unmodified lines
195210 param_ctx->addr_unit = addr_unit;
196211 param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
197212
213+ if (!is_power_of_2(param_ctx->min_region_sz)) {
214+ err = -EINVAL;
215+ goto out;
216+ }
217+
198218 if (!damon_reclaim_mon_attrs.aggr_interval) {
199219 err = -EINVAL;
200220 goto out;
201221 }
202222
203− err = damon_set_attrs(param_ctx, &damon_reclaim_mon_attrs);
223+ attrs = damon_reclaim_mon_attrs;
224+ if (autotune_monitoring_intervals) {
225+ attrs.sample_interval = 5000;
226+ attrs.aggr_interval = 100000;
227+ attrs.intervals_goal.access_bp = 40;
228+ attrs.intervals_goal.aggrs = 3;
229+ attrs.intervals_goal.min_sample_us = 5000;
230+ attrs.intervals_goal.max_sample_us = 10 * 1000 * 1000;
231+ }
232+ err = damon_set_attrs(param_ctx, &attrs);
204233 if (err)
205234 goto out;
206235
207236 err = -ENOMEM;
208− scheme = damon_reclaim_new_scheme();
237+ scheme = damon_reclaim_new_scheme(attrs.aggr_interval);
209238 if (!scheme)
210239 goto out;
211240 damon_set_schemes(param_ctx, &scheme, 1);
@@ -233,11 +262,9 @@ static int damon_reclaim_apply_parameters(void) · 21 unmodified lines
233262 damos_add_filter(scheme, filter);
234263 }
235264
236− err = damon_set_region_biggest_system_ram_default(param_target,
237− &monitor_region_start,
238− &monitor_region_end,
239− param_ctx->addr_unit,
240− param_ctx->min_region_sz);
265+ err = damon_set_region_system_rams_default(param_target,
266+ &monitor_region_start, &monitor_region_end,
267+ param_ctx->addr_unit, param_ctx->min_region_sz);
241268 if (err)
242269 goto out;
243270 err = damon_commit_ctx(ctx, param_ctx);
@@ -246,18 +273,51 @@ out: · 2 unmodified lines
246273 return err;
247274 }
248275
249−static int damon_reclaim_handle_commit_inputs(void)
276+static int damon_reclaim_commit_inputs_fn(void *arg)
250277 {
278+ return damon_reclaim_apply_parameters();
279+}
280+
281+static int damon_reclaim_commit_inputs_store(const char *val,
282+ const struct kernel_param *kp)
283+{
284+ bool commit_inputs_request;
251285 int err;
286+ struct damon_call_control control = {
287+ .fn = damon_reclaim_commit_inputs_fn,
288+ };
252289
253− if (!commit_inputs)
290+ if (!val) {
291+ commit_inputs_request = true;
292+ } else {
293+ err = kstrtobool(val, &commit_inputs_request);
294+ if (err)
295+ return err;
296+ }
297+
298+ if (!commit_inputs_request)
254299 return 0;
255300
256− err = damon_reclaim_apply_parameters();
257− commit_inputs = false;
258− return err;
301+ /*
302+ * Skip damon_call() if ctx is not initialized to avoid
303+ * NULL pointer dereference.
304+ */
305+ if (!ctx)
306+ return -EINVAL;
307+
308+ err = damon_call(ctx, &control);
309+
310+ return err ? err : control.return_code;
259311 }
260312
313+static const struct kernel_param_ops commit_inputs_param_ops = {
314+ .flags = KERNEL_PARAM_OPS_FL_NOARG,
315+ .set = damon_reclaim_commit_inputs_store,
316+ .get = param_get_bool,
317+};
318+
319+module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
320+
261321 static int damon_reclaim_damon_call_fn(void *arg)
262322 {
263323 struct damon_ctx *c = arg;
@@ -267,7 +327,7 @@ static int damon_reclaim_damon_call_fn(void *arg) · 3 unmodified lines
267327 damon_for_each_scheme(s, c)
268328 damon_reclaim_stat = s->stat;
269329
270− return damon_reclaim_handle_commit_inputs();
330+ return 0;
271331 }
272332
273333 static struct damon_call_control call_control = {
mm/damon/stat.c+42−50
@@ -148,59 +148,12 @@ static int damon_stat_damon_call_fn(void *data) · 147 unmodified lines
148148 return 0;
149149 }
150150
151−struct damon_stat_system_ram_range_walk_arg {
152− bool walked;
153− struct resource res;
154−};
155−
156−static int damon_stat_system_ram_walk_fn(struct resource *res, void *arg)
157−{
158− struct damon_stat_system_ram_range_walk_arg *a = arg;
159−
160− if (!a->walked) {
161− a->walked = true;
162− a->res.start = res->start;
163− }
164− a->res.end = res->end;
165− return 0;
166−}
167−
168−static unsigned long damon_stat_res_to_core_addr(resource_size_t ra,
169− unsigned long addr_unit)
170−{
171− /*
172− * Use div_u64() for avoiding linking errors related with __udivdi3,
173− * __aeabi_uldivmod, or similar problems. This should also improve the
174− * performance optimization (read div_u64() comment for the detail).
175− */
176− if (sizeof(ra) == 8 && sizeof(addr_unit) == 4)
177− return div_u64(ra, addr_unit);
178− return ra / addr_unit;
179−}
180−
181−static int damon_stat_set_monitoring_region(struct damon_target *t,
182− unsigned long addr_unit, unsigned long min_region_sz)
183−{
184− struct damon_addr_range addr_range;
185− struct damon_stat_system_ram_range_walk_arg arg = {};
186−
187− walk_system_ram_res(0, -1, &arg, damon_stat_system_ram_walk_fn);
188− if (!arg.walked)
189− return -EINVAL;
190− addr_range.start = damon_stat_res_to_core_addr(
191− arg.res.start, addr_unit);
192− addr_range.end = damon_stat_res_to_core_addr(
193− arg.res.end + 1, addr_unit);
194− if (addr_range.end <= addr_range.start)
195− return -EINVAL;
196− return damon_set_regions(t, &addr_range, 1, min_region_sz);
197−}
198−
199151 static struct damon_ctx *damon_stat_build_ctx(void)
200152 {
201153 struct damon_ctx *ctx;
202154 struct damon_attrs attrs;
203155 struct damon_target *target;
156+ unsigned long start = 0, end = 0;
204157
205158 ctx = damon_new_ctx();
206159 if (!ctx)
@@ -230,8 +183,8 @@ static struct damon_ctx *damon_stat_build_ctx(void) · 23 unmodified lines
230183 if (!target)
231184 goto free_out;
232185 damon_add_target(ctx, target);
233− if (damon_stat_set_monitoring_region(target, ctx->addr_unit,
234− ctx->min_region_sz))
186+ if (damon_set_region_system_rams_default(target, &start, &end,
187+ ctx->addr_unit, ctx->min_region_sz))
235188 goto free_out;
236189 return ctx;
237190 free_out:
@@ -313,6 +266,45 @@ static int damon_stat_enabled_load(char *buffer, const struct kernel_param *kp) · 75 unmodified lines
313266 return sprintf(buffer, "%c\n", damon_stat_enabled() ? 'Y' : 'N');
314267 }
315268
269+static int damon_stat_kdamond_pid_store(
270+ const char *val, const struct kernel_param *kp)
271+{
272+ /*
273+ * kdamond_pid is read-only, but kernel command line could write it.
274+ * Do nothing here.
275+ */
276+ return 0;
277+}
278+
279+static int damon_stat_kdamond_pid_load(
280+ char *buffer, const struct kernel_param *kp)
281+{
282+ int pid;
283+
284+ if (!damon_stat_context) {
285+ pid = -1;
286+ } else {
287+ pid = damon_kdamond_pid(damon_stat_context);
288+ if (pid < 1)
289+ pid = -1;
290+ }
291+ return sprintf(buffer, "%d\n", pid);
292+}
293+
294+static const struct kernel_param_ops kdamond_pid_param_ops = {
295+ .set = damon_stat_kdamond_pid_store,
296+ .get = damon_stat_kdamond_pid_load,
297+};
298+
299+/*
300+ * PID of the DAMON thread
301+ *
302+ * If DAMON_STAT is enabled, this becomes the PID of the worker thread.
303+ * Else, -1.
304+ */
305+module_param_cb(kdamond_pid, &kdamond_pid_param_ops, NULL, 0400);
306+MODULE_PARM_DESC(kdamond_pid, "pid of the kdamond");
307+
316308 static int __init damon_stat_init(void)
317309 {
318310 int err = 0;
mm/damon/sysfs-common.c+41−0
@@ -104,3 +104,44 @@ const struct kobj_type damon_sysfs_ul_range_ktype = { · 103 unmodified lines
104104 .default_groups = damon_sysfs_ul_range_groups,
105105 };
106106
107+
108+static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
109+ char *memcg_path_buf, char *path)
110+{
111+#ifdef CONFIG_MEMCG
112+ cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
113+ if (sysfs_streq(memcg_path_buf, path))
114+ return true;
115+#endif /* CONFIG_MEMCG */
116+ return false;
117+}
118+
119+int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
120+{
121+ struct mem_cgroup *memcg;
122+ char *path;
123+ bool found = false;
124+
125+ if (!memcg_path)
126+ return -EINVAL;
127+
128+ path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
129+ if (!path)
130+ return -ENOMEM;
131+
132+ for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
133+ memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
134+ /* skip offlined memcg */
135+ if (!mem_cgroup_online(memcg))
136+ continue;
137+ if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
138+ *id = mem_cgroup_id(memcg);
139+ found = true;
140+ mem_cgroup_iter_break(NULL, memcg);
141+ break;
142+ }
143+ }
144+
145+ kfree(path);
146+ return found ? 0 : -EINVAL;
147+}
mm/damon/sysfs-common.h+2−0
@@ -59,3 +59,5 @@ int damos_sysfs_set_quota_scores(struct damon_sysfs_schemes *sysfs_schemes, · 58 unmodified lines
5959 void damos_sysfs_update_effective_quotas(
6060 struct damon_sysfs_schemes *sysfs_schemes,
6161 struct damon_ctx *ctx);
62+
63+int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id);
mm/damon/sysfs-schemes.c+244−45
@@ -10,6 +10,140 @@ · 9 unmodified lines
1010
1111 #include "sysfs-common.h"
1212
13+/*
14+ * probe directory
15+ */
16+
17+struct damos_sysfs_probe {
18+ struct kobject kobj;
19+ unsigned char hits;
20+};
21+
22+static struct damos_sysfs_probe *damos_sysfs_probe_alloc(unsigned char hits)
23+{
24+ struct damos_sysfs_probe *probe;
25+
26+ probe = kzalloc_obj(*probe);
27+ if (!probe)
28+ return NULL;
29+ probe->hits = hits;
30+ return probe;
31+}
32+
33+static ssize_t hits_show(struct kobject *kobj, struct kobj_attribute *attr,
34+ char *buf)
35+{
36+ struct damos_sysfs_probe *probe = container_of(kobj,
37+ struct damos_sysfs_probe, kobj);
38+
39+ return sysfs_emit(buf, "%hhu\n", probe->hits);
40+}
41+
42+static void damos_sysfs_probe_release(struct kobject *kobj)
43+{
44+ struct damos_sysfs_probe *probe = container_of(kobj,
45+ struct damos_sysfs_probe, kobj);
46+
47+ kfree(probe);
48+}
49+
50+static struct kobj_attribute damos_sysfs_probe_hits_attr =
51+ __ATTR_RO_MODE(hits, 0400);
52+
53+static struct attribute *damos_sysfs_probe_attrs[] = {
54+ &damos_sysfs_probe_hits_attr.attr,
55+ NULL,
56+};
57+ATTRIBUTE_GROUPS(damos_sysfs_probe);
58+
59+static const struct kobj_type damos_sysfs_probe_ktype = {
60+ .release = damos_sysfs_probe_release,
61+ .sysfs_ops = &kobj_sysfs_ops,
62+ .default_groups = damos_sysfs_probe_groups,
63+};
64+
65+/*
66+ * probes directory
67+ */
68+
69+struct damos_sysfs_probes {
70+ struct kobject kobj;
71+ struct damos_sysfs_probe **probes_arr;
72+ int nr;
73+};
74+
75+static struct damos_sysfs_probes *damos_sysfs_probes_alloc(void)
76+{
77+ return kzalloc_obj(struct damos_sysfs_probes);
78+}
79+
80+static void damos_sysfs_probes_rm_dirs(struct damos_sysfs_probes *probes)
81+{
82+ struct damos_sysfs_probe **probes_arr = probes->probes_arr;
83+ int i;
84+
85+ for (i = 0; i < probes->nr; i++)
86+ kobject_put(&probes_arr[i]->kobj);
87+ probes->nr = 0;
88+ kfree(probes_arr);
89+ probes->probes_arr = NULL;
90+}
91+
92+static int damos_sysfs_probes_add_dirs(struct damos_sysfs_probes *probes,
93+ struct damon_ctx *ctx, struct damon_region *region)
94+{
95+ struct damon_probe *probe;
96+ struct damos_sysfs_probe **probes_arr;
97+ int i = 0;
98+
99+ damon_for_each_probe(probe, ctx)
100+ i++;
101+
102+ if (!i)
103+ return 0;
104+
105+ probes_arr = kmalloc_objs(*probes_arr, i);
106+ if (!probes_arr)
107+ return -ENOMEM;
108+ probes->probes_arr = probes_arr;
109+
110+ i = 0;
111+ damon_for_each_probe(probe, ctx) {
112+ struct damos_sysfs_probe *sys_probe;
113+ int err;
114+
115+ sys_probe = damos_sysfs_probe_alloc(region->probe_hits[i]);
116+ if (!sys_probe) {
117+ damos_sysfs_probes_rm_dirs(probes);
118+ return -ENOMEM;
119+ }
120+ err = kobject_init_and_add(&sys_probe->kobj,
121+ &damos_sysfs_probe_ktype, &probes->kobj, "%d",
122+ i);
123+ if (err) {
124+ kobject_put(&sys_probe->kobj);
125+ damos_sysfs_probes_rm_dirs(probes);
126+ return err;
127+ }
128+ probes_arr[i++] = sys_probe;
129+ probes->nr++;
130+ }
131+ return 0;
132+}
133+
134+static void damos_sysfs_probes_release(struct kobject *kobj)
135+{
136+ struct damos_sysfs_probes *probes = container_of(kobj,
137+ struct damos_sysfs_probes, kobj);
138+
139+ kfree(probes);
140+}
141+
142+static const struct kobj_type damos_sysfs_probes_ktype = {
143+ .release = damos_sysfs_probes_release,
144+ .sysfs_ops = &kobj_sysfs_ops,
145+};
146+
13147 /*
14148 * scheme region directory
15149 */
@@ -20,6 +154,7 @@ struct damon_sysfs_scheme_region { · 4 unmodified lines
20154 unsigned int nr_accesses;
21155 unsigned int age;
22156 unsigned long sz_filter_passed;
157+ struct damos_sysfs_probes *probes;
23158 struct list_head list;
24159 };
25160
@@ -34,10 +169,44 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc( · 8 unmodified lines
34169 sysfs_region->ar = region->ar;
35170 sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;
36171 sysfs_region->age = region->age;
172+ sysfs_region->probes = NULL;
37173 INIT_LIST_HEAD(&sysfs_region->list);
38174 return sysfs_region;
39175 }
40176
177+static int damos_sysfs_region_add_dirs(
178+ struct damon_sysfs_scheme_region *region,
179+ struct damon_ctx *ctx,
180+ struct damon_region *dregion)
181+{
182+ struct damos_sysfs_probes *probes = damos_sysfs_probes_alloc();
183+ int err;
184+
185+ if (!probes)
186+ return -ENOMEM;
187+ err = kobject_init_and_add(&probes->kobj, &damos_sysfs_probes_ktype,
188+ &region->kobj, "probes");
189+ if (err)
190+ goto fail;
191+ err = damos_sysfs_probes_add_dirs(probes, ctx, dregion);
192+ if (err)
193+ goto fail;
194+
195+ region->probes = probes;
196+ return 0;
197+
198+fail:
199+ kobject_put(&probes->kobj);
200+ return err;
201+}
202+
203+static void damos_sysfs_region_rm_dirs(
204+ struct damon_sysfs_scheme_region *region)
205+{
206+ damos_sysfs_probes_rm_dirs(region->probes);
207+ kobject_put(&region->probes->kobj);
208+}
209+
41210 static ssize_t start_show(struct kobject *kobj, struct kobj_attribute *attr,
42211 char *buf)
43212 {
@@ -163,6 +332,7 @@ static void damon_sysfs_scheme_regions_rm_dirs( · 119 unmodified lines
163332 struct damon_sysfs_scheme_region *r, *next;
164333
165334 list_for_each_entry_safe(r, next, &regions->regions_list, list) {
335+ damos_sysfs_region_rm_dirs(r);
166336 list_del(&r->list);
167337 kobject_put(&r->kobj);
168338 regions->nr_regions--;
@@ -1093,6 +1263,10 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = { · 924 unmodified lines
10931263 .metric = DAMOS_QUOTA_INACTIVE_MEM_BP,
10941264 .name = "inactive_mem_bp",
10951265 },
1266+ {
1267+ .metric = DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
1268+ .name = "node_eligible_mem_bp",
1269+ },
10961270 };
10971271
10981272 static ssize_t target_metric_show(struct kobject *kobj,
@@ -1508,6 +1682,8 @@ struct damon_sysfs_quotas { · 409 unmodified lines
15081682 unsigned long reset_interval_ms;
15091683 unsigned long effective_sz; /* Effective size quota in bytes */
15101684 enum damos_quota_goal_tuner goal_tuner;
1685+ unsigned int fail_charge_num;
1686+ unsigned int fail_charge_denom;
15111687 };
15121688
15131689 static struct damon_sysfs_quotas *damon_sysfs_quotas_alloc(void)
@@ -1682,6 +1858,48 @@ static ssize_t goal_tuner_store(struct kobject *kobj, · 168 unmodified lines
16821858 return -EINVAL;
16831859 }
16841860
1861+static ssize_t fail_charge_num_show(struct kobject *kobj,
1862+ struct kobj_attribute *attr, char *buf)
1863+{
1864+ struct damon_sysfs_quotas *quotas = container_of(kobj,
1865+ struct damon_sysfs_quotas, kobj);
1866+
1867+ return sysfs_emit(buf, "%u\n", quotas->fail_charge_num);
1868+}
1869+
1870+static ssize_t fail_charge_num_store(struct kobject *kobj,
1871+ struct kobj_attribute *attr, const char *buf, size_t count)
1872+{
1873+ struct damon_sysfs_quotas *quotas = container_of(kobj,
1874+ struct damon_sysfs_quotas, kobj);
1875+ int err = kstrtouint(buf, 0, &quotas->fail_charge_num);
1876+
1877+ if (err)
1878+ return -EINVAL;
1879+ return count;
1880+}
1881+
1882+static ssize_t fail_charge_denom_show(struct kobject *kobj,
1883+ struct kobj_attribute *attr, char *buf)
1884+{
1885+ struct damon_sysfs_quotas *quotas = container_of(kobj,
1886+ struct damon_sysfs_quotas, kobj);
1887+
1888+ return sysfs_emit(buf, "%u\n", quotas->fail_charge_denom);
1889+}
1890+
1891+static ssize_t fail_charge_denom_store(struct kobject *kobj,
1892+ struct kobj_attribute *attr, const char *buf, size_t count)
1893+{
1894+ struct damon_sysfs_quotas *quotas = container_of(kobj,
1895+ struct damon_sysfs_quotas, kobj);
1896+ int err = kstrtouint(buf, 0, &quotas->fail_charge_denom);
1897+
1898+ if (err)
1899+ return -EINVAL;
1900+ return count;
1901+}
1902+
16851903 static void damon_sysfs_quotas_release(struct kobject *kobj)
16861904 {
16871905 kfree(container_of(kobj, struct damon_sysfs_quotas, kobj));
@@ -1702,12 +1920,20 @@ static struct kobj_attribute damon_sysfs_quotas_effective_bytes_attr = · 14 unmodified lines
17021920 static struct kobj_attribute damon_sysfs_quotas_goal_tuner_attr =
17031921 __ATTR_RW_MODE(goal_tuner, 0600);
17041922
1923+static struct kobj_attribute damon_sysfs_quotas_fail_charge_num_attr =
1924+ __ATTR_RW_MODE(fail_charge_num, 0600);
1925+
1926+static struct kobj_attribute damon_sysfs_quotas_fail_charge_denom_attr =
1927+ __ATTR_RW_MODE(fail_charge_denom, 0600);
1928+
17051929 static struct attribute *damon_sysfs_quotas_attrs[] = {
17061930 &damon_sysfs_quotas_ms_attr.attr,
17071931 &damon_sysfs_quotas_sz_attr.attr,
17081932 &damon_sysfs_quotas_reset_interval_ms_attr.attr,
17091933 &damon_sysfs_quotas_effective_bytes_attr.attr,
17101934 &damon_sysfs_quotas_goal_tuner_attr.attr,
1935+ &damon_sysfs_quotas_fail_charge_num_attr.attr,
1936+ &damon_sysfs_quotas_fail_charge_denom_attr.attr,
17111937 NULL,
17121938 };
17131939 ATTRIBUTE_GROUPS(damon_sysfs_quotas);
@@ -2060,6 +2286,10 @@ static struct damos_sysfs_action_name damos_sysfs_action_names[] = { · 346 unmodified lines
20602286 .action = DAMOS_NOHUGEPAGE,
20612287 .name = "nohugepage",
20622288 },
2289+ {
2290+ .action = DAMOS_COLLAPSE,
2291+ .name = "collapse",
2292+ },
20632293 {
20642294 .action = DAMOS_LRU_PRIO,
20652295 .name = "lru_prio",
@@ -2561,47 +2791,6 @@ const struct kobj_type damon_sysfs_schemes_ktype = { · 495 unmodified lines
25612791 .default_groups = damon_sysfs_schemes_groups,
25622792 };
25632793
2564−static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
2565− char *memcg_path_buf, char *path)
2566−{
2567−#ifdef CONFIG_MEMCG
2568− cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
2569− if (sysfs_streq(memcg_path_buf, path))
2570− return true;
2571−#endif /* CONFIG_MEMCG */
2572− return false;
2573−}
2574−
2575−static int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
2576−{
2577− struct mem_cgroup *memcg;
2578− char *path;
2579− bool found = false;
2580−
2581− if (!memcg_path)
2582− return -EINVAL;
2583−
2584− path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
2585− if (!path)
2586− return -ENOMEM;
2587−
2588− for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
2589− memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
2590− /* skip offlined memcg */
2591− if (!mem_cgroup_online(memcg))
2592− continue;
2593− if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
2594− *id = mem_cgroup_id(memcg);
2595− found = true;
2596− mem_cgroup_iter_break(NULL, memcg);
2597− break;
2598− }
2599− }
2600−
2601− kfree(path);
2602− return found ? 0 : -EINVAL;
2603−}
2604−
26052794 static int damon_sysfs_add_scheme_filters(struct damos *scheme,
26062795 struct damon_sysfs_scheme_filters *sysfs_filters)
26072796 {
@@ -2685,6 +2874,9 @@ static int damos_sysfs_add_quota_score( · 77 unmodified lines
26852874 }
26862875 goal->nid = sysfs_goal->nid;
26872876 break;
2877+ case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
2878+ goal->nid = sysfs_goal->nid;
2879+ break;
26882880 default:
26892881 break;
26902882 }
@@ -2796,6 +2988,8 @@ static struct damos *damon_sysfs_mk_scheme( · 105 unmodified lines
27962988 .weight_nr_accesses = sysfs_weights->nr_accesses,
27972989 .weight_age = sysfs_weights->age,
27982990 .goal_tuner = sysfs_quotas->goal_tuner,
2991+ .fail_charge_num = sysfs_quotas->fail_charge_num,
2992+ .fail_charge_denom = sysfs_quotas->fail_charge_denom,
27992993 };
28002994 struct damos_watermarks wmarks = {
28012995 .metric = sysfs_wmarks->metric,
@@ -2930,12 +3124,17 @@ void damos_sysfs_populate_region_dir(struct damon_sysfs_schemes *sysfs_schemes, · 128 unmodified lines
29303124 if (kobject_init_and_add(&region->kobj,
29313125 &damon_sysfs_scheme_region_ktype,
29323126 &sysfs_regions->kobj, "%d",
2933− sysfs_regions->nr_regions++)) {
2934− kobject_put(&region->kobj);
2935− return;
2936− }
3127+ sysfs_regions->nr_regions))
3128+ goto out;
3129+ if (damos_sysfs_region_add_dirs(region, ctx, r))
3130+ goto out;
3131+
29373132 list_add_tail(&region->list, &sysfs_regions->regions_list);
29383133 sysfs_regions->nr_regions++;
3134+ return;
3135+
3136+out:
3137+ kobject_put(&region->kobj);
29393138 }
29403139
29413140 int damon_sysfs_schemes_clear_regions(
mm/damon/sysfs.c+588−0
@@ -747,6 +747,497 @@ static const struct kobj_type damon_sysfs_intervals_ktype = { · 746 unmodified lines
747747 .default_groups = damon_sysfs_intervals_groups,
748748 };
749749
750+/*
751+ * filter directory
752+ */
753+
754+struct damon_sysfs_filter {
755+ struct kobject kobj;
756+ enum damon_filter_type type;
757+ bool matching;
758+ bool allow;
759+ char *path;
760+};
761+
762+static struct damon_sysfs_filter *damon_sysfs_filter_alloc(void)
763+{
764+ return kzalloc_obj(struct damon_sysfs_filter);
765+}
766+
767+struct damon_sysfs_filter_type_name {
768+ enum damon_filter_type type;
769+ char *name;
770+};
771+
772+static const struct damon_sysfs_filter_type_name
773+damon_sysfs_filter_type_names[] = {
774+ {
775+ .type = DAMON_FILTER_TYPE_ANON,
776+ .name = "anon",
777+ },
778+ {
779+ .type = DAMON_FILTER_TYPE_MEMCG,
780+ .name = "memcg",
781+ },
782+};
783+
784+static ssize_t type_show(struct kobject *kobj,
785+ struct kobj_attribute *attr, char *buf)
786+{
787+ struct damon_sysfs_filter *filter = container_of(kobj,
788+ struct damon_sysfs_filter, kobj);
789+ int i;
790+
791+ for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
792+ const struct damon_sysfs_filter_type_name *type_name;
793+
794+ type_name = &damon_sysfs_filter_type_names[i];
795+ if (type_name->type == filter->type)
796+ return sysfs_emit(buf, "%s\n", type_name->name);
797+ }
798+ return -EINVAL;
799+}
800+
801+static ssize_t type_store(struct kobject *kobj,
802+ struct kobj_attribute *attr, const char *buf, size_t count)
803+{
804+ struct damon_sysfs_filter *filter = container_of(kobj,
805+ struct damon_sysfs_filter, kobj);
806+ ssize_t ret = -EINVAL;
807+ int i;
808+
809+ for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
810+ const struct damon_sysfs_filter_type_name *type_name;
811+
812+ type_name = &damon_sysfs_filter_type_names[i];
813+ if (sysfs_streq(buf, type_name->name)) {
814+ filter->type = type_name->type;
815+ ret = count;
816+ break;
817+ }
818+ }
819+ return ret;
820+}
821+
822+static ssize_t matching_show(struct kobject *kobj,
823+ struct kobj_attribute *attr, char *buf)
824+{
825+ struct damon_sysfs_filter *filter = container_of(kobj,
826+ struct damon_sysfs_filter, kobj);
827+
828+ return sysfs_emit(buf, "%c\n", filter->matching ? 'Y' : 'N');
829+}
830+
831+static ssize_t matching_store(struct kobject *kobj,
832+ struct kobj_attribute *attr, const char *buf, size_t count)
833+{
834+ struct damon_sysfs_filter *filter = container_of(kobj,
835+ struct damon_sysfs_filter, kobj);
836+ bool matching;
837+ int err = kstrtobool(buf, &matching);
838+
839+ if (err)
840+ return err;
841+
842+ filter->matching = matching;
843+ return count;
844+}
845+
846+static ssize_t allow_show(struct kobject *kobj,
847+ struct kobj_attribute *attr, char *buf)
848+{
849+ struct damon_sysfs_filter *filter = container_of(kobj,
850+ struct damon_sysfs_filter, kobj);
851+
852+ return sysfs_emit(buf, "%c\n", filter->allow ? 'Y' : 'N');
853+}
854+
855+static ssize_t allow_store(struct kobject *kobj,
856+ struct kobj_attribute *attr, const char *buf, size_t count)
857+{
858+ struct damon_sysfs_filter *filter = container_of(kobj,
859+ struct damon_sysfs_filter, kobj);
860+ bool allow;
861+ int err = kstrtobool(buf, &allow);
862+
863+ if (err)
864+ return err;
865+
866+ filter->allow = allow;
867+ return count;
868+}
869+
870+static ssize_t path_show(struct kobject *kobj,
871+ struct kobj_attribute *attr, char *buf)
872+{
873+ struct damon_sysfs_filter *filter = container_of(kobj,
874+ struct damon_sysfs_filter, kobj);
875+ int len;
876+
877+ if (!mutex_trylock(&damon_sysfs_lock))
878+ return -EBUSY;
879+ len = sysfs_emit(buf, "%s\n", filter->path ? filter->path : "");
880+ mutex_unlock(&damon_sysfs_lock);
881+ return len;
882+}
883+
884+static ssize_t path_store(struct kobject *kobj,
885+ struct kobj_attribute *attr, const char *buf, size_t count)
886+{
887+ struct damon_sysfs_filter *filter = container_of(kobj,
888+ struct damon_sysfs_filter, kobj);
889+ char *path = kmalloc_objs(*path, size_add(count, 1));
890+
891+ if (!path)
892+ return -ENOMEM;
893+ strscpy(path, buf, size_add(count, 1));
894+ if (!mutex_trylock(&damon_sysfs_lock)) {
895+ kfree(path);
896+ return -EBUSY;
897+ }
898+ kfree(filter->path);
899+ filter->path = path;
900+ mutex_unlock(&damon_sysfs_lock);
901+ return count;
902+}
903+
904+static void damon_sysfs_filter_release(struct kobject *kobj)
905+{
906+ struct damon_sysfs_filter *filter = container_of(kobj,
907+ struct damon_sysfs_filter, kobj);
908+
909+ kfree(filter->path);
910+ kfree(filter);
911+}
912+
913+static struct kobj_attribute damon_sysfs_filter_type_attr =
914+ __ATTR_RW_MODE(type, 0600);
915+
916+static struct kobj_attribute damon_sysfs_filter_matching_attr =
917+ __ATTR_RW_MODE(matching, 0600);
918+
919+static struct kobj_attribute damon_sysfs_filter_allow_attr =
920+ __ATTR_RW_MODE(allow, 0600);
921+
922+static struct kobj_attribute damon_sysfs_filter_path_attr =
923+ __ATTR_RW_MODE(path, 0600);
924+
925+static struct attribute *damon_sysfs_filter_attrs[] = {
926+ &damon_sysfs_filter_type_attr.attr,
927+ &damon_sysfs_filter_matching_attr.attr,
928+ &damon_sysfs_filter_allow_attr.attr,
929+ &damon_sysfs_filter_path_attr.attr,
930+ NULL,
931+};
932+ATTRIBUTE_GROUPS(damon_sysfs_filter);
933+
934+static const struct kobj_type damon_sysfs_filter_ktype = {
935+ .release = damon_sysfs_filter_release,
936+ .sysfs_ops = &kobj_sysfs_ops,
937+ .default_groups = damon_sysfs_filter_groups,
938+};
939+
940+/*
941+ * filters directory
942+ */
943+
944+struct damon_sysfs_filters {
945+ struct kobject kobj;
946+ struct damon_sysfs_filter **filters_arr;
947+ int nr;
948+};
949+
950+static struct damon_sysfs_filters *damon_sysfs_filters_alloc(void)
951+{
952+ return kzalloc_obj(struct damon_sysfs_filters);
953+}
954+
955+static void damon_sysfs_filters_rm_dirs(struct damon_sysfs_filters *filters)
956+{
957+ struct damon_sysfs_filter **filters_arr = filters->filters_arr;
958+ int i;
959+
960+ for (i = 0; i < filters->nr; i++)
961+ kobject_put(&filters_arr[i]->kobj);
962+ filters->nr = 0;
963+ kfree(filters_arr);
964+ filters->filters_arr = NULL;
965+}
966+
967+static int damon_sysfs_filters_add_dirs(
968+ struct damon_sysfs_filters *filters, int nr_filters)
969+{
970+ struct damon_sysfs_filter **filters_arr, *filter;
971+ int err, i;
972+
973+ damon_sysfs_filters_rm_dirs(filters);
974+ if (!nr_filters)
975+ return 0;
976+
977+ filters_arr = kmalloc_objs(*filters_arr, nr_filters,
978+ GFP_KERNEL | __GFP_NOWARN);
979+ if (!filters_arr)
980+ return -ENOMEM;
981+ filters->filters_arr = filters_arr;
982+
983+ for (i = 0; i < nr_filters; i++) {
984+ filter = damon_sysfs_filter_alloc();
985+ if (!filter) {
986+ damon_sysfs_filters_rm_dirs(filters);
987+ return -ENOMEM;
988+ }
989+
990+ err = kobject_init_and_add(&filter->kobj,
991+ &damon_sysfs_filter_ktype, &filters->kobj,
992+ "%d", i);
993+ if (err) {
994+ kobject_put(&filter->kobj);
995+ damon_sysfs_filters_rm_dirs(filters);
996+ return err;
997+ }
998+
999+ filters_arr[i] = filter;
1000+ filters->nr++;
1001+ }
1002+ return 0;
1003+}
1004+
1005+static ssize_t nr_filters_show(struct kobject *kobj,
1006+ struct kobj_attribute *attr, char *buf)
1007+{
1008+ struct damon_sysfs_filters *filters = container_of(kobj,
1009+ struct damon_sysfs_filters, kobj);
1010+
1011+ return sysfs_emit(buf, "%d\n", filters->nr);
1012+}
1013+
1014+static ssize_t nr_filters_store(struct kobject *kobj,
1015+ struct kobj_attribute *attr, const char *buf, size_t count)
1016+{
1017+ struct damon_sysfs_filters *filters;
1018+ int nr, err = kstrtoint(buf, 0, &nr);
1019+
1020+ if (err)
1021+ return err;
1022+ if (nr < 0)
1023+ return -EINVAL;
1024+
1025+ filters = container_of(kobj, struct damon_sysfs_filters, kobj);
1026+
1027+ if (!mutex_trylock(&damon_sysfs_lock))
1028+ return -EBUSY;
1029+ err = damon_sysfs_filters_add_dirs(filters, nr);
1030+ mutex_unlock(&damon_sysfs_lock);
1031+ if (err)
1032+ return err;
1033+
1034+ return count;
1035+}
1036+
1037+static void damon_sysfs_filters_release(struct kobject *kobj)
1038+{
1039+ kfree(container_of(kobj, struct damon_sysfs_filters, kobj));
1040+}
1041+
1042+static struct kobj_attribute damon_sysfs_filters_nr_attr =
1043+ __ATTR_RW_MODE(nr_filters, 0600);
1044+
1045+static struct attribute *damon_sysfs_filters_attrs[] = {
1046+ &damon_sysfs_filters_nr_attr.attr,
1047+ NULL,
1048+};
1049+ATTRIBUTE_GROUPS(damon_sysfs_filters);
1050+
1051+static const struct kobj_type damon_sysfs_filters_ktype = {
1052+ .release = damon_sysfs_filters_release,
1053+ .sysfs_ops = &kobj_sysfs_ops,
1054+ .default_groups = damon_sysfs_filters_groups,
1055+};
1056+
1057+/*
1058+ * probe directory
1059+ */
1060+
1061+struct damon_sysfs_probe {
1062+ struct kobject kobj;
1063+ struct damon_sysfs_filters *filters;
1064+};
1065+
1066+static struct damon_sysfs_probe *damon_sysfs_probe_alloc(void)
1067+{
1068+ return kzalloc_obj(struct damon_sysfs_probe);
1069+}
1070+
1071+static int damon_sysfs_probe_add_dirs(struct damon_sysfs_probe *attr)
1072+{
1073+ struct damon_sysfs_filters *filters;
1074+ int err;
1075+
1076+ filters = damon_sysfs_filters_alloc();
1077+ if (!filters)
1078+ return -ENOMEM;
1079+ attr->filters = filters;
1080+
1081+ err = kobject_init_and_add(&filters->kobj, &damon_sysfs_filters_ktype,
1082+ &attr->kobj, "filters");
1083+ if (err) {
1084+ kobject_put(&filters->kobj);
1085+ attr->filters = NULL;
1086+ }
1087+ return err;
1088+}
1089+
1090+static void damon_sysfs_probe_rm_dirs(struct damon_sysfs_probe *attr)
1091+{
1092+ if (attr->filters) {
1093+ damon_sysfs_filters_rm_dirs(attr->filters);
1094+ kobject_put(&attr->filters->kobj);
1095+ }
1096+}
1097+
1098+static void damon_sysfs_probe_release(struct kobject *kobj)
1099+{
1100+ kfree(container_of(kobj, struct damon_sysfs_probe, kobj));
1101+}
1102+
1103+static struct attribute *damon_sysfs_probe_attrs[] = {
1104+ NULL,
1105+};
1106+ATTRIBUTE_GROUPS(damon_sysfs_probe);
1107+
1108+static const struct kobj_type damon_sysfs_probe_ktype = {
1109+ .release = damon_sysfs_probe_release,
1110+ .sysfs_ops = &kobj_sysfs_ops,
1111+ .default_groups = damon_sysfs_probe_groups,
1112+};
1113+
1114+/*
1115+ * probes directory
1116+ */
1117+
1118+struct damon_sysfs_probes {
1119+ struct kobject kobj;
1120+ struct damon_sysfs_probe **probes_arr;
1121+ int nr;
1122+};
1123+
1124+static struct damon_sysfs_probes *damon_sysfs_probes_alloc(void)
1125+{
1126+ return kzalloc_obj(struct damon_sysfs_probes);
1127+}
1128+
1129+static void damon_sysfs_probes_rm_dirs(
1130+ struct damon_sysfs_probes *probes)
1131+{
1132+ struct damon_sysfs_probe **probes_arr = probes->probes_arr;
1133+ int i;
1134+
1135+ for (i = 0; i < probes->nr; i++) {
1136+ damon_sysfs_probe_rm_dirs(probes_arr[i]);
1137+ kobject_put(&probes_arr[i]->kobj);
1138+ }
1139+ probes->nr = 0;
1140+ kfree(probes_arr);
1141+ probes->probes_arr = NULL;
1142+}
1143+
1144+static int damon_sysfs_probes_add_dirs(
1145+ struct damon_sysfs_probes *probes, int nr_probes)
1146+{
1147+ struct damon_sysfs_probe **probes_arr, *probe;
1148+ int err, i;
1149+
1150+ damon_sysfs_probes_rm_dirs(probes);
1151+ if (!nr_probes)
1152+ return 0;
1153+
1154+ probes_arr = kmalloc_objs(*probes_arr, nr_probes,
1155+ GFP_KERNEL | __GFP_NOWARN);
1156+ if (!probes_arr)
1157+ return -ENOMEM;
1158+ probes->probes_arr = probes_arr;
1159+
1160+ for (i = 0; i < nr_probes; i++) {
1161+ probe = damon_sysfs_probe_alloc();
1162+ if (!probe) {
1163+ damon_sysfs_probes_rm_dirs(probes);
1164+ return -ENOMEM;
1165+ }
1166+
1167+ err = kobject_init_and_add(&probe->kobj,
1168+ &damon_sysfs_probe_ktype, &probes->kobj,
1169+ "%d", i);
1170+ if (err) {
1171+ kobject_put(&probe->kobj);
1172+ damon_sysfs_probes_rm_dirs(probes);
1173+ return err;
1174+ }
1175+
1176+ err = damon_sysfs_probe_add_dirs(probe);
1177+ if (err) {
1178+ kobject_put(&probe->kobj);
1179+ damon_sysfs_probes_rm_dirs(probes);
1180+ return err;
1181+ }
1182+
1183+ probes_arr[i] = probe;
1184+ probes->nr++;
1185+ }
1186+ return 0;
1187+}
1188+
1189+static ssize_t nr_probes_show(struct kobject *kobj,
1190+ struct kobj_attribute *attr, char *buf)
1191+{
1192+ struct damon_sysfs_probes *probes = container_of(kobj,
1193+ struct damon_sysfs_probes, kobj);
1194+
1195+ return sysfs_emit(buf, "%d\n", probes->nr);
1196+}
1197+
1198+static ssize_t nr_probes_store(struct kobject *kobj,
1199+ struct kobj_attribute *attr, const char *buf, size_t count)
1200+{
1201+ struct damon_sysfs_probes *probes;
1202+ int nr, err = kstrtoint(buf, 0, &nr);
1203+
1204+ if (err)
1205+ return err;
1206+ if (nr < 0 || nr > DAMON_MAX_PROBES)
1207+ return -EINVAL;
1208+
1209+ probes = container_of(kobj, struct damon_sysfs_probes, kobj);
1210+
1211+ if (!mutex_trylock(&damon_sysfs_lock))
1212+ return -EBUSY;
1213+ err = damon_sysfs_probes_add_dirs(probes, nr);
1214+ mutex_unlock(&damon_sysfs_lock);
1215+ if (err)
1216+ return err;
1217+
1218+ return count;
1219+}
1220+
1221+static void damon_sysfs_probes_release(struct kobject *kobj)
1222+{
1223+ kfree(container_of(kobj, struct damon_sysfs_probes, kobj));
1224+}
1225+
1226+static struct kobj_attribute damon_sysfs_probes_nr_probes =
1227+ __ATTR_RW_MODE(nr_probes, 0600);
1228+
1229+static struct attribute *damon_sysfs_probes_attrs[] = {
1230+ &damon_sysfs_probes_nr_probes.attr,
1231+ NULL,
1232+};
1233+ATTRIBUTE_GROUPS(damon_sysfs_probes);
1234+
1235+static const struct kobj_type damon_sysfs_probes_ktype = {
1236+ .release = damon_sysfs_probes_release,
1237+ .sysfs_ops = &kobj_sysfs_ops,
1238+ .default_groups = damon_sysfs_probes_groups,
1239+};
1240+
7501241 /*
7511242 * monitoring_attrs directory
7521243 */
@@ -755,6 +1246,7 @@ struct damon_sysfs_attrs { · 2 unmodified lines
7551246 struct kobject kobj;
7561247 struct damon_sysfs_intervals *intervals;
7571248 struct damon_sysfs_ul_range *nr_regions_range;
1249+ struct damon_sysfs_probes *probes;
7581250 };
7591251
7601252 static struct damon_sysfs_attrs *damon_sysfs_attrs_alloc(void)
@@ -771,6 +1263,7 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs) · 10 unmodified lines
7711263 {
7721264 struct damon_sysfs_intervals *intervals;
7731265 struct damon_sysfs_ul_range *nr_regions_range;
1266+ struct damon_sysfs_probes *probes;
7741267 int err;
7751268
7761269 intervals = damon_sysfs_intervals_alloc(5000, 100000, 60000000);
@@ -799,8 +1292,22 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs) · 22 unmodified lines
7991292 if (err)
8001293 goto put_nr_regions_intervals_out;
8011294 attrs->nr_regions_range = nr_regions_range;
1295+
1296+ probes = damon_sysfs_probes_alloc();
1297+ if (!probes) {
1298+ err = -ENOMEM;
1299+ goto put_nr_regions_intervals_out;
1300+ }
1301+ err = kobject_init_and_add(&probes->kobj,
1302+ &damon_sysfs_probes_ktype, &attrs->kobj, "probes");
1303+ if (err)
1304+ goto put_probes_out;
1305+ attrs->probes = probes;
8021306 return 0;
8031307
1308+put_probes_out:
1309+ kobject_put(&probes->kobj);
1310+ attrs->probes = NULL;
8041311 put_nr_regions_intervals_out:
8051312 kobject_put(&nr_regions_range->kobj);
8061313 attrs->nr_regions_range = NULL;
@@ -817,6 +1324,8 @@ static void damon_sysfs_attrs_rm_dirs(struct damon_sysfs_attrs *attrs) · 10 unmodified lines
8171324 kobject_put(&attrs->nr_regions_range->kobj);
8181325 damon_sysfs_intervals_rm_dirs(attrs->intervals);
8191326 kobject_put(&attrs->intervals->kobj);
1327+ damon_sysfs_probes_rm_dirs(attrs->probes);
1328+ kobject_put(&attrs->probes->kobj);
8201329 }
8211330
8221331 static void damon_sysfs_attrs_release(struct kobject *kobj)
@@ -866,6 +1375,7 @@ struct damon_sysfs_context { · 43 unmodified lines
8661375 struct damon_sysfs_attrs *attrs;
8671376 struct damon_sysfs_targets *targets;
8681377 struct damon_sysfs_schemes *schemes;
1378+ bool pause;
8691379 };
8701380
8711381 static struct damon_sysfs_context *damon_sysfs_context_alloc(
@@ -878,6 +1388,7 @@ static struct damon_sysfs_context *damon_sysfs_context_alloc( · 6 unmodified lines
8781388 context->kobj = (struct kobject){};
8791389 context->ops_id = ops_id;
8801390 context->addr_unit = 1;
1391+ context->pause = false;
8811392 return context;
8821393 }
8831394
@@ -1053,6 +1564,30 @@ static ssize_t addr_unit_store(struct kobject *kobj, · 169 unmodified lines
10531564 return count;
10541565 }
10551566
1567+static ssize_t pause_show(struct kobject *kobj, struct kobj_attribute *attr,
1568+ char *buf)
1569+{
1570+ struct damon_sysfs_context *context = container_of(kobj,
1571+ struct damon_sysfs_context, kobj);
1572+
1573+ return sysfs_emit(buf, "%c\n", context->pause ? 'Y' : 'N');
1574+}
1575+
1576+static ssize_t pause_store(struct kobject *kobj, struct kobj_attribute *attr,
1577+ const char *buf, size_t count)
1578+{
1579+ struct damon_sysfs_context *context = container_of(kobj,
1580+ struct damon_sysfs_context, kobj);
1581+ bool pause;
1582+ int err = kstrtobool(buf, &pause);
1583+
1584+ if (err)
1585+ return err;
1586+ context->pause = pause;
1587+ return count;
1588+}
1589+
1590+
10561591 static void damon_sysfs_context_release(struct kobject *kobj)
10571592 {
10581593 kfree(container_of(kobj, struct damon_sysfs_context, kobj));
@@ -1067,10 +1602,14 @@ static struct kobj_attribute damon_sysfs_context_operations_attr = · 8 unmodified lines
10671602 static struct kobj_attribute damon_sysfs_context_addr_unit_attr =
10681603 __ATTR_RW_MODE(addr_unit, 0600);
10691604
1605+static struct kobj_attribute damon_sysfs_context_pause_attr =
1606+ __ATTR_RW_MODE(pause, 0600);
1607+
10701608 static struct attribute *damon_sysfs_context_attrs[] = {
10711609 &damon_sysfs_context_avail_operations_attr.attr,
10721610 &damon_sysfs_context_operations_attr.attr,
10731611 &damon_sysfs_context_addr_unit_attr.attr,
1612+ &damon_sysfs_context_pause_attr.attr,
10741613 NULL,
10751614 };
10761615 ATTRIBUTE_GROUPS(damon_sysfs_context);
@@ -1360,6 +1899,51 @@ static int damon_sysfs_set_attrs(struct damon_ctx *ctx, · 283 unmodified lines
13601899 return damon_set_attrs(ctx, &attrs);
13611900 }
13621901
1902+static int damon_sysfs_set_probes(struct damon_ctx *ctx,
1903+ struct damon_sysfs_probes *sys_probes)
1904+{
1905+ int i;
1906+
1907+ for (i = 0; i < sys_probes->nr; i++) {
1908+ struct damon_sysfs_filters *sys_filters =
1909+ sys_probes->probes_arr[i]->filters;
1910+ struct damon_probe *c;
1911+ int j;
1912+
1913+ if (!sys_filters)
1914+ continue;
1915+ c = damon_new_probe();
1916+ if (!c)
1917+ return -ENOMEM;
1918+ damon_add_probe(ctx, c);
1919+
1920+ for (j = 0; j < sys_filters->nr; j++) {
1921+ struct damon_sysfs_filter *sys_filter =
1922+ sys_filters->filters_arr[j];
1923+ struct damon_filter *filter;
1924+
1925+ filter = damon_new_filter(sys_filter->type,
1926+ sys_filter->matching,
1927+ sys_filter->allow);
1928+ if (!filter)
1929+ return -ENOMEM;
1930+ if (filter->type == DAMON_FILTER_TYPE_MEMCG) {
1931+ int err;
1932+
1933+ err = damon_sysfs_memcg_path_to_id(
1934+ sys_filter->path,
1935+ &filter->memcg_id);
1936+ if (err) {
1937+ damon_destroy_filter(filter);
1938+ return err;
1939+ }
1940+ }
1941+ damon_add_filter(c, filter);
1942+ }
1943+ }
1944+ return 0;
1945+}
1946+
13631947 static int damon_sysfs_set_regions(struct damon_target *t,
13641948 struct damon_sysfs_regions *sysfs_regions,
13651949 unsigned long min_region_sz)
@@ -1470,7 +2054,11 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 104 unmodified lines
14702054 if (sys_ctx->ops_id == DAMON_OPS_PADDR)
14712055 ctx->min_region_sz = max(
14722056 DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1);
2057+ ctx->pause = sys_ctx->pause;
14732058 err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
2059+ if (err)
2060+ return err;
2061+ err = damon_sysfs_set_probes(ctx, sys_ctx->attrs->probes);
14742062 if (err)
14752063 return err;
14762064 err = damon_sysfs_add_targets(ctx, sys_ctx->targets);
mm/damon/tests/core-kunit.h+152−28
@@ -273,54 +273,70 @@ static void damon_test_merge_regions_of(struct kunit *test) · 272 unmodified lines
273273
274274 static void damon_test_split_regions_of(struct kunit *test)
275275 {
276+ struct damon_ctx *c;
276277 struct damon_target *t;
277278 struct damon_region *r;
278279 unsigned long sa[] = {0, 300, 500};
279280 unsigned long ea[] = {220, 400, 700};
280281 int i;
281282
283+ c = damon_new_ctx();
284+ if (!c)
285+ kunit_skip(test, "ctx alloc fail");
286+
282287 t = damon_new_target();
283− if (!t)
288+ if (!t) {
289+ damon_destroy_ctx(c);
284290 kunit_skip(test, "target alloc fail");
291+ }
285292 r = damon_new_region(0, 22);
286293 if (!r) {
287294 damon_free_target(t);
295+ damon_destroy_ctx(c);
288296 kunit_skip(test, "region alloc fail");
289297 }
290298 damon_add_region(r, t);
291− damon_split_regions_of(t, 2, 1);
299+ damon_split_regions_of(c, t, 2, 1);
292300 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 2u);
293301 damon_free_target(t);
294302
295303 t = damon_new_target();
296− if (!t)
304+ if (!t) {
305+ damon_destroy_ctx(c);
297306 kunit_skip(test, "second target alloc fail");
307+ }
298308 r = damon_new_region(0, 220);
299309 if (!r) {
300310 damon_free_target(t);
311+ damon_destroy_ctx(c);
301312 kunit_skip(test, "second region alloc fail");
302313 }
303314 damon_add_region(r, t);
304− damon_split_regions_of(t, 4, 1);
315+ damon_split_regions_of(c, t, 4, 1);
305316 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u);
306317 damon_free_target(t);
307318
308319 t = damon_new_target();
309− if (!t)
320+ if (!t) {
321+ damon_destroy_ctx(c);
310322 kunit_skip(test, "third target alloc fail");
323+ }
311324 for (i = 0; i < ARRAY_SIZE(sa); i++) {
312325 r = damon_new_region(sa[i], ea[i]);
313326 if (!r) {
314327 damon_free_target(t);
328+ damon_destroy_ctx(c);
315329 kunit_skip(test, "region alloc fail");
316330 }
317331 damon_add_region(r, t);
318332 }
319− damon_split_regions_of(t, 4, 5);
333+ damon_split_regions_of(c, t, 4, 5);
320334 KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u);
321335 damon_for_each_region(r, t)
322336 KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul);
323337 damon_free_target(t);
338+
339+ damon_destroy_ctx(c);
324340 }
325341
326342 static void damon_test_ops_registration(struct kunit *test)
@@ -374,41 +390,139 @@ static void damon_test_ops_registration(struct kunit *test) · 47 unmodified lines
374390 }
375391 }
376392
377−static void damon_test_set_regions(struct kunit *test)
393+static void damon_test_set_regions_for(struct kunit *test,
394+ struct damon_addr_range *old_ranges, int sz_old_ranges,
395+ struct damon_addr_range *new_ranges, int sz_new_ranges,
396+ unsigned long min_region_sz,
397+ struct damon_addr_range *expect_ranges, int sz_expect_ranges)
378398 {
379− struct damon_target *t = damon_new_target();
380− struct damon_region *r1, *r2;
381− struct damon_addr_range range = {.start = 8, .end = 28};
382− unsigned long expects[] = {8, 16, 16, 24, 24, 28};
383− int expect_idx = 0;
399+ struct damon_target *t;
384400 struct damon_region *r;
401+ int i;
385402
403+ t = damon_new_target();
386404 if (!t)
387405 kunit_skip(test, "target alloc fail");
388− r1 = damon_new_region(4, 16);
389− if (!r1) {
390− damon_free_target(t);
391− kunit_skip(test, "region alloc fail");
392− }
393− r2 = damon_new_region(24, 32);
394− if (!r2) {
395− damon_free_target(t);
396− damon_free_region(r1);
397− kunit_skip(test, "second region alloc fail");
406+ for (i = 0; i < sz_old_ranges; i++) {
407+ r = damon_new_region(old_ranges[i].start, old_ranges[i].end);
408+ if (!r) {
409+ damon_destroy_target(t, NULL);
410+ kunit_skip(test, "%d-th r alloc fail\n", i);
411+ }
412+ damon_add_region(r, t);
398413 }
399414
400− damon_add_region(r1, t);
401− damon_add_region(r2, t);
402− damon_set_regions(t, &range, 1, 1);
415+ damon_set_regions(t, new_ranges, sz_new_ranges, min_region_sz);
403416
404− KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 3);
417+ KUNIT_EXPECT_EQ(test, damon_nr_regions(t), sz_expect_ranges);
418+ if (damon_nr_regions(t) != sz_expect_ranges) {
419+ damon_destroy_target(t, NULL);
420+ return;
421+ }
422+ i = 0;
405423 damon_for_each_region(r, t) {
406− KUNIT_EXPECT_EQ(test, r->ar.start, expects[expect_idx++]);
407− KUNIT_EXPECT_EQ(test, r->ar.end, expects[expect_idx++]);
424+ KUNIT_EXPECT_EQ(test, r->ar.start, expect_ranges[i].start);
425+ KUNIT_EXPECT_EQ(test, r->ar.end, expect_ranges[i++].end);
408426 }
427+
409428 damon_destroy_target(t, NULL);
410429 }
411430
431+static void damon_test_set_regions(struct kunit *test)
432+{
433+ /* Initial build up on empty target. */
434+ damon_test_set_regions_for(test,
435+ (struct damon_addr_range[]){}, 0,
436+ (struct damon_addr_range[]){
437+ {.start = 5, .end = 15},
438+ {.start = 15, .end = 25},
439+ }, 2,
440+ 1,
441+ (struct damon_addr_range[]){
442+ {.start = 5, .end = 15},
443+ {.start = 15, .end = 25},
444+ }, 2);
445+ /* Un-intersecting regions should be removed. */
446+ damon_test_set_regions_for(test,
447+ (struct damon_addr_range[]){
448+ {.start = 4, .end = 16},
449+ {.start = 24, .end = 32},
450+ }, 2,
451+ (struct damon_addr_range[]){
452+ {.start = 18, .end = 23},
453+ }, 1,
454+ 1,
455+ (struct damon_addr_range[]){
456+ {.start = 18, .end = 23},
457+ }, 1);
458+ /*
459+ * Holes should be filled up with new regions.
460+ *
461+ * old: [4, 16) [24, 32)
462+ * new: [8, 28)
463+ * expect: [8, 16)[16,24),[24, 28)
464+ */
465+ damon_test_set_regions_for(test,
466+ (struct damon_addr_range[]){
467+ {.start = 4, .end = 16},
468+ {.start = 24, .end = 32},
469+ }, 2,
470+ (struct damon_addr_range[]){
471+ {.start = 8, .end = 28},
472+ }, 1,
473+ 1,
474+ (struct damon_addr_range[]){
475+ {.start = 8, .end = 16},
476+ {.start = 16, .end = 24},
477+ {.start = 24, .end = 28},
478+ }, 3);
479+ /*
480+ * New regions should be able to be appended.
481+ *
482+ * old: [0, 4)[4, 17)
483+ * new: [0, 15) [25, 40)
484+ * expect: [0, 4)[4, 15) [25, 40)
485+ */
486+ damon_test_set_regions_for(test,
487+ (struct damon_addr_range[]){
488+ {.start = 0, .end = 4},
489+ {.start = 4, .end = 17},
490+ }, 2,
491+ (struct damon_addr_range[]){
492+ {.start = 0, .end = 15},
493+ {.start = 25, .end = 40},
494+ }, 2,
495+ 1,
496+ (struct damon_addr_range[]){
497+ {.start = 0, .end = 4},
498+ {.start = 4, .end = 15},
499+ {.start = 25, .end = 40},
500+ }, 3);
501+ /*
502+ * New regions should be able to be inserted.
503+ *
504+ * old: [0, 4) [42, 52)
505+ * new: [0, 15) [25, 40) [44, 50)
506+ * expect: [0, 15) [25, 40) [44, 50)
507+ */
508+ damon_test_set_regions_for(test,
509+ (struct damon_addr_range[]){
510+ {.start = 0, .end = 4},
511+ {.start = 42, .end = 52},
512+ }, 2,
513+ (struct damon_addr_range[]){
514+ {.start = 0, .end = 15},
515+ {.start = 25, .end = 40},
516+ {.start = 44, .end = 50},
517+ }, 3,
518+ 1,
519+ (struct damon_addr_range[]){
520+ {.start = 0, .end = 15},
521+ {.start = 25, .end = 40},
522+ {.start = 44, .end = 50},
523+ }, 3);
524+}
525+
412526 static void damon_test_nr_accesses_to_accesses_bp(struct kunit *test)
413527 {
414528 struct damon_attrs attrs = {
@@ -694,6 +808,8 @@ static void damos_test_commit_quota(struct kunit *test) · 279 unmodified lines
694808 .ms = 2,
695809 .sz = 3,
696810 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST,
811+ .fail_charge_num = 2,
812+ .fail_charge_denom = 3,
697813 .weight_sz = 4,
698814 .weight_nr_accesses = 5,
699815 .weight_age = 6,
@@ -703,6 +819,8 @@ static void damos_test_commit_quota(struct kunit *test) · 3 unmodified lines
703819 .ms = 8,
704820 .sz = 9,
705821 .goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL,
822+ .fail_charge_num = 1,
823+ .fail_charge_denom = 1024,
706824 .weight_sz = 10,
707825 .weight_nr_accesses = 11,
708826 .weight_age = 12,
@@ -717,6 +835,8 @@ static void damos_test_commit_quota(struct kunit *test) · 8 unmodified lines
717835 KUNIT_EXPECT_EQ(test, dst.ms, src.ms);
718836 KUNIT_EXPECT_EQ(test, dst.sz, src.sz);
719837 KUNIT_EXPECT_EQ(test, dst.goal_tuner, src.goal_tuner);
838+ KUNIT_EXPECT_EQ(test, dst.fail_charge_num, src.fail_charge_num);
839+ KUNIT_EXPECT_EQ(test, dst.fail_charge_denom, src.fail_charge_denom);
720840 KUNIT_EXPECT_EQ(test, dst.weight_sz, src.weight_sz);
721841 KUNIT_EXPECT_EQ(test, dst.weight_nr_accesses, src.weight_nr_accesses);
722842 KUNIT_EXPECT_EQ(test, dst.weight_age, src.weight_age);
@@ -1077,6 +1197,10 @@ static void damon_test_commit_ctx(struct kunit *test) · 354 unmodified lines
10771197 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
10781198 src->min_region_sz = 4095;
10791199 KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), -EINVAL);
1200+ src->min_region_sz = 4096;
1201+ src->pause = true;
1202+ KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
1203+ KUNIT_EXPECT_TRUE(test, dst->pause);
10801204 damon_destroy_ctx(src);
10811205 damon_destroy_ctx(dst);
10821206 }
mm/damon/tests/vaddr-kunit.h+20−7
@@ -132,22 +132,35 @@ static void damon_do_test_apply_three_regions(struct kunit *test, · 131 unmodified lines
132132 unsigned long *expected, int nr_expected)
133133 {
134134 struct damon_target *t;
135+ struct damon_addr_range *ranges;
135136 struct damon_region *r;
136137 int i;
137138
138139 t = damon_new_target();
139140 if (!t)
140141 kunit_skip(test, "target alloc fail");
142+
143+ ranges = kmalloc_array(nr_regions / 2, sizeof(*ranges), GFP_KERNEL);
144+ if (!ranges) {
145+ damon_destroy_target(t, NULL);
146+ kunit_skip(test, "ranges alloc fail");
147+ }
141148 for (i = 0; i < nr_regions / 2; i++) {
142− r = damon_new_region(regions[i * 2], regions[i * 2 + 1]);
143− if (!r) {
144− damon_destroy_target(t, NULL);
145− kunit_skip(test, "region alloc fail");
146− }
147− damon_add_region(r, t);
149+ ranges[i].start = regions[i * 2];
150+ ranges[i].end = regions[i * 2 + 1];
148151 }
152+ if (damon_set_regions(t, ranges, nr_regions / 2,
153+ DAMON_MIN_REGION_SZ)) {
154+ kfree(ranges);
155+ damon_destroy_target(t, NULL);
156+ kunit_skip(test, "damon_set_regions() fail");
157+ }
158+ kfree(ranges);
149159
150− damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ);
160+ if (damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ)) {
161+ damon_destroy_target(t, NULL);
162+ kunit_skip(test, "second damon_set_regions() fail");
163+ }
151164
152165 for (i = 0; i < nr_expected / 2; i++) {
153166 r = __nth_region_of(t, i);
mm/damon/vaddr.c+50−29
@@ -237,6 +237,35 @@ static void damon_va_update(struct damon_ctx *ctx) · 236 unmodified lines
237237 }
238238 }
239239
240+static void damon_va_walk_page_range(struct mm_struct *mm, unsigned long start,
241+ unsigned long end, struct mm_walk_ops *ops, void *private)
242+{
243+ struct vm_area_struct *vma;
244+
245+ vma = lock_vma_under_rcu(mm, start);
246+ if (!vma)
247+ goto lock_mmap;
248+
249+ if (end > vma->vm_end) {
250+ vma_end_read(vma);
251+ goto lock_mmap;
252+ }
253+
254+ if (!(vma->vm_flags & VM_PFNMAP)) {
255+ ops->walk_lock = PGWALK_VMA_RDLOCK_VERIFY;
256+ walk_page_range_vma(vma, start, end, ops, private);
257+ }
258+
259+ vma_end_read(vma);
260+ return;
261+
262+lock_mmap:
263+ mmap_read_lock(mm);
264+ ops->walk_lock = PGWALK_RDLOCK;
265+ walk_page_range(mm, start, end, ops, private);
266+ mmap_read_unlock(mm);
267+}
268+
240269 static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
241270 unsigned long next, struct mm_walk *walk)
242271 {
@@ -315,17 +344,14 @@ out: · 72 unmodified lines
315344 #define damon_mkold_hugetlb_entry NULL
316345 #endif /* CONFIG_HUGETLB_PAGE */
317346
318−static const struct mm_walk_ops damon_mkold_ops = {
319− .pmd_entry = damon_mkold_pmd_entry,
320− .hugetlb_entry = damon_mkold_hugetlb_entry,
321− .walk_lock = PGWALK_RDLOCK,
322−};
323−
324347 static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
325348 {
326− mmap_read_lock(mm);
327− walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
328− mmap_read_unlock(mm);
349+ struct mm_walk_ops damon_mkold_ops = {
350+ .pmd_entry = damon_mkold_pmd_entry,
351+ .hugetlb_entry = damon_mkold_hugetlb_entry,
352+ };
353+
354+ damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
329355 }
330356
331357 /*
@@ -333,9 +359,10 @@ static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) · 1 unmodified line
333359 */
334360
335361 static void __damon_va_prepare_access_check(struct mm_struct *mm,
336− struct damon_region *r)
362+ struct damon_region *r,
363+ struct damon_ctx *ctx)
337364 {
338− r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
365+ r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
339366
340367 damon_va_mkold(mm, r->sampling_addr);
341368 }
@@ -351,7 +378,7 @@ static void damon_va_prepare_access_checks(struct damon_ctx *ctx) · 9 unmodified lines
351378 if (!mm)
352379 continue;
353380 damon_for_each_region(r, t)
354− __damon_va_prepare_access_check(mm, r);
381+ __damon_va_prepare_access_check(mm, r, ctx);
355382 mmput(mm);
356383 }
357384 }
@@ -444,12 +471,6 @@ out: · 86 unmodified lines
444471 #define damon_young_hugetlb_entry NULL
445472 #endif /* CONFIG_HUGETLB_PAGE */
446473
447−static const struct mm_walk_ops damon_young_ops = {
448− .pmd_entry = damon_young_pmd_entry,
449− .hugetlb_entry = damon_young_hugetlb_entry,
450− .walk_lock = PGWALK_RDLOCK,
451−};
452−
453474 static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
454475 unsigned long *folio_sz)
455476 {
@@ -458,9 +479,12 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines
458479 .young = false,
459480 };
460481
461− mmap_read_lock(mm);
462− walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);
463− mmap_read_unlock(mm);
482+ struct mm_walk_ops damon_young_ops = {
483+ .pmd_entry = damon_young_pmd_entry,
484+ .hugetlb_entry = damon_young_hugetlb_entry,
485+ };
486+
487+ damon_va_walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);
464488 return arg.young;
465489 }
466490
@@ -749,7 +773,6 @@ static unsigned long damos_va_migrate(struct damon_target *target, · 282 unmodified lines
749773 struct mm_walk_ops walk_ops = {
750774 .pmd_entry = damos_va_migrate_pmd_entry,
751775 .pte_entry = NULL,
752− .walk_lock = PGWALK_RDLOCK,
753776 };
754777
755778 use_target_nid = dests->nr_dests == 0;
@@ -767,9 +790,7 @@ static unsigned long damos_va_migrate(struct damon_target *target, · 11 unmodified lines
767790 if (!mm)
768791 goto free_lists;
769792
770− mmap_read_lock(mm);
771− walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
772− mmap_read_unlock(mm);
793+ damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
773794 mmput(mm);
774795
775796 for (int i = 0; i < nr_dests; i++) {
@@ -861,7 +882,6 @@ static unsigned long damos_va_stat(struct damon_target *target, · 85 unmodified lines
861882 struct mm_struct *mm;
862883 struct mm_walk_ops walk_ops = {
863884 .pmd_entry = damos_va_stat_pmd_entry,
864− .walk_lock = PGWALK_RDLOCK,
865885 };
866886
867887 priv.scheme = s;
@@ -874,9 +894,7 @@ static unsigned long damos_va_stat(struct damon_target *target, · 6 unmodified lines
874894 if (!mm)
875895 return 0;
876896
877− mmap_read_lock(mm);
878− walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
879− mmap_read_unlock(mm);
897+ damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
880898 mmput(mm);
881899 return 0;
882900 }
@@ -903,6 +921,9 @@ static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx, · 20 unmodified lines
903921 case DAMOS_NOHUGEPAGE:
904922 madv_action = MADV_NOHUGEPAGE;
905923 break;
924+ case DAMOS_COLLAPSE:
925+ madv_action = MADV_COLLAPSE;
926+ break;
906927 case DAMOS_MIGRATE_HOT:
907928 case DAMOS_MIGRATE_COLD:
908929 return damos_va_migrate(t, r, scheme, sz_filter_passed);
mm/filemap.c+80−53
@@ -1808,9 +1808,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1807 unmodified lines
18081808 pgoff_t index, unsigned long max_scan)
18091809 {
18101810 XA_STATE(xas, &mapping->i_pages, index);
1811− unsigned long nr = max_scan;
18121811
1813− while (nr--) {
1812+ while (max_scan--) {
18141813 void *entry = xas_next(&xas);
18151814 if (!entry || xa_is_value(entry))
18161815 return xas.xa_index;
@@ -1818,7 +1817,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1 unmodified line
18181817 return 0;
18191818 }
18201819
1821− return index + max_scan;
1820+ /* Return end of the range + 1 when no hole is found */
1821+ return xas.xa_index + 1;
18221822 }
18231823 EXPORT_SYMBOL(page_cache_next_miss);
18241824
@@ -1849,12 +1849,13 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, · 24 unmodified lines
18491849 while (max_scan--) {
18501850 void *entry = xas_prev(&xas);
18511851 if (!entry || xa_is_value(entry))
1852− break;
1852+ return xas.xa_index;
18531853 if (xas.xa_index == ULONG_MAX)
1854− break;
1854+ return ULONG_MAX;
18551855 }
18561856
1857− return xas.xa_index;
1857+ /* Return start of the range - 1 when no hole is found */
1858+ return xas.xa_index - 1;
18581859 }
18591860 EXPORT_SYMBOL(page_cache_prev_miss);
18601861
@@ -2294,8 +2295,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, · 433 unmodified lines
22942295 goto put_folio;
22952296
22962297 if (!folio_batch_add(fbatch, folio)) {
2297− nr = folio_nr_pages(folio);
2298− *start = folio->index + nr;
2298+ *start = folio_next_index(folio);
22992299 goto out;
23002300 }
23012301 xas_advance(&xas, folio_next_index(folio) - 1);
@@ -2355,8 +2355,7 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, · 53 unmodified lines
23552355 if (xa_is_value(folio))
23562356 continue;
23572357 if (!folio_batch_add(fbatch, folio)) {
2358− unsigned long nr = folio_nr_pages(folio);
2359− *start = folio->index + nr;
2358+ *start = folio_next_index(folio);
23602359 goto out;
23612360 }
23622361 }
@@ -2414,8 +2413,7 @@ unsigned filemap_get_folios_dirty(struct address_space *mapping, pgoff_t *start, · 51 unmodified lines
24142413 }
24152414 }
24162415 if (!folio_batch_add(fbatch, folio)) {
2417− unsigned long nr = folio_nr_pages(folio);
2418− *start = folio->index + nr;
2416+ *start = folio_next_index(folio);
24192417 goto out;
24202418 }
24212419 }
@@ -3323,12 +3321,26 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 901 unmodified lines
33233321 struct file *fpin = NULL;
33243322 vm_flags_t vm_flags = vmf->vma->vm_flags;
33253323 bool force_thp_readahead = false;
3324+ unsigned int thp_order = 0;
33263325 unsigned short mmap_miss;
33273326
3327+ ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
3328+
33283329 /* Use the readahead code, even if readahead is disabled */
3329− if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
3330− (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)
3331− force_thp_readahead = true;
3330+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
3331+ /*
3332+ * Cap max THP order at 2MB: this is the common PMD-sized
3333+ * hugepage size, and it avoids memory pressure from very
3334+ * large forced readahead when mapping_max_folio_order() is
3335+ * high (for example, 128MB with 64K base pages on arm64).
3336+ */
3337+ if (mapping_large_folio_support(mapping)) {
3338+ force_thp_readahead = true;
3339+ thp_order = min_t(unsigned int,
3340+ mapping_max_folio_order(mapping),
3341+ get_order(SZ_2M));
3342+ }
3343+ }
33323344
33333345 if (!force_thp_readahead) {
33343346 /*
@@ -3348,7 +3360,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 13 unmodified lines
33483360 }
33493361 }
33503362
3351− if (!(vm_flags & VM_SEQ_READ)) {
3363+ if (!(vm_flags & (VM_SEQ_READ | VM_EXEC))) {
33523364 /* Avoid banging the cache line if not needed */
33533365 mmap_miss = READ_ONCE(ra->mmap_miss);
33543366 if (mmap_miss < MMAP_LOTSAMISS * 10)
@@ -3363,17 +3375,19 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 8 unmodified lines
33633375 }
33643376
33653377 if (force_thp_readahead) {
3378+ unsigned long folio_nr_pages = 1UL << thp_order;
3379+
33663380 fpin = maybe_unlock_mmap_for_io(vmf, fpin);
3367− ractl._index &= ~((unsigned long)HPAGE_PMD_NR - 1);
3368− ra->size = HPAGE_PMD_NR;
3381+ ractl._index &= ~(folio_nr_pages - 1);
3382+ ra->size = folio_nr_pages;
33693383 /*
3370− * Fetch two PMD folios, so we get the chance to actually
3384+ * Fetch two folios so we get the chance to actually
33713385 * readahead, unless we've been told not to.
33723386 */
33733387 if (!(vm_flags & VM_RAND_READ))
33743388 ra->size *= 2;
3375− ra->async_size = HPAGE_PMD_NR;
3376− ra->order = HPAGE_PMD_ORDER;
3389+ ra->async_size = folio_nr_pages;
3390+ ra->order = thp_order;
33773391 page_cache_ra_order(&ractl, ra);
33783392 return fpin;
33793393 }
@@ -3407,6 +3421,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · 27 unmodified lines
34073421 * mmap read-around
34083422 */
34093423 ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);
3424+ ra->start = max(ra->start, vmf->vma->vm_pgoff);
34103425 ra->size = ra->ra_pages;
34113426 ra->async_size = ra->ra_pages / 4;
34123427 ra->order = 0;
@@ -3441,14 +3456,20 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · 28 unmodified lines
34413456 * Don't touch the mmap_miss counter to avoid decreasing it multiple
34423457 * times for a single folio and break the balance with mmap_miss
34433458 * increase in do_sync_mmap_readahead().
3459+ *
3460+ * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss increment in
3461+ * do_sync_mmap_readahead(), so skip the decrement here as well to
3462+ * keep the counter symmetric.
34443463 */
3445− if (likely(!folio_test_locked(folio))) {
3464+ if (likely(!folio_test_locked(folio)) &&
3465+ !(vmf->vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
34463466 mmap_miss = READ_ONCE(ra->mmap_miss);
34473467 if (mmap_miss)
34483468 WRITE_ONCE(ra->mmap_miss, --mmap_miss);
34493469 }
34503470
34513471 if (folio_test_readahead(folio)) {
3472+ ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
34523473 fpin = maybe_unlock_mmap_for_io(vmf, fpin);
34533474 page_cache_async_ra(&ractl, folio, ra->ra_pages);
34543475 }
@@ -3758,8 +3779,7 @@ skip: · 303 unmodified lines
37583779 static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,
37593780 struct folio *folio, unsigned long start,
37603781 unsigned long addr, unsigned int nr_pages,
3761− unsigned long *rss, unsigned short *mmap_miss,
3762− pgoff_t file_end)
3782+ unsigned long *rss, pgoff_t file_end)
37633783 {
37643784 struct address_space *mapping = folio->mapping;
37653785 unsigned int ref_from_caller = 1;
@@ -3791,16 +3811,6 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, · 25 unmodified lines
37913811 if (PageHWPoison(page + count))
37923812 goto skip;
37933813
3794− /*
3795− * If there are too many folios that are recently evicted
3796− * in a file, they will probably continue to be evicted.
3797− * In such situation, read-ahead is only a waste of IO.
3798− * Don't decrease mmap_miss in this scenario to make sure
3799− * we can stop read-ahead.
3800− */
3801− if (!folio_test_workingset(folio))
3802− (*mmap_miss)++;
3803−
38043814 /*
38053815 * NOTE: If there're PTE markers, we'll leave them to be
38063816 * handled in the specific fault path, and it'll prohibit the
@@ -3847,7 +3857,7 @@ skip: · 40 unmodified lines
38473857
38483858 static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
38493859 struct folio *folio, unsigned long addr,
3850− unsigned long *rss, unsigned short *mmap_miss)
3860+ unsigned long *rss)
38513861 {
38523862 vm_fault_t ret = 0;
38533863 struct page *page = &folio->page;
@@ -3855,10 +3865,6 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf, · 1 unmodified line
38553865 if (PageHWPoison(page))
38563866 goto out;
38573867
3858− /* See comment of filemap_map_folio_range() */
3859− if (!folio_test_workingset(folio))
3860− (*mmap_miss)++;
3861−
38623868 /*
38633869 * NOTE: If there're PTE markers, we'll leave them to be
38643870 * handled in the specific fault path, and it'll prohibit
@@ -3893,7 +3899,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 28 unmodified lines
38933899 vm_fault_t ret = 0;
38943900 unsigned long rss = 0;
38953901 unsigned int nr_pages = 0, folio_type;
3896− unsigned short mmap_miss = 0, mmap_miss_saved;
38973902
38983903 /*
38993904 * Recalculate end_pgoff based on file_end before calling
@@ -3932,6 +3937,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 32 unmodified lines
39323937 folio_type = mm_counter_file(folio);
39333938 do {
39343939 unsigned long end;
3940+ vm_fault_t map_ret;
39353941
39363942 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
39373943 vmf->pte += xas.xa_index - last_pgoff;
@@ -3939,13 +3945,40 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 1 unmodified line
39393945 end = folio_next_index(folio) - 1;
39403946 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;
39413947
3942− if (!folio_test_large(folio))
3943− ret |= filemap_map_order0_folio(vmf,
3944− folio, addr, &rss, &mmap_miss);
3945− else
3946− ret |= filemap_map_folio_range(vmf, folio,
3947− xas.xa_index - folio->index, addr,
3948− nr_pages, &rss, &mmap_miss, file_end);
3948+ if (!folio_test_large(folio)) {
3949+ map_ret = filemap_map_order0_folio(vmf, folio, addr,
3950+ &rss);
3951+ } else {
3952+ unsigned long start = xas.xa_index - folio->index;
3953+
3954+ map_ret = filemap_map_folio_range(vmf, folio, start,
3955+ addr, nr_pages, &rss,
3956+ file_end);
3957+ }
3958+ ret |= map_ret;
3959+
3960+ /*
3961+ * If there are too many folios that are recently evicted
3962+ * in a file, they will probably continue to be evicted.
3963+ * In such situation, read-ahead is only a waste of IO.
3964+ * Don't decrease mmap_miss in this scenario to make sure
3965+ * we can stop read-ahead.
3966+ *
3967+ * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss
3968+ * increment in do_sync_mmap_readahead(), so skip the
3969+ * decrement here as well to keep the counter symmetric.
3970+ */
3971+ if ((map_ret & VM_FAULT_NOPAGE) &&
3972+ !(vmf->flags & FAULT_FLAG_TRIED) &&
3973+ !folio_test_workingset(folio) &&
3974+ !(vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
3975+ unsigned short mmap_miss;
3976+
3977+ mmap_miss = READ_ONCE(file->f_ra.mmap_miss);
3978+ if (mmap_miss)
3979+ WRITE_ONCE(file->f_ra.mmap_miss,
3980+ mmap_miss - 1);
3981+ }
39493982
39503983 folio_unlock(folio);
39513984 } while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL);
@@ -3955,12 +3988,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 3 unmodified lines
39553988 out:
39563989 rcu_read_unlock();
39573990
3958− mmap_miss_saved = READ_ONCE(file->f_ra.mmap_miss);
3959− if (mmap_miss >= mmap_miss_saved)
3960− WRITE_ONCE(file->f_ra.mmap_miss, 0);
3961− else
3962− WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss_saved - mmap_miss);
3963−
39643991 return ret;
39653992 }
39663993 EXPORT_SYMBOL(filemap_map_pages);
mm/gup.c+4−4
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, · 2864 unmodified lines
28652865 if (!folio)
28662866 goto pte_unmap;
28672867
2868− if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) ||
2869− unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {
2868+ if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
2869+ unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
28702870 gup_put_folio(folio, 1, flags);
28712871 goto pte_unmap;
28722872 }
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, · 69 unmodified lines
29422942 if (!folio)
29432943 return 0;
29442944
2945− if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
2945+ if (unlikely(pmd_val(orig) != pmd_val(pmdp_get_lockless(pmdp)))) {
29462946 gup_put_folio(folio, refs, flags);
29472947 return 0;
29482948 }
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, · 36 unmodified lines
29852985 if (!folio)
29862986 return 0;
29872987
2988− if (unlikely(pud_val(orig) != pud_val(*pudp))) {
2988+ if (unlikely(pud_val(orig) != pud_val(pudp_get(pudp)))) {
29892989 gup_put_folio(folio, refs, flags);
29902990 return 0;
29912991 }
mm/huge_memory.c+190−289
@@ -14,6 +14,7 @@ · 13 unmodified lines
1414 #include <linux/mmu_notifier.h>
1515 #include <linux/rmap.h>
1616 #include <linux/swap.h>
17+#include <linux/list_lru.h>
1718 #include <linux/shrinker.h>
1819 #include <linux/mm_inline.h>
1920 #include <linux/swapops.h>
@@ -67,6 +68,8 @@ unsigned long transparent_hugepage_flags __read_mostly = · 47 unmodified lines
6768 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
6869 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
6970
71+static struct lock_class_key deferred_split_key;
72+static struct list_lru deferred_split_lru;
7073 static struct shrinker *deferred_split_shrinker;
7174 static unsigned long deferred_split_count(struct shrinker *shrink,
7275 struct shrink_control *sc);
@@ -429,61 +432,75 @@ ssize_t single_hugepage_flag_store(struct kobject *kobj, · 356 unmodified lines
429432 return count;
430433 }
431434
435+enum defrag_mode {
436+ DEFRAG_ALWAYS = 0,
437+ DEFRAG_DEFER,
438+ DEFRAG_DEFER_MADVISE,
439+ DEFRAG_MADVISE,
440+ DEFRAG_NEVER,
441+};
442+
443+static const char * const defrag_mode_strings[] = {
444+ [DEFRAG_ALWAYS] = "always",
445+ [DEFRAG_DEFER] = "defer",
446+ [DEFRAG_DEFER_MADVISE] = "defer+madvise",
447+ [DEFRAG_MADVISE] = "madvise",
448+ [DEFRAG_NEVER] = "never",
449+};
450+
451+static const enum transparent_hugepage_flag defrag_flags[] = {
452+ [DEFRAG_ALWAYS] = TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,
453+ [DEFRAG_DEFER] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,
454+ [DEFRAG_DEFER_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
455+ [DEFRAG_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
456+};
457+
432458 static ssize_t defrag_show(struct kobject *kobj,
433459 struct kobj_attribute *attr, char *buf)
434460 {
435− const char *output;
461+ int active = DEFRAG_NEVER;
462+ int len = 0;
463+ int i;
436464
437− if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,
438− &transparent_hugepage_flags))
439− output = "[always] defer defer+madvise madvise never";
440− else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,
441− &transparent_hugepage_flags))
442− output = "always [defer] defer+madvise madvise never";
443− else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
444− &transparent_hugepage_flags))
445− output = "always defer [defer+madvise] madvise never";
446− else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
447− &transparent_hugepage_flags))
448− output = "always defer defer+madvise [madvise] never";
449− else
450− output = "always defer defer+madvise madvise [never]";
465+ for (i = 0; i < ARRAY_SIZE(defrag_flags); i++) {
466+ if (test_bit(defrag_flags[i], &transparent_hugepage_flags)) {
467+ active = i;
468+ break;
469+ }
470+ }
451471
452− return sysfs_emit(buf, "%s\n", output);
472+ for (i = 0; i < ARRAY_SIZE(defrag_mode_strings); i++) {
473+ if (i == active)
474+ len += sysfs_emit_at(buf, len, "[%s] ",
475+ defrag_mode_strings[i]);
476+ else
477+ len += sysfs_emit_at(buf, len, "%s ",
478+ defrag_mode_strings[i]);
479+ }
480+
481+ /* Replace trailing space with newline */
482+ buf[len - 1] = '\n';
483+
484+ return len;
453485 }
454486
455487 static ssize_t defrag_store(struct kobject *kobj,
456488 struct kobj_attribute *attr,
457489 const char *buf, size_t count)
458490 {
459− if (sysfs_streq(buf, "always")) {
460− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
461− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
462− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
463− set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
464− } else if (sysfs_streq(buf, "defer+madvise")) {
465− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
466− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
467− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
468− set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
469− } else if (sysfs_streq(buf, "defer")) {
470− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
471− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
472− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
473− set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
474− } else if (sysfs_streq(buf, "madvise")) {
475− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
476− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
477− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
478− set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
479− } else if (sysfs_streq(buf, "never")) {
480− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
481− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
482− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
483− clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
484− } else
491+ int mode, m;
492+
493+ mode = sysfs_match_string(defrag_mode_strings, buf);
494+ if (mode < 0)
485495 return -EINVAL;
486496
497+ for (m = 0; m < ARRAY_SIZE(defrag_flags); m++) {
498+ if (m == mode)
499+ set_bit(defrag_flags[m], &transparent_hugepage_flags);
500+ else
501+ clear_bit(defrag_flags[m], &transparent_hugepage_flags);
502+ }
503+
487504 return count;
488505 }
489506 static struct kobj_attribute defrag_attr = __ATTR_RW(defrag);
@@ -918,15 +935,28 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · 428 unmodified lines
918935 }
919936 #endif /* CONFIG_SYSFS */
920937
938+int folio_memcg_alloc_deferred(struct folio *folio)
939+{
940+ if (mem_cgroup_disabled())
941+ return 0;
942+ return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
943+}
944+
921945 static int __init thp_shrinker_init(void)
922946 {
923947 deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
924− SHRINKER_MEMCG_AWARE |
925− SHRINKER_NONSLAB,
948+ SHRINKER_MEMCG_AWARE,
926949 "thp-deferred_split");
927950 if (!deferred_split_shrinker)
928951 return -ENOMEM;
929952
953+ if (list_lru_init_memcg_key(&deferred_split_lru,
954+ deferred_split_shrinker,
955+ &deferred_split_key)) {
956+ shrinker_free(deferred_split_shrinker);
957+ return -ENOMEM;
958+ }
959+
930960 deferred_split_shrinker->count_objects = deferred_split_count;
931961 deferred_split_shrinker->scan_objects = deferred_split_scan;
932962 shrinker_register(deferred_split_shrinker);
@@ -948,6 +978,7 @@ static int __init thp_shrinker_init(void) · 15 unmodified lines
948978 huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
949979 if (!huge_zero_folio_shrinker) {
950980 shrinker_free(deferred_split_shrinker);
981+ list_lru_destroy(&deferred_split_lru);
951982 return -ENOMEM;
952983 }
953984
@@ -962,6 +993,7 @@ static void __init thp_shrinker_exit(void) · 8 unmodified lines
962993 {
963994 shrinker_free(huge_zero_folio_shrinker);
964995 shrinker_free(deferred_split_shrinker);
996+ list_lru_destroy(&deferred_split_lru);
965997 }
966998
967999 static int __init hugepage_init(void)
@@ -1141,119 +1173,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) · 173 unmodified lines
11411173 return pmd;
11421174 }
11431175
1144−static struct deferred_split *split_queue_node(int nid)
1145−{
1146− struct pglist_data *pgdata = NODE_DATA(nid);
1147−
1148− return &pgdata->deferred_split_queue;
1149−}
1150−
1151−#ifdef CONFIG_MEMCG
1152−static inline
1153−struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
1154− struct deferred_split *queue)
1155−{
1156− if (mem_cgroup_disabled())
1157− return NULL;
1158− if (split_queue_node(folio_nid(folio)) == queue)
1159− return NULL;
1160− return container_of(queue, struct mem_cgroup, deferred_split_queue);
1161−}
1162−
1163−static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
1164−{
1165− return memcg ? &memcg->deferred_split_queue : split_queue_node(nid);
1166−}
1167−#else
1168−static inline
1169−struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
1170− struct deferred_split *queue)
1171−{
1172− return NULL;
1173−}
1174−
1175−static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
1176−{
1177− return split_queue_node(nid);
1178−}
1179−#endif
1180−
1181−static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup *memcg)
1182−{
1183− struct deferred_split *queue;
1184−
1185−retry:
1186− queue = memcg_split_queue(nid, memcg);
1187− spin_lock(&queue->split_queue_lock);
1188− /*
1189− * There is a period between setting memcg to dying and reparenting
1190− * deferred split queue, and during this period the THPs in the deferred
1191− * split queue will be hidden from the shrinker side.
1192− */
1193− if (unlikely(memcg_is_dying(memcg))) {
1194− spin_unlock(&queue->split_queue_lock);
1195− memcg = parent_mem_cgroup(memcg);
1196− goto retry;
1197− }
1198−
1199− return queue;
1200−}
1201−
1202−static struct deferred_split *
1203−split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flags)
1204−{
1205− struct deferred_split *queue;
1206−
1207−retry:
1208− queue = memcg_split_queue(nid, memcg);
1209− spin_lock_irqsave(&queue->split_queue_lock, *flags);
1210− if (unlikely(memcg_is_dying(memcg))) {
1211− spin_unlock_irqrestore(&queue->split_queue_lock, *flags);
1212− memcg = parent_mem_cgroup(memcg);
1213− goto retry;
1214− }
1215−
1216− return queue;
1217−}
1218−
1219−static struct deferred_split *folio_split_queue_lock(struct folio *folio)
1220−{
1221− struct deferred_split *queue;
1222−
1223− rcu_read_lock();
1224− queue = split_queue_lock(folio_nid(folio), folio_memcg(folio));
1225− /*
1226− * The memcg destruction path is acquiring the split queue lock for
1227− * reparenting. Once you have it locked, it's safe to drop the rcu lock.
1228− */
1229− rcu_read_unlock();
1230−
1231− return queue;
1232−}
1233−
1234−static struct deferred_split *
1235−folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags)
1236−{
1237− struct deferred_split *queue;
1238−
1239− rcu_read_lock();
1240− queue = split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
1241− rcu_read_unlock();
1242−
1243− return queue;
1244−}
1245−
1246−static inline void split_queue_unlock(struct deferred_split *queue)
1247−{
1248− spin_unlock(&queue->split_queue_lock);
1249−}
1250−
1251−static inline void split_queue_unlock_irqrestore(struct deferred_split *queue,
1252− unsigned long flags)
1253−{
1254− spin_unlock_irqrestore(&queue->split_queue_lock, flags);
1255−}
1256−
12571176 static inline bool is_transparent_hugepage(const struct folio *folio)
12581177 {
12591178 if (!folio_test_large(folio))
@@ -1354,6 +1273,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, · 94 unmodified lines
13541273 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
13551274 return NULL;
13561275 }
1276+
1277+ if (folio_memcg_alloc_deferred(folio)) {
1278+ folio_put(folio);
1279+ count_vm_event(THP_FAULT_FALLBACK);
1280+ count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
1281+ return NULL;
1282+ }
1283+
13571284 folio_throttle_swaprate(folio, gfp);
13581285
13591286 /*
@@ -2638,6 +2565,8 @@ static void change_non_present_huge_pmd(struct mm_struct *mm, · 1278 unmodified lines
26382565 } else if (softleaf_is_device_private_write(entry)) {
26392566 entry = make_readable_device_private_entry(swp_offset(entry));
26402567 newpmd = swp_entry_to_pmd(entry);
2568+ if (pmd_swp_uffd_wp(*pmd))
2569+ newpmd = pmd_swp_mkuffd_wp(newpmd);
26412570 } else {
26422571 newpmd = *pmd;
26432572 }
@@ -3890,34 +3819,43 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n · 1246 unmodified lines
38903819 struct folio *end_folio = folio_next(folio);
38913820 struct folio *new_folio, *next;
38923821 int old_order = folio_order(folio);
3822+ struct list_lru_one *lru;
3823+ bool dequeue_deferred;
38933824 int ret = 0;
3894− struct deferred_split *ds_queue;
38953825
38963826 VM_WARN_ON_ONCE(!mapping && end);
3897− /* Prevent deferred_split_scan() touching ->_refcount */
3898− ds_queue = folio_split_queue_lock(folio);
3827+ /*
3828+ * If this folio can be on the deferred split queue, lock out
3829+ * the shrinker before freezing the ref. If the shrinker sees
3830+ * a 0-ref folio, it assumes it beat folio_put() to the list
3831+ * lock and must clean up the LRU state - the same dequeue we
3832+ * will do below as part of the split.
3833+ */
3834+ dequeue_deferred = folio_test_anon(folio) && old_order > 1;
3835+ if (dequeue_deferred) {
3836+ struct mem_cgroup *memcg;
3837+
3838+ rcu_read_lock();
3839+ memcg = folio_memcg(folio);
3840+ lru = list_lru_lock(&deferred_split_lru,
3841+ folio_nid(folio), &memcg);
3842+ }
38993843 if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
39003844 struct swap_cluster_info *ci = NULL;
39013845 struct lruvec *lruvec;
39023846
3903− if (old_order > 1) {
3904− if (!list_empty(&folio->_deferred_list)) {
3905− ds_queue->split_queue_len--;
3906− /*
3907− * Reinitialize page_deferred_list after removing the
3908− * page from the split_queue, otherwise a subsequent
3909− * split will see list corruption when checking the
3910− * page_deferred_list.
3911− */
3912− list_del_init(&folio->_deferred_list);
3913− }
3847+ if (dequeue_deferred) {
3848+ __list_lru_del(&deferred_split_lru, lru,
3849+ &folio->_deferred_list, folio_nid(folio));
39143850 if (folio_test_partially_mapped(folio)) {
39153851 folio_clear_partially_mapped(folio);
39163852 mod_mthp_stat(old_order,
39173853 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
39183854 }
3855+ list_lru_unlock(lru);
3856+ rcu_read_unlock();
39193857 }
3920− split_queue_unlock(ds_queue);
3858+
39213859 if (mapping) {
39223860 int nr = folio_nr_pages(folio);
39233861
@@ -4018,7 +3956,10 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n · 94 unmodified lines
40183956 if (ci)
40193957 swap_cluster_unlock(ci);
40203958 } else {
4021− split_queue_unlock(ds_queue);
3959+ if (dequeue_deferred) {
3960+ list_lru_unlock(lru);
3961+ rcu_read_unlock();
3962+ }
40223963 return -EAGAIN;
40233964 }
40243965
@@ -4193,11 +4134,10 @@ fail: · 168 unmodified lines
41934134
41944135 folio_unlock(new_folio);
41954136 /*
4196− * Subpages may be freed if there wasn't any mapping
4197− * like if add_to_swap() is running on a lru page that
4198− * had its mapping zapped. And freeing these pages
4199− * requires taking the lru_lock so we do the put_page
4200− * of the tail pages after the split is complete.
4137+ * Subpages whose mapping has been zapped may be freed
4138+ * earlier, but freeing them requires taking the
4139+ * lru_lock, so we defer put_page() on tail pages until
4140+ * after the split completes.
42014141 */
42024142 free_folio_and_swap_cache(new_folio);
42034143 }
@@ -4385,33 +4325,37 @@ int split_folio_to_list(struct folio *folio, struct list_head *list) · 181 unmodified lines
43854325 * queueing THP splits, and that list is (racily observed to be) non-empty.
43864326 *
43874327 * It is unsafe to call folio_unqueue_deferred_split() until folio refcount is
4388− * zero: because even when split_queue_lock is held, a non-empty _deferred_list
4389− * might be in use on deferred_split_scan()'s unlocked on-stack list.
4328+ * zero: because even when the list_lru lock is held, a non-empty
4329+ * _deferred_list might be in use on deferred_split_scan()'s unlocked
4330+ * on-stack list.
43904331 *
4391− * If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: it is
4392− * therefore important to unqueue deferred split before changing folio memcg.
4332+ * The list_lru sublist is determined by folio's memcg: it is therefore
4333+ * important to unqueue deferred split before changing folio memcg.
43934334 */
43944335 bool __folio_unqueue_deferred_split(struct folio *folio)
43954336 {
4396− struct deferred_split *ds_queue;
4337+ struct mem_cgroup *memcg;
4338+ struct list_lru_one *lru;
4339+ int nid = folio_nid(folio);
43974340 unsigned long flags;
43984341 bool unqueued = false;
43994342
44004343 WARN_ON_ONCE(folio_ref_count(folio));
44014344 WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio));
44024345
4403− ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
4404− if (!list_empty(&folio->_deferred_list)) {
4405− ds_queue->split_queue_len--;
4346+ rcu_read_lock();
4347+ memcg = folio_memcg(folio);
4348+ lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
4349+ if (__list_lru_del(&deferred_split_lru, lru, &folio->_deferred_list, nid)) {
44064350 if (folio_test_partially_mapped(folio)) {
44074351 folio_clear_partially_mapped(folio);
44084352 mod_mthp_stat(folio_order(folio),
44094353 MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
44104354 }
4411− list_del_init(&folio->_deferred_list);
44124355 unqueued = true;
44134356 }
4414− split_queue_unlock_irqrestore(ds_queue, flags);
4357+ list_lru_unlock_irqrestore(lru, &flags);
4358+ rcu_read_unlock();
44154359
44164360 return unqueued; /* useful for debug warnings */
44174361 }
@@ -4419,7 +4363,9 @@ bool __folio_unqueue_deferred_split(struct folio *folio) · 1 unmodified line
44194363 /* partially_mapped=false won't clear PG_partially_mapped folio flag */
44204364 void deferred_split_folio(struct folio *folio, bool partially_mapped)
44214365 {
4422− struct deferred_split *ds_queue;
4366+ struct list_lru_one *lru;
4367+ int nid;
4368+ struct mem_cgroup *memcg;
44234369 unsigned long flags;
44244370
44254371 /*
@@ -4434,7 +4380,7 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 8 unmodified lines
44344380
44354381 /*
44364382 * Exclude swapcache: originally to avoid a corrupt deferred split
4437− * queue. Nowadays that is fully prevented by memcg1_swapout();
4383+ * queue. Nowadays that is fully prevented by __memcg1_swapout();
44384384 * but if page reclaim is already handling the same folio, it is
44394385 * unnecessary to handle it again in the shrinker, so excluding
44404386 * swapcache here may still be a useful optimization.
@@ -4442,7 +4388,11 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 1 unmodified line
44424388 if (folio_test_swapcache(folio))
44434389 return;
44444390
4445− ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
4391+ nid = folio_nid(folio);
4392+
4393+ rcu_read_lock();
4394+ memcg = folio_memcg(folio);
4395+ lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
44464396 if (partially_mapped) {
44474397 if (!folio_test_partially_mapped(folio)) {
44484398 folio_set_partially_mapped(folio);
@@ -4450,36 +4400,23 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) · 1 unmodified line
44504400 count_vm_event(THP_DEFERRED_SPLIT_PAGE);
44514401 count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
44524402 mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);
4453−
44544403 }
44554404 } else {
44564405 /* partially mapped folios cannot become non-partially mapped */
44574406 VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);
44584407 }
4459− if (list_empty(&folio->_deferred_list)) {
4460− struct mem_cgroup *memcg;
4461−
4462− memcg = folio_split_queue_memcg(folio, ds_queue);
4463− list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
4464− ds_queue->split_queue_len++;
4465− if (memcg)
4466− set_shrinker_bit(memcg, folio_nid(folio),
4467− shrinker_id(deferred_split_shrinker));
4468− }
4469− split_queue_unlock_irqrestore(ds_queue, flags);
4408+ __list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, memcg);
4409+ list_lru_unlock_irqrestore(lru, &flags);
4410+ rcu_read_unlock();
44704411 }
44714412
44724413 static unsigned long deferred_split_count(struct shrinker *shrink,
44734414 struct shrink_control *sc)
44744415 {
4475− struct pglist_data *pgdata = NODE_DATA(sc->nid);
4476− struct deferred_split *ds_queue = &pgdata->deferred_split_queue;
4416+ unsigned long count;
44774417
4478−#ifdef CONFIG_MEMCG
4479− if (sc->memcg)
4480− ds_queue = &sc->memcg->deferred_split_queue;
4481−#endif
4482− return READ_ONCE(ds_queue->split_queue_len);
4418+ count = list_lru_shrink_count(&deferred_split_lru, sc);
4419+ return count ?: SHRINK_EMPTY;
44834420 }
44844421
44854422 static bool thp_underused(struct folio *folio)
@@ -4509,45 +4446,49 @@ static bool thp_underused(struct folio *folio) · 23 unmodified lines
45094446 return false;
45104447 }
45114448
4449+static enum lru_status deferred_split_isolate(struct list_head *item,
4450+ struct list_lru_one *lru,
4451+ void *cb_arg)
4452+{
4453+ struct folio *folio = container_of(item, struct folio, _deferred_list);
4454+ struct list_head *freeable = cb_arg;
4455+
4456+ if (folio_try_get(folio)) {
4457+ list_lru_isolate_move(lru, item, freeable);
4458+ return LRU_REMOVED;
4459+ }
4460+
4461+ /*
4462+ * We lost race with folio_put(). Read folio state before the
4463+ * isolate: folio_unqueue_deferred_split() checks list_empty()
4464+ * locklessly, so once removed the folio can be freed any time.
4465+ */
4466+ if (folio_test_partially_mapped(folio)) {
4467+ folio_clear_partially_mapped(folio);
4468+ mod_mthp_stat(folio_order(folio),
4469+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
4470+ }
4471+ list_lru_isolate(lru, item);
4472+ return LRU_REMOVED;
4473+}
4474+
45124475 static unsigned long deferred_split_scan(struct shrinker *shrink,
45134476 struct shrink_control *sc)
45144477 {
4515− struct deferred_split *ds_queue;
4516− unsigned long flags;
4478+ LIST_HEAD(dispose);
45174479 struct folio *folio, *next;
4518− int split = 0, i;
4519− struct folio_batch fbatch;
4520−
4521− folio_batch_init(&fbatch);
4480+ int split = 0;
4481+ unsigned long isolated;
45224482
4523−retry:
4524− ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);
4525− /* Take pin on all head pages to avoid freeing them under us */
4526− list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
4527− _deferred_list) {
4528− if (folio_try_get(folio)) {
4529− folio_batch_add(&fbatch, folio);
4530− } else if (folio_test_partially_mapped(folio)) {
4531− /* We lost race with folio_put() */
4532− folio_clear_partially_mapped(folio);
4533− mod_mthp_stat(folio_order(folio),
4534− MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
4535− }
4536− list_del_init(&folio->_deferred_list);
4537− ds_queue->split_queue_len--;
4538− if (!--sc->nr_to_scan)
4539− break;
4540− if (!folio_batch_space(&fbatch))
4541− break;
4542− }
4543− split_queue_unlock_irqrestore(ds_queue, flags);
4483+ isolated = list_lru_shrink_walk_irq(&deferred_split_lru, sc,
4484+ deferred_split_isolate, &dispose);
45444485
4545− for (i = 0; i < folio_batch_count(&fbatch); i++) {
4486+ list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
45464487 bool did_split = false;
45474488 bool underused = false;
4548− struct deferred_split *fqueue;
45494489
4550− folio = fbatch.folios[i];
4490+ list_del_init(&folio->_deferred_list);
4491+
45514492 if (!folio_test_partially_mapped(folio)) {
45524493 /*
45534494 * See try_to_map_unused_to_zeropage(): we cannot
@@ -4576,63 +4517,23 @@ next: · 22 unmodified lines
45764517 * underused, then consider it used and don't add it back to
45774518 * split_queue.
45784519 */
4579− if (did_split || !folio_test_partially_mapped(folio))
4580− continue;
4520+ if (!did_split && folio_test_partially_mapped(folio)) {
45814521 requeue:
4582− /*
4583− * Add back partially mapped folios, or underused folios that
4584− * we could not lock this round.
4585− */
4586− fqueue = folio_split_queue_lock_irqsave(folio, &flags);
4587− if (list_empty(&folio->_deferred_list)) {
4588− list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
4589− fqueue->split_queue_len++;
4522+ rcu_read_lock();
4523+ list_lru_add_irq(&deferred_split_lru,
4524+ &folio->_deferred_list,
4525+ folio_nid(folio),
4526+ folio_memcg(folio));
4527+ rcu_read_unlock();
45904528 }
4591− split_queue_unlock_irqrestore(fqueue, flags);
4592− }
4593− folios_put(&fbatch);
4594−
4595− if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
4596− cond_resched();
4597− goto retry;
4529+ folio_put(folio);
45984530 }
45994531
4600− /*
4601− * Stop shrinker if we didn't split any page, but the queue is empty.
4602− * This can happen if pages were freed under us.
4603− */
4604− if (!split && list_empty(&ds_queue->split_queue))
4532+ if (!split && !isolated)
46054533 return SHRINK_STOP;
46064534 return split;
46074535 }
46084536
4609−#ifdef CONFIG_MEMCG
4610−void reparent_deferred_split_queue(struct mem_cgroup *memcg)
4611−{
4612− struct mem_cgroup *parent = parent_mem_cgroup(memcg);
4613− struct deferred_split *ds_queue = &memcg->deferred_split_queue;
4614− struct deferred_split *parent_ds_queue = &parent->deferred_split_queue;
4615− int nid;
4616−
4617− spin_lock_irq(&ds_queue->split_queue_lock);
4618− spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING);
4619−
4620− if (!ds_queue->split_queue_len)
4621− goto unlock;
4622−
4623− list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_queue);
4624− parent_ds_queue->split_queue_len += ds_queue->split_queue_len;
4625− ds_queue->split_queue_len = 0;
4626−
4627− for_each_node(nid)
4628− set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker));
4629−
4630−unlock:
4631− spin_unlock(&parent_ds_queue->split_queue_lock);
4632− spin_unlock_irq(&ds_queue->split_queue_lock);
4633−}
4634−#endif
4635−
46364537 #ifdef CONFIG_DEBUG_FS
46374538 static void split_huge_pages_all(void)
46384539 {
mm/hugetlb.c+4−3
@@ -2862,6 +2862,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 2861 unmodified lines
28622862 map_chg_state map_chg;
28632863 int ret, idx;
28642864 struct hugetlb_cgroup *h_cg = NULL;
2865+ struct hugetlb_cgroup *h_cg_rsvd = NULL;
28652866 gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
28662867
28672868 idx = hstate_index(h);
@@ -2912,7 +2913,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 44 unmodified lines
29122913 */
29132914 if (map_chg) {
29142915 ret = hugetlb_cgroup_charge_cgroup_rsvd(
2915− idx, pages_per_huge_page(h), &h_cg);
2916+ idx, pages_per_huge_page(h), &h_cg_rsvd);
29162917 if (ret)
29172918 goto out_subpool_put;
29182919 }
@@ -2954,7 +2955,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 35 unmodified lines
29542955 */
29552956 if (map_chg) {
29562957 hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h),
2957− h_cg, folio);
2958+ h_cg_rsvd, folio);
29582959 }
29592960
29602961 spin_unlock_irq(&hugetlb_lock);
@@ -3006,7 +3007,7 @@ out_uncharge_cgroup: · 45 unmodified lines
30063007 out_uncharge_cgroup_reservation:
30073008 if (map_chg)
30083009 hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
3009− h_cg);
3010+ h_cg_rsvd);
30103011 out_subpool_put:
30113012 /*
30123013 * put page to subpool iff the quota of subpool's rsv_hpages is used
mm/hugetlb_cma.c+16−19
@@ -142,7 +142,7 @@ unsigned int __weak arch_hugetlb_cma_order(void) · 141 unmodified lines
142142
143143 void __init hugetlb_cma_reserve(void)
144144 {
145− unsigned long size, reserved, per_node, order;
145+ unsigned long size, reserved, per_node, order, gigantic_page_size;
146146 bool node_specific_cma_alloc = false;
147147 int nid;
148148
@@ -162,37 +162,36 @@ void __init hugetlb_cma_reserve(void) · 13 unmodified lines
162162 * breaking this assumption.
163163 */
164164 VM_WARN_ON(order <= MAX_PAGE_ORDER);
165+ gigantic_page_size = PAGE_SIZE << order;
165166
166167 hugetlb_bootmem_set_nodes();
167168
168169 for (nid = 0; nid < MAX_NUMNODES; nid++) {
169− if (hugetlb_cma_size_in_node[nid] == 0)
170+ size = hugetlb_cma_size_in_node[nid];
171+ if (size == 0)
170172 continue;
171173
172174 if (!node_isset(nid, hugetlb_bootmem_nodes)) {
173175 pr_warn("hugetlb_cma: invalid node %d specified\n", nid);
174− hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
175− hugetlb_cma_size_in_node[nid] = 0;
176− continue;
177− }
178−
179− if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) {
180− pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n",
181− nid, (PAGE_SIZE << order) / SZ_1M);
182− hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
183− hugetlb_cma_size_in_node[nid] = 0;
176+ } else if (!IS_ALIGNED(size, gigantic_page_size)) {
177+ pr_warn("hugetlb_cma: cma area of node %d must be a multiple of %lu MiB\n",
178+ nid, gigantic_page_size / SZ_1M);
184179 } else {
185180 node_specific_cma_alloc = true;
181+ continue;
186182 }
183+
184+ hugetlb_cma_size -= size;
185+ hugetlb_cma_size_in_node[nid] = 0;
187186 }
188187
189188 /* Validate the CMA size again in case some invalid nodes specified. */
190189 if (!hugetlb_cma_size)
191190 return;
192191
193− if (hugetlb_cma_size < (PAGE_SIZE << order)) {
194− pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n",
195− (PAGE_SIZE << order) / SZ_1M);
192+ if (!IS_ALIGNED(hugetlb_cma_size, gigantic_page_size)) {
193+ pr_warn("hugetlb_cma: cma area must be a multiple of %lu MiB\n",
194+ gigantic_page_size / SZ_1M);
196195 hugetlb_cma_size = 0;
197196 return;
198197 }
@@ -204,7 +203,7 @@ void __init hugetlb_cma_reserve(void) · 5 unmodified lines
204203 */
205204 per_node = DIV_ROUND_UP(hugetlb_cma_size,
206205 nodes_weight(hugetlb_bootmem_nodes));
207− per_node = round_up(per_node, PAGE_SIZE << order);
206+ per_node = round_up(per_node, gigantic_page_size);
208207 pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
209208 hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
210209 }
@@ -223,15 +222,13 @@ void __init hugetlb_cma_reserve(void) · 12 unmodified lines
223222 size = min(per_node, hugetlb_cma_size - reserved);
224223 }
225224
226− size = round_up(size, PAGE_SIZE << order);
227−
228225 snprintf(name, sizeof(name), "hugetlb%d", nid);
229226 /*
230227 * Note that 'order per bit' is based on smallest size that
231228 * may be returned to CMA allocator in the case of
232229 * huge page demotion.
233230 */
234− res = cma_declare_contiguous_multi(size, PAGE_SIZE << order,
231+ res = cma_declare_contiguous_multi(size, gigantic_page_size,
235232 HUGETLB_PAGE_ORDER, name,
236233 &hugetlb_cma[nid], nid);
237234 if (res) {
mm/internal.h+13−14
@@ -17,7 +17,6 @@ · 16 unmodified lines
1717 #include <linux/rmap.h>
1818 #include <linux/swap.h>
1919 #include <linux/leafops.h>
20−#include <linux/swap_cgroup.h>
2120 #include <linux/tracepoint-defs.h>
2221
2322 /* Internal core VMA manipulation functions. */
@@ -451,24 +450,16 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) · 427 unmodified lines
451450 {
452451 pte_t expected_pte = pte_next_swp_offset(pte);
453452 const pte_t *end_ptep = start_ptep + max_nr;
454− const softleaf_t entry = softleaf_from_pte(pte);
455453 pte_t *ptep = start_ptep + 1;
456− unsigned short cgroup_id;
457454
458455 VM_WARN_ON(max_nr < 1);
459− VM_WARN_ON(!softleaf_is_swap(entry));
456+ VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte)));
460457
461− cgroup_id = lookup_swap_cgroup_id(entry);
462458 while (ptep < end_ptep) {
463− softleaf_t entry;
464−
465459 pte = ptep_get(ptep);
466460
467461 if (!pte_same(pte, expected_pte))
468462 break;
469− entry = softleaf_from_pte(pte);
470− if (lookup_swap_cgroup_id(entry) != cgroup_id)
471− break;
472463 expected_pte = pte_next_swp_offset(expected_pte);
473464 ptep++;
474465 }
@@ -861,7 +852,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio) · 386 unmodified lines
861852 /*
862853 * At this point, there is no one trying to add the folio to
863854 * deferred_list. If folio is not in deferred_list, it's safe
864− * to check without acquiring the split_queue_lock.
855+ * to check without acquiring the list_lru lock.
865856 */
866857 if (data_race(list_empty(&folio->_deferred_list)))
867858 return false;
@@ -1104,9 +1095,17 @@ static inline void init_cma_pageblock(struct page *page) · 236 unmodified lines
11041095 }
11051096 #endif
11061097
1107−
1108−int find_suitable_fallback(struct free_area *area, unsigned int order,
1109− int migratetype, bool claimable);
1098+enum fallback_result {
1099+ /* Found suitable migratetype, *mt_out is valid. */
1100+ FALLBACK_FOUND,
1101+ /* No fallback found in requested order. */
1102+ FALLBACK_EMPTY,
1103+ /* Passed @claimable, but claiming whole block is a bad idea. */
1104+ FALLBACK_NOCLAIM,
1105+};
1106+enum fallback_result
1107+find_suitable_fallback(struct free_area *area, unsigned int order,
1108+ int migratetype, bool claimable, int *mt_out);
11101109
11111110 static inline bool free_area_empty(struct free_area *area, int migratetype)
11121111 {
mm/kasan/kasan_test_c.c+10−0
@@ -874,6 +874,16 @@ static void kmalloc_double_kzfree(struct kunit *test) · 873 unmodified lines
874874 char *ptr;
875875 size_t size = 16;
876876
877+ /*
878+ * With the tag-based KASAN modes, if the memory happens to be
879+ * reallocated between the two frees and the new allocation tag happens
880+ * to match the old one, the second free will cause a memory corruption.
881+ * Resolving https://bugzilla.kernel.org/show_bug.cgi?id=212177 would
882+ * help to deal with this. With Generic KASAN, it's effectively
883+ * impossible for the memory to get reallocated due to the quarantine.
884+ */
885+ KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
886+
877887 ptr = kmalloc(size, GFP_KERNEL);
878888 KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
879889
mm/kfence/kfence_test.c+1−1
@@ -263,7 +263,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat · 262 unmodified lines
263263 break;
264264 }
265265
266− kunit_info(test, "%s: size=%zu, gfp=%x, policy=%s, cache=%i\n", __func__, size, gfp,
266+ kunit_info(test, "%s: size=%zu, gfp=%pGg, policy=%s, cache=%i\n", __func__, size, &gfp,
267267 policy_name, !!test_cache);
268268
269269 /*
mm/khugepaged.c+15−6
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm) · 436 unmodified lines
437437
438438 /* __khugepaged_exit() must not run from under us */
439439 VM_BUG_ON_MM(collapse_test_exit(mm), mm);
440− if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
441− return;
442440
443441 slot = mm_slot_alloc(mm_slot_cache);
444442 if (!slot)
445443 return;
446444
445+ if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) {
446+ mm_slot_free(mm_slot_cache, slot);
447+ return;
448+ }
449+
447450 spin_lock(&khugepaged_mm_lock);
448451 mm_slot_insert(mm_slots_hash, mm, slot);
449452 /*
@@ -1120,6 +1123,11 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a · 670 unmodified lines
11201123 if (result != SCAN_SUCCEED)
11211124 goto out_nolock;
11221125
1126+ if (folio_memcg_alloc_deferred(folio)) {
1127+ result = SCAN_ALLOC_HUGE_PAGE_FAIL;
1128+ goto out_nolock;
1129+ }
1130+
11231131 mmap_read_lock(mm);
11241132 result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
11251133 if (result != SCAN_SUCCEED) {
@@ -2528,8 +2536,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, · 1402 unmodified lines
25282536 cc->progress++;
25292537 continue;
25302538 }
2531− hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE);
2532− hend = round_down(vma->vm_end, HPAGE_PMD_SIZE);
2539+ hstart = ALIGN(vma->vm_start, HPAGE_PMD_SIZE);
2540+ hend = ALIGN_DOWN(vma->vm_end, HPAGE_PMD_SIZE);
25332541 if (khugepaged_scan.address > hend) {
25342542 cc->progress++;
25352543 continue;
@@ -2808,6 +2816,7 @@ static int madvise_collapse_errno(enum scan_result r) · 272 unmodified lines
28082816 case SCAN_PAGE_LRU:
28092817 case SCAN_DEL_PAGE_LRU:
28102818 case SCAN_PAGE_FILLED:
2819+ case SCAN_PAGE_HAS_PRIVATE:
28112820 case SCAN_PAGE_DIRTY_OR_WRITEBACK:
28122821 return -EAGAIN;
28132822 /*
@@ -2845,8 +2854,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · 31 unmodified lines
28452854 mmgrab(mm);
28462855 lru_add_drain_all();
28472856
2848− hstart = (start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
2849− hend = end & HPAGE_PMD_MASK;
2857+ hstart = ALIGN(start, HPAGE_PMD_SIZE);
2858+ hend = ALIGN_DOWN(end, HPAGE_PMD_SIZE);
28502859
28512860 for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
28522861 enum scan_result result = SCAN_FAIL;
mm/kmemleak.c+140−8
@@ -92,6 +92,7 @@ · 91 unmodified lines
9292 #include <linux/nodemask.h>
9393 #include <linux/mm.h>
9494 #include <linux/workqueue.h>
95+#include <linux/xarray.h>
9596 #include <linux/crc32.h>
9697
9798 #include <asm/sections.h>
@@ -157,6 +158,8 @@ struct kmemleak_object { · 59 unmodified lines
157158 struct hlist_head area_list;
158159 unsigned long jiffies; /* creation timestamp */
159160 pid_t pid; /* pid of the current task */
161+ /* per-scan dedup count, valid only while in scan-local dedup xarray */
162+ unsigned int dup_count;
160163 char comm[TASK_COMM_LEN]; /* executable name */
161164 };
162165
@@ -360,8 +363,9 @@ static const char *__object_type_str(struct kmemleak_object *object) · 197 unmodified lines
360363 * Printing of the unreferenced objects information to the seq file. The
361364 * print_unreferenced function must be called with the object->lock held.
362365 */
363−static void print_unreferenced(struct seq_file *seq,
364− struct kmemleak_object *object)
366+static void __print_unreferenced(struct seq_file *seq,
367+ struct kmemleak_object *object,
368+ bool hex_dump)
365369 {
366370 int i;
367371 unsigned long *entries;
@@ -373,7 +377,8 @@ static void print_unreferenced(struct seq_file *seq, · 5 unmodified lines
373377 object->pointer, object->size);
374378 warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu\n",
375379 object->comm, object->pid, object->jiffies);
376− hex_dump_object(seq, object);
380+ if (hex_dump)
381+ hex_dump_object(seq, object);
377382 warn_or_seq_printf(seq, " backtrace (crc %x):\n", object->checksum);
378383
379384 for (i = 0; i < nr_entries; i++) {
@@ -382,6 +387,12 @@ static void print_unreferenced(struct seq_file *seq, · 2 unmodified lines
382387 }
383388 }
384389
390+static void print_unreferenced(struct seq_file *seq,
391+ struct kmemleak_object *object)
392+{
393+ __print_unreferenced(seq, object, true);
394+}
395+
385396 /*
386397 * Print the kmemleak_object information. This function is used mainly for
387398 * debugging special cases when kmemleak operations. It must be called with
@@ -1684,6 +1695,103 @@ unlock_put: · 1296 unmodified lines
16841695 put_object(object);
16851696 }
16861697
1698+/*
1699+ * Print one leak inline. The hex dump is gated on OBJECT_ALLOCATED so it
1700+ * does not touch user memory that was freed concurrently; the rest of the
1701+ * report (backtrace, comm, pid) is always emitted since the kmemleak_object
1702+ * metadata is pinned by the caller.
1703+ */
1704+static void print_leak_locked(struct kmemleak_object *object, bool hex_dump)
1705+{
1706+ raw_spin_lock_irq(&object->lock);
1707+ __print_unreferenced(NULL, object,
1708+ hex_dump && (object->flags & OBJECT_ALLOCATED));
1709+ raw_spin_unlock_irq(&object->lock);
1710+}
1711+
1712+/*
1713+ * Per-scan dedup table for verbose leak printing. The xarray is keyed by
1714+ * stackdepot trace_handle and stores a pointer to the representative
1715+ * kmemleak_object. The per-scan repeat count lives in object->dup_count.
1716+ *
1717+ * dedup_record() must run outside object->lock: xa_store() may take
1718+ * mutexes (xa_node slab allocation) which lockdep would flag against the
1719+ * raw spinlock object->lock.
1720+ */
1721+static void dedup_record(struct xarray *dedup, struct kmemleak_object *object,
1722+ depot_stack_handle_t trace_handle)
1723+{
1724+ struct kmemleak_object *rep;
1725+ void *old;
1726+
1727+ /*
1728+ * No stack trace to dedup against: early-boot allocation tracked
1729+ * before kmemleak_init() set up object_cache, or stack_depot_save()
1730+ * failure under memory pressure.
1731+ */
1732+ if (!trace_handle) {
1733+ print_leak_locked(object, true);
1734+ return;
1735+ }
1736+
1737+ /* stack is available, now we can de-dup */
1738+ rep = xa_load(dedup, trace_handle);
1739+ if (rep) {
1740+ rep->dup_count++;
1741+ return;
1742+ }
1743+
1744+ /*
1745+ * Object is being torn down (use_count already hit zero); the
1746+ * tracked memory at object->pointer is unsafe to read, so skip.
1747+ */
1748+ if (!get_object(object))
1749+ return;
1750+
1751+ object->dup_count = 1;
1752+ old = xa_store(dedup, trace_handle, object, GFP_ATOMIC);
1753+ if (xa_is_err(old)) {
1754+ /* xa_node allocation failed; fall back to inline print. */
1755+ print_leak_locked(object, true);
1756+ put_object(object);
1757+ return;
1758+ }
1759+ /*
1760+ * scan_mutex serialises all writers to the dedup xarray, so xa_store()
1761+ * after a NULL xa_load() must always overwrite an empty slot.
1762+ */
1763+ WARN_ON_ONCE(old);
1764+}
1765+
1766+/*
1767+ * Drain the dedup table. Re-acquires object->lock and re-checks
1768+ * OBJECT_ALLOCATED before printing: while get_object() pins the
1769+ * kmemleak_object metadata, the underlying tracked allocation may have
1770+ * been freed since the scan walked it (kmemleak_free clears
1771+ * OBJECT_ALLOCATED under object->lock before the user memory goes away).
1772+ * The hex dump is skipped for coalesced entries since the bytes would
1773+ * differ across objects anyway.
1774+ */
1775+static void dedup_flush(struct xarray *dedup)
1776+{
1777+ struct kmemleak_object *object;
1778+ unsigned long idx;
1779+ unsigned int dup;
1780+ bool coalesced;
1781+
1782+ xa_for_each(dedup, idx, object) {
1783+ dup = object->dup_count;
1784+ coalesced = dup > 1;
1785+
1786+ print_leak_locked(object, !coalesced);
1787+ if (coalesced)
1788+ pr_warn(" ... and %u more object(s) with the same backtrace\n",
1789+ dup - 1);
1790+ put_object(object);
1791+ xa_erase(dedup, idx);
1792+ }
1793+}
1794+
16871795 /*
16881796 * Scan data sections and all the referenced memory blocks allocated via the
16891797 * kernel's standard allocators. This function must be called with the
@@ -1694,6 +1802,7 @@ static void kmemleak_scan(void) · 4 unmodified lines
16941802 struct kmemleak_object *object;
16951803 struct zone *zone;
16961804 int __maybe_unused i;
1805+ struct xarray dedup;
16971806 int new_leaks = 0;
16981807
16991808 jiffies_last_scan = jiffies;
@@ -1834,10 +1943,18 @@ static void kmemleak_scan(void) · 134 unmodified lines
18341943 return;
18351944
18361945 /*
1837− * Scanning result reporting.
1946+ * Scanning result reporting. When verbose printing is enabled, dedupe
1947+ * by stackdepot trace_handle so each unique backtrace is logged once
1948+ * per scan, annotated with the number of objects that share it. The
1949+ * per-leak count below still reflects every object, and
1950+ * /sys/kernel/debug/kmemleak still lists them individually.
18381951 */
1952+ xa_init(&dedup);
18391953 rcu_read_lock();
18401954 list_for_each_entry_rcu(object, &object_list, object_list) {
1955+ depot_stack_handle_t trace_handle;
1956+ bool dedup_print;
1957+
18411958 if (need_resched())
18421959 kmemleak_cond_resched(object);
18431960
@@ -1849,18 +1966,33 @@ static void kmemleak_scan(void) · 5 unmodified lines
18491966 if (!color_white(object))
18501967 continue;
18511968 raw_spin_lock_irq(&object->lock);
1969+ trace_handle = 0;
1970+ dedup_print = false;
18521971 if (unreferenced_object(object) &&
18531972 !(object->flags & OBJECT_REPORTED)) {
18541973 object->flags |= OBJECT_REPORTED;
1855−
1856− if (kmemleak_verbose)
1857− print_unreferenced(NULL, object);
1858−
1974+ if (kmemleak_verbose) {
1975+ trace_handle = object->trace_handle;
1976+ dedup_print = true;
1977+ }
18591978 new_leaks++;
18601979 }
18611980 raw_spin_unlock_irq(&object->lock);
1981+
1982+ /*
1983+ * Defer the verbose print outside object->lock: xa_store()
1984+ * may take xa_node slab locks at a higher wait-context level
1985+ * which lockdep would flag against the raw_spinlock_t
1986+ * object->lock. rcu_read_lock() keeps the kmemleak_object
1987+ * alive across the call.
1988+ */
1989+ if (dedup_print)
1990+ dedup_record(&dedup, object, trace_handle);
18621991 }
18631992 rcu_read_unlock();
1993+ /* Flush'em all */
1994+ dedup_flush(&dedup);
1995+ xa_destroy(&dedup);
18641996
18651997 if (new_leaks) {
18661998 kmemleak_found_leaks = true;
mm/list_lru.c+162−76
@@ -15,6 +15,28 @@ · 14 unmodified lines
1515 #include "slab.h"
1616 #include "internal.h"
1717
18+static inline void lock_list_lru(struct list_lru_one *l, bool irq,
19+ unsigned long *irq_flags)
20+{
21+ if (irq_flags)
22+ spin_lock_irqsave(&l->lock, *irq_flags);
23+ else if (irq)
24+ spin_lock_irq(&l->lock);
25+ else
26+ spin_lock(&l->lock);
27+}
28+
29+static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
30+ unsigned long *irq_flags)
31+{
32+ if (irq_flags)
33+ spin_unlock_irqrestore(&l->lock, *irq_flags);
34+ else if (irq_off)
35+ spin_unlock_irq(&l->lock);
36+ else
37+ spin_unlock(&l->lock);
38+}
39+
1840 #ifdef CONFIG_MEMCG
1941 static LIST_HEAD(memcg_list_lrus);
2042 static DEFINE_MUTEX(list_lrus_mutex);
@@ -60,34 +82,23 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 39 unmodified lines
6082 return &lru->node[nid].lru;
6183 }
6284
63−static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
64−{
65− if (irq)
66− spin_lock_irq(&l->lock);
67− else
68− spin_lock(&l->lock);
69− if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
70− if (irq)
71− spin_unlock_irq(&l->lock);
72− else
73− spin_unlock(&l->lock);
74− return false;
75− }
76− return true;
77−}
78−
7985 static inline struct list_lru_one *
80−lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
81− bool irq, bool skip_empty)
86+lock_list_lru_of_memcg(struct list_lru *lru, int nid,
87+ struct mem_cgroup **memcg, bool irq,
88+ unsigned long *irq_flags, bool skip_empty)
8289 {
8390 struct list_lru_one *l;
8491
8592 rcu_read_lock();
8693 again:
87− l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
88− if (likely(l) && lock_list_lru(l, irq)) {
89− rcu_read_unlock();
90− return l;
94+ l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(*memcg));
95+ if (likely(l)) {
96+ lock_list_lru(l, irq, irq_flags);
97+ if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
98+ rcu_read_unlock();
99+ return l;
100+ }
101+ unlock_list_lru(l, irq, irq_flags);
91102 }
92103 /*
93104 * Caller may simply bail out if raced with reparenting or
@@ -97,18 +108,10 @@ again: · 3 unmodified lines
97108 rcu_read_unlock();
98109 return NULL;
99110 }
100− VM_WARN_ON(!css_is_dying(&memcg->css));
101− memcg = parent_mem_cgroup(memcg);
111+ VM_WARN_ON(!css_is_dying(&(*memcg)->css));
112+ *memcg = parent_mem_cgroup(*memcg);
102113 goto again;
103114 }
104−
105−static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
106−{
107− if (irq_off)
108− spin_unlock_irq(&l->lock);
109− else
110− spin_unlock(&l->lock);
111−}
112115 #else
113116 static void list_lru_register(struct list_lru *lru)
114117 {
@@ -135,52 +138,112 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 20 unmodified lines
135138 }
136139
137140 static inline struct list_lru_one *
138−lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
139− bool irq, bool skip_empty)
141+lock_list_lru_of_memcg(struct list_lru *lru, int nid,
142+ struct mem_cgroup **memcg, bool irq,
143+ unsigned long *irq_flags, bool skip_empty)
140144 {
141145 struct list_lru_one *l = &lru->node[nid].lru;
142146
143− if (irq)
144− spin_lock_irq(&l->lock);
145− else
146− spin_lock(&l->lock);
147+ lock_list_lru(l, irq, irq_flags);
147148
148149 return l;
149150 }
151+#endif /* CONFIG_MEMCG */
150152
151−static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
153+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
154+ struct mem_cgroup **memcg)
152155 {
153− if (irq_off)
154− spin_unlock_irq(&l->lock);
155− else
156− spin_unlock(&l->lock);
156+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
157+ /*irq_flags=*/NULL, /*skip_empty=*/false);
157158 }
158−#endif /* CONFIG_MEMCG */
159159
160−/* The caller must ensure the memcg lifetime. */
161−bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
162− struct mem_cgroup *memcg)
160+void list_lru_unlock(struct list_lru_one *l)
163161 {
164− struct list_lru_node *nlru = &lru->node[nid];
165− struct list_lru_one *l;
162+ unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
163+}
166164
167− l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
168− if (!l)
169− return false;
165+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
166+ struct mem_cgroup **memcg)
167+{
168+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
169+ /*irq_flags=*/NULL, /*skip_empty=*/false);
170+}
171+
172+void list_lru_unlock_irq(struct list_lru_one *l)
173+{
174+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
175+}
176+
177+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
178+ struct mem_cgroup **memcg,
179+ unsigned long *flags)
180+{
181+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
182+ /*irq_flags=*/flags, /*skip_empty=*/false);
183+}
184+
185+void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
186+{
187+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
188+}
189+
190+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
191+ struct list_head *item, int nid,
192+ struct mem_cgroup *memcg)
193+{
170194 if (list_empty(item)) {
171195 list_add_tail(item, &l->list);
172− /* Set shrinker bit if the first element was added */
196+ /*
197+ * Set shrinker bit on the memcg that owns the locked
198+ * sublist - lock_list_lru_of_memcg() may have walked up
199+ * past a dying memcg, and the bit must be set there.
200+ */
173201 if (!l->nr_items++)
174202 set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
175− unlock_list_lru(l, false);
176− atomic_long_inc(&nlru->nr_items);
203+ atomic_long_inc(&lru->node[nid].nr_items);
177204 return true;
178205 }
179− unlock_list_lru(l, false);
180206 return false;
181207 }
182208 EXPORT_SYMBOL_GPL(list_lru_add);
183209
210+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
211+ struct list_head *item, int nid)
212+{
213+ if (!list_empty(item)) {
214+ list_del_init(item);
215+ l->nr_items--;
216+ atomic_long_dec(&lru->node[nid].nr_items);
217+ return true;
218+ }
219+ return false;
220+}
221+
222+/* The caller must ensure the memcg lifetime. */
223+bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
224+ struct mem_cgroup *memcg)
225+{
226+ struct list_lru_one *l;
227+ bool ret;
228+
229+ l = list_lru_lock(lru, nid, &memcg);
230+ ret = __list_lru_add(lru, l, item, nid, memcg);
231+ list_lru_unlock(l);
232+ return ret;
233+}
234+
235+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
236+ int nid, struct mem_cgroup *memcg)
237+{
238+ struct list_lru_one *l;
239+ bool ret;
240+
241+ l = list_lru_lock_irq(lru, nid, &memcg);
242+ ret = __list_lru_add(lru, l, item, nid, memcg);
243+ list_lru_unlock_irq(l);
244+ return ret;
245+}
246+
184247 bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
185248 {
186249 bool ret;
@@ -202,20 +265,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); · 15 unmodified lines
202265 bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
203266 struct mem_cgroup *memcg)
204267 {
205− struct list_lru_node *nlru = &lru->node[nid];
206268 struct list_lru_one *l;
207− l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
208− if (!l)
209− return false;
210− if (!list_empty(item)) {
211− list_del_init(item);
212− l->nr_items--;
213− unlock_list_lru(l, false);
214− atomic_long_dec(&nlru->nr_items);
215− return true;
216− }
217− unlock_list_lru(l, false);
218− return false;
269+ bool ret;
270+
271+ l = list_lru_lock(lru, nid, &memcg);
272+ ret = __list_lru_del(lru, l, item, nid);
273+ list_lru_unlock(l);
274+ return ret;
219275 }
220276
221277 bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
@@ -288,7 +344,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, · 66 unmodified lines
288344 unsigned long isolated = 0;
289345
290346 restart:
291− l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);
347+ l = lock_list_lru_of_memcg(lru, nid, &memcg, /*irq=*/irq_off,
348+ /*irq_flags=*/NULL, /*skip_empty=*/true);
292349 if (!l)
293350 return isolated;
294351 list_for_each_safe(item, n, &l->list) {
@@ -329,7 +386,7 @@ restart: · 34 unmodified lines
329386 BUG();
330387 }
331388 }
332− unlock_list_lru(l, irq_off);
389+ unlock_list_lru(l, irq_off, NULL);
333390 out:
334391 return isolated;
335392 }
@@ -514,17 +571,14 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg, · 178 unmodified lines
514571 return idx < 0 || xa_load(&lru->xa, idx);
515572 }
516573
517−int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
518− gfp_t gfp)
574+static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
575+ struct list_lru *lru, gfp_t gfp)
519576 {
520577 unsigned long flags;
521578 struct list_lru_memcg *mlru = NULL;
522579 struct mem_cgroup *pos, *parent;
523580 XA_STATE(xas, &lru->xa, 0);
524581
525− if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
526− return 0;
527−
528582 gfp &= GFP_RECLAIM_MASK;
529583 /*
530584 * Because the list_lru can be reparented to the parent cgroup's
@@ -565,6 +619,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, · 34 unmodified lines
565619
566620 return xas_error(&xas);
567621 }
622+
623+int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
624+ gfp_t gfp)
625+{
626+ if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
627+ return 0;
628+ return __memcg_list_lru_alloc(memcg, lru, gfp);
629+}
630+
631+int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
632+ gfp_t gfp)
633+{
634+ struct mem_cgroup *memcg;
635+ int res;
636+
637+ if (!list_lru_memcg_aware(lru))
638+ return 0;
639+
640+ /* Fast path when list_lru heads already exist */
641+ rcu_read_lock();
642+ memcg = folio_memcg(folio);
643+ res = memcg_list_lru_allocated(memcg, lru);
644+ rcu_read_unlock();
645+ if (likely(res))
646+ return 0;
647+
648+ /* Allocation may block, pin the memcg */
649+ memcg = get_mem_cgroup_from_folio(folio);
650+ res = __memcg_list_lru_alloc(memcg, lru, gfp);
651+ mem_cgroup_put(memcg);
652+ return res;
653+}
568654 #else
569655 static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware)
570656 {
mm/madvise.c+25−35
@@ -1834,50 +1834,29 @@ static void madvise_finish_tlb(struct madvise_behavior *madv_behavior) · 1833 unmodified lines
18341834 tlb_finish_mmu(madv_behavior->tlb);
18351835 }
18361836
1837−static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior)
1837+/**
1838+ * check_input_range() - Check if the requested range is valid.
1839+ * @start: Start address of madvise-requested address range.
1840+ * @len_in: Length of madvise-requested address range.
1841+ *
1842+ * Returns: 0 if the input range is valid, otherwise an error code.
1843+ */
1844+static int check_input_range(unsigned long start, size_t len_in)
18381845 {
18391846 size_t len;
18401847
1841− if (!madvise_behavior_valid(behavior))
1842− return false;
1843−
18441848 if (!PAGE_ALIGNED(start))
1845− return false;
1849+ return -EINVAL;
18461850 len = PAGE_ALIGN(len_in);
18471851
18481852 /* Check to see whether len was rounded up from small -ve to zero */
18491853 if (len_in && !len)
1850− return false;
1854+ return -EINVAL;
18511855
18521856 if (start + len < start)
1853− return false;
1854−
1855− return true;
1856−}
1857+ return -EINVAL;
18571858
1858−/*
1859− * madvise_should_skip() - Return if the request is invalid or nothing.
1860− * @start: Start address of madvise-requested address range.
1861− * @len_in: Length of madvise-requested address range.
1862− * @behavior: Requested madvise behavior.
1863− * @err: Pointer to store an error code from the check.
1864− *
1865− * If the specified behaviour is invalid or nothing would occur, we skip the
1866− * operation. This function returns true in the cases, otherwise false. In
1867− * the former case we store an error on @err.
1868− */
1869−static bool madvise_should_skip(unsigned long start, size_t len_in,
1870− int behavior, int *err)
1871−{
1872− if (!is_valid_madvise(start, len_in, behavior)) {
1873− *err = -EINVAL;
1874− return true;
1875− }
1876− if (start + PAGE_ALIGN(len_in) == start) {
1877− *err = 0;
1878− return true;
1879− }
1880− return false;
1859+ return 0;
18811860 }
18821861
18831862 static bool is_madvise_populate(struct madvise_behavior *madv_behavior)
@@ -2013,8 +1992,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh · 129 unmodified lines
20131992 .tlb = &tlb,
20141993 };
20151994
2016− if (madvise_should_skip(start, len_in, behavior, &error))
1995+ if (!madvise_behavior_valid(behavior))
1996+ return -EINVAL;
1997+
1998+ error = check_input_range(start, len_in);
1999+ if (error || !len_in)
20172000 return error;
2001+
20182002 error = madvise_lock(&madv_behavior);
20192003 if (error)
20202004 return error;
@@ -2056,7 +2040,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, · 35 unmodified lines
20562040 size_t len_in = iter_iov_len(iter);
20572041 int error;
20582042
2059− if (madvise_should_skip(start, len_in, behavior, &error))
2043+ error = check_input_range(start, len_in);
2044+ if (error || !len_in)
20602045 ret = error;
20612046 else
20622047 ret = madvise_do_behavior(start, len_in, &madv_behavior);
@@ -2131,6 +2116,11 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, · 68 unmodified lines
21312116 goto release_task;
21322117 }
21332118
2119+ if (!madvise_behavior_valid(behavior)) {
2120+ ret = -EINVAL;
2121+ goto release_mm;
2122+ }
2123+
21342124 /*
21352125 * We need only perform this check if we are attempting to manipulate a
21362126 * remote process's address space.
mm/memcontrol-v1.c+44−24
@@ -5,7 +5,6 @@ · 4 unmodified lines
55 #include <linux/mm_inline.h>
66 #include <linux/pagewalk.h>
77 #include <linux/backing-dev.h>
8−#include <linux/swap_cgroup.h>
98 #include <linux/eventfd.h>
109 #include <linux/poll.h>
1110 #include <linux/sort.h>
@@ -14,6 +13,7 @@ · 2 unmodified lines
1413
1514 #include "internal.h"
1615 #include "swap.h"
16+#include "swap_table.h"
1717 #include "memcontrol-v1.h"
1818
1919 /*
@@ -603,19 +603,26 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 583 unmodified lines
603603 local_irq_restore(flags);
604604 }
605605
606+#ifdef CONFIG_SWAP
606607 /**
607− * memcg1_swapout - transfer a memsw charge to swap
608+ * __memcg1_swapout - transfer a memsw charge to swap
608609 * @folio: folio whose memsw charge to transfer
609− * @entry: swap entry to move the charge to
610+ * @ci: the locked swap cluster holding the swap entries
611+ *
612+ * Transfer the memsw charge of @folio to the swap entry stored in
613+ * folio->swap.
610614 *
611− * Transfer the memsw charge of @folio to @entry.
615+ * Context: folio must be isolated, unmapped, locked and is just about to
616+ * be freed, and caller must disable IRQs and hold the swap cluster lock.
612617 */
613−void memcg1_swapout(struct folio *folio, swp_entry_t entry)
618+void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
614619 {
615620 struct mem_cgroup *memcg, *swap_memcg;
616621 struct obj_cgroup *objcg;
617622 unsigned int nr_entries;
618623
624+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
625+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
619626 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
620627 VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);
621628
@@ -641,7 +648,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 19 unmodified lines
641648 swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries);
642649 mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);
643650
644− swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry);
651+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries,
652+ mem_cgroup_private_id(swap_memcg));
645653
646654 folio_unqueue_deferred_split(folio);
647655 folio->memcg_data = 0;
@@ -656,8 +664,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 8 unmodified lines
656664 }
657665
658666 /*
659− * Interrupts should be disabled here because the caller holds the
660− * i_pages lock which is taken with interrupts-off. It is
667+ * The caller must hold the swap cluster lock with IRQ off. It is
661668 * important here to have the interrupts disabled because it is the
662669 * only synchronisation we have for updating the per-CPU variables.
663670 */
@@ -671,18 +678,24 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 7 unmodified lines
671678 obj_cgroup_put(objcg);
672679 }
673680
674−/*
675− * memcg1_swapin - uncharge swap slot
676− * @entry: the first swap entry for which the pages are charged
677− * @nr_pages: number of pages which will be uncharged
681+/**
682+ * memcg1_swapin - uncharge swap slot on swapin
683+ * @folio: folio being swapped in
678684 *
679− * Call this function after successfully adding the charged page to swapcache.
685+ * Call this function after successfully adding the charged
686+ * folio to swapcache.
680687 *
681− * Note: This function assumes the page for which swap slot is being uncharged
682− * is order 0 page.
688+ * Context: The folio has to be in swap cache and locked.
683689 */
684−void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
690+void memcg1_swapin(struct folio *folio)
685691 {
692+ struct swap_cluster_info *ci;
693+ unsigned long nr_pages;
694+ unsigned short id;
695+
696+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
697+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
698+
686699 /*
687700 * Cgroup1's unified memory+swap counter has been charged with the
688701 * new swapcache page, finish the transfer by uncharging the swap
@@ -695,15 +708,22 @@ void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages) · 6 unmodified lines
695708 * correspond 1:1 to page and swap slot lifetimes: we charge the
696709 * page to memory here, and uncharge swap when the slot is freed.
697710 */
698− if (do_memsw_account()) {
699− /*
700− * The swap entry might not get freed for a long time,
701− * let's not wait for it. The page already received a
702− * memory+swap charge, drop the swap entry duplicate.
703− */
704− mem_cgroup_uncharge_swap(entry, nr_pages);
705− }
711+ if (!do_memsw_account())
712+ return;
713+
714+ /*
715+ * The swap entry might not get freed for a long time,
716+ * let's not wait for it. The page already received a
717+ * memory+swap charge, drop the swap entry duplicate.
718+ */
719+ nr_pages = folio_nr_pages(folio);
720+ ci = swap_cluster_get_and_lock(folio);
721+ id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
722+ nr_pages);
723+ swap_cluster_unlock(ci);
724+ mem_cgroup_uncharge_swap(id, nr_pages);
706725 }
726+#endif
707727
708728 void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
709729 unsigned long nr_memory, int nid)
mm/memcontrol.c+197−97
@@ -54,7 +54,6 @@ · 53 unmodified lines
5454 #include <linux/vmpressure.h>
5555 #include <linux/memremap.h>
5656 #include <linux/mm_inline.h>
57−#include <linux/swap_cgroup.h>
5857 #include <linux/cpu.h>
5958 #include <linux/oom.h>
6059 #include <linux/lockdep.h>
@@ -64,6 +63,7 @@ · 3 unmodified lines
6463 #include <linux/sched/isolation.h>
6564 #include <linux/kmemleak.h>
6665 #include "internal.h"
66+#include "swap_table.h"
6767 #include <net/sock.h>
6868 #include <net/ip.h>
6969 #include "slab.h"
@@ -150,15 +150,15 @@ static void obj_cgroup_release(struct percpu_ref *ref) · 80 unmodified lines
150150 * However, it can be PAGE_SIZE or (x * PAGE_SIZE).
151151 *
152152 * The following sequence can lead to it:
153− * 1) CPU0: objcg == stock->cached_objcg
153+ * 1) CPU0: objcg cached in one of stock->cached[i]
154154 * 2) CPU1: we do a small allocation (e.g. 92 bytes),
155155 * PAGE_SIZE bytes are charged
156156 * 3) CPU1: a process from another memcg is allocating something,
157157 * the stock if flushed,
158158 * objcg->nr_charged_bytes = PAGE_SIZE - 92
159− * 5) CPU0: we do release this object,
160− * 92 bytes are added to stock->nr_bytes
161− * 6) CPU0: stock is flushed,
159+ * 4) CPU0: we do release this object,
160+ * 92 bytes are added to stock->nr_bytes[i]
161+ * 5) CPU0: stock is flushed,
162162 * 92 bytes are added to objcg->nr_charged_bytes
163163 *
164164 * In the result, nr_charged_bytes == PAGE_SIZE.
@@ -2018,24 +2018,49 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { · 1853 unmodified lines
20182018 .lock = INIT_LOCAL_TRYLOCK(lock),
20192019 };
20202020
2021+/*
2022+ * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
2023+ * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
2024+ * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
2025+ * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
2026+ * + pad(6) + cached(40) == 64 bytes.
2027+ * A CPU can thus consume/refill/account against five different objcgs
2028+ * (typically per-node variants of the same memcg) while incurring at
2029+ * most one cache miss on the stock.
2030+ */
2031+#define NR_OBJ_STOCK 5
20212032 struct obj_stock_pcp {
20222033 local_trylock_t lock;
2023− unsigned int nr_bytes;
2024− struct obj_cgroup *cached_objcg;
2025− struct pglist_data *cached_pgdat;
2026− int nr_slab_reclaimable_b;
2027− int nr_slab_unreclaimable_b;
2034+ int8_t index;
2035+ int16_t node_id;
2036+ int16_t nr_slab_reclaimable_b;
2037+ int16_t nr_slab_unreclaimable_b;
2038+#if PAGE_SHIFT > 16
2039+ /*
2040+ * On rare archs with 256KiB base page size (hexagon and powerpc 44x)
2041+ * keep nr_bytes to unsigned int as uint16_t cannot represent the full
2042+e patches/memcg-uint16_t-for-nr_bytes-in-obj_stock_pcp.patch * sub-page remainder. Such archs are not cacheline optimization target.
2043+ */
2044+ unsigned int nr_bytes[NR_OBJ_STOCK];
2045+#else
2046+ uint16_t nr_bytes[NR_OBJ_STOCK];
2047+#endif
2048+ struct obj_cgroup *cached[NR_OBJ_STOCK];
20282049
20292050 struct work_struct work;
20302051 unsigned long flags;
2052+ uint8_t drain_idx;
20312053 };
20322054
20332055 static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
20342056 .lock = INIT_LOCAL_TRYLOCK(lock),
2057+ .index = -1,
2058+ .node_id = NUMA_NO_NODE,
20352059 };
20362060
20372061 static DEFINE_MUTEX(percpu_charge_mutex);
20382062
2063+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
20392064 static void drain_obj_stock(struct obj_stock_pcp *stock);
20402065 static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
20412066 struct mem_cgroup *root_memcg);
@@ -3155,54 +3180,73 @@ static void unlock_stock(struct obj_stock_pcp *stock) · 1113 unmodified lines
31553180 local_unlock(&obj_stock.lock);
31563181 }
31573182
3158−/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
3183+/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
31593184 static void __account_obj_stock(struct obj_cgroup *objcg,
31603185 struct obj_stock_pcp *stock, int nr,
31613186 struct pglist_data *pgdat, enum node_stat_item idx)
31623187 {
3163− int *bytes;
3188+ int16_t *bytes;
3189+ int i;
31643190
3165− if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
3191+ /*
3192+ * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
3193+ * sure it does not exceed S16_MAX otherwise we need to fix node_id type
3194+ * in struct obj_stock_pcp.
3195+ */
3196+ BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
3197+
3198+ if (!stock)
3199+ goto direct;
3200+
3201+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
3202+ if (READ_ONCE(stock->cached[i]) == objcg)
3203+ break;
3204+ }
3205+ if (i == NR_OBJ_STOCK)
31663206 goto direct;
31673207
31683208 /*
31693209 * Save vmstat data in stock and skip vmstat array update unless
3170− * accumulating over a page of vmstat data or when pgdat changes.
3210+ * accumulating over a page of vmstat data or when the objcg slot or
3211+ * pgdat the stats belong to changes.
31713212 */
3172− if (stock->cached_pgdat != pgdat) {
3173− /* Flush the existing cached vmstat data */
3174− struct pglist_data *oldpg = stock->cached_pgdat;
3213+ if (stock->index < 0) {
3214+ stock->index = i;
3215+ stock->node_id = pgdat->node_id;
3216+ } else if (stock->index != i || stock->node_id != pgdat->node_id) {
3217+ struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
3218+ struct pglist_data *oldpg = NODE_DATA(stock->node_id);
31753219
31763220 if (stock->nr_slab_reclaimable_b) {
3177− mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
3221+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
31783222 stock->nr_slab_reclaimable_b);
31793223 stock->nr_slab_reclaimable_b = 0;
31803224 }
31813225 if (stock->nr_slab_unreclaimable_b) {
3182− mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
3226+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
31833227 stock->nr_slab_unreclaimable_b);
31843228 stock->nr_slab_unreclaimable_b = 0;
31853229 }
3186− stock->cached_pgdat = pgdat;
3230+ stock->index = i;
3231+ stock->node_id = pgdat->node_id;
31873232 }
31883233
31893234 bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
31903235 : &stock->nr_slab_unreclaimable_b;
3236+
31913237 /*
3192− * Even for large object >= PAGE_SIZE, the vmstat data will still be
3193− * cached locally at least once before pushing it out.
3238+ * Fold @nr into the cached value and decide whether to keep it cached
3239+ * or flush it directly. Cache the combined value when it fits in the
3240+ * int16_t storage and either the cache was empty (so even a value
3241+ * above PAGE_SIZE gets a chance to be canceled by a paired delta) or
3242+ * the combined value is within the PAGE_SIZE flush threshold.
31943243 */
3195− if (!*bytes) {
3244+ nr += *bytes;
3245+ if (abs(nr) <= S16_MAX && (!*bytes || abs(nr) <= PAGE_SIZE)) {
31963246 *bytes = nr;
31973247 nr = 0;
31983248 } else {
3199− *bytes += nr;
3200− if (abs(*bytes) > PAGE_SIZE) {
3201− nr = *bytes;
3202− *bytes = 0;
3203− } else {
3204− nr = 0;
3205− }
3249+ *bytes = 0;
32063250 }
32073251 direct:
32083252 if (nr)
@@ -3213,10 +3257,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg, · 4 unmodified lines
32133257 struct obj_stock_pcp *stock,
32143258 unsigned int nr_bytes)
32153259 {
3216− if (objcg == READ_ONCE(stock->cached_objcg) &&
3217− stock->nr_bytes >= nr_bytes) {
3218− stock->nr_bytes -= nr_bytes;
3219− return true;
3260+ int i;
3261+
3262+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
3263+ if (READ_ONCE(stock->cached[i]) != objcg)
3264+ continue;
3265+ if (stock->nr_bytes[i] >= nr_bytes) {
3266+ stock->nr_bytes[i] -= nr_bytes;
3267+ return true;
3268+ }
3269+ return false;
32203270 }
32213271
32223272 return false;
@@ -3237,16 +3287,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) · 14 unmodified lines
32373287 return ret;
32383288 }
32393289
3240−static void drain_obj_stock(struct obj_stock_pcp *stock)
3290+/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
3291+static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
32413292 {
3242− struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);
3293+ struct obj_cgroup *old;
3294+ struct pglist_data *oldpg;
3295+
3296+ if (stock->index < 0)
3297+ return;
3298+
3299+ old = READ_ONCE(stock->cached[stock->index]);
3300+ oldpg = NODE_DATA(stock->node_id);
3301+
3302+ if (stock->nr_slab_reclaimable_b) {
3303+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
3304+ stock->nr_slab_reclaimable_b);
3305+ stock->nr_slab_reclaimable_b = 0;
3306+ }
3307+ if (stock->nr_slab_unreclaimable_b) {
3308+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
3309+ stock->nr_slab_unreclaimable_b);
3310+ stock->nr_slab_unreclaimable_b = 0;
3311+ }
3312+ stock->index = -1;
3313+ stock->node_id = NUMA_NO_NODE;
3314+}
3315+
3316+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
3317+{
3318+ struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
32433319
32443320 if (!old)
32453321 return;
32463322
3247− if (stock->nr_bytes) {
3248− unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
3249− unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);
3323+ if (stock->nr_bytes[i]) {
3324+ unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
3325+ unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
32503326
32513327 if (nr_pages) {
32523328 struct mem_cgroup *memcg;
@@ -3272,44 +3348,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock) · 19 unmodified lines
32723348 * so it might be changed in the future.
32733349 */
32743350 atomic_add(nr_bytes, &old->nr_charged_bytes);
3275− stock->nr_bytes = 0;
3351+ stock->nr_bytes[i] = 0;
32763352 }
32773353
3278− /*
3279− * Flush the vmstat data in current stock
3280− */
3281− if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
3282− if (stock->nr_slab_reclaimable_b) {
3283− mod_objcg_mlstate(old, stock->cached_pgdat,
3284− NR_SLAB_RECLAIMABLE_B,
3285− stock->nr_slab_reclaimable_b);
3286− stock->nr_slab_reclaimable_b = 0;
3287− }
3288− if (stock->nr_slab_unreclaimable_b) {
3289− mod_objcg_mlstate(old, stock->cached_pgdat,
3290− NR_SLAB_UNRECLAIMABLE_B,
3291− stock->nr_slab_unreclaimable_b);
3292− stock->nr_slab_unreclaimable_b = 0;
3293− }
3294− stock->cached_pgdat = NULL;
3295− }
3354+ /* Flush vmstat data when its owning slot is being drained. */
3355+ if (stock->index == i)
3356+ drain_obj_stock_stats(stock);
32963357
3297− WRITE_ONCE(stock->cached_objcg, NULL);
3358+ WRITE_ONCE(stock->cached[i], NULL);
32983359 obj_cgroup_put(old);
32993360 }
33003361
3362+static void drain_obj_stock(struct obj_stock_pcp *stock)
3363+{
3364+ int i;
3365+
3366+ for (i = 0; i < NR_OBJ_STOCK; ++i)
3367+ drain_obj_stock_slot(stock, i);
3368+}
3369+
33013370 static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
33023371 struct mem_cgroup *root_memcg)
33033372 {
3304− struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
3373+ struct obj_cgroup *objcg;
33053374 struct mem_cgroup *memcg;
33063375 bool flush = false;
3376+ int i;
33073377
33083378 rcu_read_lock();
3309− if (objcg) {
3379+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
3380+ objcg = READ_ONCE(stock->cached[i]);
3381+ if (!objcg)
3382+ continue;
33103383 memcg = obj_cgroup_memcg(objcg);
3311− if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
3384+ if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
33123385 flush = true;
3386+ break;
3387+ }
33133388 }
33143389 rcu_read_unlock();
33153390
@@ -3322,6 +3397,8 @@ static void __refill_obj_stock(struct obj_cgroup *objcg, · 6 unmodified lines
33223397 bool allow_uncharge)
33233398 {
33243399 unsigned int nr_pages = 0;
3400+ unsigned int stock_nr_bytes;
3401+ int i, slot = -1, empty_slot = -1;
33253402
33263403 if (!stock) {
33273404 nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3330,21 +3407,52 @@ static void __refill_obj_stock(struct obj_cgroup *objcg, · 2 unmodified lines
33303407 goto out;
33313408 }
33323409
3333− if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
3334− drain_obj_stock(stock);
3410+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
3411+ struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
3412+
3413+ if (!cached) {
3414+ if (empty_slot == -1)
3415+ empty_slot = i;
3416+ continue;
3417+ }
3418+ if (cached == objcg) {
3419+ slot = i;
3420+ break;
3421+ }
3422+ }
3423+
3424+ if (slot == -1) {
3425+ slot = empty_slot;
3426+ if (slot == -1) {
3427+ slot = stock->drain_idx++;
3428+ if (stock->drain_idx == NR_OBJ_STOCK)
3429+ stock->drain_idx = 0;
3430+ drain_obj_stock_slot(stock, slot);
3431+ }
33353432 obj_cgroup_get(objcg);
3336− stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
3433+ /*
3434+ * Keep the xchg result in the unsigned int local; storing
3435+ * it directly into stock->nr_bytes[slot] (uint16_t) would
3436+ * silently truncate values >= U16_MAX and bypass the flush
3437+ * guard below, leaking page-counter charges.
3438+ */
3439+ stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
33373440 ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
3338− WRITE_ONCE(stock->cached_objcg, objcg);
3441+ WRITE_ONCE(stock->cached[slot], objcg);
33393442
33403443 allow_uncharge = true; /* Allow uncharge when objcg changes */
3444+ } else {
3445+ stock_nr_bytes = stock->nr_bytes[slot];
33413446 }
3342− stock->nr_bytes += nr_bytes;
33433447
3344− if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
3345− nr_pages = stock->nr_bytes >> PAGE_SHIFT;
3346− stock->nr_bytes &= (PAGE_SIZE - 1);
3448+ stock_nr_bytes += nr_bytes;
3449+
3450+ if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
3451+ stock_nr_bytes > U16_MAX) {
3452+ nr_pages = stock_nr_bytes >> PAGE_SHIFT;
3453+ stock_nr_bytes &= (PAGE_SIZE - 1);
33473454 }
3455+ stock->nr_bytes[slot] = stock_nr_bytes;
33483456
33493457 out:
33503458 if (nr_pages)
@@ -4005,11 +4113,10 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 654 unmodified lines
40054113 if (!memcg1_alloc_events(memcg))
40064114 goto fail;
40074115
4116+ pstatc_pcpu = parent ? parent->vmstats_percpu : NULL;
40084117 for_each_possible_cpu(cpu) {
4009− if (parent)
4010− pstatc_pcpu = parent->vmstats_percpu;
40114118 statc = per_cpu_ptr(memcg->vmstats_percpu, cpu);
4012− statc->parent_pcpu = parent ? pstatc_pcpu : NULL;
4119+ statc->parent_pcpu = pstatc_pcpu;
40134120 statc->vmstats = memcg->vmstats;
40144121 }
40154122
@@ -4036,11 +4143,6 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 20 unmodified lines
40364143 for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)
40374144 memcg->cgwb_frn[i].done =
40384145 __WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq);
4039−#endif
4040−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
4041− spin_lock_init(&memcg->deferred_split_queue.split_queue_lock);
4042− INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue);
4043− memcg->deferred_split_queue.split_queue_len = 0;
40444146 #endif
40454147 lru_gen_init_memcg(memcg);
40464148 return memcg;
@@ -4192,11 +4294,10 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) · 145 unmodified lines
41924294 zswap_memcg_offline_cleanup(memcg);
41934295
41944296 memcg_offline_kmem(memcg);
4195− reparent_deferred_split_queue(memcg);
41964297 /*
4197− * The reparenting of objcg must be after the reparenting of the
4198− * list_lru and deferred_split_queue above, which ensures that they will
4199− * not mistakenly get the parent list_lru and deferred_split_queue.
4298+ * The reparenting of objcg must be after the reparenting of
4299+ * the list_lru in memcg_offline_kmem(), which ensures that
4300+ * they will not mistakenly get the parent list_lru.
42004301 */
42014302 memcg_reparent_objcgs(memcg);
42024303 reparent_shrinker_deferred(memcg);
@@ -5080,27 +5181,25 @@ out: · 877 unmodified lines
50805181
50815182 /**
50825183 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
5083− * @folio: folio to charge.
5184+ * @folio: the folio to charge
5185+ * @id: memory cgroup id
50845186 * @mm: mm context of the victim
50855187 * @gfp: reclaim mode
5086− * @entry: swap entry for which the folio is allocated
50875188 *
50885189 * This function charges a folio allocated for swapin. Please call this before
50895190 * adding the folio to the swapcache.
50905191 *
50915192 * Returns 0 on success. Otherwise, an error code is returned.
50925193 */
5093−int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
5094− gfp_t gfp, swp_entry_t entry)
5194+int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
5195+ struct mm_struct *mm, gfp_t gfp)
50955196 {
50965197 struct mem_cgroup *memcg;
5097− unsigned short id;
50985198 int ret;
50995199
51005200 if (mem_cgroup_disabled())
51015201 return 0;
51025202
5103− id = lookup_swap_cgroup_id(entry);
51045203 rcu_read_lock();
51055204 memcg = mem_cgroup_from_private_id(id);
51065205 if (!memcg || !css_tryget_online(&memcg->css))
@@ -5474,15 +5573,15 @@ int __init mem_cgroup_init(void) · 367 unmodified lines
54745573 /**
54755574 * __mem_cgroup_try_charge_swap - try charging swap space for a folio
54765575 * @folio: folio being added to swap
5477− * @entry: swap entry to charge
54785576 *
5479− * Try to charge @folio's memcg for the swap space at @entry.
5577+ * Try to charge @folio's memcg for the swap space at folio->swap.
54805578 *
54815579 * Returns 0 on success, -ENOMEM on failure.
54825580 */
5483−int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
5581+int __mem_cgroup_try_charge_swap(struct folio *folio)
54845582 {
54855583 unsigned int nr_pages = folio_nr_pages(folio);
5584+ struct swap_cluster_info *ci;
54865585 struct page_counter *counter;
54875586 struct mem_cgroup *memcg;
54885587 struct obj_cgroup *objcg;
@@ -5497,7 +5596,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) · 8 unmodified lines
54975596
54985597 rcu_read_lock();
54995598 memcg = obj_cgroup_memcg(objcg);
5500− if (!entry.val) {
5599+ if (!folio_test_swapcache(folio)) {
55015600 memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
55025601 rcu_read_unlock();
55035602 return 0;
@@ -5516,22 +5615,23 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) · 12 unmodified lines
55165615 }
55175616 mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
55185617
5519− swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry);
5618+ ci = swap_cluster_get_and_lock(folio);
5619+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
5620+ mem_cgroup_private_id(memcg));
5621+ swap_cluster_unlock(ci);
55205622
55215623 return 0;
55225624 }
55235625
55245626 /**
55255627 * __mem_cgroup_uncharge_swap - uncharge swap space
5526− * @entry: swap entry to uncharge
5628+ * @id: cgroup id to uncharge
55275629 * @nr_pages: the amount of swap space to uncharge
55285630 */
5529−void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
5631+void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
55305632 {
55315633 struct mem_cgroup *memcg;
5532− unsigned short id;
55335634
5534− id = swap_cgroup_clear(entry, nr_pages);
55355635 rcu_read_lock();
55365636 memcg = mem_cgroup_from_private_id(id);
55375637 if (memcg) {
mm/memory-failure.c+18−29
@@ -172,23 +172,11 @@ static int __page_handle_poison(struct page *page) · 171 unmodified lines
172172 {
173173 int ret;
174174
175− /*
176− * zone_pcp_disable() can't be used here. It will
177− * hold pcp_batch_high_lock and dissolve_free_hugetlb_folio() might hold
178− * cpu_hotplug_lock via static_key_slow_dec() when hugetlb vmemmap
179− * optimization is enabled. This will break current lock dependency
180− * chain and leads to deadlock.
181− * Disabling pcp before dissolving the page was a deterministic
182− * approach because we made sure that those pages cannot end up in any
183− * PCP list. Draining PCP lists expels those pages to the buddy system,
184− * but nothing guarantees that those pages do not get back to a PCP
185− * queue if we need to refill those.
186− */
175+ zone_pcp_disable(page_zone(page));
187176 ret = dissolve_free_hugetlb_folio(page_folio(page));
188− if (!ret) {
189− drain_all_pages(page_zone(page));
177+ if (!ret)
190178 ret = take_page_off_buddy(page);
191− }
179+ zone_pcp_enable(page_zone(page));
192180
193181 return ret;
194182 }
@@ -459,7 +447,7 @@ void add_to_kill_ksm(struct task_struct *tsk, const struct page *p, · 264 unmodified lines
459447 * Only do anything when FORCEKILL is set, otherwise just free the
460448 * list (this is used for clean pages which do not need killing)
461449 */
462−static void kill_procs(struct list_head *to_kill, int forcekill,
450+static void kill_procs(struct list_head *to_kill, bool forcekill,
463451 unsigned long pfn, int flags)
464452 {
465453 struct to_kill *tk, *next;
@@ -1418,7 +1406,7 @@ try_again: · 952 unmodified lines
14181406 * We raced with (possibly temporary) unhandlable
14191407 * page, retry.
14201408 */
1421− if (pass++ < 3) {
1409+ if (pass++ < GET_PAGE_MAX_RETRY_NUM) {
14221410 shake_page(p);
14231411 goto try_again;
14241412 }
@@ -1582,7 +1570,7 @@ static bool hwpoison_user_mappings(struct folio *folio, struct page *p, · 157 unmodified lines
15821570 {
15831571 LIST_HEAD(tokill);
15841572 bool unmap_success;
1585− int forcekill;
1573+ bool forcekill;
15861574 bool mlocked = folio_test_mlocked(folio);
15871575
15881576 /*
@@ -1703,7 +1691,7 @@ static void unmap_and_kill(struct list_head *to_kill, unsigned long pfn, · 114 unmodified lines
17031691 unmap_mapping_range(mapping, start, size, 0);
17041692 }
17051693
1706− kill_procs(to_kill, flags & MF_MUST_KILL, pfn, flags);
1694+ kill_procs(to_kill, !!(flags & MF_MUST_KILL), pfn, flags);
17071695 }
17081696
17091697 /*
@@ -2027,13 +2015,14 @@ out_unlock: · 317 unmodified lines
20272015 * So some of prechecks for hwpoison (pinning, and testing/setting
20282016 * PageHWPoison) should be done in single hugetlb_lock range.
20292017 * Returns:
2030− * 0 - not hugetlb, or recovered
2018+ * 0 - recovered
2019+ * -ENOENT - no hugetlb page
20312020 * -EBUSY - not recovered
20322021 * -EOPNOTSUPP - hwpoison_filter'ed
20332022 * -EHWPOISON - folio or exact page already poisoned
20342023 * -EFAULT - kill_accessing_process finds current->mm null
20352024 */
2036−static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb)
2025+static int try_memory_failure_hugetlb(unsigned long pfn, int flags)
20372026 {
20382027 int res, rv;
20392028 struct page *p = pfn_to_page(pfn);
@@ -2041,13 +2030,11 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb · 1 unmodified line
20412030 unsigned long page_flags;
20422031 bool migratable_cleared = false;
20432032
2044− *hugetlb = 1;
20452033 retry:
20462034 res = get_huge_page_for_hwpoison(pfn, flags, &migratable_cleared);
20472035 switch (res) {
20482036 case MF_HUGETLB_NON_HUGEPAGE: /* fallback to normal page handling */
2049− *hugetlb = 0;
2050− return 0;
2037+ return -ENOENT;
20512038 case MF_HUGETLB_RETRY:
20522039 if (!(flags & MF_NO_RETRY)) {
20532040 flags |= MF_NO_RETRY;
@@ -2108,9 +2095,9 @@ retry: · 54 unmodified lines
21082095 }
21092096
21102097 #else
2111−static inline int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb)
2098+static inline int try_memory_failure_hugetlb(unsigned long pfn, int flags)
21122099 {
2113− return 0;
2100+ return -ENOENT;
21142101 }
21152102
21162103 static inline unsigned long folio_free_raw_hwp(struct folio *folio, bool flag)
@@ -2348,7 +2335,6 @@ int memory_failure(unsigned long pfn, int flags) · 231 unmodified lines
23482335 int res = 0;
23492336 unsigned long page_flags;
23502337 bool retry = true;
2351− int hugetlb = 0;
23522338
23532339 if (!sysctl_memory_failure_recovery)
23542340 panic("Memory failure on page %lx", pfn);
@@ -2387,8 +2373,11 @@ int memory_failure(unsigned long pfn, int flags) · 32 unmodified lines
23872373 }
23882374
23892375 try_again:
2390− res = try_memory_failure_hugetlb(pfn, flags, &hugetlb);
2391− if (hugetlb)
2376+ res = try_memory_failure_hugetlb(pfn, flags);
2377+ /*
2378+ * -ENOENT means the page we found is not hugetlb, so proceed with normal page handling
2379+ */
2380+ if (res != -ENOENT)
23922381 goto unlock_mutex;
23932382
23942383 if (TestSetPageHWPoison(p)) {
mm/memory.c+79−127
@@ -3837,8 +3837,8 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf) · 3836 unmodified lines
38373837 * Handle the case of a page which we actually need to copy to a new page,
38383838 * either due to COW or unsharing.
38393839 *
3840− * Called with mmap_lock locked and the old page referenced, but
3841− * without the ptl held.
3840+ * Called with either the VMA lock or the mmap_lock held (see FAULT_FLAG_VMA_LOCK)
3841+ * and the old page referenced, but without the ptl held.
38423842 *
38433843 * High level logic flow:
38443844 *
@@ -4237,9 +4237,9 @@ static bool wp_can_reuse_anon_folio(struct folio *folio, · 392 unmodified lines
42374237 * though the page will change only once the write actually happens. This
42384238 * avoids a few races, and potentially makes it more efficient.
42394239 *
4240− * We enter with non-exclusive mmap_lock (to exclude vma changes,
4241− * but allow concurrent faults), with pte both mapped and locked.
4242− * We return with mmap_lock still held, but pte unmapped and unlocked.
4240+ * We enter with either the VMA lock or the mmap_lock held (see
4241+ * FAULT_FLAG_VMA_LOCK) and pte both mapped and locked. We return with
4242+ * the same lock still held, but pte unmapped and unlocked.
42434243 */
42444244 static vm_fault_t do_wp_page(struct vm_fault *vmf)
42454245 __releases(vmf->ptl)
@@ -4609,35 +4609,13 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · 363 unmodified lines
46094609 return VM_FAULT_SIGBUS;
46104610 }
46114611
4612−static struct folio *__alloc_swap_folio(struct vm_fault *vmf)
4613−{
4614− struct vm_area_struct *vma = vmf->vma;
4615− struct folio *folio;
4616− softleaf_t entry;
4617−
4618− folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address);
4619− if (!folio)
4620− return NULL;
4621−
4622− entry = softleaf_from_pte(vmf->orig_pte);
4623− if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
4624− GFP_KERNEL, entry)) {
4625− folio_put(folio);
4626− return NULL;
4627− }
4628−
4629− return folio;
4630−}
4631−
46324612 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
46334613 /*
4634− * Check if the PTEs within a range are contiguous swap entries
4635− * and have consistent swapcache, zeromap.
4614+ * Check if the PTEs within a range are contiguous swap entries.
46364615 */
46374616 static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
46384617 {
46394618 unsigned long addr;
4640− softleaf_t entry;
46414619 int idx;
46424620 pte_t pte;
46434621
@@ -4647,20 +4625,13 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages) · 3 unmodified lines
46474625
46484626 if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
46494627 return false;
4650− entry = softleaf_from_pte(pte);
4651− if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
4652− return false;
4653−
46544628 /*
46554629 * swap_read_folio() can't handle the case a large folio is hybridly
46564630 * from different backends. And they are likely corner cases. Similar
46574631 * things might be added once zswap support large folios.
46584632 */
4659− if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
4660− return false;
4661− if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
4633+ if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
46624634 return false;
4663−
46644635 return true;
46654636 }
46664637
@@ -4687,16 +4658,14 @@ static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset, · 20 unmodified lines
46874658 return orders;
46884659 }
46894660
4690−static struct folio *alloc_swap_folio(struct vm_fault *vmf)
4661+static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
46914662 {
46924663 struct vm_area_struct *vma = vmf->vma;
46934664 unsigned long orders;
4694− struct folio *folio;
46954665 unsigned long addr;
46964666 softleaf_t entry;
46974667 spinlock_t *ptl;
46984668 pte_t *pte;
4699− gfp_t gfp;
47004669 int order;
47014670
47024671 /*
@@ -4704,7 +4673,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 1 unmodified line
47044673 * maintain the uffd semantics.
47054674 */
47064675 if (unlikely(userfaultfd_armed(vma)))
4707− goto fallback;
4676+ return 0;
47084677
47094678 /*
47104679 * A large swapped out folio could be partially or fully in zswap. We
@@ -4712,7 +4681,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 1 unmodified line
47124681 * folio.
47134682 */
47144683 if (!zswap_never_enabled())
4715− goto fallback;
4684+ return 0;
47164685
47174686 entry = softleaf_from_pte(vmf->orig_pte);
47184687 /*
@@ -4726,12 +4695,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 7 unmodified lines
47264695 vmf->address, orders);
47274696
47284697 if (!orders)
4729− goto fallback;
4698+ return 0;
47304699
47314700 pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
47324701 vmf->address & PMD_MASK, &ptl);
47334702 if (unlikely(!pte))
4734− goto fallback;
4703+ return 0;
47354704
47364705 /*
47374706 * For do_swap_page, find the highest order where the aligned range is
@@ -4747,29 +4716,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 9 unmodified lines
47474716
47484717 pte_unmap_unlock(pte, ptl);
47494718
4750− /* Try allocating the highest of the remaining orders. */
4751− gfp = vma_thp_gfp_mask(vma);
4752− while (orders) {
4753− addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
4754− folio = vma_alloc_folio(gfp, order, vma, addr);
4755− if (folio) {
4756− if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
4757− gfp, entry))
4758− return folio;
4759− count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
4760− folio_put(folio);
4761− }
4762− count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
4763− order = next_order(&orders, order);
4764− }
4765−
4766−fallback:
4767− return __alloc_swap_folio(vmf);
4719+ return orders;
47684720 }
47694721 #else /* !CONFIG_TRANSPARENT_HUGEPAGE */
4770−static struct folio *alloc_swap_folio(struct vm_fault *vmf)
4722+static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
47714723 {
4772− return __alloc_swap_folio(vmf);
4724+ return 0;
47734725 }
47744726 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
47754727
@@ -4785,12 +4737,12 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · 9 unmodified lines
47854737 }
47864738
47874739 /*
4788− * We enter with non-exclusive mmap_lock (to exclude vma changes,
4789− * but allow concurrent faults), and pte mapped but not yet locked.
4740+ * We enter with either the VMA lock or the mmap_lock held (see
4741+ * FAULT_FLAG_VMA_LOCK), and pte mapped but not yet locked.
47904742 * We return with pte unmapped and unlocked.
47914743 *
4792− * We return with the mmap_lock locked or unlocked in the same cases
4793− * as does filemap_fault().
4744+ * When returning, the lock may have been released in the same cases
4745+ * as done by filemap_fault().
47944746 */
47954747 vm_fault_t do_swap_page(struct vm_fault *vmf)
47964748 {
@@ -4875,23 +4827,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 78 unmodified lines
48754827 if (folio)
48764828 swap_update_readahead(folio, vma, vmf->address);
48774829 if (!folio) {
4878− if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
4879− folio = alloc_swap_folio(vmf);
4880− if (folio) {
4881− /*
4882− * folio is charged, so swapin can only fail due
4883− * to raced swapin and return NULL.
4884− */
4885− swapcache = swapin_folio(entry, folio);
4886− if (swapcache != folio)
4887− folio_put(folio);
4888− folio = swapcache;
4889− }
4890− } else {
4830+ /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
4831+ if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
4832+ folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
4833+ thp_swapin_suitable_orders(vmf) | BIT(0),
4834+ vmf, NULL, 0);
4835+ else
48914836 folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
4892− }
48934837
4894− if (!folio) {
4838+ if (IS_ERR_OR_NULL(folio)) {
48954839 /*
48964840 * Back out if somebody else faulted in this pte
48974841 * while we released the pte lock.
@@ -4901,6 +4845,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 3 unmodified lines
49014845 if (likely(vmf->pte &&
49024846 pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
49034847 ret = VM_FAULT_OOM;
4848+ folio = NULL;
49044849 goto unlock;
49054850 }
49064851
@@ -5270,24 +5215,28 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) · 363 unmodified lines
52705215 while (orders) {
52715216 addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
52725217 folio = vma_alloc_folio(gfp, order, vma, addr);
5273− if (folio) {
5274− if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
5275− count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
5276− folio_put(folio);
5277− goto next;
5278− }
5279− folio_throttle_swaprate(folio, gfp);
5280− /*
5281− * When a folio is not zeroed during allocation
5282− * (__GFP_ZERO not used) or user folios require special
5283− * handling, folio_zero_user() is used to make sure
5284− * that the page corresponding to the faulting address
5285− * will be hot in the cache after zeroing.
5286− */
5287− if (user_alloc_needs_zeroing())
5288− folio_zero_user(folio, vmf->address);
5289− return folio;
5218+ if (!folio)
5219+ goto next;
5220+ if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
5221+ count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
5222+ folio_put(folio);
5223+ goto next;
52905224 }
5225+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
5226+ folio_put(folio);
5227+ goto fallback;
5228+ }
5229+ folio_throttle_swaprate(folio, gfp);
5230+ /*
5231+ * When a folio is not zeroed during allocation
5232+ * (__GFP_ZERO not used) or user folios require special
5233+ * handling, folio_zero_user() is used to make sure
5234+ * that the page corresponding to the faulting address
5235+ * will be hot in the cache after zeroing.
5236+ */
5237+ if (user_alloc_needs_zeroing())
5238+ folio_zero_user(folio, vmf->address);
5239+ return folio;
52915240 next:
52925241 count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
52935242 order = next_order(&orders, order);
@@ -5330,9 +5279,10 @@ static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte, · 36 unmodified lines
53305279 }
53315280
53325281 /*
5333− * We enter with non-exclusive mmap_lock (to exclude vma changes,
5334− * but allow concurrent faults), and pte mapped but not yet locked.
5335− * We return with mmap_lock still held, but pte unmapped and unlocked.
5282+ * We enter with either the VMA lock or the mmap_lock held (see
5283+ * FAULT_FLAG_VMA_LOCK), and pte unmapped and unlocked.
5284+ * We return with the lock still held, but pte unmapped and unlocked.
5285+ * If VM_FAULT_RETRY is returned, the lock may have been released.
53365286 */
53375287 static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
53385288 {
@@ -5440,9 +5390,10 @@ oom: · 101 unmodified lines
54405390 }
54415391
54425392 /*
5443− * The mmap_lock must have been held on entry, and may have been
5444− * released depending on flags and vma->vm_ops->fault() return value.
5445− * See filemap_fault() and __lock_page_retry().
5393+ * Either the VMA lock or the mmap_lock must have been held on entry
5394+ * (see FAULT_FLAG_VMA_LOCK) and may have been released depending on
5395+ * flags and vma->vm_ops->fault() return value.
5396+ * See filemap_fault() and __folio_lock_or_retry().
54465397 */
54475398 static vm_fault_t __do_fault(struct vm_fault *vmf)
54485399 {
@@ -5451,18 +5402,18 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) · 2 unmodified lines
54515402 vm_fault_t ret;
54525403
54535404 /*
5454− * Preallocate pte before we take page_lock because this might lead to
5455− * deadlocks for memcg reclaim which waits for pages under writeback:
5456− * lock_page(A)
5457− * SetPageWriteback(A)
5458− * unlock_page(A)
5459− * lock_page(B)
5460− * lock_page(B)
5405+ * Preallocate pte before we take folio lock because this might lead to
5406+ * deadlocks for memcg reclaim which waits for folios under writeback:
5407+ * folio_lock(A)
5408+ * folio_set_writeback(A)
5409+ * folio_unlock(A)
5410+ * folio_lock(B)
5411+ * folio_lock(B)
54615412 * pte_alloc_one
54625413 * shrink_folio_list
5463− * wait_on_page_writeback(A)
5464− * SetPageWriteback(B)
5465− * unlock_page(B)
5414+ * folio_wait_writeback(A)
5415+ * folio_set_writeback(B)
5416+ * folio_unlock(B)
54665417 * # flush A, B to clear the writeback
54675418 */
54685419 if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) {
@@ -5480,7 +5431,7 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) · 11 unmodified lines
54805431 if (unlikely(PageHWPoison(vmf->page))) {
54815432 vm_fault_t poisonret = VM_FAULT_HWPOISON;
54825433 if (ret & VM_FAULT_LOCKED) {
5483− if (page_mapped(vmf->page))
5434+ if (folio_mapped(folio))
54845435 unmap_mapping_folio(folio);
54855436 /* Retry if a clean folio was removed from the cache. */
54865437 if (mapping_evict_folio(folio->mapping, folio))
@@ -6003,11 +5954,11 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) · 516 unmodified lines
60035954 }
60045955
60055956 /*
6006− * We enter with non-exclusive mmap_lock (to exclude vma changes,
6007− * but allow concurrent faults).
6008− * The mmap_lock may have been released depending on flags and our
5957+ * We enter with either the VMA lock or the mmap_lock held (see
5958+ * FAULT_FLAG_VMA_LOCK).
5959+ * The lock may have been released depending on flags and our
60095960 * return value. See filemap_fault() and __folio_lock_or_retry().
6010− * If mmap_lock is released, vma may become invalid (for example
5961+ * If the lock is released, vma may become invalid (for example
60115962 * by other thread calling munmap()).
60125963 */
60135964 static vm_fault_t do_fault(struct vm_fault *vmf)
@@ -6374,10 +6325,11 @@ static void fix_spurious_fault(struct vm_fault *vmf, · 360 unmodified lines
63746325 * with external mmu caches can use to update those (ie the Sparc or
63756326 * PowerPC hashed page tables that act as extended TLBs).
63766327 *
6377− * We enter with non-exclusive mmap_lock (to exclude vma changes, but allow
6378− * concurrent faults).
6328+ * On entry, we hold either the VMA lock or the mmap_lock
6329+ * (see FAULT_FLAG_VMA_LOCK).
63796330 *
6380− * The mmap_lock may have been released depending on flags and our return value.
6331+ * The mmap_lock or VMA lock may have been released depending on flags
6332+ * and our return value.
63816333 * See filemap_fault() and __folio_lock_or_retry().
63826334 */
63836335 static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
@@ -6458,8 +6410,8 @@ unlock: · 74 unmodified lines
64586410
64596411 /*
64606412 * On entry, we hold either the VMA lock or the mmap_lock
6461− * (FAULT_FLAG_VMA_LOCK tells you which). If VM_FAULT_RETRY is set in
6462− * the result, the mmap_lock is not held on exit. See filemap_fault()
6413+ * (see FAULT_FLAG_VMA_LOCK). If VM_FAULT_RETRY is set in
6414+ * the result, the lock is not held on exit. See filemap_fault()
64636415 * and __folio_lock_or_retry().
64646416 */
64656417 static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
@@ -6691,9 +6643,9 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, · 225 unmodified lines
66916643
66926644 /*
66936645 * By the time we get here, we already hold either the VMA lock or the
6694− * mmap_lock (FAULT_FLAG_VMA_LOCK tells you which).
6646+ * mmap_lock (see FAULT_FLAG_VMA_LOCK).
66956647 *
6696− * The mmap_lock may have been released depending on flags and our
6648+ * The lock may have been released depending on flags and our
66976649 * return value. See filemap_fault() and __folio_lock_or_retry().
66986650 */
66996651 vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
mm/memory_hotplug.c+18−16
@@ -576,6 +576,7 @@ void remove_pfn_range_from_zone(struct zone *zone, · 575 unmodified lines
576576 * @pfn: starting pageframe (must be aligned to start of a section)
577577 * @nr_pages: number of pages to remove (must be multiple of section size)
578578 * @altmap: alternative device page map or %NULL if default memmap is used
579+ * @pgmap: device page map or %NULL if not ZONE_DEVICE
579580 *
580581 * Generic helper function to remove section mappings and sysfs entries
581582 * for the section of the memory we are removing. Caller needs to make
@@ -583,7 +584,7 @@ void remove_pfn_range_from_zone(struct zone *zone, · 1 unmodified line
583584 * calling offline_pages().
584585 */
585586 void __remove_pages(unsigned long pfn, unsigned long nr_pages,
586− struct vmem_altmap *altmap)
587+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
587588 {
588589 const unsigned long end_pfn = pfn + nr_pages;
589590 unsigned long cur_nr_pages;
@@ -598,7 +599,7 @@ void __remove_pages(unsigned long pfn, unsigned long nr_pages, · 8 unmodified lines
598599 /* Select all remaining pages up to the next section boundary */
599600 cur_nr_pages = min(end_pfn - pfn,
600601 SECTION_ALIGN_UP(pfn + 1) - pfn);
601− sparse_remove_section(pfn, cur_nr_pages, altmap);
602+ sparse_remove_section(pfn, cur_nr_pages, altmap, pgmap);
602603 }
603604 }
604605
@@ -1402,6 +1403,12 @@ bool mhp_supports_memmap_on_memory(void) · 797 unmodified lines
14021403 }
14031404 EXPORT_SYMBOL_GPL(mhp_supports_memmap_on_memory);
14041405
1406+static void altmap_free(struct vmem_altmap *altmap)
1407+{
1408+ WARN_ONCE(altmap->alloc, "Altmap not fully unmapped");
1409+ kfree(altmap);
1410+}
1411+
14051412 static void remove_memory_blocks_and_altmaps(u64 start, u64 size)
14061413 {
14071414 unsigned long memblock_size = memory_block_size_bytes();
@@ -1416,22 +1423,17 @@ static void remove_memory_blocks_and_altmaps(u64 start, u64 size) · 8 unmodified lines
14161423 struct vmem_altmap *altmap = NULL;
14171424 struct memory_block *mem;
14181425
1419− mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(cur_start)));
1426+ mem = memory_block_get(phys_to_block_id(cur_start));
14201427 if (WARN_ON_ONCE(!mem))
14211428 continue;
14221429
14231430 altmap = mem->altmap;
14241431 mem->altmap = NULL;
1425− /* drop the ref. we got via find_memory_block() */
1426− put_device(&mem->dev);
1432+ memory_block_put(mem);
14271433
14281434 remove_memory_block_devices(cur_start, memblock_size);
1429−
1430− arch_remove_memory(cur_start, memblock_size, altmap);
1431−
1432− /* Verify that all vmemmap pages have actually been freed. */
1433− WARN(altmap->alloc, "Altmap not fully unmapped");
1434− kfree(altmap);
1435+ arch_remove_memory(cur_start, memblock_size, altmap, NULL);
1436+ altmap_free(altmap);
14351437 }
14361438 }
14371439
@@ -1462,7 +1464,7 @@ static int create_altmaps_and_memory_blocks(int nid, struct memory_group *group, · 24 unmodified lines
14621464 /* call arch's memory hotadd */
14631465 ret = arch_add_memory(nid, cur_start, memblock_size, &params);
14641466 if (ret < 0) {
1465− kfree(params.altmap);
1467+ altmap_free(params.altmap);
14661468 goto out;
14671469 }
14681470
@@ -1470,8 +1472,8 @@ static int create_altmaps_and_memory_blocks(int nid, struct memory_group *group, · 1 unmodified line
14701472 ret = create_memory_block_devices(cur_start, memblock_size, nid,
14711473 params.altmap, group);
14721474 if (ret) {
1473− arch_remove_memory(cur_start, memblock_size, NULL);
1474− kfree(params.altmap);
1475+ arch_remove_memory(cur_start, memblock_size, params.altmap, NULL);
1476+ altmap_free(params.altmap);
14751477 goto out;
14761478 }
14771479 }
@@ -1556,7 +1558,7 @@ int add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags) · 78 unmodified lines
15561558 /* create memory block devices after memory was added */
15571559 ret = create_memory_block_devices(start, size, nid, NULL, group);
15581560 if (ret) {
1559− arch_remove_memory(start, size, params.altmap);
1561+ arch_remove_memory(start, size, params.altmap, NULL);
15601562 goto error;
15611563 }
15621564 }
@@ -2268,7 +2270,7 @@ static int try_remove_memory(u64 start, u64 size) · 705 unmodified lines
22682270 * No altmaps present, do the removal directly
22692271 */
22702272 remove_memory_block_devices(start, size);
2271− arch_remove_memory(start, size, NULL);
2273+ arch_remove_memory(start, size, NULL, NULL);
22722274 } else {
22732275 /* all memblocks in the range have altmaps */
22742276 remove_memory_blocks_and_altmaps(start, size);
mm/mempolicy.c+1−1
@@ -2865,7 +2865,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 2864 unmodified lines
28652865 case MPOL_PREFERRED:
28662866 case MPOL_PREFERRED_MANY:
28672867 case MPOL_WEIGHTED_INTERLEAVE:
2868− return !!nodes_equal(a->nodes, b->nodes);
2868+ return nodes_equal(a->nodes, b->nodes);
28692869 case MPOL_LOCAL:
28702870 return true;
28712871 default:
mm/memremap.c+2−2
@@ -97,10 +97,10 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) · 96 unmodified lines
9797 PHYS_PFN(range_len(range)));
9898 if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
9999 __remove_pages(PHYS_PFN(range->start),
100− PHYS_PFN(range_len(range)), NULL);
100+ PHYS_PFN(range_len(range)), NULL, pgmap);
101101 } else {
102102 arch_remove_memory(range->start, range_len(range),
103− pgmap_altmap(pgmap));
103+ pgmap_altmap(pgmap), pgmap);
104104 kasan_remove_zero_shadow(__va(range->start), range_len(range));
105105 }
106106 mem_hotplug_done();
mm/migrate.c+33−30
@@ -1135,26 +1135,24 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 1134 unmodified lines
11351135 * This is safe because nobody is using it except us.
11361136 */
11371137 enum {
1138− PAGE_WAS_MAPPED = BIT(0),
1139− PAGE_WAS_MLOCKED = BIT(1),
1140− PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED,
1138+ FOLIO_WAS_MAPPED = BIT(0),
1139+ FOLIO_WAS_MLOCKED = BIT(1),
1140+ FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
11411141 };
11421142
11431143 static void __migrate_folio_record(struct folio *dst,
1144− int old_page_state,
1145− struct anon_vma *anon_vma)
1144+ int old_folio_state, struct anon_vma *anon_vma)
11461145 {
1147− dst->private = (void *)anon_vma + old_page_state;
1146+ dst->private = (void *)anon_vma + old_folio_state;
11481147 }
11491148
11501149 static void __migrate_folio_extract(struct folio *dst,
1151− int *old_page_state,
1152− struct anon_vma **anon_vmap)
1150+ int *old_folio_state, struct anon_vma **anon_vmap)
11531151 {
11541152 unsigned long private = (unsigned long)dst->private;
11551153
1156− *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES);
1157− *old_page_state = private & PAGE_OLD_STATES;
1154+ *anon_vmap = (struct anon_vma *)(private & ~FOLIO_OLD_STATES);
1155+ *old_folio_state = private & FOLIO_OLD_STATES;
11581156 dst->private = NULL;
11591157 }
11601158
@@ -1209,7 +1207,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 48 unmodified lines
12091207 {
12101208 struct folio *dst;
12111209 int rc = -EAGAIN;
1212− int old_page_state = 0;
1210+ int old_folio_state = 0;
12131211 struct anon_vma *anon_vma = NULL;
12141212 bool locked = false;
12151213 bool dst_locked = false;
@@ -1253,12 +1251,12 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 37 unmodified lines
12531251 }
12541252 locked = true;
12551253 if (folio_test_mlocked(src))
1256− old_page_state |= PAGE_WAS_MLOCKED;
1254+ old_folio_state |= FOLIO_WAS_MLOCKED;
12571255
12581256 if (folio_test_writeback(src)) {
12591257 /*
12601258 * Only in the case of a full synchronous migration is it
1261− * necessary to wait for PageWriteback. In the async case,
1259+ * necessary to wait for writeback. In the async case,
12621260 * the retry loop is too short and in the sync-light case,
12631261 * the overhead of stalling is too much
12641262 */
@@ -1302,7 +1300,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 37 unmodified lines
13021300 dst_locked = true;
13031301
13041302 if (unlikely(page_has_movable_ops(&src->page))) {
1305− __migrate_folio_record(dst, old_page_state, anon_vma);
1303+ __migrate_folio_record(dst, old_folio_state, anon_vma);
13061304 return 0;
13071305 }
13081306
@@ -1328,11 +1326,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 19 unmodified lines
13281326 VM_BUG_ON_FOLIO(folio_test_anon(src) &&
13291327 !folio_test_ksm(src) && !anon_vma, src);
13301328 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
1331− old_page_state |= PAGE_WAS_MAPPED;
1329+ old_folio_state |= FOLIO_WAS_MAPPED;
13321330 }
13331331
13341332 if (!folio_mapped(src)) {
1335− __migrate_folio_record(dst, old_page_state, anon_vma);
1333+ __migrate_folio_record(dst, old_folio_state, anon_vma);
13361334 return 0;
13371335 }
13381336
@@ -1344,7 +1342,7 @@ out: · 5 unmodified lines
13441342 if (rc == -EAGAIN)
13451343 ret = NULL;
13461344
1347− migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1345+ migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
13481346 anon_vma, locked, ret);
13491347 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
13501348
@@ -1358,13 +1356,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 7 unmodified lines
13581356 struct list_head *ret)
13591357 {
13601358 int rc;
1361− int old_page_state = 0;
1359+ int old_folio_state = 0;
13621360 struct anon_vma *anon_vma = NULL;
13631361 bool src_deferred_split = false;
13641362 bool src_partially_mapped = false;
13651363 struct list_head *prev;
13661364
1367− __migrate_folio_extract(dst, &old_page_state, &anon_vma);
1365+ __migrate_folio_extract(dst, &old_folio_state, &anon_vma);
13681366 prev = dst->lru.prev;
13691367 list_del(&dst->lru);
13701368
@@ -1404,10 +1402,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 33 unmodified lines
14041402 * isolated from the unevictable LRU: but this case is the easiest.
14051403 */
14061404 folio_add_lru(dst);
1407− if (old_page_state & PAGE_WAS_MLOCKED)
1405+ if (old_folio_state & FOLIO_WAS_MLOCKED)
14081406 lru_add_drain();
14091407
1410− if (old_page_state & PAGE_WAS_MAPPED)
1408+ if (old_folio_state & FOLIO_WAS_MAPPED)
14111409 remove_migration_ptes(src, dst, 0);
14121410
14131411 out_unlock_both:
@@ -1439,11 +1437,11 @@ out: · 25 unmodified lines
14391437 */
14401438 if (rc == -EAGAIN) {
14411439 list_add(&dst->lru, prev);
1442− __migrate_folio_record(dst, old_page_state, anon_vma);
1440+ __migrate_folio_record(dst, old_folio_state, anon_vma);
14431441 return rc;
14441442 }
14451443
1446− migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1444+ migrate_folio_undo_src(src, old_folio_state & FOLIO_WAS_MAPPED,
14471445 anon_vma, true, ret);
14481446 migrate_folio_undo_dst(dst, true, put_new_folio, private);
14491447
@@ -1777,11 +1775,11 @@ static void migrate_folios_undo(struct list_head *src_folios, · 327 unmodified lines
17771775 dst = list_first_entry(dst_folios, struct folio, lru);
17781776 dst2 = list_next_entry(dst, lru);
17791777 list_for_each_entry_safe(folio, folio2, src_folios, lru) {
1780− int old_page_state = 0;
1778+ int old_folio_state = 0;
17811779 struct anon_vma *anon_vma = NULL;
17821780
1783− __migrate_folio_extract(dst, &old_page_state, &anon_vma);
1784− migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
1781+ __migrate_folio_extract(dst, &old_folio_state, &anon_vma);
1782+ migrate_folio_undo_src(folio, old_folio_state & FOLIO_WAS_MAPPED,
17851783 anon_vma, true, ret_folios);
17861784 list_del(&dst->lru);
17871785 migrate_folio_undo_dst(dst, true, put_new_folio, private);
@@ -2557,24 +2555,29 @@ static struct mm_struct *find_mm_struct(pid_t pid, nodemask_t *mem_nodes) · 769 unmodified lines
25572555 }
25582556
25592557 task = find_get_task_by_vpid(pid);
2560− if (!task) {
2558+ if (!task)
25612559 return ERR_PTR(-ESRCH);
2562− }
25632560
2561+ if (down_read_killable(&task->signal->exec_update_lock)) {
2562+ mm = ERR_PTR(-EINTR);
2563+ goto out;
2564+ }
25642565 /*
25652566 * Check if this process has the right to modify the specified
25662567 * process. Use the regular "ptrace_may_access()" checks.
25672568 */
25682569 if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {
25692570 mm = ERR_PTR(-EPERM);
2570− goto out;
2571+ goto unlock;
25712572 }
25722573
25732574 mm = ERR_PTR(security_task_movememory(task));
25742575 if (IS_ERR(mm))
2575− goto out;
2576+ goto unlock;
25762577 *mem_nodes = cpuset_mems_allowed(task);
25772578 mm = get_task_mm(task);
2579+unlock:
2580+ up_read(&task->signal->exec_update_lock);
25782581 out:
25792582 put_task_struct(task);
25802583 if (!mm)
mm/migrate_device.c+3−6
@@ -801,8 +801,7 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, · 800 unmodified lines
801801 bool flush = false;
802802 unsigned long i;
803803
804− VM_WARN_ON_FOLIO(!folio, folio);
805− VM_WARN_ON_ONCE(!pmd_none(*pmdp) && !is_huge_zero_pmd(*pmdp));
804+ VM_WARN_ON_ONCE(!folio);
806805
807806 if (!thp_vma_suitable_order(vma, addr, HPAGE_PMD_ORDER))
808807 return -EINVAL;
@@ -859,11 +858,9 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, · 50 unmodified lines
859858 if (userfaultfd_missing(vma))
860859 goto unlock_abort;
861860
862− if (!pmd_none(*pmdp)) {
863− if (!is_huge_zero_pmd(*pmdp))
864− goto unlock_abort;
861+ if (is_huge_zero_pmd(*pmdp))
865862 flush = true;
866− } else if (!pmd_none(*pmdp))
863+ else if (!pmd_none(*pmdp))
867864 goto unlock_abort;
868865
869866 add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
mm/mm_init.c+36−37
@@ -674,6 +674,20 @@ static inline void fixup_hashdist(void) · 673 unmodified lines
674674 static inline void fixup_hashdist(void) {}
675675 #endif /* CONFIG_NUMA */
676676
677+#ifdef CONFIG_ZONE_DEVICE
678+static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
679+ unsigned long nr_pages, int migratetype)
680+{
681+ const unsigned long end = pfn + nr_pages;
682+
683+ for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) {
684+ init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false);
685+ if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
686+ cond_resched();
687+ }
688+}
689+#endif
690+
677691 /*
678692 * Initialize a reserved page unconditionally, finding its zone first.
679693 */
@@ -1011,21 +1025,6 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, · 331 unmodified lines
10111025 page_folio(page)->pgmap = pgmap;
10121026 page->zone_device_data = NULL;
10131027
1014− /*
1015− * Mark the block movable so that blocks are reserved for
1016− * movable at startup. This will force kernel allocations
1017− * to reserve their blocks rather than leaking throughout
1018− * the address space during boot when many long-lived
1019− * kernel allocations are made.
1020− *
1021− * Please note that MEMINIT_HOTPLUG path doesn't clear memmap
1022− * because this is done early in section_activate()
1023− */
1024− if (pageblock_aligned(pfn)) {
1025− init_pageblock_migratetype(page, MIGRATE_MOVABLE, false);
1026− cond_resched();
1027− }
1028−
10291028 /*
10301029 * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released
10311030 * directly to the driver page allocator which will set the page count
@@ -1056,10 +1055,17 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, · 24 unmodified lines
10561055 * of how the sparse_vmemmap internals handle compound pages in the lack
10571056 * of an altmap. See vmemmap_populate_compound_pages().
10581057 */
1059−static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,
1058+static inline unsigned long compound_nr_pages(unsigned long pfn,
1059+ struct vmem_altmap *altmap,
10601060 struct dev_pagemap *pgmap)
10611061 {
1062− if (!vmemmap_can_optimize(altmap, pgmap))
1062+ /*
1063+ * If DAX memory is hot-plugged into an unoccupied subsection
1064+ * of an early section, the unoptimized boot memmap is reused.
1065+ * See section_activate().
1066+ */
1067+ if (early_section(__pfn_to_section(pfn)) ||
1068+ !vmemmap_can_optimize(altmap, pgmap))
10631069 return pgmap_vmemmap_nr(pgmap);
10641070
10651071 return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page));
@@ -1122,13 +1128,18 @@ void __ref memmap_init_zone_device(struct zone *zone, · 56 unmodified lines
11221128
11231129 __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
11241130
1131+ if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
1132+ cond_resched();
1133+
11251134 if (pfns_per_compound == 1)
11261135 continue;
11271136
11281137 memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
1129− compound_nr_pages(altmap, pgmap));
1138+ compound_nr_pages(pfn, altmap, pgmap));
11301139 }
11311140
1141+ pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE);
1142+
11321143 pr_debug("%s initialised %lu pages in %ums\n", __func__,
11331144 nr_pages, jiffies_to_msecs(jiffies - start));
11341145 }
@@ -1362,19 +1373,6 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, · 227 unmodified lines
13621373 pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages);
13631374 }
13641375
1365−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1366−static void pgdat_init_split_queue(struct pglist_data *pgdat)
1367−{
1368− struct deferred_split *ds_queue = &pgdat->deferred_split_queue;
1369−
1370− spin_lock_init(&ds_queue->split_queue_lock);
1371− INIT_LIST_HEAD(&ds_queue->split_queue);
1372− ds_queue->split_queue_len = 0;
1373−}
1374−#else
1375−static void pgdat_init_split_queue(struct pglist_data *pgdat) {}
1376−#endif
1377−
13781376 #ifdef CONFIG_COMPACTION
13791377 static void pgdat_init_kcompactd(struct pglist_data *pgdat)
13801378 {
@@ -1390,8 +1388,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) · 9 unmodified lines
13901388
13911389 pgdat_resize_init(pgdat);
13921390 pgdat_kswapd_lock_init(pgdat);
1393−
1394− pgdat_init_split_queue(pgdat);
13951391 pgdat_init_kcompactd(pgdat);
13961392
13971393 init_waitqueue_head(&pgdat->kswapd_wait);
@@ -1418,11 +1414,14 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, · 20 unmodified lines
14181414
14191415 static void __meminit zone_init_free_lists(struct zone *zone)
14201416 {
1421− unsigned int order, t;
1422− for_each_migratetype_order(order, t) {
1423− INIT_LIST_HEAD(&zone->free_area[order].free_list[t]);
1417+ struct list_head *list;
1418+ unsigned int order;
1419+
1420+ for_each_free_list(list, zone, order)
1421+ INIT_LIST_HEAD(list);
1422+
1423+ for (order = 0; order < NR_PAGE_ORDERS; order++)
14241424 zone->free_area[order].nr_free = 0;
1425− }
14261425
14271426 #ifdef CONFIG_UNACCEPTED_MEMORY
14281427 INIT_LIST_HEAD(&zone->unaccepted_pages);
mm/mmap.c+1−1
@@ -504,7 +504,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, · 503 unmodified lines
504504 break;
505505 case MAP_DROPPABLE:
506506 if (VM_DROPPABLE == VM_NONE)
507− return -ENOTSUPP;
507+ return -EOPNOTSUPP;
508508 /*
509509 * A locked or stack area makes no sense to be droppable.
510510 *
mm/mseal.c+3−2
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 */
99
1010 #include <linux/mempolicy.h>
11+#include <linux/minmax.h>
1112 #include <linux/mman.h>
1213 #include <linux/mm.h>
1314 #include <linux/mm_inline.h>
@@ -65,8 +66,8 @@ static int mseal_apply(struct mm_struct *mm, · 51 unmodified lines
6566 prev = vma;
6667
6768 for_each_vma_range(vmi, vma, end) {
68− const unsigned long curr_start = MAX(vma->vm_start, start);
69− const unsigned long curr_end = MIN(vma->vm_end, end);
69+ const unsigned long curr_start = max(vma->vm_start, start);
70+ const unsigned long curr_end = min(vma->vm_end, end);
7071
7172 if (!vma_test(vma, VMA_SEALED_BIT)) {
7273 vma_flags_t vma_flags = vma->flags;
mm/page_alloc.c+277−106
@@ -90,6 +90,9 @@ typedef int __bitwise fpi_t; · 89 unmodified lines
9090 /* Free the page without taking locks. Rely on trylock only. */
9191 #define FPI_TRYLOCK ((__force fpi_t)BIT(2))
9292
93+/* free_pages_prepare() has already been called for page(s) being freed. */
94+#define FPI_PREPARED ((__force fpi_t)BIT(3))
95+
9396 /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */
9497 static DEFINE_MUTEX(pcp_batch_high_lock);
9598 #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)
@@ -282,6 +285,14 @@ EXPORT_SYMBOL(nr_node_ids); · 186 unmodified lines
282285 EXPORT_SYMBOL(nr_online_nodes);
283286 #endif
284287
288+/*
289+ * When page allocations stall for longer than a threshold,
290+ * ALLOC_STALL_WARN_MSECS, leave a warning in the kernel log. Only one warning
291+ * will be printed during this duration for the entire system.
292+ */
293+#define ALLOC_STALL_WARN_MSECS (10 * 1000UL)
294+static unsigned long alloc_stall_warn_jiffies = INITIAL_JIFFIES;
295+
285296 static bool page_contains_unaccepted(struct page *page, unsigned int order);
286297 static bool cond_accept_memory(struct zone *zone, unsigned int order,
287298 int alloc_flags);
@@ -353,7 +364,7 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn, · 65 unmodified lines
353364 #else
354365 BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4);
355366 #endif
356− BUILD_BUG_ON(__MIGRATE_TYPE_END > MIGRATETYPE_MASK);
367+ BUILD_BUG_ON(__MIGRATE_TYPE_END > PAGEBLOCK_MIGRATETYPE_MASK);
357368 VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page);
358369
359370 bitmap = get_pageblock_bitmap(page, pfn);
@@ -423,10 +434,10 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn, · 63 unmodified lines
423434 * Use get_pfnblock_migratetype() if caller already has both @page and @pfn
424435 * to save a call to page_to_pfn().
425436 */
426−__always_inline enum migratetype
437+enum migratetype
427438 get_pfnblock_migratetype(const struct page *page, unsigned long pfn)
428439 {
429− unsigned long mask = MIGRATETYPE_AND_ISO_MASK;
440+ unsigned long mask = PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK;
430441 unsigned long flags;
431442
432443 flags = __get_pfnblock_flags_mask(page, pfn, mask);
@@ -435,7 +446,7 @@ get_pfnblock_migratetype(const struct page *page, unsigned long pfn) · 2 unmodified lines
435446 if (flags & BIT(PB_migrate_isolate))
436447 return MIGRATE_ISOLATE;
437448 #endif
438− return flags & MIGRATETYPE_MASK;
449+ return flags & PAGEBLOCK_MIGRATETYPE_MASK;
439450 }
440451
441452 /**
@@ -523,11 +534,11 @@ static void set_pageblock_migratetype(struct page *page, · 81 unmodified lines
523534 }
524535 VM_WARN_ONCE(get_pageblock_isolate(page),
525536 "Use clear_pageblock_isolate() to unisolate pageblock");
526− /* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */
537+ /* PAGEBLOCK_ISO_MASK clears PB_migrate_isolate if it is set */
527538 #endif
528539 __set_pfnblock_flags_mask(page, page_to_pfn(page),
529540 (unsigned long)migratetype,
530− MIGRATETYPE_AND_ISO_MASK);
541+ PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK);
531542 }
532543
533544 void __meminit init_pageblock_migratetype(struct page *page,
@@ -553,7 +564,7 @@ void __meminit init_pageblock_migratetype(struct page *page, · 19 unmodified lines
553564 flags |= BIT(PB_migrate_isolate);
554565 #endif
555566 __set_pfnblock_flags_mask(page, page_to_pfn(page), flags,
556− MIGRATETYPE_AND_ISO_MASK);
567+ PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK);
557568 }
558569
559570 #ifdef CONFIG_DEBUG_VM
@@ -639,19 +650,12 @@ out: · 79 unmodified lines
639650
640651 static inline unsigned int order_to_pindex(int migratetype, int order)
641652 {
653+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
654+ bool movable = migratetype == MIGRATE_MOVABLE;
642655
643−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
644− bool movable;
645− if (order > PAGE_ALLOC_COSTLY_ORDER) {
646− VM_BUG_ON(!is_pmd_order(order));
647−
648− movable = migratetype == MIGRATE_MOVABLE;
649−
650− return NR_LOWORDER_PCP_LISTS + movable;
656+ if (order > PAGE_ALLOC_COSTLY_ORDER)
657+ return NR_LOWORDER_PCP_LISTS + movable;
651658 }
652−#else
653− VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
654−#endif
655659
656660 return (MIGRATE_PCPTYPES * order) + migratetype;
657661 }
@@ -660,12 +664,10 @@ static inline int pindex_to_order(unsigned int pindex) · 2 unmodified lines
660664 {
661665 int order = pindex / MIGRATE_PCPTYPES;
662666
663−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
664− if (pindex >= NR_LOWORDER_PCP_LISTS)
665− order = HPAGE_PMD_ORDER;
666−#else
667− VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
668−#endif
667+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
668+ if (pindex >= NR_LOWORDER_PCP_LISTS)
669+ order = HPAGE_PMD_ORDER;
670+ }
669671
670672 return order;
671673 }
@@ -1211,14 +1213,18 @@ static inline bool should_skip_kasan_poison(struct page *page) · 539 unmodified lines
12111213 return page_kasan_tag(page) == KASAN_TAG_KERNEL;
12121214 }
12131215
1214−static void kernel_init_pages(struct page *page, int numpages)
1216+static void clear_highpages_kasan_tagged(struct page *page, int numpages)
12151217 {
1216− int i;
1217−
12181218 /* s390's use of memset() could override KASAN redzones. */
12191219 kasan_disable_current();
1220− for (i = 0; i < numpages; i++)
1221− clear_highpage_kasan_tagged(page + i);
1220+ if (!IS_ENABLED(CONFIG_HIGHMEM)) {
1221+ clear_pages(kasan_reset_tag(page_address(page)), numpages);
1222+ } else {
1223+ int i;
1224+
1225+ for (i = 0; i < numpages; i++)
1226+ clear_highpage_kasan_tagged(page + i);
1227+ }
12221228 kasan_enable_current();
12231229 }
12241230
@@ -1303,8 +1309,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) · 78 unmodified lines
13031309
13041310 #endif /* CONFIG_MEM_ALLOC_PROFILING */
13051311
1306−__always_inline bool __free_pages_prepare(struct page *page,
1307− unsigned int order, fpi_t fpi_flags)
1312+static __always_inline bool __free_pages_prepare(struct page *page,
1313+ unsigned int order, fpi_t fpi_flags)
13081314 {
13091315 int bad = 0;
13101316 bool skip_kasan_poison = should_skip_kasan_poison(page);
@@ -1312,6 +1318,9 @@ __always_inline bool __free_pages_prepare(struct page *page, · 1 unmodified line
13121318 bool compound = PageCompound(page);
13131319 struct folio *folio = page_folio(page);
13141320
1321+ if (fpi_flags & FPI_PREPARED)
1322+ return true;
1323+
13151324 VM_BUG_ON_PAGE(PageTail(page), page);
13161325
13171326 trace_mm_page_free(page, order);
@@ -1423,7 +1432,7 @@ __always_inline bool __free_pages_prepare(struct page *page, · 105 unmodified lines
14231432 init = false;
14241433 }
14251434 if (init)
1426− kernel_init_pages(page, 1 << order);
1435+ clear_highpages_kasan_tagged(page, 1 << order);
14271436
14281437 /*
14291438 * arch_free_page() can make the page's contents inaccessible. s390
@@ -1451,7 +1460,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 21 unmodified lines
14511460 struct per_cpu_pages *pcp,
14521461 int pindex)
14531462 {
1454− unsigned long flags;
14551463 unsigned int order;
14561464 struct page *page;
14571465
@@ -1464,7 +1472,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 6 unmodified lines
14641472 /* Ensure requested pindex is drained first. */
14651473 pindex = pindex - 1;
14661474
1467− spin_lock_irqsave(&zone->lock, flags);
1475+ guard(spinlock_irqsave)(&zone->lock);
14681476
14691477 while (count > 0) {
14701478 struct list_head *list;
@@ -1496,8 +1504,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, · 25 unmodified lines
14961504 trace_mm_page_pcpu_drain(page, order, mt);
14971505 } while (count > 0 && !list_empty(list));
14981506 }
1499−
1500− spin_unlock_irqrestore(&zone->lock, flags);
15011507 }
15021508
15031509 /* Split a multi-block free page into its individual pageblocks. */
@@ -1848,7 +1854,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, · 344 unmodified lines
18481854 }
18491855 /* If memory is still not initialized, initialize it now. */
18501856 if (init)
1851− kernel_init_pages(page, 1 << order);
1857+ clear_highpages_kasan_tagged(page, 1 << order);
18521858
18531859 set_page_owner(page, order, gfp_flags);
18541860 page_table_check_alloc(page, order);
@@ -2125,15 +2131,15 @@ static bool __move_freepages_block_isolate(struct zone *zone, · 270 unmodified lines
21252131 }
21262132
21272133 move:
2128− /* Use MIGRATETYPE_MASK to get non-isolate migratetype */
2134+ /* Use PAGEBLOCK_MIGRATETYPE_MASK to get non-isolate migratetype */
21292135 if (isolate) {
21302136 from_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),
2131− MIGRATETYPE_MASK);
2137+ PAGEBLOCK_MIGRATETYPE_MASK);
21322138 to_mt = MIGRATE_ISOLATE;
21332139 } else {
21342140 from_mt = MIGRATE_ISOLATE;
21352141 to_mt = __get_pfnblock_flags_mask(page, page_to_pfn(page),
2136− MIGRATETYPE_MASK);
2142+ PAGEBLOCK_MIGRATETYPE_MASK);
21372143 }
21382144
21392145 __move_freepages_block(zone, start_pfn, from_mt, to_mt);
@@ -2244,25 +2250,29 @@ static bool should_try_claim_block(unsigned int order, int start_mt) · 104 unmodified lines
22442250 * we would do this whole-block claiming. This would help to reduce
22452251 * fragmentation due to mixed migratetype pages in one pageblock.
22462252 */
2247−int find_suitable_fallback(struct free_area *area, unsigned int order,
2248− int migratetype, bool claimable)
2253+enum fallback_result
2254+find_suitable_fallback(struct free_area *area, unsigned int order,
2255+ int migratetype, bool claimable, int *mt_out)
22492256 {
22502257 int i;
22512258
22522259 if (claimable && !should_try_claim_block(order, migratetype))
2253− return -2;
2260+ return FALLBACK_NOCLAIM;
22542261
22552262 if (area->nr_free == 0)
2256− return -1;
2263+ return FALLBACK_EMPTY;
22572264
22582265 for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) {
22592266 int fallback_mt = fallbacks[migratetype][i];
22602267
2261− if (!free_area_empty(area, fallback_mt))
2262− return fallback_mt;
2268+ if (!free_area_empty(area, fallback_mt)) {
2269+ if (mt_out)
2270+ *mt_out = fallback_mt;
2271+ return FALLBACK_FOUND;
2272+ }
22632273 }
22642274
2265− return -1;
2275+ return FALLBACK_EMPTY;
22662276 }
22672277
22682278 /*
@@ -2372,16 +2382,16 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype, · 103 unmodified lines
23722382 */
23732383 for (current_order = MAX_PAGE_ORDER; current_order >= min_order;
23742384 --current_order) {
2385+ enum fallback_result result;
2386+
23752387 area = &(zone->free_area[current_order]);
2376− fallback_mt = find_suitable_fallback(area, current_order,
2377− start_migratetype, true);
2388+ result = find_suitable_fallback(area, current_order,
2389+ start_migratetype, true, &fallback_mt);
23782390
2379− /* No block in that order */
2380− if (fallback_mt == -1)
2391+ if (result == FALLBACK_EMPTY)
23812392 continue;
23822393
2383− /* Advanced into orders too low to claim, abort */
2384− if (fallback_mt == -2)
2394+ if (result == FALLBACK_NOCLAIM)
23852395 break;
23862396
23872397 page = get_page_from_free_area(area, fallback_mt);
@@ -2411,10 +2421,12 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype) · 23 unmodified lines
24112421 int fallback_mt;
24122422
24132423 for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) {
2424+ enum fallback_result result;
2425+
24142426 area = &(zone->free_area[current_order]);
2415− fallback_mt = find_suitable_fallback(area, current_order,
2416− start_migratetype, false);
2417− if (fallback_mt == -1)
2427+ result = find_suitable_fallback(area, current_order, start_migratetype,
2428+ false, &fallback_mt);
2429+ if (result == FALLBACK_EMPTY)
24182430 continue;
24192431
24202432 page = get_page_from_free_area(area, fallback_mt);
@@ -3424,7 +3436,7 @@ static void reserve_highatomic_pageblock(struct page *page, int order, · 1003 unmodified lines
34243436 struct zone *zone)
34253437 {
34263438 int mt;
3427− unsigned long max_managed, flags;
3439+ unsigned long max_managed;
34283440
34293441 /*
34303442 * The number reserved as: minimum is 1 pageblock, maximum is
@@ -3438,29 +3450,26 @@ static void reserve_highatomic_pageblock(struct page *page, int order, · 7 unmodified lines
34383450 if (zone->nr_reserved_highatomic >= max_managed)
34393451 return;
34403452
3441− spin_lock_irqsave(&zone->lock, flags);
3453+ guard(spinlock_irqsave)(&zone->lock);
34423454
34433455 /* Recheck the nr_reserved_highatomic limit under the lock */
34443456 if (zone->nr_reserved_highatomic >= max_managed)
3445− goto out_unlock;
3457+ return;
34463458
34473459 /* Yoink! */
34483460 mt = get_pageblock_migratetype(page);
34493461 /* Only reserve normal pageblocks (i.e., they can merge with others) */
34503462 if (!migratetype_is_mergeable(mt))
3451− goto out_unlock;
3463+ return;
34523464
34533465 if (order < pageblock_order) {
34543466 if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) == -1)
3455− goto out_unlock;
3467+ return;
34563468 zone->nr_reserved_highatomic += pageblock_nr_pages;
34573469 } else {
34583470 change_pageblock_range(page, order, MIGRATE_HIGHATOMIC);
34593471 zone->nr_reserved_highatomic += 1 << order;
34603472 }
3461−
3462−out_unlock:
3463− spin_unlock_irqrestore(&zone->lock, flags);
34643473 }
34653474
34663475 /*
@@ -3476,7 +3485,6 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 9 unmodified lines
34763485 bool force)
34773486 {
34783487 struct zonelist *zonelist = ac->zonelist;
3479− unsigned long flags;
34803488 struct zoneref *z;
34813489 struct zone *zone;
34823490 struct page *page;
@@ -3493,7 +3501,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 10 unmodified lines
34933501 pageblock_nr_pages)
34943502 continue;
34953503
3496− spin_lock_irqsave(&zone->lock, flags);
3504+ guard(spinlock_irqsave)(&zone->lock);
34973505 for (order = 0; order < NR_PAGE_ORDERS; order++) {
34983506 struct free_area *area = &(zone->free_area[order]);
34993507 unsigned long size;
@@ -3540,12 +3548,9 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, · 40 unmodified lines
35403548 * so this should not fail on zone boundaries.
35413549 */
35423550 WARN_ON_ONCE(ret == -1);
3543− if (ret > 0) {
3544− spin_unlock_irqrestore(&zone->lock, flags);
3551+ if (ret > 0)
35453552 return ret;
3546− }
35473553 }
3548− spin_unlock_irqrestore(&zone->lock, flags);
35493554 }
35503555
35513556 return false;
@@ -4156,7 +4161,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 604 unmodified lines
41564161 psi_memstall_leave(&pflags);
41574162 delayacct_compact_end();
41584163
4159− if (*compact_result == COMPACT_SKIPPED)
4164+ if (*compact_result == COMPACT_SKIPPED ||
4165+ *compact_result == COMPACT_DEFERRED)
41604166 return NULL;
41614167 /*
41624168 * At least in one zone compaction wasn't deferred or skipped, so let's
@@ -4193,7 +4199,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 30 unmodified lines
41934199 }
41944200
41954201 static inline bool
4196−should_compact_retry(struct alloc_context *ac, int order, int alloc_flags,
4202+should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order,
4203+ int alloc_flags,
41974204 enum compact_result compact_result,
41984205 enum compact_priority *compact_priority,
41994206 int *compaction_retries)
@@ -4215,7 +4222,8 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags, · 15 unmodified lines
42154222 * migration targets. Continue if reclaim can help.
42164223 */
42174224 if (compact_result == COMPACT_SKIPPED) {
4218− ret = compaction_zonelist_suitable(ac, order, alloc_flags);
4225+ ret = compaction_zonelist_suitable(ac, order, alloc_flags,
4226+ gfp_mask);
42194227 goto out;
42204228 }
42214229
@@ -4268,7 +4276,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, · 46 unmodified lines
42684276 }
42694277
42704278 static inline bool
4271−should_compact_retry(struct alloc_context *ac, int order, int alloc_flags,
4279+should_compact_retry(gfp_t gfp_mask, struct alloc_context *ac, int order,
4280+ int alloc_flags,
42724281 enum compact_result compact_result,
42734282 enum compact_priority *compact_priority,
42744283 int *compaction_retries)
@@ -4678,6 +4687,40 @@ check_retry_cpuset(int cpuset_mems_cookie, struct alloc_context *ac) · 403 unmodified lines
46784687 return false;
46794688 }
46804689
4690+static void check_alloc_stall_warn(gfp_t gfp_mask, nodemask_t *nodemask,
4691+ unsigned int order, unsigned long alloc_start_time)
4692+{
4693+ static DEFINE_SPINLOCK(alloc_stall_lock);
4694+ unsigned long stall_msecs = jiffies_to_msecs(jiffies - alloc_start_time);
4695+
4696+ if (likely(stall_msecs < ALLOC_STALL_WARN_MSECS))
4697+ return;
4698+ if (time_is_after_jiffies(READ_ONCE(alloc_stall_warn_jiffies)))
4699+ return;
4700+ if (gfp_mask & __GFP_NOWARN)
4701+ return;
4702+
4703+ if (!spin_trylock(&alloc_stall_lock))
4704+ return;
4705+
4706+ /* Check again, this time under the lock */
4707+ if (time_is_after_jiffies(alloc_stall_warn_jiffies)) {
4708+ spin_unlock(&alloc_stall_lock);
4709+ return;
4710+ }
4711+
4712+ WRITE_ONCE(alloc_stall_warn_jiffies, jiffies + msecs_to_jiffies(ALLOC_STALL_WARN_MSECS));
4713+ spin_unlock(&alloc_stall_lock);
4714+
4715+ pr_warn("%s: page allocation stall for %lu secs: order:%d, mode:%#x(%pGg) nodemask=%*pbl",
4716+ current->comm, stall_msecs / MSEC_PER_SEC, order, gfp_mask, &gfp_mask,
4717+ nodemask_pr_args(nodemask));
4718+ cpuset_print_current_mems_allowed();
4719+ pr_cont("\n");
4720+ dump_stack();
4721+ warn_alloc_show_mem(gfp_mask, nodemask);
4722+}
4723+
46814724 static inline struct page *
46824725 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
46834726 struct alloc_context *ac)
@@ -4698,6 +4741,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 14 unmodified lines
46984741 int reserve_flags;
46994742 bool compact_first = false;
47004743 bool can_retry_reserves = true;
4744+ unsigned long alloc_start_time = jiffies;
47014745
47024746 if (unlikely(nofail)) {
47034747 /*
@@ -4806,13 +4850,27 @@ retry: · 102 unmodified lines
48064850 }
48074851
48084852 /* Caller is not willing to reclaim, we can't balance anything */
4809− if (!can_direct_reclaim)
4853+ if (!can_direct_reclaim) {
4854+ /*
4855+ * Reclaim/compaction cannot run, so defrag_mode's strategy
4856+ * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
4857+ * fallbacks rather than failing the allocation outright.
4858+ */
4859+ if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
4860+ (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
4861+ alloc_flags &= ~ALLOC_NOFRAGMENT;
4862+ goto retry;
4863+ }
48104864 goto nopage;
4865+ }
48114866
48124867 /* Avoid recursion of direct reclaim */
48134868 if (current->flags & PF_MEMALLOC)
48144869 goto nopage;
48154870
4871+ /* If allocation has taken excessively long, warn about it */
4872+ check_alloc_stall_warn(gfp_mask, ac->nodemask, order, alloc_start_time);
4873+
48164874 /* Try direct reclaim and then allocating */
48174875 if (!compact_first) {
48184876 page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
@@ -4886,9 +4944,9 @@ retry: · 67 unmodified lines
48864944 * of free memory (see __compaction_suitable)
48874945 */
48884946 if (did_some_progress > 0 && can_compact &&
4889− should_compact_retry(ac, order, alloc_flags,
4890− compact_result, &compact_priority,
4891− &compaction_retries))
4947+ should_compact_retry(gfp_mask, ac, order, alloc_flags,
4948+ compact_result, &compact_priority,
4949+ &compaction_retries))
48924950 goto retry;
48934951
48944952 /* Reclaim/compaction failed to prevent the fallback */
@@ -5044,7 +5102,6 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 149 unmodified lines
50445102 struct per_cpu_pages *pcp;
50455103 struct list_head *pcp_list;
50465104 struct alloc_context ac;
5047− gfp_t alloc_gfp;
50485105 unsigned int alloc_flags = ALLOC_WMARK_LOW;
50495106 int nr_populated = 0, nr_account = 0;
50505107
@@ -5085,10 +5142,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 34 unmodified lines
50855142
50865143 /* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
50875144 gfp &= gfp_allowed_mask;
5088− alloc_gfp = gfp;
5089− if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &alloc_gfp, &alloc_flags))
5145+ if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
50905146 goto out;
5091− gfp = alloc_gfp;
50925147
50935148 /* Find an allowed local zone that meets the low watermark. */
50945149 z = ac.preferred_zoneref;
@@ -5179,6 +5234,34 @@ failed: · 84 unmodified lines
51795234 }
51805235 EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
51815236
5237+/*
5238+ * free_pages_bulk - Free an array of order-0 pages
5239+ * @page_array: Array of pages to free
5240+ * @nr_pages: The number of pages in the array
5241+ *
5242+ * Free the order-0 pages. Adjacent entries whose PFNs form a contiguous
5243+ * run are released with a single __free_contig_range() call.
5244+ *
5245+ * This assumes page_array is sorted in ascending PFN order. Without that,
5246+ * the function still frees all pages, but contiguous runs may not be
5247+ * detected and the freeing pattern can degrade to freeing one page at a
5248+ * time.
5249+ *
5250+ * Context: Sleepable process context only; calls cond_resched()
5251+ */
5252+void free_pages_bulk(struct page **page_array, unsigned long nr_pages)
5253+{
5254+ while (nr_pages) {
5255+ unsigned long nr_contig = num_pages_contiguous(page_array, nr_pages);
5256+
5257+ __free_contig_range(page_to_pfn(*page_array), nr_contig);
5258+
5259+ nr_pages -= nr_contig;
5260+ page_array += nr_contig;
5261+ cond_resched();
5262+ }
5263+}
5264+
51825265 /*
51835266 * This is the 'heart' of the zoned buddy allocator.
51845267 */
@@ -6758,6 +6841,105 @@ void __init page_alloc_sysctl_init(void) · 1573 unmodified lines
67586841 register_sysctl_init("vm", page_alloc_sysctl_table);
67596842 }
67606843
6844+static void free_prepared_contig_range(struct page *page,
6845+ unsigned long nr_pages)
6846+{
6847+ unsigned long pfn = page_to_pfn(page);
6848+
6849+ while (nr_pages) {
6850+ unsigned int order;
6851+
6852+ /* We are limited by the largest buddy order. */
6853+ order = pfn ? __ffs(pfn) : MAX_PAGE_ORDER;
6854+ /* Don't exceed the number of pages to free. */
6855+ order = min_t(unsigned int, order, ilog2(nr_pages));
6856+ order = min_t(unsigned int, order, MAX_PAGE_ORDER);
6857+
6858+ /*
6859+ * Free the chunk as a single block. Our caller has already
6860+ * called free_pages_prepare() for each order-0 page.
6861+ */
6862+ __free_frozen_pages(page, order, FPI_PREPARED);
6863+
6864+ pfn += 1UL << order;
6865+ page += 1UL << order;
6866+ nr_pages -= 1UL << order;
6867+ }
6868+}
6869+
6870+static void __free_contig_range_common(unsigned long pfn, unsigned long nr_pages,
6871+ bool is_frozen)
6872+{
6873+ struct page *page, *start = NULL;
6874+ unsigned long nr_start = 0;
6875+ unsigned long start_sec;
6876+ unsigned long i;
6877+
6878+ for (i = 0; i < nr_pages; i++) {
6879+ bool can_free = true;
6880+
6881+ /*
6882+ * Contiguous PFNs might not have contiguous "struct pages"
6883+ * in some kernel configs: page++ across a section boundary
6884+ * is undefined. Use pfn_to_page() for each PFN.
6885+ */
6886+ page = pfn_to_page(pfn + i);
6887+
6888+ VM_WARN_ON_ONCE(PageHead(page));
6889+ VM_WARN_ON_ONCE(PageTail(page));
6890+
6891+ if (!is_frozen)
6892+ can_free = put_page_testzero(page);
6893+
6894+ if (can_free)
6895+ can_free = free_pages_prepare(page, 0);
6896+
6897+ if (!can_free) {
6898+ if (start) {
6899+ free_prepared_contig_range(start, i - nr_start);
6900+ start = NULL;
6901+ }
6902+ continue;
6903+ }
6904+
6905+ if (start && memdesc_section(page->flags) != start_sec) {
6906+ free_prepared_contig_range(start, i - nr_start);
6907+ start = page;
6908+ nr_start = i;
6909+ start_sec = memdesc_section(page->flags);
6910+ } else if (!start) {
6911+ start = page;
6912+ nr_start = i;
6913+ start_sec = memdesc_section(page->flags);
6914+ }
6915+ }
6916+
6917+ if (start)
6918+ free_prepared_contig_range(start, nr_pages - nr_start);
6919+}
6920+
6921+/**
6922+ * __free_contig_range - Free contiguous range of order-0 pages.
6923+ * @pfn: Page frame number of the first page in the range.
6924+ * @nr_pages: Number of pages to free.
6925+ *
6926+ * For each order-0 struct page in the physically contiguous range, put a
6927+ * reference. Free any page who's reference count falls to zero. The
6928+ * implementation is functionally equivalent to, but significantly faster than
6929+ * calling __free_page() for each struct page in a loop.
6930+ *
6931+ * Memory allocated with alloc_pages(order>=1) then subsequently split to
6932+ * order-0 with split_page() is an example of appropriate contiguous pages that
6933+ * can be freed with this API.
6934+ *
6935+ * Context: May be called in interrupt context or while holding a normal
6936+ * spinlock, but not in NMI context or while holding a raw spinlock.
6937+ */
6938+void __free_contig_range(unsigned long pfn, unsigned long nr_pages)
6939+{
6940+ __free_contig_range_common(pfn, nr_pages, /* is_frozen= */ false);
6941+}
6942+
67616943 #ifdef CONFIG_CONTIG_ALLOC
67626944 /* Usage: See admin-guide/dynamic-debug-howto.rst */
67636945 static void alloc_contig_dump_pages(struct list_head *page_list)
@@ -6895,8 +7077,7 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 131 unmodified lines
68957077
68967078 static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)
68977079 {
6898− for (; nr_pages--; pfn++)
6899− free_frozen_pages(pfn_to_page(pfn), 0);
7080+ __free_contig_range_common(pfn, nr_pages, /* is_frozen= */ true);
69007081 }
69017082
69027083 /**
@@ -7304,8 +7485,7 @@ void free_contig_range(unsigned long pfn, unsigned long nr_pages) · 401 unmodified lines
73047485 if (WARN_ON_ONCE(PageHead(pfn_to_page(pfn))))
73057486 return;
73067487
7307− for (; nr_pages--; pfn++)
7308− __free_page(pfn_to_page(pfn));
7488+ __free_contig_range(pfn, nr_pages);
73097489 }
73107490 EXPORT_SYMBOL(free_contig_range);
73117491 #endif /* CONFIG_CONTIG_ALLOC */
@@ -7363,7 +7543,7 @@ void zone_pcp_reset(struct zone *zone) · 51 unmodified lines
73637543 unsigned long __offline_isolated_pages(unsigned long start_pfn,
73647544 unsigned long end_pfn)
73657545 {
7366− unsigned long already_offline = 0, flags;
7546+ unsigned long already_offline = 0;
73677547 unsigned long pfn = start_pfn;
73687548 struct page *page;
73697549 struct zone *zone;
@@ -7371,7 +7551,7 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn, · 1 unmodified line
73717551
73727552 offline_mem_sections(pfn, end_pfn);
73737553 zone = page_zone(pfn_to_page(pfn));
7374− spin_lock_irqsave(&zone->lock, flags);
7554+ guard(spinlock_irqsave)(&zone->lock);
73757555 while (pfn < end_pfn) {
73767556 page = pfn_to_page(pfn);
73777557 /*
@@ -7401,7 +7581,6 @@ unsigned long __offline_isolated_pages(unsigned long start_pfn, · 23 unmodified lines
74017581 del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE);
74027582 pfn += (1 << order);
74037583 }
7404− spin_unlock_irqrestore(&zone->lock, flags);
74057584
74067585 return end_pfn - start_pfn - already_offline;
74077586 }
@@ -7473,11 +7652,9 @@ bool take_page_off_buddy(struct page *page) · 65 unmodified lines
74737652 {
74747653 struct zone *zone = page_zone(page);
74757654 unsigned long pfn = page_to_pfn(page);
7476− unsigned long flags;
74777655 unsigned int order;
7478− bool ret = false;
74797656
7480− spin_lock_irqsave(&zone->lock, flags);
7657+ guard(spinlock_irqsave)(&zone->lock);
74817658 for (order = 0; order < NR_PAGE_ORDERS; order++) {
74827659 struct page *page_head = page - (pfn & ((1 << order) - 1));
74837660 int page_order = buddy_order(page_head);
@@ -7492,14 +7669,12 @@ bool take_page_off_buddy(struct page *page) · 8 unmodified lines
74927669 break_down_buddy_pages(zone, page_head, page, 0,
74937670 page_order, migratetype);
74947671 SetPageHWPoisonTakenOff(page);
7495− ret = true;
7496− break;
7672+ return true;
74977673 }
74987674 if (page_count(page_head) > 0)
74997675 break;
75007676 }
7501− spin_unlock_irqrestore(&zone->lock, flags);
7502− return ret;
7677+ return false;
75037678 }
75047679
75057680 /*
@@ -7508,23 +7683,19 @@ bool take_page_off_buddy(struct page *page) · 2 unmodified lines
75087683 bool put_page_back_buddy(struct page *page)
75097684 {
75107685 struct zone *zone = page_zone(page);
7511− unsigned long flags;
7512− bool ret = false;
75137686
7514− spin_lock_irqsave(&zone->lock, flags);
7687+ guard(spinlock_irqsave)(&zone->lock);
75157688 if (put_page_testzero(page)) {
75167689 unsigned long pfn = page_to_pfn(page);
75177690 int migratetype = get_pfnblock_migratetype(page, pfn);
75187691
75197692 ClearPageHWPoisonTakenOff(page);
75207693 __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE);
7521− if (TestClearPageHWPoison(page)) {
7522− ret = true;
7523− }
7694+ if (TestClearPageHWPoison(page))
7695+ return true;
75247696 }
7525− spin_unlock_irqrestore(&zone->lock, flags);
75267697
7527− return ret;
7698+ return false;
75287699 }
75297700 #endif
75307701
@@ -7774,8 +7945,8 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned · 243 unmodified lines
77747945 * @order: allocation order size
77757946 *
77767947 * Allocates pages of a given order from the given node. This is safe to
7777− * call from any context (from atomic, NMI, and also reentrant
7778− * allocator -> tracepoint -> alloc_pages_nolock_noprof).
7948+ * call from any context where RCU is watching (from atomic, NMI, and also
7949+ * reentrant allocator -> tracepoint -> alloc_pages_nolock_noprof).
77797950 * Allocation is best effort and to be expected to fail easily so nobody should
77807951 * rely on the success. Failures are not reported via warn_alloc().
77817952 * See always fail conditions below.
mm/page_io.c+73−30
@@ -26,6 +26,7 @@ · 25 unmodified lines
2626 #include <linux/delayacct.h>
2727 #include <linux/zswap.h>
2828 #include "swap.h"
29+#include "swap_table.h"
2930
3031 static void __end_swap_bio_write(struct bio *bio)
3132 {
@@ -204,15 +205,20 @@ static bool is_folio_zero_filled(struct folio *folio) · 172 unmodified lines
204205 static void swap_zeromap_folio_set(struct folio *folio)
205206 {
206207 struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
207− struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
208208 int nr_pages = folio_nr_pages(folio);
209+ struct swap_cluster_info *ci;
209210 swp_entry_t entry;
210211 unsigned int i;
211212
213+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
214+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
215+
216+ ci = swap_cluster_get_and_lock(folio);
212217 for (i = 0; i < folio_nr_pages(folio); i++) {
213218 entry = page_swap_entry(folio_page(folio, i));
214− set_bit(swp_offset(entry), sis->zeromap);
219+ __swap_table_set_zero(ci, swp_cluster_offset(entry));
215220 }
221+ swap_cluster_unlock(ci);
216222
217223 count_vm_events(SWPOUT_ZERO, nr_pages);
218224 if (objcg) {
@@ -223,14 +229,19 @@ static void swap_zeromap_folio_set(struct folio *folio) · 4 unmodified lines
223229
224230 static void swap_zeromap_folio_clear(struct folio *folio)
225231 {
226− struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
232+ struct swap_cluster_info *ci;
227233 swp_entry_t entry;
228234 unsigned int i;
229235
236+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
237+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
238+
239+ ci = swap_cluster_get_and_lock(folio);
230240 for (i = 0; i < folio_nr_pages(folio); i++) {
231241 entry = page_swap_entry(folio_page(folio, i));
232− clear_bit(swp_offset(entry), sis->zeromap);
242+ __swap_table_clear_zero(ci, swp_cluster_offset(entry));
233243 }
244+ swap_cluster_unlock(ci);
234245 }
235246
236247 /*
@@ -255,10 +266,9 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug) · 18 unmodified lines
255266 }
256267
257268 /*
258− * Use a bitmap (zeromap) to avoid doing IO for zero-filled pages.
259− * The bits in zeromap are protected by the locked swapcache folio
260− * and atomic updates are used to protect against read-modify-write
261− * corruption due to other zero swap entries seeing concurrent updates.
269+ * Use the swap table zero mark to avoid doing IO for zero-filled
270+ * pages. The zero mark is protected by the cluster lock, which is
271+ * acquired internally by swap_zeromap_folio_set/clear.
262272 */
263273 if (is_folio_zero_filled(folio)) {
264274 swap_zeromap_folio_set(folio);
@@ -326,8 +336,8 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) · 61 unmodified lines
326336
327337 struct swap_iocb {
328338 struct kiocb iocb;
329− struct bio_vec bvec[SWAP_CLUSTER_MAX];
330− int pages;
339+ struct bio_vec bvecs[SWAP_CLUSTER_MAX];
340+ int nr_bvecs;
331341 int len;
332342 };
333343 static mempool_t *sio_pool;
@@ -348,7 +358,7 @@ int sio_pool_init(void) · 14 unmodified lines
348358 static void sio_write_complete(struct kiocb *iocb, long ret)
349359 {
350360 struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
351− struct page *page = sio->bvec[0].bv_page;
361+ struct page *page = sio->bvecs[0].bv_page;
352362 int p;
353363
354364 if (ret != sio->len) {
@@ -362,15 +372,15 @@ static void sio_write_complete(struct kiocb *iocb, long ret) · 7 unmodified lines
362372 */
363373 pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
364374 ret, swap_dev_pos(page_swap_entry(page)));
365− for (p = 0; p < sio->pages; p++) {
366− page = sio->bvec[p].bv_page;
375+ for (p = 0; p < sio->nr_bvecs; p++) {
376+ page = sio->bvecs[p].bv_page;
367377 set_page_dirty(page);
368378 ClearPageReclaim(page);
369379 }
370380 }
371381
372− for (p = 0; p < sio->pages; p++)
373− end_page_writeback(sio->bvec[p].bv_page);
382+ for (p = 0; p < sio->nr_bvecs; p++)
383+ end_page_writeback(sio->bvecs[p].bv_page);
374384
375385 mempool_free(sio, sio_pool);
376386 }
@@ -397,13 +407,13 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) · 20 unmodified lines
397407 init_sync_kiocb(&sio->iocb, swap_file);
398408 sio->iocb.ki_complete = sio_write_complete;
399409 sio->iocb.ki_pos = pos;
400− sio->pages = 0;
410+ sio->nr_bvecs = 0;
401411 sio->len = 0;
402412 }
403− bvec_set_folio(&sio->bvec[sio->pages], folio, folio_size(folio), 0);
413+ bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
404414 sio->len += folio_size(folio);
405− sio->pages += 1;
406− if (sio->pages == ARRAY_SIZE(sio->bvec) || !swap_plug) {
415+ sio->nr_bvecs += 1;
416+ if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !swap_plug) {
407417 swap_write_unplug(sio);
408418 sio = NULL;
409419 }
@@ -477,7 +487,7 @@ void swap_write_unplug(struct swap_iocb *sio) · 67 unmodified lines
477487 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
478488 int ret;
479489
480− iov_iter_bvec(&from, ITER_SOURCE, sio->bvec, sio->pages, sio->len);
490+ iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len);
481491 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
482492 if (ret != -EIOCBQUEUED)
483493 sio_write_complete(&sio->iocb, ret);
@@ -489,8 +499,8 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 5 unmodified lines
489499 int p;
490500
491501 if (ret == sio->len) {
492− for (p = 0; p < sio->pages; p++) {
493− struct folio *folio = bvec_folio(&sio->bvec[p]);
502+ for (p = 0; p < sio->nr_bvecs; p++) {
503+ struct folio *folio = bvec_folio(&sio->bvecs[p]);
494504
495505 count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
496506 count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
@@ -499,8 +509,8 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 2 unmodified lines
499509 }
500510 count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
501511 } else {
502− for (p = 0; p < sio->pages; p++) {
503− struct folio *folio = bvec_folio(&sio->bvec[p]);
512+ for (p = 0; p < sio->nr_bvecs; p++) {
513+ struct folio *folio = bvec_folio(&sio->bvecs[p]);
504514
505515 folio_unlock(folio);
506516 }
@@ -509,19 +519,52 @@ static void sio_read_complete(struct kiocb *iocb, long ret) · 2 unmodified lines
509519 mempool_free(sio, sio_pool);
510520 }
511521
522+/*
523+ * Return the count of contiguous swap entries that share the same
524+ * zeromap status as the starting entry. If is_zerop is not NULL,
525+ * it will return the zeromap status of the starting entry.
526+ *
527+ * Context: Caller must ensure the cluster containing the entries
528+ * that will be checked won't be freed.
529+ */
530+static int swap_zeromap_batch(swp_entry_t entry, int max_nr,
531+ bool *is_zerop)
532+{
533+ int i;
534+ bool is_zero;
535+ unsigned int ci_start = swp_cluster_offset(entry);
536+ struct swap_cluster_info *ci = __swap_entry_to_cluster(entry);
537+
538+ VM_WARN_ON_ONCE(ci_start + max_nr > SWAPFILE_CLUSTER);
539+
540+ rcu_read_lock();
541+ is_zero = __swap_table_test_zero(ci, ci_start);
542+ for (i = 1; i < max_nr; i++)
543+ if (is_zero != __swap_table_test_zero(ci, ci_start + i))
544+ break;
545+ rcu_read_unlock();
546+ if (is_zerop)
547+ *is_zerop = is_zero;
548+
549+ return i;
550+}
551+
512552 static bool swap_read_folio_zeromap(struct folio *folio)
513553 {
514554 int nr_pages = folio_nr_pages(folio);
515555 struct obj_cgroup *objcg;
516556 bool is_zeromap;
517557
558+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
559+
518560 /*
519561 * Swapping in a large folio that is partially in the zeromap is not
520562 * currently handled. Return true without marking the folio uptodate so
521563 * that an IO error is emitted (e.g. do_swap_page() will sigbus).
564+ * Folio lock stabilizes the cluster and map, so the check is safe.
522565 */
523566 if (WARN_ON_ONCE(swap_zeromap_batch(folio->swap, nr_pages,
524− &is_zeromap) != nr_pages))
567+ &is_zeromap) != nr_pages))
525568 return true;
526569
527570 if (!is_zeromap)
@@ -559,13 +602,13 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug) · 31 unmodified lines
559602 init_sync_kiocb(&sio->iocb, sis->swap_file);
560603 sio->iocb.ki_pos = pos;
561604 sio->iocb.ki_complete = sio_read_complete;
562− sio->pages = 0;
605+ sio->nr_bvecs = 0;
563606 sio->len = 0;
564607 }
565− bvec_set_folio(&sio->bvec[sio->pages], folio, folio_size(folio), 0);
608+ bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
566609 sio->len += folio_size(folio);
567− sio->pages += 1;
568− if (sio->pages == ARRAY_SIZE(sio->bvec) || !plug) {
610+ sio->nr_bvecs += 1;
611+ if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !plug) {
569612 swap_read_unplug(sio);
570613 sio = NULL;
571614 }
@@ -666,7 +709,7 @@ void __swap_read_unplug(struct swap_iocb *sio) · 94 unmodified lines
666709 struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
667710 int ret;
668711
669− iov_iter_bvec(&from, ITER_DEST, sio->bvec, sio->pages, sio->len);
712+ iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len);
670713 ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
671714 if (ret != -EIOCBQUEUED)
672715 sio_read_complete(&sio->iocb, ret);
mm/page_isolation.c+28−39
@@ -167,48 +167,40 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode, · 166 unmodified lines
167167 {
168168 struct zone *zone = page_zone(page);
169169 struct page *unmovable;
170− unsigned long flags;
171170 unsigned long check_unmovable_start, check_unmovable_end;
172171
173172 if (PageUnaccepted(page))
174173 accept_page(page);
175174
176− spin_lock_irqsave(&zone->lock, flags);
177−
178− /*
179− * We assume the caller intended to SET migrate type to isolate.
180− * If it is already set, then someone else must have raced and
181− * set it before us.
182− */
183− if (is_migrate_isolate_page(page)) {
184− spin_unlock_irqrestore(&zone->lock, flags);
185− return -EBUSY;
186− }
187−
188− /*
189− * FIXME: Now, memory hotplug doesn't call shrink_slab() by itself.
190− * We just check MOVABLE pages.
191− *
192− * Pass the intersection of [start_pfn, end_pfn) and the page's pageblock
193− * to avoid redundant checks.
194− */
195− check_unmovable_start = max(page_to_pfn(page), start_pfn);
196− check_unmovable_end = min(pageblock_end_pfn(page_to_pfn(page)),
197− end_pfn);
198−
199− unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end,
200− mode);
201− if (!unmovable) {
202− if (!pageblock_isolate_and_move_free_pages(zone, page)) {
203− spin_unlock_irqrestore(&zone->lock, flags);
175+ scoped_guard(spinlock_irqsave, &zone->lock) {
176+ /*
177+ * We assume the caller intended to SET migrate type to
178+ * isolate. If it is already set, then someone else must have
179+ * raced and set it before us.
180+ */
181+ if (is_migrate_isolate_page(page))
204182 return -EBUSY;
183+
184+ /*
185+ * FIXME: Now, memory hotplug doesn't call shrink_slab() by
186+ * itself. We just check MOVABLE pages.
187+ *
188+ * Pass the intersection of [start_pfn, end_pfn) and the page's
189+ * pageblock to avoid redundant checks.
190+ */
191+ check_unmovable_start = max(page_to_pfn(page), start_pfn);
192+ check_unmovable_end = min(pageblock_end_pfn(page_to_pfn(page)),
193+ end_pfn);
194+
195+ unmovable = has_unmovable_pages(check_unmovable_start,
196+ check_unmovable_end, mode);
197+ if (!unmovable) {
198+ if (!pageblock_isolate_and_move_free_pages(zone, page))
199+ return -EBUSY;
200+ zone->nr_isolate_pageblock++;
201+ return 0;
205202 }
206− zone->nr_isolate_pageblock++;
207− spin_unlock_irqrestore(&zone->lock, flags);
208− return 0;
209203 }
210−
211− spin_unlock_irqrestore(&zone->lock, flags);
212204 if (mode == PB_ISOLATE_MODE_MEM_OFFLINE) {
213205 /*
214206 * printk() with zone->lock held will likely trigger a
@@ -223,15 +215,14 @@ static int set_migratetype_isolate(struct page *page, enum pb_isolate_mode mode, · 8 unmodified lines
223215 static void unset_migratetype_isolate(struct page *page)
224216 {
225217 struct zone *zone;
226− unsigned long flags;
227218 bool isolated_page = false;
228219 unsigned int order;
229220 struct page *buddy;
230221
231222 zone = page_zone(page);
232− spin_lock_irqsave(&zone->lock, flags);
223+ guard(spinlock_irqsave)(&zone->lock);
233224 if (!is_migrate_isolate_page(page))
234− goto out;
225+ return;
235226
236227 /*
237228 * Because freepage with more than pageblock_order on isolated
@@ -279,8 +270,6 @@ static void unset_migratetype_isolate(struct page *page) · 41 unmodified lines
279270 __putback_isolated_page(page, order, get_pageblock_migratetype(page));
280271 }
281272 zone->nr_isolate_pageblock--;
282−out:
283− spin_unlock_irqrestore(&zone->lock, flags);
284273 }
285274
286275 static inline struct page *
mm/page_owner.c+1−1
@@ -573,7 +573,7 @@ print_page_owner(char __user *buf, size_t count, unsigned long pfn, · 572 unmodified lines
573573 migratetype_names[page_mt],
574574 pfn >> pageblock_order,
575575 migratetype_names[pageblock_mt],
576− &page->flags);
576+ &page->flags.f);
577577
578578 ret += stack_depot_snprint(handle, kbuf + ret, count - ret, 0);
579579 if (ret >= count)
mm/page_vma_mapped.c+7−2
@@ -41,7 +41,7 @@ again: · 40 unmodified lines
4141 if (!pvmw->pte)
4242 return false;
4343
44− ptent = ptep_get(pvmw->pte);
44+ ptent = ptep_get_lockless(pvmw->pte);
4545
4646 if (pte_none(ptent)) {
4747 return false;
@@ -183,6 +183,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) · 135 unmodified lines
183183 struct mm_struct *mm = vma->vm_mm;
184184 unsigned long end;
185185 spinlock_t *ptl;
186+ pte_t pteval;
186187 pgd_t *pgd;
187188 p4d_t *p4d;
188189 pud_t *pud;
@@ -310,7 +311,11 @@ next_pte: · 121 unmodified lines
310311 goto restart;
311312 }
312313 pvmw->pte++;
313− } while (pte_none(ptep_get(pvmw->pte)));
314+ if (!pvmw->ptl)
315+ pteval = ptep_get_lockless(pvmw->pte);
316+ else
317+ pteval = ptep_get(pvmw->pte);
318+ } while (pte_none(pteval));
314319
315320 if (!pvmw->ptl) {
316321 spin_lock(ptl);
mm/percpu-internal.h+3−3
@@ -77,13 +77,13 @@ struct pcpu_chunk { · 76 unmodified lines
7777 int end_offset; /* additional area required to
7878 have the region end page
7979 aligned */
80+ int nr_pages; /* # of pages served by this chunk */
81+ int nr_populated; /* # of populated pages */
82+ int nr_empty_pop_pages; /* # of empty populated pages */
8083 #ifdef NEED_PCPUOBJ_EXT
8184 struct pcpuobj_ext *obj_exts; /* vector of object cgroups */
8285 #endif
8386
84− int nr_pages; /* # of pages served by this chunk */
85− int nr_populated; /* # of populated pages */
86− int nr_empty_pop_pages; /* # of empty populated pages */
8787 unsigned long populated[]; /* populated bitmap */
8888 };
8989
mm/readahead.c+30−6
@@ -146,6 +146,17 @@ file_ra_state_init(struct file_ra_state *ra, struct address_space *mapping) · 145 unmodified lines
146146 }
147147 EXPORT_SYMBOL_GPL(file_ra_state_init);
148148
149+/**
150+ * read_pages() - Start IO for a contiguous range of allocated folios in the
151+ * page cache.
152+ * @rac: Readahead control.
153+ *
154+ * When read_pages() returns, it is guaranteed that all of the folios will have
155+ * been processed or removed so that ``readahead_count(rac) == 0``. However,
156+ * that does not imply that ``readahead_index(rac)`` will be updated to point
157+ * to the end of the originally requested range because, for example, the
158+ * filesystem may expand the range upwards.
159+ */
149160 static void read_pages(struct readahead_control *rac)
150161 {
151162 const struct address_space_operations *aops = rac->mapping->a_ops;
@@ -270,7 +281,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, · 118 unmodified lines
270281 */
271282 read_pages(ractl);
272283 ractl->_index += min_nrpages;
273− i = ractl->_index + ractl->_nr_pages - index;
284+ i = ractl->_index - index;
274285 continue;
275286 }
276287
@@ -286,7 +297,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, · 9 unmodified lines
286297 break;
287298 read_pages(ractl);
288299 ractl->_index += min_nrpages;
289− i = ractl->_index + ractl->_nr_pages - index;
300+ i = ractl->_index - index;
290301 continue;
291302 }
292303 if (i == mark)
@@ -324,11 +335,16 @@ static void do_page_cache_ra(struct readahead_control *ractl, · 31 unmodified lines
324335 return;
325336
326337 end_index = (isize - 1) >> PAGE_SHIFT;
338+ if (end_index > ractl->_max_index)
339+ end_index = ractl->_max_index;
327340 if (index > end_index)
328341 return;
329342 /* Don't read past the page containing the last byte of the file */
330− if (nr_to_read > end_index - index)
343+ if (nr_to_read > end_index - index) {
331344 nr_to_read = end_index - index + 1;
345+ /* We've reached the end, so don't set a readahead marker. */
346+ lookahead_size = 0;
347+ }
332348
333349 filemap_invalidate_lock_shared(mapping);
334350 page_cache_ra_unbounded(ractl, nr_to_read, lookahead_size);
@@ -471,8 +487,8 @@ void page_cache_ra_order(struct readahead_control *ractl, · 136 unmodified lines
471487 pgoff_t start = readahead_index(ractl);
472488 pgoff_t index = start;
473489 unsigned int min_order = mapping_min_folio_order(mapping);
474− pgoff_t limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
475− pgoff_t mark = index + ra->size - ra->async_size;
490+ pgoff_t limit;
491+ pgoff_t mark;
476492 unsigned int nofs;
477493 int err = 0;
478494 gfp_t gfp = readahead_gfp_mask(mapping);
@@ -484,7 +500,15 @@ void page_cache_ra_order(struct readahead_control *ractl, · 5 unmodified lines
484500 goto fallback;
485501 }
486502
487− limit = min(limit, index + ra->size - 1);
503+ limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT;
504+ limit = min(limit, ractl->_max_index);
505+ if (limit > index + ra->size - 1) {
506+ limit = index + ra->size - 1;
507+ mark = index + ra->size - ra->async_size;
508+ } else {
509+ /* We've reached the end, so don't set a readahead marker. */
510+ mark = ULONG_MAX;
511+ }
488512
489513 new_order = min(mapping_max_folio_order(mapping), new_order);
490514 new_order = min_t(unsigned int, new_order, ilog2(ra->size));
mm/rmap.c+4−4
@@ -571,7 +571,7 @@ void __init anon_vma_init(void) · 570 unmodified lines
571571 * In case it was remapped to a different anon_vma, the new anon_vma will be a
572572 * child of the old anon_vma, and the anon_vma lifetime rules will therefore
573573 * ensure that any anon_vma obtained from the page will still be valid for as
574− * long as we observe page_mapped() [ hence all those page_mapped() tests ].
574+ * long as we observe folio_mapped() [ hence all those folio_mapped() tests ].
575575 *
576576 * All users of this function must be very careful when walking the anon_vma
577577 * chain and verify that the page in question is indeed mapped in it
@@ -1999,7 +1999,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · 1421 unmodified lines
19991999 /*
20002000 * When racing against e.g. zap_pte_range() on another cpu,
20012001 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),
2002− * try_to_unmap() may return before page_mapped() has become false,
2002+ * try_to_unmap() may return before folio_mapped() has become false,
20032003 * if page table locking is skipped: use TTU_SYNC to wait for that.
20042004 */
20052005 if (flags & TTU_SYNC)
@@ -2428,7 +2428,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · 422 unmodified lines
24282428 /*
24292429 * When racing against e.g. zap_pte_range() on another cpu,
24302430 * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(),
2431− * try_to_migrate() may return before page_mapped() has become false,
2431+ * try_to_migrate() may return before folio_mapped() has become false,
24322432 * if page table locking is skipped: use TTU_SYNC to wait for that.
24332433 */
24342434 if (flags & TTU_SYNC)
@@ -2929,7 +2929,7 @@ static struct anon_vma *rmap_walk_anon_lock(const struct folio *folio, · 494 unmodified lines
29292929
29302930 /*
29312931 * Note: remove_migration_ptes() cannot use folio_lock_anon_vma_read()
2932− * because that depends on page_mapped(); but not all its usages
2932+ * because that depends on folio_mapped(); but not all its usages
29332933 * are holding mmap_lock. Users without mmap_lock are required to
29342934 * take a reference count to prevent the anon_vma disappearing
29352935 */
mm/shmem.c+119−159
@@ -159,7 +159,7 @@ static unsigned long shmem_default_max_inodes(void) · 158 unmodified lines
159159
160160 static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
161161 struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
162− struct vm_area_struct *vma, vm_fault_t *fault_type);
162+ struct vm_fault *vmf, vm_fault_t *fault_type);
163163
164164 static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
165165 {
@@ -1789,30 +1789,6 @@ static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, · 1623 unmodified lines
17891789 return folio;
17901790 }
17911791
1792−/*
1793− * Make sure huge_gfp is always more limited than limit_gfp.
1794− * Some of the flags set permissions, while others set limitations.
1795− */
1796−static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
1797−{
1798− gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
1799− gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
1800− gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
1801− gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
1802−
1803− /* Allow allocations only from the originally specified zones. */
1804− result |= zoneflags;
1805−
1806− /*
1807− * Minimize the result gfp by taking the union with the deny flags,
1808− * and the intersection of the allow flags.
1809− */
1810− result |= (limit_gfp & denyflags);
1811− result |= (huge_gfp & limit_gfp) & allowflags;
1812−
1813− return result;
1814−}
1815−
18161792 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
18171793 bool shmem_hpage_pmd_enabled(void)
18181794 {
@@ -2039,68 +2015,32 @@ unlock: · 220 unmodified lines
20392015 }
20402016
20412017 static struct folio *shmem_swap_alloc_folio(struct inode *inode,
2042− struct vm_area_struct *vma, pgoff_t index,
2018+ struct vm_fault *vmf, pgoff_t index,
20432019 swp_entry_t entry, int order, gfp_t gfp)
20442020 {
2021+ pgoff_t ilx;
2022+ struct folio *folio;
2023+ struct mempolicy *mpol;
20452024 struct shmem_inode_info *info = SHMEM_I(inode);
2046− struct folio *new, *swapcache;
2047− int nr_pages = 1 << order;
2048− gfp_t alloc_gfp = gfp;
2049−
2050− if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
2051− if (WARN_ON_ONCE(order))
2052− return ERR_PTR(-EINVAL);
2053− } else if (order) {
2054− /*
2055− * If uffd is active for the vma, we need per-page fault
2056− * fidelity to maintain the uffd semantics, then fallback
2057− * to swapin order-0 folio, as well as for zswap case.
2058− * Any existing sub folio in the swap cache also blocks
2059− * mTHP swapin.
2060− */
2061− if ((vma && unlikely(userfaultfd_armed(vma))) ||
2062− !zswap_never_enabled() ||
2063− non_swapcache_batch(entry, nr_pages) != nr_pages)
2064− goto fallback;
20652025
2066− alloc_gfp = limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
2067− }
2068−retry:
2069− new = shmem_alloc_folio(alloc_gfp, order, info, index);
2070− if (!new) {
2071− new = ERR_PTR(-ENOMEM);
2072− goto fallback;
2073− }
2026+ if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
2027+ !zswap_never_enabled())
2028+ order = 0;
20742029
2075− if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,
2076− alloc_gfp, entry)) {
2077− folio_put(new);
2078− new = ERR_PTR(-ENOMEM);
2079− goto fallback;
2080− }
2030+again:
2031+ mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
2032+ folio = swapin_sync(entry, gfp, BIT(order), vmf, mpol, ilx);
2033+ mpol_cond_put(mpol);
20812034
2082− swapcache = swapin_folio(entry, new);
2083− if (swapcache != new) {
2084− folio_put(new);
2085− if (!swapcache) {
2086− /*
2087− * The new folio is charged already, swapin can
2088− * only fail due to another raced swapin.
2089− */
2090− new = ERR_PTR(-EEXIST);
2091− goto fallback;
2092− }
2035+ if (!IS_ERR(folio))
2036+ return folio;
2037+
2038+ if (order) {
2039+ order = 0;
2040+ goto again;
20932041 }
2094− return swapcache;
2095−fallback:
2096− /* Order 0 swapin failed, nothing to fallback to, abort */
2097− if (!order)
2098− return new;
2099− entry.val += index - round_down(index, nr_pages);
2100− alloc_gfp = gfp;
2101− nr_pages = 1;
2102− order = 0;
2103− goto retry;
2042+
2043+ return folio;
21042044 }
21052045
21062046 /*
@@ -2139,7 +2079,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, · 32 unmodified lines
21392079 if (nr_pages > 1) {
21402080 gfp_t huge_gfp = vma_thp_gfp_mask(vma);
21412081
2142− gfp = limit_gfp_mask(huge_gfp, gfp);
2082+ gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
21432083 }
21442084 #endif
21452085
@@ -2287,11 +2227,12 @@ unlock: · 141 unmodified lines
22872227 */
22882228 static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
22892229 struct folio **foliop, enum sgp_type sgp,
2290− gfp_t gfp, struct vm_area_struct *vma,
2230+ gfp_t gfp, struct vm_fault *vmf,
22912231 vm_fault_t *fault_type)
22922232 {
22932233 struct address_space *mapping = inode->i_mapping;
2294− struct mm_struct *fault_mm = vma ? vma->vm_mm : NULL;
2234+ struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
2235+ struct mm_struct *fault_mm = vmf ? vmf->vma->vm_mm : NULL;
22952236 struct shmem_inode_info *info = SHMEM_I(inode);
22962237 swp_entry_t swap;
22972238 softleaf_t index_entry;
@@ -2332,20 +2273,19 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 34 unmodified lines
23322273 if (!folio) {
23332274 if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
23342275 /* Direct swapin skipping swap cache & readahead */
2335− folio = shmem_swap_alloc_folio(inode, vma, index,
2336− index_entry, order, gfp);
2337− if (IS_ERR(folio)) {
2338− error = PTR_ERR(folio);
2339− folio = NULL;
2340− goto failed;
2341− }
2276+ folio = shmem_swap_alloc_folio(inode, vmf, index,
2277+ swap, order, gfp);
23422278 } else {
23432279 /* Cached swapin only supports order 0 folio */
23442280 folio = shmem_swapin_cluster(swap, gfp, info, index);
2345− if (!folio) {
2281+ }
2282+ if (IS_ERR_OR_NULL(folio)) {
2283+ if (IS_ERR(folio))
2284+ error = PTR_ERR(folio);
2285+ else
23462286 error = -ENOMEM;
2347− goto failed;
2348− }
2287+ folio = NULL;
2288+ goto failed;
23492289 }
23502290 if (fault_type) {
23512291 *fault_type |= VM_FAULT_MAJOR;
@@ -2493,7 +2433,7 @@ repeat: · 141 unmodified lines
24932433
24942434 if (xa_is_value(folio)) {
24952435 error = shmem_swapin_folio(inode, index, &folio,
2496− sgp, gfp, vma, fault_type);
2436+ sgp, gfp, vmf, fault_type);
24972437 if (error == -EEXIST)
24982438 goto repeat;
24992439
@@ -2546,7 +2486,7 @@ repeat: · 46 unmodified lines
25462486 gfp_t huge_gfp;
25472487
25482488 huge_gfp = vma_thp_gfp_mask(vma);
2549− huge_gfp = limit_gfp_mask(huge_gfp, gfp);
2489+ huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
25502490 folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
25512491 inode, index, fault_mm, orders);
25522492 if (!IS_ERR(folio)) {
@@ -3100,10 +3040,7 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap, · 547 unmodified lines
31003040 cache_no_acl(inode);
31013041 if (sbinfo->noswap)
31023042 mapping_set_unevictable(inode->i_mapping);
3103−
3104− /* Don't consider 'deny' for emergencies and 'force' for testing */
3105− if (sbinfo->huge)
3106− mapping_set_large_folios(inode->i_mapping);
3043+ mapping_set_large_folios(inode->i_mapping);
31073044
31083045 switch (mode & S_IFMT) {
31093046 default:
@@ -5510,24 +5447,74 @@ static ssize_t shmem_enabled_store(struct kobject *kobj, · 2400 unmodified lines
55105447 struct kobj_attribute shmem_enabled_attr = __ATTR_RW(shmem_enabled);
55115448 static DEFINE_SPINLOCK(huge_shmem_orders_lock);
55125449
5450+enum huge_mode {
5451+ HUGE_SHMEM_ENABLED_ALWAYS = 0,
5452+ HUGE_SHMEM_ENABLED_INHERIT,
5453+ HUGE_SHMEM_ENABLED_WITHIN_SIZE,
5454+ HUGE_SHMEM_ENABLED_ADVISE,
5455+ HUGE_SHMEM_ENABLED_NEVER,
5456+};
5457+
5458+static const char * const huge_mode_strings[] = {
5459+ [HUGE_SHMEM_ENABLED_ALWAYS] = "always",
5460+ [HUGE_SHMEM_ENABLED_INHERIT] = "inherit",
5461+ [HUGE_SHMEM_ENABLED_WITHIN_SIZE] = "within_size",
5462+ [HUGE_SHMEM_ENABLED_ADVISE] = "advise",
5463+ [HUGE_SHMEM_ENABLED_NEVER] = "never",
5464+};
5465+
5466+static unsigned long * const huge_mode_orders[] = {
5467+ [HUGE_SHMEM_ENABLED_ALWAYS] = &huge_shmem_orders_always,
5468+ [HUGE_SHMEM_ENABLED_INHERIT] = &huge_shmem_orders_inherit,
5469+ [HUGE_SHMEM_ENABLED_WITHIN_SIZE] = &huge_shmem_orders_within_size,
5470+ [HUGE_SHMEM_ENABLED_ADVISE] = &huge_shmem_orders_madvise,
5471+};
5472+
55135473 static ssize_t thpsize_shmem_enabled_show(struct kobject *kobj,
55145474 struct kobj_attribute *attr, char *buf)
55155475 {
55165476 int order = to_thpsize(kobj)->order;
5517− const char *output;
5518−
5519− if (test_bit(order, &huge_shmem_orders_always))
5520− output = "[always] inherit within_size advise never";
5521− else if (test_bit(order, &huge_shmem_orders_inherit))
5522− output = "always [inherit] within_size advise never";
5523− else if (test_bit(order, &huge_shmem_orders_within_size))
5524− output = "always inherit [within_size] advise never";
5525− else if (test_bit(order, &huge_shmem_orders_madvise))
5526− output = "always inherit within_size [advise] never";
5527− else
5528− output = "always inherit within_size advise [never]";
5477+ int active = HUGE_SHMEM_ENABLED_NEVER;
5478+ int len = 0;
5479+ int i;
5480+
5481+ for (i = 0; i < ARRAY_SIZE(huge_mode_orders); i++) {
5482+ if (test_bit(order, huge_mode_orders[i])) {
5483+ active = i;
5484+ break;
5485+ }
5486+ }
5487+
5488+ for (i = 0; i < ARRAY_SIZE(huge_mode_strings); i++) {
5489+ if (i == active)
5490+ len += sysfs_emit_at(buf, len, "[%s] ",
5491+ huge_mode_strings[i]);
5492+ else
5493+ len += sysfs_emit_at(buf, len, "%s ",
5494+ huge_mode_strings[i]);
5495+ }
5496+
5497+ /* Replace trailing space with newline */
5498+ buf[len - 1] = '\n';
5499+
5500+ return len;
5501+}
5502+
5503+static bool set_shmem_enabled_mode(int order, enum huge_mode mode)
5504+{
5505+ bool changed = false;
5506+ enum huge_mode idx;
5507+
5508+ spin_lock(&huge_shmem_orders_lock);
5509+ for (idx = 0; idx < ARRAY_SIZE(huge_mode_orders); idx++) {
5510+ if (idx == mode)
5511+ changed |= !__test_and_set_bit(order, huge_mode_orders[idx]);
5512+ else
5513+ changed |= __test_and_clear_bit(order, huge_mode_orders[idx]);
5514+ }
5515+ spin_unlock(&huge_shmem_orders_lock);
55295516
5530− return sysfs_emit(buf, "%s\n", output);
5517+ return changed;
55315518 }
55325519
55335520 static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,
@@ -5535,58 +5522,31 @@ static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj, · 1 unmodified line
55355522 const char *buf, size_t count)
55365523 {
55375524 int order = to_thpsize(kobj)->order;
5538− ssize_t ret = count;
5539−
5540− if (sysfs_streq(buf, "always")) {
5541− spin_lock(&huge_shmem_orders_lock);
5542− clear_bit(order, &huge_shmem_orders_inherit);
5543− clear_bit(order, &huge_shmem_orders_madvise);
5544− clear_bit(order, &huge_shmem_orders_within_size);
5545− set_bit(order, &huge_shmem_orders_always);
5546− spin_unlock(&huge_shmem_orders_lock);
5547− } else if (sysfs_streq(buf, "inherit")) {
5548− /* Do not override huge allocation policy with non-PMD sized mTHP */
5549− if (shmem_huge == SHMEM_HUGE_FORCE && !is_pmd_order(order))
5550− return -EINVAL;
5525+ int mode;
55515526
5552− spin_lock(&huge_shmem_orders_lock);
5553− clear_bit(order, &huge_shmem_orders_always);
5554− clear_bit(order, &huge_shmem_orders_madvise);
5555− clear_bit(order, &huge_shmem_orders_within_size);
5556− set_bit(order, &huge_shmem_orders_inherit);
5557− spin_unlock(&huge_shmem_orders_lock);
5558− } else if (sysfs_streq(buf, "within_size")) {
5559− spin_lock(&huge_shmem_orders_lock);
5560− clear_bit(order, &huge_shmem_orders_always);
5561− clear_bit(order, &huge_shmem_orders_inherit);
5562− clear_bit(order, &huge_shmem_orders_madvise);
5563− set_bit(order, &huge_shmem_orders_within_size);
5564− spin_unlock(&huge_shmem_orders_lock);
5565− } else if (sysfs_streq(buf, "advise")) {
5566− spin_lock(&huge_shmem_orders_lock);
5567− clear_bit(order, &huge_shmem_orders_always);
5568− clear_bit(order, &huge_shmem_orders_inherit);
5569− clear_bit(order, &huge_shmem_orders_within_size);
5570− set_bit(order, &huge_shmem_orders_madvise);
5571− spin_unlock(&huge_shmem_orders_lock);
5572− } else if (sysfs_streq(buf, "never")) {
5573− spin_lock(&huge_shmem_orders_lock);
5574− clear_bit(order, &huge_shmem_orders_always);
5575− clear_bit(order, &huge_shmem_orders_inherit);
5576− clear_bit(order, &huge_shmem_orders_within_size);
5577− clear_bit(order, &huge_shmem_orders_madvise);
5578− spin_unlock(&huge_shmem_orders_lock);
5579− } else {
5580− ret = -EINVAL;
5581− }
5527+ mode = sysfs_match_string(huge_mode_strings, buf);
5528+ if (mode < 0)
5529+ return mode;
55825530
5583− if (ret > 0) {
5584− int err = start_stop_khugepaged();
5531+ /* Do not override huge allocation policy with non-PMD sized mTHP */
5532+ if (mode == HUGE_SHMEM_ENABLED_INHERIT &&
5533+ shmem_huge == SHMEM_HUGE_FORCE && !is_pmd_order(order))
5534+ return -EINVAL;
55855535
5536+ if (set_shmem_enabled_mode(order, mode)) {
5537+ int err = start_stop_khugepaged();
55865538 if (err)
5587− ret = err;
5539+ return err;
5540+ } else {
5541+ /*
5542+ * Recalculate watermarks even when the mode hasn't changed
5543+ * to preserve the legacy behavior, as this is always called
5544+ * inside start_stop_khugepaged().
5545+ */
5546+ set_recommended_min_free_kbytes();
55885547 }
5589− return ret;
5548+
5549+ return count;
55905550 }
55915551
55925552 struct kobj_attribute thpsize_shmem_enabled_attr =
mm/shrinker.c+8−10
@@ -197,12 +197,13 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) · 196 unmodified lines
197197 {
198198 if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {
199199 struct shrinker_info *info;
200− struct shrinker_info_unit *unit;
201200
202201 rcu_read_lock();
203202 info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
204− unit = info->unit[shrinker_id_to_index(shrinker_id)];
205203 if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {
204+ struct shrinker_info_unit *unit;
205+
206+ unit = info->unit[shrinker_id_to_index(shrinker_id)];
206207 /* Pairs with smp mb in shrink_slab() */
207208 smp_mb__before_atomic();
208209 set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
@@ -215,29 +216,26 @@ static DEFINE_IDR(shrinker_idr); · 6 unmodified lines
215216
216217 static int shrinker_memcg_alloc(struct shrinker *shrinker)
217218 {
218− int id, ret = -ENOMEM;
219+ int id;
219220
220221 if (mem_cgroup_disabled())
221222 return -ENOSYS;
222223 if (mem_cgroup_kmem_disabled() && !(shrinker->flags & SHRINKER_NONSLAB))
223224 return -ENOSYS;
224225
225− mutex_lock(&shrinker_mutex);
226+ guard(mutex)(&shrinker_mutex);
226227 id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL);
227228 if (id < 0)
228− goto unlock;
229+ return id;
229230
230231 if (id >= shrinker_nr_max) {
231232 if (expand_shrinker_info(id)) {
232233 idr_remove(&shrinker_idr, id);
233− goto unlock;
234+ return -ENOMEM;
234235 }
235236 }
236237 shrinker->id = id;
237− ret = 0;
238−unlock:
239− mutex_unlock(&shrinker_mutex);
240− return ret;
238+ return 0;
241239 }
242240
243241 static void shrinker_memcg_remove(struct shrinker *shrinker)
mm/sparse-vmemmap.c+56−26
@@ -87,15 +87,10 @@ static void * __meminit altmap_alloc_block_buf(unsigned long size, · 86 unmodified lines
8787 void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node,
8888 struct vmem_altmap *altmap)
8989 {
90− void *ptr;
91−
9290 if (altmap)
9391 return altmap_alloc_block_buf(size, altmap);
9492
95− ptr = sparse_buffer_alloc(size);
96− if (!ptr)
97− ptr = vmemmap_alloc_block(size, node);
98− return ptr;
93+ return vmemmap_alloc_block(size, node);
9994 }
10095
10196 static unsigned long __meminit vmem_altmap_next_pfn(struct vmem_altmap *altmap)
@@ -151,7 +146,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, · 49 unmodified lines
151146 start, end - 1);
152147 }
153148
154−pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
149+static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
155150 struct vmem_altmap *altmap,
156151 unsigned long ptpfn, unsigned long flags)
157152 {
@@ -195,7 +190,7 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) · 37 unmodified lines
195190 return p;
196191 }
197192
198−pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
193+static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)
199194 {
200195 pmd_t *pmd = pmd_offset(pud, addr);
201196 if (pmd_none(*pmd)) {
@@ -208,7 +203,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) · 6 unmodified lines
208203 return pmd;
209204 }
210205
211−pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
206+static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)
212207 {
213208 pud_t *pud = pud_offset(p4d, addr);
214209 if (pud_none(*pud)) {
@@ -221,7 +216,7 @@ pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node) · 6 unmodified lines
221216 return pud;
222217 }
223218
224−p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
219+static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node)
225220 {
226221 p4d_t *p4d = p4d_offset(pgd, addr);
227222 if (p4d_none(*p4d)) {
@@ -234,7 +229,7 @@ p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node) · 6 unmodified lines
234229 return p4d;
235230 }
236231
237−pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
232+static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
238233 {
239234 pgd_t *pgd = pgd_offset_k(addr);
240235 if (pgd_none(*pgd)) {
@@ -391,12 +386,17 @@ int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, · 150 unmodified lines
391386 void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
392387 unsigned long addr, unsigned long next)
393388 {
389+ WARN_ON_ONCE(!pmd_set_huge(pmd, virt_to_phys(p), PAGE_KERNEL));
394390 }
395391
396392 int __weak __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
397393 unsigned long addr, unsigned long next)
398394 {
399− return 0;
395+ if (!pmd_leaf(pmdp_get(pmd)))
396+ return 0;
397+ vmemmap_verify((pte_t *)pmd, node, addr, next);
398+
399+ return 1;
400400 }
401401
402402 int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,
@@ -652,26 +652,61 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) · 249 unmodified lines
652652 }
653653 }
654654
655+static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages,
656+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
657+{
658+ const unsigned int order = pgmap ? pgmap->vmemmap_shift : 0;
659+ const unsigned long pages_per_compound = 1UL << order;
660+
661+ VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION));
662+ VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION);
663+
664+ if (!vmemmap_can_optimize(altmap, pgmap))
665+ return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE);
666+
667+ if (order < PFN_SECTION_SHIFT) {
668+ VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound));
669+ return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound;
670+ }
671+
672+ VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION));
673+
674+ if (IS_ALIGNED(pfn, pages_per_compound))
675+ return VMEMMAP_RESERVE_NR;
676+
677+ return 0;
678+}
679+
655680 static struct page * __meminit populate_section_memmap(unsigned long pfn,
656681 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
657682 struct dev_pagemap *pgmap)
658683 {
659− return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap);
684+ struct page *page = __populate_section_memmap(pfn, nr_pages, nid, altmap,
685+ pgmap);
686+
687+ memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap));
688+
689+ return page;
660690 }
661691
662692 static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages,
663− struct vmem_altmap *altmap)
693+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
664694 {
665695 unsigned long start = (unsigned long) pfn_to_page(pfn);
666696 unsigned long end = start + nr_pages * sizeof(struct page);
667697
698+ memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap));
668699 vmemmap_free(start, end, altmap);
669700 }
701+
670702 static void free_map_bootmem(struct page *memmap)
671703 {
672704 unsigned long start = (unsigned long)memmap;
673705 unsigned long end = (unsigned long)(memmap + PAGES_PER_SECTION);
706+ unsigned long pfn = page_to_pfn(memmap);
674707
708+ memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION,
709+ NULL, NULL));
675710 vmemmap_free(start, end, NULL);
676711 }
677712
@@ -737,7 +772,7 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) · 59 unmodified lines
737772 * usage map, but still need to free the vmemmap range.
738773 */
739774 static void section_deactivate(unsigned long pfn, unsigned long nr_pages,
740− struct vmem_altmap *altmap)
775+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
741776 {
742777 struct mem_section *ms = __pfn_to_section(pfn);
743778 bool section_is_early = early_section(ms);
@@ -774,14 +809,10 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, · 30 unmodified lines
774809 * The memmap of early sections is always fully populated. See
775810 * section_activate() and pfn_valid() .
776811 */
777− if (!section_is_early) {
778− memmap_pages_add(-1L * (DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE)));
779− depopulate_section_memmap(pfn, nr_pages, altmap);
780− } else if (memmap) {
781− memmap_boot_pages_add(-1L * (DIV_ROUND_UP(nr_pages * sizeof(struct page),
782− PAGE_SIZE)));
812+ if (!section_is_early)
813+ depopulate_section_memmap(pfn, nr_pages, altmap, pgmap);
814+ else if (memmap)
783815 free_map_bootmem(memmap);
784− }
785816
786817 if (empty)
787818 ms->section_mem_map = (unsigned long)NULL;
@@ -823,10 +854,9 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, · 35 unmodified lines
823854
824855 memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap);
825856 if (!memmap) {
826− section_deactivate(pfn, nr_pages, altmap);
857+ section_deactivate(pfn, nr_pages, altmap, pgmap);
827858 return ERR_PTR(-ENOMEM);
828859 }
829− memmap_pages_add(DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE));
830860
831861 return memmap;
832862 }
@@ -885,13 +915,13 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, · 52 unmodified lines
885915 }
886916
887917 void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
888− struct vmem_altmap *altmap)
918+ struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
889919 {
890920 struct mem_section *ms = __pfn_to_section(pfn);
891921
892922 if (WARN_ON_ONCE(!valid_section(ms)))
893923 return;
894924
895− section_deactivate(pfn, nr_pages, altmap);
925+ section_deactivate(pfn, nr_pages, altmap, pgmap);
896926 }
897927 #endif /* CONFIG_MEMORY_HOTPLUG */
mm/sparse.c+5−63
@@ -201,13 +201,11 @@ static void __init memblocks_present(void) · 200 unmodified lines
201201 int i, nid;
202202
203203 #ifdef CONFIG_SPARSEMEM_EXTREME
204− if (unlikely(!mem_section)) {
205− unsigned long size, align;
204+ unsigned long size, align;
206205
207− size = sizeof(struct mem_section *) * NR_SECTION_ROOTS;
208− align = 1 << (INTERNODE_CACHE_SHIFT);
209− mem_section = memblock_alloc_or_panic(size, align);
210− }
206+ size = sizeof(struct mem_section *) * NR_SECTION_ROOTS;
207+ align = 1 << (INTERNODE_CACHE_SHIFT);
208+ mem_section = memblock_alloc_or_panic(size, align);
211209 #endif
212210
213211 for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid)
@@ -241,12 +239,9 @@ struct page __init *__populate_section_memmap(unsigned long pfn, · 27 unmodified lines
241239 struct dev_pagemap *pgmap)
242240 {
243241 unsigned long size = section_map_size();
244− struct page *map = sparse_buffer_alloc(size);
242+ struct page *map;
245243 phys_addr_t addr = __pa(MAX_DMA_ADDRESS);
246244
247− if (map)
248− return map;
249−
250245 map = memmap_alloc(size, size, addr, nid, false);
251246 if (!map)
252247 panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%pa\n",
@@ -256,55 +251,6 @@ struct page __init *__populate_section_memmap(unsigned long pfn, · 3 unmodified lines
256251 }
257252 #endif /* !CONFIG_SPARSEMEM_VMEMMAP */
258253
259−static void *sparsemap_buf __meminitdata;
260−static void *sparsemap_buf_end __meminitdata;
261−
262−static inline void __meminit sparse_buffer_free(unsigned long size)
263−{
264− WARN_ON(!sparsemap_buf || size == 0);
265− memblock_free(sparsemap_buf, size);
266−}
267−
268−static void __init sparse_buffer_init(unsigned long size, int nid)
269−{
270− phys_addr_t addr = __pa(MAX_DMA_ADDRESS);
271− WARN_ON(sparsemap_buf); /* forgot to call sparse_buffer_fini()? */
272− /*
273− * Pre-allocated buffer is mainly used by __populate_section_memmap
274− * and we want it to be properly aligned to the section size - this is
275− * especially the case for VMEMMAP which maps memmap to PMDs
276− */
277− sparsemap_buf = memmap_alloc(size, section_map_size(), addr, nid, true);
278− sparsemap_buf_end = sparsemap_buf + size;
279−}
280−
281−static void __init sparse_buffer_fini(void)
282−{
283− unsigned long size = sparsemap_buf_end - sparsemap_buf;
284−
285− if (sparsemap_buf && size > 0)
286− sparse_buffer_free(size);
287− sparsemap_buf = NULL;
288−}
289−
290−void * __meminit sparse_buffer_alloc(unsigned long size)
291−{
292− void *ptr = NULL;
293−
294− if (sparsemap_buf) {
295− ptr = (void *) roundup((unsigned long)sparsemap_buf, size);
296− if (ptr + size > sparsemap_buf_end)
297− ptr = NULL;
298− else {
299− /* Free redundant aligned space */
300− if ((unsigned long)(ptr - sparsemap_buf) > 0)
301− sparse_buffer_free((unsigned long)(ptr - sparsemap_buf));
302− sparsemap_buf = ptr + size;
303− }
304− }
305− return ptr;
306−}
307−
308254 void __weak __meminit vmemmap_populate_print_last(void)
309255 {
310256 }
@@ -362,8 +308,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 51 unmodified lines
362308 goto failed;
363309 }
364310
365− sparse_buffer_init(map_count * section_map_size(), nid);
366−
367311 sparse_vmemmap_init_nid_early(nid);
368312
369313 for_each_present_section_nr(pnum_begin, pnum) {
@@ -381,7 +325,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 11 unmodified lines
381325 __func__, nid);
382326 pnum_begin = pnum;
383327 sparse_usage_fini();
384− sparse_buffer_fini();
385328 goto failed;
386329 }
387330 memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page),
@@ -390,7 +333,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, · 2 unmodified lines
390333 }
391334 }
392335 sparse_usage_fini();
393− sparse_buffer_fini();
394336 return;
395337 failed:
396338 /*
mm/swap.c+53−4
@@ -160,13 +160,41 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · 159 unmodified lines
160160 int i;
161161 struct lruvec *lruvec = NULL;
162162 unsigned long flags = 0;
163+ struct folio_batch free_fbatch;
164+ bool is_lru_add = (move_fn == lru_add);
165+
166+ /*
167+ * If we're adding to the LRU, preemptively filter dead folios. Use
168+ * this dedicated folio batch for temp storage and deferred cleanup.
169+ */
170+ if (is_lru_add)
171+ folio_batch_init(&free_fbatch);
163172
164173 for (i = 0; i < folio_batch_count(fbatch); i++) {
165174 struct folio *folio = fbatch->folios[i];
166175
167176 /* block memcg migration while the folio moves between lru */
168− if (move_fn != lru_add && !folio_test_clear_lru(folio))
177+ if (!is_lru_add && !folio_test_clear_lru(folio))
178+ continue;
179+
180+ /*
181+ * Filter dead folios by moving them from the add batch to the temp
182+ * batch for freeing after this loop.
183+ *
184+ * We're bypassing normal cleanup. Clear flags that are not
185+ * applicable to dead folios.
186+ *
187+ * Since the folio may be part of a huge page, unqueue from
188+ * deferred split list to avoid a dangling list entry.
189+ */
190+ if (is_lru_add && folio_ref_freeze(folio, 1)) {
191+ __folio_clear_active(folio);
192+ __folio_clear_unevictable(folio);
193+ folio_unqueue_deferred_split(folio);
194+ fbatch->folios[i] = NULL;
195+ folio_batch_add(&free_fbatch, folio);
169196 continue;
197+ }
170198
171199 folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
172200 move_fn(lruvec, folio);
@@ -176,6 +204,13 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · 3 unmodified lines
176204
177205 if (lruvec)
178206 lruvec_unlock_irqrestore(lruvec, flags);
207+
208+ /* Cleanup filtered dead folios. */
209+ if (is_lru_add) {
210+ mem_cgroup_uncharge_folios(&free_fbatch);
211+ free_unref_folios(&free_fbatch);
212+ }
213+
179214 folios_put(fbatch);
180215 }
181216
@@ -509,10 +544,20 @@ void folio_add_lru(struct folio *folio) · 327 unmodified lines
509544 folio_test_unevictable(folio), folio);
510545 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
511546
512− /* see the comment in lru_gen_folio_seq() */
547+ /*
548+ * For refaulted workingset folios, set PG_active so they
549+ * can be added to active generations.
550+ * For prefaulted file folios, folio_mark_accessed() sets
551+ * PG_referenced so lru_gen_folio_seq() places them into
552+ * the second oldest generation.
553+ */
513554 if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
514− lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
515− folio_set_active(folio);
555+ lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
556+ if (folio_test_workingset(folio))
557+ folio_set_active(folio);
558+ else if (!folio_test_referenced(folio))
559+ folio_mark_accessed(folio);
560+ }
516561
517562 folio_batch_add_and_move(folio, lru_add);
518563 }
@@ -964,6 +1009,10 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) · 445 unmodified lines
9641009 struct folio *folio = folios->folios[i];
9651010 unsigned int nr_refs = refs ? refs[i] : 1;
9661011
1012+ /* Folio batch entry may have been preemptively removed during drain. */
1013+ if (!folio)
1014+ continue;
1015+
9671016 if (is_huge_zero_folio(folio))
9681017 continue;
9691018
mm/swap.h+32−58
@@ -3,11 +3,29 @@ · 2 unmodified lines
33 #define _MM_SWAP_H
44
55 #include <linux/atomic.h> /* for atomic_long_t */
6+#include <linux/mm.h> /* for PAGE_SHIFT */
67 struct mempolicy;
78 struct swap_iocb;
9+struct swap_memcg_table;
810
911 extern int page_cluster;
1012
13+#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
14+#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
15+#elif defined(MAX_PHYSMEM_BITS)
16+#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
17+#else
18+#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
19+#endif
20+
21+/* Swap table marker, 0x1 means shadow, 0x2 means PFN (SWP_TB_PFN_MARK) */
22+#define SWAP_CACHE_PFN_MARK_BITS 2
23+/* At least 2 bits are needed to distinguish SWP_TB_COUNT_MAX, 1 and 0 */
24+#define SWAP_COUNT_MIN_BITS 2
25+/* If there are enough bits besides PFN and marker, store zero flag inline */
26+#define SWAP_TABLE_HAS_ZEROFLAG ((BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - \
27+ SWAP_CACHE_PFN_BITS) > SWAP_COUNT_MIN_BITS)
28+
1129 #ifdef CONFIG_THP_SWAP
1230 #define SWAPFILE_CLUSTER HPAGE_PMD_NR
1331 #define swap_entry_order(order) (order)
@@ -38,6 +56,12 @@ struct swap_cluster_info { · 24 unmodified lines
3856 u8 order;
3957 atomic_long_t __rcu *table; /* Swap table entries, see mm/swap_table.h */
4058 unsigned int *extend_table; /* For large swap count, protected by ci->lock */
59+#ifdef CONFIG_MEMCG
60+ struct swap_memcg_table *memcg_table; /* Swap table entries' cgroup record */
61+#endif
62+#if !SWAP_TABLE_HAS_ZEROFLAG
63+ unsigned long *zero_bitmap;
64+#endif
4165 struct list_head list;
4266 };
4367
@@ -280,9 +304,9 @@ bool swap_cache_has_folio(swp_entry_t entry); · 236 unmodified lines
280304 struct folio *swap_cache_get_folio(swp_entry_t entry);
281305 void *swap_cache_get_shadow(swp_entry_t entry);
282306 void swap_cache_del_folio(struct folio *folio);
283−struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
284− struct mempolicy *mpol, pgoff_t ilx,
285− bool *alloced);
307+struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
308+ unsigned long orders, struct vm_fault *vmf,
309+ struct mempolicy *mpol, pgoff_t ilx);
286310 /* Below helpers require the caller to lock and pass in the swap cluster. */
287311 void __swap_cache_add_folio(struct swap_cluster_info *ci,
288312 struct folio *folio, swp_entry_t entry);
@@ -300,7 +324,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, · 11 unmodified lines
300324 struct mempolicy *mpol, pgoff_t ilx);
301325 struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
302326 struct vm_fault *vmf);
303−struct folio *swapin_folio(swp_entry_t entry, struct folio *folio);
327+struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders,
328+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx);
304329 void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
305330 unsigned long addr);
306331
@@ -309,49 +334,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio) · 2 unmodified lines
309334 return __swap_entry_to_info(folio->swap)->flags;
310335 }
311336
312−/*
313− * Return the count of contiguous swap entries that share the same
314− * zeromap status as the starting entry. If is_zeromap is not NULL,
315− * it will return the zeromap status of the starting entry.
316− */
317−static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
318− bool *is_zeromap)
319−{
320− struct swap_info_struct *sis = __swap_entry_to_info(entry);
321− unsigned long start = swp_offset(entry);
322− unsigned long end = start + max_nr;
323− bool first_bit;
324−
325− first_bit = test_bit(start, sis->zeromap);
326− if (is_zeromap)
327− *is_zeromap = first_bit;
328−
329− if (max_nr <= 1)
330− return max_nr;
331− if (first_bit)
332− return find_next_zero_bit(sis->zeromap, end, start) - start;
333− else
334− return find_next_bit(sis->zeromap, end, start) - start;
335−}
336−
337−static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
338−{
339− int i;
340−
341− /*
342− * While allocating a large folio and doing mTHP swapin, we need to
343− * ensure all entries are not cached, otherwise, the mTHP folio will
344− * be in conflict with the folio in swap cache.
345− */
346− for (i = 0; i < max_nr; i++) {
347− if (swap_cache_has_folio(entry))
348− return i;
349− entry.val++;
350− }
351−
352− return i;
353−}
354−
355337 #else /* CONFIG_SWAP */
356338 struct swap_iocb;
357339 static inline struct swap_cluster_info *swap_cluster_lock(
@@ -433,7 +415,9 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, · 75 unmodified lines
433415 return NULL;
434416 }
435417
436−static inline struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
418+static inline struct folio *swapin_sync(
419+ swp_entry_t entry, gfp_t flag, unsigned long orders,
420+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
437421 {
438422 return NULL;
439423 }
@@ -488,15 +472,5 @@ static inline unsigned int folio_swap_flags(struct folio *folio) · 48 unmodified lines
488472 return 0;
489473 }
490474
491−static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
492− bool *has_zeromap)
493−{
494− return 0;
495−}
496−
497−static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
498−{
499− return 0;
500−}
501475 #endif /* CONFIG_SWAP */
502476 #endif /* _MM_SWAP_H */
mm/swap_cgroup.c+0−172
@@ -1,172 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−#include <linux/swap_cgroup.h>
3−#include <linux/vmalloc.h>
4−#include <linux/mm.h>
5−
6−#include <linux/swapops.h> /* depends on mm.h include */
7−
8−static DEFINE_MUTEX(swap_cgroup_mutex);
9−
10−/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t) */
11−#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short))
12−#define ID_SHIFT (BITS_PER_TYPE(unsigned short))
13−#define ID_MASK (BIT(ID_SHIFT) - 1)
14−struct swap_cgroup {
15− atomic_t ids;
16−};
17−
18−struct swap_cgroup_ctrl {
19− struct swap_cgroup *map;
20−};
21−
22−static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES];
23−
24−static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map,
25− pgoff_t offset)
26−{
27− unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
28− unsigned int old_ids = atomic_read(&map[offset / ID_PER_SC].ids);
29−
30− BUILD_BUG_ON(!is_power_of_2(ID_PER_SC));
31− BUILD_BUG_ON(sizeof(struct swap_cgroup) != sizeof(atomic_t));
32−
33− return (old_ids >> shift) & ID_MASK;
34−}
35−
36−static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map,
37− pgoff_t offset,
38− unsigned short new_id)
39−{
40− unsigned short old_id;
41− struct swap_cgroup *sc = &map[offset / ID_PER_SC];
42− unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
43− unsigned int new_ids, old_ids = atomic_read(&sc->ids);
44−
45− do {
46− old_id = (old_ids >> shift) & ID_MASK;
47− new_ids = (old_ids & ~(ID_MASK << shift));
48− new_ids |= ((unsigned int)new_id) << shift;
49− } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids));
50−
51− return old_id;
52−}
53−
54−/**
55− * swap_cgroup_record - record mem_cgroup for a set of swap entries.
56− * These entries must belong to one single folio, and that folio
57− * must be being charged for swap space (swap out), and these
58− * entries must not have been charged
59− *
60− * @folio: the folio that the swap entry belongs to
61− * @id: mem_cgroup ID to be recorded
62− * @ent: the first swap entry to be recorded
63− */
64−void swap_cgroup_record(struct folio *folio, unsigned short id,
65− swp_entry_t ent)
66−{
67− unsigned int nr_ents = folio_nr_pages(folio);
68− struct swap_cgroup *map;
69− pgoff_t offset, end;
70− unsigned short old;
71−
72− offset = swp_offset(ent);
73− end = offset + nr_ents;
74− map = swap_cgroup_ctrl[swp_type(ent)].map;
75−
76− do {
77− old = __swap_cgroup_id_xchg(map, offset, id);
78− VM_BUG_ON(old);
79− } while (++offset != end);
80−}
81−
82−/**
83− * swap_cgroup_clear - clear mem_cgroup for a set of swap entries.
84− * These entries must be being uncharged from swap. They either
85− * belongs to one single folio in the swap cache (swap in for
86− * cgroup v1), or no longer have any users (slot freeing).
87− *
88− * @ent: the first swap entry to be recorded into
89− * @nr_ents: number of swap entries to be recorded
90− *
91− * Returns the existing old value.
92− */
93−unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
94−{
95− pgoff_t offset, end;
96− struct swap_cgroup *map;
97− unsigned short old, iter = 0;
98−
99− offset = swp_offset(ent);
100− end = offset + nr_ents;
101− map = swap_cgroup_ctrl[swp_type(ent)].map;
102−
103− do {
104− old = __swap_cgroup_id_xchg(map, offset, 0);
105− if (!iter)
106− iter = old;
107− VM_BUG_ON(iter != old);
108− } while (++offset != end);
109−
110− return old;
111−}
112−
113−/**
114− * lookup_swap_cgroup_id - lookup mem_cgroup id tied to swap entry
115− * @ent: swap entry to be looked up.
116− *
117− * Returns ID of mem_cgroup at success. 0 at failure. (0 is invalid ID)
118− */
119−unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
120−{
121− struct swap_cgroup_ctrl *ctrl;
122−
123− if (mem_cgroup_disabled())
124− return 0;
125−
126− ctrl = &swap_cgroup_ctrl[swp_type(ent)];
127− return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent));
128−}
129−
130−int swap_cgroup_swapon(int type, unsigned long max_pages)
131−{
132− struct swap_cgroup *map;
133− struct swap_cgroup_ctrl *ctrl;
134−
135− if (mem_cgroup_disabled())
136− return 0;
137−
138− BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=
139− sizeof(struct swap_cgroup));
140− map = vzalloc(DIV_ROUND_UP(max_pages, ID_PER_SC) *
141− sizeof(struct swap_cgroup));
142− if (!map)
143− goto nomem;
144−
145− ctrl = &swap_cgroup_ctrl[type];
146− mutex_lock(&swap_cgroup_mutex);
147− ctrl->map = map;
148− mutex_unlock(&swap_cgroup_mutex);
149−
150− return 0;
151−nomem:
152− pr_info("couldn't allocate enough memory for swap_cgroup\n");
153− pr_info("swap_cgroup can be disabled by swapaccount=0 boot option\n");
154− return -ENOMEM;
155−}
156−
157−void swap_cgroup_swapoff(int type)
158−{
159− struct swap_cgroup *map;
160− struct swap_cgroup_ctrl *ctrl;
161−
162− if (mem_cgroup_disabled())
163− return;
164−
165− mutex_lock(&swap_cgroup_mutex);
166− ctrl = &swap_cgroup_ctrl[type];
167− map = ctrl->map;
168− ctrl->map = NULL;
169− mutex_unlock(&swap_cgroup_mutex);
170−
171− vfree(map);
172−}
mm/swap_state.c+308−225
@@ -137,8 +137,71 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 136 unmodified lines
137137 return NULL;
138138 }
139139
140−void __swap_cache_add_folio(struct swap_cluster_info *ci,
141− struct folio *folio, swp_entry_t entry)
140+/**
141+ * __swap_cache_add_check - Check if a range is suitable for adding a folio.
142+ * @ci: The locked swap cluster
143+ * @targ_entry: The target swap entry to check, will be rounded down by @nr
144+ * @nr: Number of slots to check, must be a power of 2
145+ * @shadowp: Returns the shadow value if one exists in the range
146+ * @memcg_id: Returns the memory cgroup id, NULL to ignore cgroup check
147+ *
148+ * Check if all slots covered by given range have a swap count >= 1.
149+ * Retrieves the shadow if there is one. If @memcg_id is not NULL, also
150+ * checks if all slots belong to the same cgroup and return the cgroup
151+ * private id.
152+ *
153+ * Context: Caller must lock the cluster.
154+ * Return: 0 if success, error code if failed.
155+ */
156+static int __swap_cache_add_check(struct swap_cluster_info *ci,
157+ swp_entry_t targ_entry,
158+ unsigned long nr, void **shadowp,
159+ unsigned short *memcg_id)
160+{
161+ unsigned int ci_off, ci_end;
162+ unsigned long old_tb;
163+ bool is_zero;
164+
165+ lockdep_assert_held(&ci->lock);
166+
167+ /*
168+ * If the target slot is not swapped out or already cached, return
169+ * -ENOENT or -EEXIST. If the batch is not suitable, could be a
170+ * race with concurrent free or cache add, return -EBUSY.
171+ */
172+ if (unlikely(!ci->table))
173+ return -ENOENT;
174+ ci_off = swp_cluster_offset(targ_entry);
175+ old_tb = __swap_table_get(ci, ci_off);
176+ if (swp_tb_is_folio(old_tb))
177+ return -EEXIST;
178+ if (!__swp_tb_get_count(old_tb))
179+ return -ENOENT;
180+ if (shadowp && swp_tb_is_shadow(old_tb))
181+ *shadowp = swp_tb_to_shadow(old_tb);
182+ if (memcg_id)
183+ *memcg_id = __swap_cgroup_get(ci, ci_off);
184+
185+ if (nr == 1)
186+ return 0;
187+
188+ is_zero = __swap_table_test_zero(ci, ci_off);
189+ ci_off = round_down(ci_off, nr);
190+ ci_end = ci_off + nr;
191+ do {
192+ old_tb = __swap_table_get(ci, ci_off);
193+ if (unlikely(swp_tb_is_folio(old_tb) ||
194+ !__swp_tb_get_count(old_tb) ||
195+ is_zero != __swap_table_test_zero(ci, ci_off) ||
196+ (memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
197+ return -EBUSY;
198+ } while (++ci_off < ci_end);
199+
200+ return 0;
201+}
202+
203+static void __swap_cache_do_add_folio(struct swap_cluster_info *ci,
204+ struct folio *folio, swp_entry_t entry)
142205 {
143206 unsigned int ci_off = swp_cluster_offset(entry), ci_end;
144207 unsigned long nr_pages = folio_nr_pages(folio);
@@ -153,88 +216,42 @@ void __swap_cache_add_folio(struct swap_cluster_info *ci, · 8 unmodified lines
153216 do {
154217 old_tb = __swap_table_get(ci, ci_off);
155218 VM_WARN_ON_ONCE(swp_tb_is_folio(old_tb));
156− __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));
219+ __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
157220 } while (++ci_off < ci_end);
158221
159222 folio_ref_add(folio, nr_pages);
160223 folio_set_swapcache(folio);
161224 folio->swap = entry;
162−
163− node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
164− lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
165225 }
166226
167227 /**
168− * swap_cache_add_folio - Add a folio into the swap cache.
228+ * __swap_cache_add_folio - Add a folio to the swap cache and update stats.
229+ * @ci: The locked swap cluster.
169230 * @folio: The folio to be added.
170231 * @entry: The swap entry corresponding to the folio.
171− * @gfp: gfp_mask for XArray node allocation.
172− * @shadowp: If a shadow is found, return the shadow.
173232 *
174− * Context: Caller must ensure @entry is valid and protect the swap device
175− * with reference count or locks.
233+ * Unconditionally add a folio to the swap cache. The caller must ensure
234+ * all slots are usable and have no conflicts. This assigns entry to
235+ * @folio->swap, increases folio refcount by the number of pages, and
236+ * updates swap cache stats.
237+ *
238+ * Context: Caller must ensure the folio is locked and lock the cluster
239+ * that holds the entries.
176240 */
177−static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
178− void **shadowp)
241+void __swap_cache_add_folio(struct swap_cluster_info *ci,
242+ struct folio *folio, swp_entry_t entry)
179243 {
180− int err;
181− void *shadow = NULL;
182− unsigned long old_tb;
183− struct swap_info_struct *si;
184− struct swap_cluster_info *ci;
185− unsigned int ci_start, ci_off, ci_end;
186244 unsigned long nr_pages = folio_nr_pages(folio);
187245
188− si = __swap_entry_to_info(entry);
189− ci_start = swp_cluster_offset(entry);
190− ci_end = ci_start + nr_pages;
191− ci_off = ci_start;
192− ci = swap_cluster_lock(si, swp_offset(entry));
193− if (unlikely(!ci->table)) {
194− err = -ENOENT;
195− goto failed;
196− }
197− do {
198− old_tb = __swap_table_get(ci, ci_off);
199− if (unlikely(swp_tb_is_folio(old_tb))) {
200− err = -EEXIST;
201− goto failed;
202− }
203− if (unlikely(!__swp_tb_get_count(old_tb))) {
204− err = -ENOENT;
205− goto failed;
206− }
207− if (swp_tb_is_shadow(old_tb))
208− shadow = swp_tb_to_shadow(old_tb);
209− } while (++ci_off < ci_end);
210− __swap_cache_add_folio(ci, folio, entry);
211− swap_cluster_unlock(ci);
212− if (shadowp)
213− *shadowp = shadow;
214− return 0;
215−
216−failed:
217− swap_cluster_unlock(ci);
218− return err;
246+ __swap_cache_do_add_folio(ci, folio, entry);
247+ node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
248+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
219249 }
220250
221−/**
222− * __swap_cache_del_folio - Removes a folio from the swap cache.
223− * @ci: The locked swap cluster.
224− * @folio: The folio.
225− * @entry: The first swap entry that the folio corresponds to.
226− * @shadow: shadow value to be filled in the swap cache.
227− *
228− * Removes a folio from the swap cache and fills a shadow in place.
229− * This won't put the folio's refcount. The caller has to do that.
230− *
231− * Context: Caller must ensure the folio is locked and in the swap cache
232− * using the index of @entry, and lock the cluster that holds the entries.
233− */
234−void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
235− swp_entry_t entry, void *shadow)
251+static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
252+ struct folio *folio,
253+ swp_entry_t entry, void *shadow)
236254 {
237− int count;
238255 unsigned long old_tb;
239256 struct swap_info_struct *si;
240257 unsigned int ci_start, ci_off, ci_end;
@@ -254,19 +271,17 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 13 unmodified lines
254271 old_tb = __swap_table_get(ci, ci_off);
255272 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) ||
256273 swp_tb_to_folio(old_tb) != folio);
257− count = __swp_tb_get_count(old_tb);
258− if (count)
274+ if (__swp_tb_get_count(old_tb))
259275 folio_swapped = true;
260276 else
261277 need_free = true;
262− /* If shadow is NULL, we sets an empty shadow. */
263− __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow, count));
278+ /* If shadow is NULL, we set an empty shadow. */
279+ __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow,
280+ __swp_tb_get_flags(old_tb)));
264281 } while (++ci_off < ci_end);
265282
266283 folio->swap.val = 0;
267284 folio_clear_swapcache(folio);
268− node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
269− lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
270285
271286 if (!folio_swapped) {
272287 __swap_cluster_free_entries(si, ci, ci_start, nr_pages);
@@ -279,6 +294,29 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 6 unmodified lines
279294 }
280295 }
281296
297+/**
298+ * __swap_cache_del_folio - Removes a folio from the swap cache.
299+ * @ci: The locked swap cluster.
300+ * @folio: The folio.
301+ * @entry: The first swap entry that the folio corresponds to.
302+ * @shadow: shadow value to be filled in the swap cache.
303+ *
304+ * Removes a folio from the swap cache and fills a shadow in place.
305+ * This won't put the folio's refcount. The caller has to do that.
306+ *
307+ * Context: Caller must ensure the folio is locked and in the swap cache
308+ * using the index of @entry, and lock the cluster that holds the entries.
309+ */
310+void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
311+ swp_entry_t entry, void *shadow)
312+{
313+ unsigned long nr_pages = folio_nr_pages(folio);
314+
315+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
316+ node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
317+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
318+}
319+
282320 /**
283321 * swap_cache_del_folio - Removes a folio from the swap cache.
284322 * @folio: The folio.
@@ -333,7 +371,7 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 48 unmodified lines
333371 do {
334372 old_tb = __swap_table_get(ci, ci_off);
335373 WARN_ON_ONCE(!swp_tb_is_folio(old_tb) || swp_tb_to_folio(old_tb) != old);
336− __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));
374+ __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
337375 } while (++ci_off < ci_end);
338376
339377 /*
@@ -350,6 +388,153 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 10 unmodified lines
350388 }
351389 }
352390
391+/*
392+ * Try to allocate a folio of given order in the swap cache.
393+ *
394+ * This helper resolves the potential races of swap allocation
395+ * and prepares a folio to be used for swap IO. May return following
396+ * value:
397+ *
398+ * -ENOMEM / -EBUSY: Order is too large or in conflict with sub slot,
399+ * caller should shrink the order and retry
400+ * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
401+ * should abort or try to use the cached folio instead
402+ */
403+static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
404+ swp_entry_t targ_entry, gfp_t gfp,
405+ unsigned int order, struct vm_fault *vmf,
406+ struct mempolicy *mpol, pgoff_t ilx)
407+{
408+ int err;
409+ swp_entry_t entry;
410+ struct folio *folio;
411+ void *shadow = NULL;
412+ unsigned short memcg_id;
413+ unsigned long address, nr_pages = 1UL << order;
414+ struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
415+
416+ VM_WARN_ON_ONCE(nr_pages > SWAPFILE_CLUSTER);
417+ entry.val = round_down(targ_entry.val, nr_pages);
418+
419+ /* Check if the slot and range are available, skip allocation if not */
420+ spin_lock(&ci->lock);
421+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
422+ spin_unlock(&ci->lock);
423+ if (unlikely(err))
424+ return ERR_PTR(err);
425+
426+ /*
427+ * Limit THP gfp. The limitation is a no-op for typical
428+ * GFP_HIGHUSER_MOVABLE but matters for shmem.
429+ */
430+ if (order)
431+ gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
432+
433+ if (mpol || !vmf) {
434+ folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
435+ } else {
436+ address = round_down(vmf->address, PAGE_SIZE << order);
437+ folio = vma_alloc_folio(gfp, order, vmf->vma, address);
438+ }
439+ if (unlikely(!folio))
440+ return ERR_PTR(-ENOMEM);
441+
442+ /* Double check the range is still not in conflict */
443+ spin_lock(&ci->lock);
444+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
445+ if (unlikely(err)) {
446+ spin_unlock(&ci->lock);
447+ folio_put(folio);
448+ return ERR_PTR(err);
449+ }
450+
451+ __folio_set_locked(folio);
452+ __folio_set_swapbacked(folio);
453+ __swap_cache_do_add_folio(ci, folio, entry);
454+ spin_unlock(&ci->lock);
455+
456+ if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
457+ vmf ? vmf->vma->vm_mm : NULL, gfp)) {
458+ spin_lock(&ci->lock);
459+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
460+ spin_unlock(&ci->lock);
461+ folio_unlock(folio);
462+ /* nr_pages refs from swap cache, 1 from allocation */
463+ folio_put_refs(folio, nr_pages + 1);
464+ count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
465+ return ERR_PTR(-ENOMEM);
466+ }
467+
468+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
469+ spin_lock(&ci->lock);
470+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
471+ spin_unlock(&ci->lock);
472+ folio_unlock(folio);
473+ /* nr_pages refs from swap cache, 1 from allocation */
474+ folio_put_refs(folio, nr_pages + 1);
475+ return ERR_PTR(-ENOMEM);
476+ }
477+
478+ /* memsw uncharges swap when folio is added to swap cache */
479+ memcg1_swapin(folio);
480+ if (shadow)
481+ workingset_refault(folio, shadow);
482+
483+ node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
484+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
485+
486+ /* Caller will initiate read into locked new_folio */
487+ folio_add_lru(folio);
488+ return folio;
489+}
490+
491+/**
492+ * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
493+ * @targ_entry: swap entry indicating the target slot
494+ * @gfp: memory allocation flags
495+ * @orders: allocation orders, must be non zero
496+ * @vmf: fault information
497+ * @mpol: NUMA memory allocation policy to be applied
498+ * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
499+ *
500+ * Allocate a folio in the swap cache for one swap slot, typically before
501+ * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
502+ * @targ_entry must have a non-zero swap count (swapped out).
503+ *
504+ * Context: Caller must protect the swap device with reference count or locks.
505+ * Return: Returns the folio if allocation succeeded and folio is in the swap
506+ * cache. Returns error code if failed due to race, OOM or invalid arguments.
507+ */
508+struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
509+ unsigned long orders, struct vm_fault *vmf,
510+ struct mempolicy *mpol, pgoff_t ilx)
511+{
512+ int order, err;
513+ struct folio *ret;
514+ struct swap_cluster_info *ci;
515+
516+ ci = __swap_entry_to_cluster(targ_entry);
517+ order = highest_order(orders);
518+
519+ /* orders must be non-zero, and must not exceed cluster size. */
520+ if (WARN_ON_ONCE(!orders || (1UL << order) > SWAPFILE_CLUSTER))
521+ return ERR_PTR(-EINVAL);
522+
523+ do {
524+ ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
525+ vmf, mpol, ilx);
526+ if (!IS_ERR(ret))
527+ break;
528+ err = PTR_ERR(ret);
529+ if (!order || (err && err != -EBUSY && err != -ENOMEM))
530+ break;
531+ count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
532+ order = next_order(&orders, order);
533+ } while (orders);
534+
535+ return ret;
536+}
537+
353538 /*
354539 * If we are the only user, then try to free up the swap cache.
355540 *
@@ -448,140 +633,64 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, · 92 unmodified lines
448633 }
449634 }
450635
451−/**
452− * __swap_cache_prepare_and_add - Prepare the folio and add it to swap cache.
453− * @entry: swap entry to be bound to the folio.
454− * @folio: folio to be added.
455− * @gfp: memory allocation flags for charge, can be 0 if @charged if true.
456− * @charged: if the folio is already charged.
457− *
458− * Update the swap_map and add folio as swap cache, typically before swapin.
459− * All swap slots covered by the folio must have a non-zero swap count.
460− *
461− * Context: Caller must protect the swap device with reference count or locks.
462− * Return: Returns the folio being added on success. Returns the existing folio
463− * if @entry is already cached. Returns NULL if raced with swapin or swapoff.
464− */
465−static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
466− struct folio *folio,
467− gfp_t gfp, bool charged)
636+static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
637+ struct mempolicy *mpol, pgoff_t ilx,
638+ struct swap_iocb **plug, bool readahead)
468639 {
469− struct folio *swapcache = NULL;
470− void *shadow;
471− int ret;
472−
473− __folio_set_locked(folio);
474− __folio_set_swapbacked(folio);
475−
476− if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry))
477− goto failed;
640+ struct folio *folio;
478641
479− for (;;) {
480− ret = swap_cache_add_folio(folio, entry, &shadow);
481− if (!ret)
482− break;
642+ do {
643+ folio = swap_cache_get_folio(entry);
644+ if (folio)
645+ return folio;
646+ folio = swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
647+ } while (PTR_ERR(folio) == -EEXIST);
483648
484− /*
485− * Large order allocation needs special handling on
486− * race: if a smaller folio exists in cache, swapin needs
487− * to fallback to order 0, and doing a swap cache lookup
488− * might return a folio that is irrelevant to the faulting
489− * entry because @entry is aligned down. Just return NULL.
490− */
491− if (ret != -EEXIST || folio_test_large(folio))
492− goto failed;
649+ if (IS_ERR_OR_NULL(folio))
650+ return NULL;
493651
494− swapcache = swap_cache_get_folio(entry);
495− if (swapcache)
496− goto failed;
652+ swap_read_folio(folio, plug);
653+ if (readahead) {
654+ folio_set_readahead(folio);
655+ count_vm_event(SWAP_RA);
497656 }
498657
499− memcg1_swapin(entry, folio_nr_pages(folio));
500− if (shadow)
501− workingset_refault(folio, shadow);
502−
503− /* Caller will initiate read into locked folio */
504− folio_add_lru(folio);
505658 return folio;
506−
507−failed:
508− folio_unlock(folio);
509− return swapcache;
510659 }
511660
512661 /**
513− * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
514− * @entry: the swapped out swap entry to be binded to the folio.
515− * @gfp_mask: memory allocation flags
662+ * swapin_sync - swap-in one or multiple entries skipping readahead.
663+ * @entry: swap entry indicating the target slot
664+ * @gfp: memory allocation flags
665+ * @orders: allocation orders
666+ * @vmf: fault information
516667 * @mpol: NUMA memory allocation policy to be applied
517668 * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
518− * @new_page_allocated: sets true if allocation happened, false otherwise
519669 *
520− * Allocate a folio in the swap cache for one swap slot, typically before
521− * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
522− * @entry must have a non-zero swap count (swapped out).
523− * Currently only supports order 0.
670+ * This allocates a folio suitable for given @orders, or returns the
671+ * existing folio in the swap cache for @entry. This initiates the IO, too,
672+ * if needed. @entry is rounded down if @orders allow large allocation.
524673 *
525− * Context: Caller must protect the swap device with reference count or locks.
526− * Return: Returns the existing folio if @entry is cached already. Returns
527− * NULL if failed due to -ENOMEM or @entry have a swap count < 1.
674+ * Context: Caller must ensure @entry is valid and pin the swap device with refcount.
675+ * Return: Returns the folio on success, error code if failed.
528676 */
529−struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
530− struct mempolicy *mpol, pgoff_t ilx,
531− bool *new_page_allocated)
677+struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
678+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
532679 {
533− struct swap_info_struct *si = __swap_entry_to_info(entry);
534680 struct folio *folio;
535− struct folio *result = NULL;
536681
537− *new_page_allocated = false;
538− /* Check the swap cache again for readahead path. */
539− folio = swap_cache_get_folio(entry);
540− if (folio)
541− return folio;
542−
543− /* Skip allocation for unused and bad swap slot for readahead. */
544− if (!swap_entry_swapped(si, entry))
545− return NULL;
546−
547− /* Allocate a new folio to be added into the swap cache. */
548− folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
549− if (!folio)
550− return NULL;
551− /* Try add the new folio, returns existing folio or NULL on failure. */
552− result = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
553− if (result == folio)
554− *new_page_allocated = true;
555− else
556− folio_put(folio);
557− return result;
558−}
682+ do {
683+ folio = swap_cache_get_folio(entry);
684+ if (folio)
685+ return folio;
686+ folio = swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
687+ } while (PTR_ERR(folio) == -EEXIST);
559688
560−/**
561− * swapin_folio - swap-in one or multiple entries skipping readahead.
562− * @entry: starting swap entry to swap in
563− * @folio: a new allocated and charged folio
564− *
565− * Reads @entry into @folio, @folio will be added to the swap cache.
566− * If @folio is a large folio, the @entry will be rounded down to align
567− * with the folio size.
568− *
569− * Return: returns pointer to @folio on success. If folio is a large folio
570− * and this raced with another swapin, NULL will be returned to allow fallback
571− * to order 0. Else, if another folio was already added to the swap cache,
572− * return that swap cache folio instead.
573− */
574−struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
575−{
576− struct folio *swapcache;
577− pgoff_t offset = swp_offset(entry);
578− unsigned long nr_pages = folio_nr_pages(folio);
689+ if (IS_ERR(folio))
690+ return folio;
579691
580− entry = swp_entry(swp_type(entry), round_down(offset, nr_pages));
581− swapcache = __swap_cache_prepare_and_add(entry, folio, 0, true);
582− if (swapcache == folio)
583− swap_read_folio(folio, NULL);
584− return swapcache;
692+ swap_read_folio(folio, NULL);
693+ return folio;
585694 }
586695
587696 /*
@@ -595,7 +704,6 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 7 unmodified lines
595704 struct swap_iocb **plug)
596705 {
597706 struct swap_info_struct *si;
598− bool page_allocated;
599707 struct mempolicy *mpol;
600708 pgoff_t ilx;
601709 struct folio *folio;
@@ -605,13 +713,9 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 3 unmodified lines
605713 return NULL;
606714
607715 mpol = get_vma_policy(vma, addr, 0, &ilx);
608− folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
609− &page_allocated);
716+ folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false);
610717 mpol_cond_put(mpol);
611718
612− if (page_allocated)
613− swap_read_folio(folio, plug);
614−
615719 put_swap_device(si);
616720 return folio;
617721 }
@@ -696,7 +800,7 @@ static unsigned long swapin_nr_pages(unsigned long offset) · 78 unmodified lines
696800 * are fairly likely to have been swapped out from the same node.
697801 */
698802 struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
699− struct mempolicy *mpol, pgoff_t ilx)
803+ struct mempolicy *mpol, pgoff_t ilx)
700804 {
701805 struct folio *folio;
702806 unsigned long entry_offset = swp_offset(entry);
@@ -706,7 +810,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 3 unmodified lines
706810 struct swap_info_struct *si = __swap_entry_to_info(entry);
707811 struct blk_plug plug;
708812 struct swap_iocb *splug = NULL;
709− bool page_allocated;
813+ swp_entry_t ra_entry;
710814
711815 mask = swapin_nr_pages(offset) - 1;
712816 if (!mask)
@@ -723,18 +827,11 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 10 unmodified lines
723827 blk_start_plug(&plug);
724828 for (offset = start_offset; offset <= end_offset ; offset++) {
725829 /* Ok, do the async read-ahead now */
726− folio = swap_cache_alloc_folio(
727− swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx,
728− &page_allocated);
830+ ra_entry = swp_entry(swp_type(entry), offset);
831+ folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx,
832+ &splug, offset != entry_offset);
729833 if (!folio)
730834 continue;
731− if (page_allocated) {
732− swap_read_folio(folio, &splug);
733− if (offset != entry_offset) {
734− folio_set_readahead(folio);
735− count_vm_event(SWAP_RA);
736− }
737− }
738835 folio_put(folio);
739836 }
740837 blk_finish_plug(&plug);
@@ -742,11 +839,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 1 unmodified line
742839 lru_add_drain(); /* Push any new pages onto the LRU now */
743840 skip:
744841 /* The page was likely read above, so no need for plugging here */
745− folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
746− &page_allocated);
747− if (unlikely(page_allocated))
748− swap_read_folio(folio, NULL);
749− return folio;
842+ return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
750843 }
751844
752845 static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
@@ -812,8 +905,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 59 unmodified lines
812905 pte_t *pte = NULL, pentry;
813906 int win;
814907 unsigned long start, end, addr;
815− pgoff_t ilx;
816− bool page_allocated;
908+ pgoff_t ilx = targ_ilx;
817909
818910 win = swap_vma_ra_win(vmf, &start, &end);
819911 if (win == 1)
@@ -847,19 +939,12 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 27 unmodified lines
847939 if (!si)
848940 continue;
849941 }
850− folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
851− &page_allocated);
942+ folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx,
943+ &splug, addr != vmf->address);
852944 if (si)
853945 put_swap_device(si);
854946 if (!folio)
855947 continue;
856− if (page_allocated) {
857− swap_read_folio(folio, &splug);
858− if (addr != vmf->address) {
859− folio_set_readahead(folio);
860− count_vm_event(SWAP_RA);
861− }
862− }
863948 folio_put(folio);
864949 }
865950 if (pte)
@@ -869,10 +954,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 3 unmodified lines
869954 lru_add_drain();
870955 skip:
871956 /* The folio was likely read above, so no need for plugging here */
872− folio = swap_cache_alloc_folio(targ_entry, gfp_mask, mpol, targ_ilx,
873− &page_allocated);
874− if (unlikely(page_allocated))
875− swap_read_folio(folio, NULL);
957+ folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
958+ NULL, false);
876959 return folio;
877960 }
878961
mm/swap_table.h+149−30
@@ -11,6 +11,11 @@ struct swap_table { · 10 unmodified lines
1111 atomic_long_t entries[SWAPFILE_CLUSTER];
1212 };
1313
14+/* For storing memcg private id */
15+struct swap_memcg_table {
16+ unsigned short id[SWAPFILE_CLUSTER];
17+};
18+
1419 #define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)
1520
1621 /*
@@ -21,12 +26,14 @@ struct swap_table { · 4 unmodified lines
2126 * Swap table entry type and bits layouts:
2227 *
2328 * NULL: |---------------- 0 ---------------| - Free slot
24− * Shadow: | SWAP_COUNT |---- SHADOW_VAL ---|1| - Swapped out slot
25− * PFN: | SWAP_COUNT |------ PFN -------|10| - Cached slot
29+ * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
30+ * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
2631 * Pointer: |----------- Pointer ----------|100| - (Unused)
2732 * Bad: |------------- 1 -------------|1000| - Bad slot
2833 *
29− * SWAP_COUNT is `SWP_TB_COUNT_BITS` long, each entry is an atomic long.
34+ * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
35+ * and together they form the `SWP_TB_FLAGS_BITS` wide flags field.
36+ * Each entry is an atomic long.
3037 *
3138 * Usages:
3239 *
@@ -49,14 +56,6 @@ struct swap_table { · 16 unmodified lines
4956 * - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
5057 */
5158
52−#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
53−#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
54−#elif defined(MAX_PHYSMEM_BITS)
55−#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
56−#else
57−#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
58−#endif
59−
6059 /* NULL Entry, all 0 */
6160 #define SWP_TB_NULL 0UL
6261
@@ -64,22 +63,26 @@ struct swap_table { · 1 unmodified line
6463 #define SWP_TB_SHADOW_MARK 0b1UL
6564
6665 /* Cached: PFN */
67−#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWP_TB_PFN_MARK_BITS)
66+#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWAP_CACHE_PFN_MARK_BITS)
6867 #define SWP_TB_PFN_MARK 0b10UL
69−#define SWP_TB_PFN_MARK_BITS 2
70−#define SWP_TB_PFN_MARK_MASK (BIT(SWP_TB_PFN_MARK_BITS) - 1)
68+#define SWP_TB_PFN_MARK_MASK (BIT(SWAP_CACHE_PFN_MARK_BITS) - 1)
7169
72−/* SWAP_COUNT part for PFN or shadow, the width can be shrunk or extended */
73−#define SWP_TB_COUNT_BITS min(4, BITS_PER_LONG - SWP_TB_PFN_BITS)
70+/* Flags: For PFN or shadow, contains SWAP_COUNT, width changes */
71+#define SWP_TB_FLAGS_BITS min(5, BITS_PER_LONG - SWP_TB_PFN_BITS)
72+#define SWP_TB_COUNT_BITS (SWP_TB_FLAGS_BITS - SWAP_TABLE_HAS_ZEROFLAG)
73+#define SWP_TB_FLAGS_MASK (~((~0UL) >> SWP_TB_FLAGS_BITS))
7474 #define SWP_TB_COUNT_MASK (~((~0UL) >> SWP_TB_COUNT_BITS))
75+#define SWP_TB_FLAGS_SHIFT (BITS_PER_LONG - SWP_TB_FLAGS_BITS)
7576 #define SWP_TB_COUNT_SHIFT (BITS_PER_LONG - SWP_TB_COUNT_BITS)
7677 #define SWP_TB_COUNT_MAX ((1 << SWP_TB_COUNT_BITS) - 1)
78+/* The first flag is zero bit (SWAP_TABLE_HAS_ZEROFLAG) */
79+#define SWP_TB_ZERO_FLAG BIT(BITS_PER_LONG - SWP_TB_FLAGS_BITS)
7780
7881 /* Bad slot: ends with 0b1000 and rests of bits are all 1 */
7982 #define SWP_TB_BAD ((~0UL) << 3)
8083
8184 /* Macro for shadow offset calculation */
82−#define SWAP_COUNT_SHIFT SWP_TB_COUNT_BITS
85+#define SWAP_COUNT_SHIFT SWP_TB_FLAGS_BITS
8386
8487 /*
8588 * Helpers for casting one type of info into a swap table entry.
@@ -97,40 +100,47 @@ static inline unsigned long __count_to_swp_tb(unsigned char count) · 11 unmodified lines
97100 * used (count > 0 && count < SWP_TB_COUNT_MAX), and
98101 * overflow (count == SWP_TB_COUNT_MAX).
99102 */
100− BUILD_BUG_ON(SWP_TB_COUNT_MAX < 2 || SWP_TB_COUNT_BITS < 2);
103+ BUILD_BUG_ON(SWP_TB_COUNT_BITS < SWAP_COUNT_MIN_BITS);
101104 VM_WARN_ON(count > SWP_TB_COUNT_MAX);
102105 return ((unsigned long)count) << SWP_TB_COUNT_SHIFT;
103106 }
104107
105−static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned int count)
108+static inline unsigned long __flags_to_swp_tb(unsigned char flags)
109+{
110+ BUILD_BUG_ON(SWP_TB_FLAGS_BITS > BITS_PER_BYTE);
111+ VM_WARN_ON(flags >> SWP_TB_FLAGS_BITS);
112+ return ((unsigned long)flags) << SWP_TB_FLAGS_SHIFT;
113+}
114+
115+static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned char flags)
106116 {
107117 unsigned long swp_tb;
108118
109119 BUILD_BUG_ON(sizeof(unsigned long) != sizeof(void *));
110120 BUILD_BUG_ON(SWAP_CACHE_PFN_BITS >
111− (BITS_PER_LONG - SWP_TB_PFN_MARK_BITS - SWP_TB_COUNT_BITS));
121+ (BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - SWP_TB_FLAGS_BITS));
112122
113− swp_tb = (pfn << SWP_TB_PFN_MARK_BITS) | SWP_TB_PFN_MARK;
114− VM_WARN_ON_ONCE(swp_tb & SWP_TB_COUNT_MASK);
123+ swp_tb = (pfn << SWAP_CACHE_PFN_MARK_BITS) | SWP_TB_PFN_MARK;
124+ VM_WARN_ON_ONCE(swp_tb & SWP_TB_FLAGS_MASK);
115125
116− return swp_tb | __count_to_swp_tb(count);
126+ return swp_tb | __flags_to_swp_tb(flags);
117127 }
118128
119−static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned int count)
129+static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned char flags)
120130 {
121− return pfn_to_swp_tb(folio_pfn(folio), count);
131+ return pfn_to_swp_tb(folio_pfn(folio), flags);
122132 }
123133
124−static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned int count)
134+static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned char flags)
125135 {
126136 BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) !=
127137 BITS_PER_BYTE * sizeof(unsigned long));
128138 BUILD_BUG_ON((unsigned long)xa_mk_value(0) != SWP_TB_SHADOW_MARK);
129139
130140 VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow));
131− VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_COUNT_MASK));
141+ VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_FLAGS_MASK));
132142
133− return (unsigned long)shadow | __count_to_swp_tb(count) | SWP_TB_SHADOW_MARK;
143+ return (unsigned long)shadow | SWP_TB_SHADOW_MARK | __flags_to_swp_tb(flags);
134144 }
135145
136146 /*
@@ -168,14 +178,14 @@ static inline bool swp_tb_is_countable(unsigned long swp_tb) · 31 unmodified lines
168178 static inline struct folio *swp_tb_to_folio(unsigned long swp_tb)
169179 {
170180 VM_WARN_ON(!swp_tb_is_folio(swp_tb));
171− return pfn_folio((swp_tb & ~SWP_TB_COUNT_MASK) >> SWP_TB_PFN_MARK_BITS);
181+ return pfn_folio((swp_tb & ~SWP_TB_FLAGS_MASK) >> SWAP_CACHE_PFN_MARK_BITS);
172182 }
173183
174184 static inline void *swp_tb_to_shadow(unsigned long swp_tb)
175185 {
176186 VM_WARN_ON(!swp_tb_is_shadow(swp_tb));
177187 /* No shift needed, xa_value is stored as it is in the lower bits. */
178− return (void *)(swp_tb & ~SWP_TB_COUNT_MASK);
188+ return (void *)(swp_tb & ~SWP_TB_FLAGS_MASK);
179189 }
180190
181191 static inline unsigned char __swp_tb_get_count(unsigned long swp_tb)
@@ -184,6 +194,12 @@ static inline unsigned char __swp_tb_get_count(unsigned long swp_tb) · 2 unmodified lines
184194 return ((swp_tb & SWP_TB_COUNT_MASK) >> SWP_TB_COUNT_SHIFT);
185195 }
186196
197+static inline unsigned char __swp_tb_get_flags(unsigned long swp_tb)
198+{
199+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
200+ return ((swp_tb & SWP_TB_FLAGS_MASK) >> SWP_TB_FLAGS_SHIFT);
201+}
202+
187203 static inline int swp_tb_get_count(unsigned long swp_tb)
188204 {
189205 if (swp_tb_is_countable(swp_tb))
@@ -247,4 +263,107 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci, · 57 unmodified lines
247263
248264 return swp_tb;
249265 }
266+
267+static inline void __swap_table_set_zero(struct swap_cluster_info *ci,
268+ unsigned int ci_off)
269+{
270+#if SWAP_TABLE_HAS_ZEROFLAG
271+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
272+
273+ BUILD_BUG_ON(SWP_TB_ZERO_FLAG & ~SWP_TB_FLAGS_MASK);
274+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
275+ swp_tb |= SWP_TB_ZERO_FLAG;
276+ __swap_table_set(ci, ci_off, swp_tb);
277+#else
278+ lockdep_assert_held(&ci->lock);
279+ __set_bit(ci_off, ci->zero_bitmap);
280+#endif
281+}
282+
283+static inline bool __swap_table_test_zero(struct swap_cluster_info *ci,
284+ unsigned int ci_off)
285+{
286+#if SWAP_TABLE_HAS_ZEROFLAG
287+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
288+
289+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
290+ return !!(swp_tb & SWP_TB_ZERO_FLAG);
291+#else
292+ return test_bit(ci_off, ci->zero_bitmap);
293+#endif
294+}
295+
296+static inline void __swap_table_clear_zero(struct swap_cluster_info *ci,
297+ unsigned int ci_off)
298+{
299+#if SWAP_TABLE_HAS_ZEROFLAG
300+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
301+
302+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
303+ swp_tb &= ~SWP_TB_ZERO_FLAG;
304+ __swap_table_set(ci, ci_off, swp_tb);
305+#else
306+ lockdep_assert_held(&ci->lock);
307+ __clear_bit(ci_off, ci->zero_bitmap);
308+#endif
309+}
310+
311+#ifdef CONFIG_MEMCG
312+static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
313+ unsigned int ci_off, unsigned long nr, unsigned short id)
314+{
315+ lockdep_assert_held(&ci->lock);
316+ VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
317+ if (WARN_ON_ONCE(!ci->memcg_table))
318+ return;
319+ do {
320+ ci->memcg_table->id[ci_off++] = id;
321+ } while (--nr);
322+}
323+
324+static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
325+ unsigned int ci_off)
326+{
327+ lockdep_assert_held(&ci->lock);
328+ VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
329+ if (unlikely(!ci->memcg_table))
330+ return 0;
331+ return ci->memcg_table->id[ci_off];
332+}
333+
334+static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
335+ unsigned int ci_off,
336+ unsigned long nr)
337+{
338+ unsigned short old = __swap_cgroup_get(ci, ci_off);
339+
340+ if (!old)
341+ return 0;
342+ do {
343+ VM_WARN_ON_ONCE(ci->memcg_table->id[ci_off] != old);
344+ ci->memcg_table->id[ci_off++] = 0;
345+ } while (--nr);
346+
347+ return old;
348+}
349+#else
350+static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
351+ unsigned int ci_off, unsigned long nr, unsigned short id)
352+{
353+}
354+
355+static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
356+ unsigned int ci_off)
357+{
358+ return 0;
359+}
360+
361+static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
362+ unsigned int ci_off,
363+ unsigned long nr)
364+{
365+ return 0;
366+}
367+#endif
368+
250369 #endif
mm/swapfile.c+302−158
@@ -45,7 +45,6 @@ · 44 unmodified lines
4545
4646 #include <asm/tlbflush.h>
4747 #include <linux/leafops.h>
48−#include <linux/swap_cgroup.h>
4948 #include "swap_table.h"
5049 #include "internal.h"
5150 #include "swap.h"
@@ -133,7 +132,7 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = { · 81 unmodified lines
133132 /* May return NULL on invalid type, caller must check for NULL return */
134133 static struct swap_info_struct *swap_type_to_info(int type)
135134 {
136− if (type >= MAX_SWAPFILES)
135+ if (type < 0 || type >= MAX_SWAPFILES)
137136 return NULL;
138137 return READ_ONCE(swap_info[type]); /* rcu_dereference() */
139138 }
@@ -411,20 +410,7 @@ static inline unsigned int cluster_offset(struct swap_info_struct *si, · 271 unmodified lines
411410 return cluster_index(si, ci) * SWAPFILE_CLUSTER;
412411 }
413412
414−static struct swap_table *swap_table_alloc(gfp_t gfp)
415−{
416− struct folio *folio;
417−
418− if (!SWP_TABLE_USE_PAGE)
419− return kmem_cache_zalloc(swap_table_cachep, gfp);
420−
421− folio = folio_alloc(gfp | __GFP_ZERO, 0);
422− if (folio)
423− return folio_address(folio);
424− return NULL;
425−}
426−
427−static void swap_table_free_folio_rcu_cb(struct rcu_head *head)
413+static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
428414 {
429415 struct folio *folio;
430416
@@ -432,15 +418,76 @@ static void swap_table_free_folio_rcu_cb(struct rcu_head *head) · 1 unmodified line
432418 folio_put(folio);
433419 }
434420
435−static void swap_table_free(struct swap_table *table)
421+static void swap_cluster_free_table(struct swap_cluster_info *ci)
436422 {
423+ struct swap_table *table;
424+
425+#ifdef CONFIG_MEMCG
426+ kfree(ci->memcg_table);
427+ ci->memcg_table = NULL;
428+#endif
429+
430+#if !SWAP_TABLE_HAS_ZEROFLAG
431+ kfree(ci->zero_bitmap);
432+ ci->zero_bitmap = NULL;
433+#endif
434+
435+ table = (struct swap_table *)rcu_access_pointer(ci->table);
436+ if (!table)
437+ return;
438+
439+ rcu_assign_pointer(ci->table, NULL);
437440 if (!SWP_TABLE_USE_PAGE) {
438441 kmem_cache_free(swap_table_cachep, table);
439442 return;
440443 }
441444
442445 call_rcu(&(folio_page(virt_to_folio(table), 0)->rcu_head),
443− swap_table_free_folio_rcu_cb);
446+ swap_cluster_free_table_folio_rcu_cb);
447+}
448+
449+static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
450+{
451+ struct swap_table *table = NULL;
452+ struct folio *folio;
453+
454+ /* The cluster must be empty and not on any list during allocation. */
455+ VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
456+ if (rcu_access_pointer(ci->table))
457+ return 0;
458+
459+ if (SWP_TABLE_USE_PAGE) {
460+ folio = folio_alloc(gfp | __GFP_ZERO, 0);
461+ if (folio)
462+ table = folio_address(folio);
463+ } else {
464+ table = kmem_cache_zalloc(swap_table_cachep, gfp);
465+ }
466+ if (!table)
467+ return -ENOMEM;
468+
469+ rcu_assign_pointer(ci->table, table);
470+
471+#ifdef CONFIG_MEMCG
472+ if (!mem_cgroup_disabled()) {
473+ VM_WARN_ON_ONCE(ci->memcg_table);
474+ ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
475+ if (!ci->memcg_table) {
476+ swap_cluster_free_table(ci);
477+ return -ENOMEM;
478+ }
479+ }
480+#endif
481+
482+#if !SWAP_TABLE_HAS_ZEROFLAG
483+ VM_WARN_ON_ONCE(ci->zero_bitmap);
484+ ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
485+ if (!ci->zero_bitmap) {
486+ swap_cluster_free_table(ci);
487+ return -ENOMEM;
488+ }
489+#endif
490+ return 0;
444491 }
445492
446493 /*
@@ -465,33 +512,22 @@ static void swap_cluster_assert_empty(struct swap_cluster_info *ci, · 18 unmodified lines
465512 bad_slots++;
466513 else
467514 WARN_ON_ONCE(!swp_tb_is_null(swp_tb));
515+ WARN_ON_ONCE(__swap_cgroup_get(ci, ci_off));
468516 } while (++ci_off < ci_end);
469517
470518 WARN_ON_ONCE(bad_slots != (swapoff ? ci->count : 0));
471519 WARN_ON_ONCE(nr == SWAPFILE_CLUSTER && ci->extend_table);
472520 }
473521
474−static void swap_cluster_free_table(struct swap_cluster_info *ci)
475−{
476− struct swap_table *table;
477−
478− /* Only empty cluster's table is allow to be freed */
479− lockdep_assert_held(&ci->lock);
480− table = (void *)rcu_dereference_protected(ci->table, true);
481− rcu_assign_pointer(ci->table, NULL);
482−
483− swap_table_free(table);
484−}
485−
486522 /*
487523 * Allocate swap table for one cluster. Attempt an atomic allocation first,
488524 * then fallback to sleeping allocation.
489525 */
490526 static struct swap_cluster_info *
491−swap_cluster_alloc_table(struct swap_info_struct *si,
527+swap_cluster_populate(struct swap_info_struct *si,
492528 struct swap_cluster_info *ci)
493529 {
494− struct swap_table *table;
530+ int ret;
495531
496532 /*
497533 * Only cluster isolation from the allocator does table allocation.
@@ -502,14 +538,9 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 4 unmodified lines
502538 lockdep_assert_held(&si->global_cluster_lock);
503539 lockdep_assert_held(&ci->lock);
504540
505− /* The cluster must be free and was just isolated from the free list. */
506− VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
507−
508− table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
509− if (table) {
510− rcu_assign_pointer(ci->table, table);
541+ if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
542+ __GFP_NOWARN))
511543 return ci;
512− }
513544
514545 /*
515546 * Try a sleep allocation. Each isolated free cluster may cause
@@ -521,7 +552,8 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 5 unmodified lines
521552 spin_unlock(&si->global_cluster_lock);
522553 local_unlock(&percpu_swap_cluster.lock);
523554
524− table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL);
555+ ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
556+ GFP_KERNEL);
525557
526558 /*
527559 * Back to atomic context. We might have migrated to a new CPU with a
@@ -536,20 +568,11 @@ swap_cluster_alloc_table(struct swap_info_struct *si, · 8 unmodified lines
536568 spin_lock(&si->global_cluster_lock);
537569 spin_lock(&ci->lock);
538570
539− /* Nothing except this helper should touch a dangling empty cluster. */
540− if (WARN_ON_ONCE(cluster_table_is_alloced(ci))) {
541− if (table)
542− swap_table_free(table);
543− return ci;
544− }
545−
546− if (!table) {
571+ if (ret) {
547572 move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
548573 spin_unlock(&ci->lock);
549574 return NULL;
550575 }
551−
552− rcu_assign_pointer(ci->table, table);
553576 return ci;
554577 }
555578
@@ -621,12 +644,11 @@ static struct swap_cluster_info *isolate_lock_cluster( · 65 unmodified lines
621644 }
622645 spin_unlock(&si->lock);
623646
624− if (found && !cluster_table_is_alloced(found)) {
625− /* Only an empty free cluster's swap table can be freed. */
626− VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);
647+ /* Cluster's table is freed when and only when it's on the free list. */
648+ if (found && flags == CLUSTER_FLAG_FREE) {
627649 VM_WARN_ON_ONCE(list != &si->free_clusters);
628− VM_WARN_ON_ONCE(!cluster_is_empty(found));
629− return swap_cluster_alloc_table(si, found);
650+ VM_WARN_ON_ONCE(cluster_table_is_alloced(found));
651+ return swap_cluster_populate(si, found);
630652 }
631653
632654 return found;
@@ -769,7 +791,6 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si, · 136 unmodified lines
769791 unsigned int ci_off = offset % SWAPFILE_CLUSTER;
770792 unsigned long idx = offset / SWAPFILE_CLUSTER;
771793 struct swap_cluster_info *ci;
772− struct swap_table *table;
773794 int ret = 0;
774795
775796 /* si->max may got shrunk by swap swap_activate() */
@@ -790,12 +811,9 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si, · 14 unmodified lines
790811 }
791812
792813 ci = cluster_info + idx;
793− if (!ci->table) {
794− table = swap_table_alloc(GFP_KERNEL);
795− if (!table)
796− return -ENOMEM;
797− rcu_assign_pointer(ci->table, table);
798− }
814+ /* Need to allocate swap table first for initial bad slot marking. */
815+ if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
816+ return -ENOMEM;
799817 spin_lock(&ci->lock);
800818 /* Check for duplicated bad swap slots. */
801819 if (__swap_table_xchg(ci, ci_off, SWP_TB_BAD) != SWP_TB_NULL) {
@@ -922,8 +940,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si, · 120 unmodified lines
922940 order = 0;
923941 nr_pages = 1;
924942 swap_cluster_assert_empty(ci, ci_off, 1, false);
925− /* Sets a fake shadow as placeholder */
926− __swap_table_set(ci, ci_off, shadow_to_swp_tb(NULL, 1));
943+ /* Fake shadow placeholder with no flag, hibernation does not use the zeromap */
944+ __swap_table_set(ci, ci_off, __swp_tb_mk_count(shadow_to_swp_tb(NULL, 0), 1));
927945 } else {
928946 /* Allocation without folio is only possible with hibernation */
929947 WARN_ON_ONCE(1);
@@ -1054,6 +1072,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) · 124 unmodified lines
10541072 swap_cluster_unlock(ci);
10551073 if (to_scan <= 0)
10561074 break;
1075+ cond_resched();
10571076 }
10581077 }
10591078
@@ -1295,14 +1314,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, · 235 unmodified lines
12951314 void (*swap_slot_free_notify)(struct block_device *, unsigned long);
12961315 unsigned int i;
12971316
1298− /*
1299− * Use atomic clear_bit operations only on zeromap instead of non-atomic
1300− * bitmap_clear to prevent adjacent bits corruption due to simultaneous writes.
1301− */
1302− for (i = 0; i < nr_entries; i++) {
1303− clear_bit(offset + i, si->zeromap);
1317+ for (i = 0; i < nr_entries; i++)
13041318 zswap_invalidate(swp_entry(si->type, offset + i));
1305− }
13061319
13071320 if (si->flags & SWP_BLKDEV)
13081321 swap_slot_free_notify =
@@ -1442,8 +1455,10 @@ start_over: · 133 unmodified lines
14421455 }
14431456
14441457 static int swap_extend_table_alloc(struct swap_info_struct *si,
1445− struct swap_cluster_info *ci, gfp_t gfp)
1458+ struct swap_cluster_info *ci,
1459+ unsigned int ci_off, gfp_t gfp)
14461460 {
1461+ int count;
14471462 void *table;
14481463
14491464 table = kzalloc(sizeof(ci->extend_table[0]) * SWAPFILE_CLUSTER, gfp);
@@ -1451,11 +1466,27 @@ static int swap_extend_table_alloc(struct swap_info_struct *si, · 1 unmodified line
14511466 return -ENOMEM;
14521467
14531468 spin_lock(&ci->lock);
1454− if (!ci->extend_table)
1455− ci->extend_table = table;
1456− else
1457− kfree(table);
1469+ /*
1470+ * Extend table allocation requires releasing ci lock first so it's
1471+ * possible that the slot has been freed, no longer overflowed, or
1472+ * a concurrent extend table allocation has already succeeded, so
1473+ * the allocation is no longer needed.
1474+ */
1475+ if (!cluster_table_is_alloced(ci))
1476+ goto out_free;
1477+ count = swp_tb_get_count(__swap_table_get(ci, ci_off));
1478+ if (count < (SWP_TB_COUNT_MAX - 1))
1479+ goto out_free;
1480+ if (ci->extend_table)
1481+ goto out_free;
1482+
1483+ ci->extend_table = table;
1484+ spin_unlock(&ci->lock);
1485+ return 0;
1486+
1487+out_free:
14581488 spin_unlock(&ci->lock);
1489+ kfree(table);
14591490 return 0;
14601491 }
14611492
@@ -1471,7 +1502,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp) · 9 unmodified lines
14711502 return 0;
14721503
14731504 ci = __swap_offset_to_cluster(si, offset);
1474− ret = swap_extend_table_alloc(si, ci, gfp);
1505+ ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
14751506
14761507 put_swap_device(si);
14771508 return ret;
@@ -1518,13 +1549,21 @@ static void __swap_cluster_put_entry(struct swap_cluster_info *ci, · 40 unmodified lines
15181549 if (count == (SWP_TB_COUNT_MAX - 1)) {
15191550 ci->extend_table[ci_off] = 0;
15201551 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, count));
1521− swap_extend_table_try_free(ci);
15221552 } else {
15231553 ci->extend_table[ci_off] = count;
15241554 }
15251555 } else {
15261556 __swap_table_set(ci, ci_off, __swp_tb_mk_count(swp_tb, --count));
15271557 }
1558+
1559+ /*
1560+ * `SWP_TB_COUNT_MAX - 1` triggers extend table allocation. If the
1561+ * count was above that, then the extend table is no longer needed,
1562+ * so free it. And if we just put the count value from MAX - 1, it's
1563+ * also possible that a pending dup just attached an extend table.
1564+ */
1565+ if (unlikely(count == SWP_TB_COUNT_MAX - 2 || count == SWP_TB_COUNT_MAX - 1))
1566+ swap_extend_table_try_free(ci);
15281567 }
15291568
15301569 /**
@@ -1664,7 +1703,7 @@ restart: · 133 unmodified lines
16641703 if (unlikely(err)) {
16651704 if (err == -ENOMEM) {
16661705 spin_unlock(&ci->lock);
1667− err = swap_extend_table_alloc(si, ci, GFP_ATOMIC);
1706+ err = swap_extend_table_alloc(si, ci, ci_off, GFP_ATOMIC);
16681707 spin_lock(&ci->lock);
16691708 if (!err)
16701709 goto restart;
@@ -1730,7 +1769,7 @@ again: · 59 unmodified lines
17301769 }
17311770
17321771 /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
1733− if (unlikely(mem_cgroup_try_charge_swap(folio, folio->swap)))
1772+ if (unlikely(mem_cgroup_try_charge_swap(folio)))
17341773 swap_cache_del_folio(folio);
17351774
17361775 if (unlikely(!folio_test_swapcache(folio)))
@@ -1826,8 +1865,7 @@ void folio_put_swap(struct folio *folio, struct page *subpage) · 89 unmodified lines
18261865 * do_swap_page()
18271866 * ... swapoff+swapon
18281867 * swap_cache_alloc_folio()
1829− * swap_cache_add_folio()
1830− * // check swap_map
1868+ * // check swap_map
18311869 * // verify PTE not changed
18321870 *
18331871 * In __swap_duplicate(), the swap_map need to be checked before
@@ -1873,21 +1911,44 @@ void __swap_cluster_free_entries(struct swap_info_struct *si, · 39 unmodified lines
18731911 unsigned int ci_start, unsigned int nr_pages)
18741912 {
18751913 unsigned long old_tb;
1914+ unsigned short batch_id = 0, id_cur;
18761915 unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
1877− unsigned long offset = cluster_offset(si, ci) + ci_start;
1916+ unsigned long ci_head = cluster_offset(si, ci);
1917+ unsigned int batch_off = ci_off;
18781918
18791919 VM_WARN_ON(ci->count < nr_pages);
18801920
18811921 ci->count -= nr_pages;
18821922 do {
18831923 old_tb = __swap_table_get(ci, ci_off);
1884− /* Release the last ref, or after swap cache is dropped */
1924+ /*
1925+ * Freeing is done after release of the last swap count
1926+ * ref, or after swap cache is dropped
1927+ */
18851928 VM_WARN_ON(!swp_tb_is_shadow(old_tb) || __swp_tb_get_count(old_tb) > 1);
1929+
1930+ /* Resetting the slot to NULL also clears the inline flags. */
18861931 __swap_table_set(ci, ci_off, null_to_swp_tb());
1932+ if (!SWAP_TABLE_HAS_ZEROFLAG)
1933+ __swap_table_clear_zero(ci, ci_off);
1934+
1935+ /*
1936+ * Uncharge swap slots by memcg in batches. Consecutive
1937+ * slots with the same cgroup id are uncharged together.
1938+ */
1939+ id_cur = __swap_cgroup_clear(ci, ci_off, 1);
1940+ if (batch_id != id_cur) {
1941+ if (batch_id)
1942+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
1943+ batch_id = id_cur;
1944+ batch_off = ci_off;
1945+ }
18871946 } while (++ci_off < ci_end);
18881947
1889− mem_cgroup_uncharge_swap(swp_entry(si->type, offset), nr_pages);
1890− swap_range_free(si, offset, nr_pages);
1948+ if (batch_id)
1949+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
1950+
1951+ swap_range_free(si, ci_head + ci_start, nr_pages);
18911952 swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
18921953
18931954 if (!ci->count)
@@ -2077,7 +2138,16 @@ out: · 183 unmodified lines
20772138 }
20782139
20792140 #ifdef CONFIG_HIBERNATION
2080−/* Allocate a slot for hibernation */
2141+/**
2142+ * swap_alloc_hibernation_slot() - Allocate a swap slot for hibernation.
2143+ * @type: swap device type index to allocate from.
2144+ *
2145+ * The caller must ensure the swap device is stable, either by pinning
2146+ * it (SWP_HIBERNATION) or by freezing user-space.
2147+ *
2148+ * Return: a valid swp_entry_t on success, or an empty entry (val == 0)
2149+ * on failure.
2150+ */
20812151 swp_entry_t swap_alloc_hibernation_slot(int type)
20822152 {
20832153 struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);
@@ -2088,46 +2158,42 @@ swp_entry_t swap_alloc_hibernation_slot(int type) · 4 unmodified lines
20882158 if (!si)
20892159 goto fail;
20902160
2091− /* This is called for allocating swap entry, not cache */
2092− if (get_swap_device_info(si)) {
2093− if (si->flags & SWP_WRITEOK) {
2094− /*
2095− * Try the local cluster first if it matches the device. If
2096− * not, try grab a new cluster and override local cluster.
2097− */
2098− local_lock(&percpu_swap_cluster.lock);
2099− pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
2100− pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
2101− if (pcp_si == si && pcp_offset) {
2102− ci = swap_cluster_lock(si, pcp_offset);
2103− if (cluster_is_usable(ci, 0))
2104− offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
2105− else
2106− swap_cluster_unlock(ci);
2107− }
2108− if (!offset)
2109− offset = cluster_alloc_swap_entry(si, NULL);
2110− local_unlock(&percpu_swap_cluster.lock);
2111− if (offset)
2112− entry = swp_entry(si->type, offset);
2113− }
2114− put_swap_device(si);
2161+ /*
2162+ * Try the local cluster first if it matches the device. If
2163+ * not, try grab a new cluster and override local cluster.
2164+ */
2165+ local_lock(&percpu_swap_cluster.lock);
2166+ pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
2167+ pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
2168+ if (pcp_si == si && pcp_offset) {
2169+ ci = swap_cluster_lock(si, pcp_offset);
2170+ if (cluster_is_usable(ci, 0))
2171+ offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
2172+ else
2173+ swap_cluster_unlock(ci);
21152174 }
2175+ if (!offset)
2176+ offset = cluster_alloc_swap_entry(si, NULL);
2177+ local_unlock(&percpu_swap_cluster.lock);
2178+ if (offset)
2179+ entry = swp_entry(si->type, offset);
2180+
21162181 fail:
21172182 return entry;
21182183 }
21192184
2120−/* Free a slot allocated by swap_alloc_hibernation_slot */
2185+/**
2186+ * swap_free_hibernation_slot() - Free a swap slot allocated for hibernation.
2187+ * @entry: swap entry to free.
2188+ *
2189+ * The caller must ensure the swap device is stable.
2190+ */
21212191 void swap_free_hibernation_slot(swp_entry_t entry)
21222192 {
2123− struct swap_info_struct *si;
2193+ struct swap_info_struct *si = __swap_entry_to_info(entry);
21242194 struct swap_cluster_info *ci;
21252195 pgoff_t offset = swp_offset(entry);
21262196
2127− si = get_swap_device(entry);
2128− if (WARN_ON(!si))
2129− return;
2130−
21312197 ci = swap_cluster_lock(si, offset);
21322198 __swap_cluster_put_entry(ci, offset % SWAPFILE_CLUSTER);
21332199 __swap_cluster_free_entries(si, ci, offset % SWAPFILE_CLUSTER, 1);
@@ -2135,25 +2201,17 @@ void swap_free_hibernation_slot(swp_entry_t entry) · 1 unmodified line
21352201
21362202 /* In theory readahead might add it to the swap cache by accident */
21372203 __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
2138− put_swap_device(si);
21392204 }
21402205
2141−/*
2142− * Find the swap type that corresponds to given device (if any).
2143− *
2144− * @offset - number of the PAGE_SIZE-sized block of the device, starting
2145− * from 0, in which the swap header is expected to be located.
2146− *
2147− * This is needed for the suspend to disk (aka swsusp).
2148− */
2149−int swap_type_of(dev_t device, sector_t offset)
2206+static int __find_hibernation_swap_type(dev_t device, sector_t offset)
21502207 {
21512208 int type;
21522209
2210+ lockdep_assert_held(&swap_lock);
2211+
21532212 if (!device)
2154− return -1;
2213+ return -EINVAL;
21552214
2156− spin_lock(&swap_lock);
21572215 for (type = 0; type < nr_swapfiles; type++) {
21582216 struct swap_info_struct *sis = swap_info[type];
21592217
@@ -2163,16 +2221,118 @@ int swap_type_of(dev_t device, sector_t offset) · 3 unmodified lines
21632221 if (device == sis->bdev->bd_dev) {
21642222 struct swap_extent *se = first_se(sis);
21652223
2166− if (se->start_block == offset) {
2167− spin_unlock(&swap_lock);
2224+ if (se->start_block == offset)
21682225 return type;
2169− }
21702226 }
21712227 }
2172− spin_unlock(&swap_lock);
21732228 return -ENODEV;
21742229 }
21752230
2231+/**
2232+ * pin_hibernation_swap_type - Pin the swap device for hibernation
2233+ * @device: Block device containing the resume image
2234+ * @offset: Offset identifying the swap area
2235+ *
2236+ * Locate the swap device for @device/@offset and mark it as pinned
2237+ * for hibernation. While pinned, swapoff() is prevented.
2238+ *
2239+ * Only one uswsusp context may pin a swap device at a time.
2240+ * If already pinned, this function returns -EBUSY.
2241+ *
2242+ * Return:
2243+ * >= 0 on success (swap type).
2244+ * -EINVAL if @device is invalid.
2245+ * -ENODEV if the swap device is not found.
2246+ * -EBUSY if the device is already pinned for hibernation.
2247+ */
2248+int pin_hibernation_swap_type(dev_t device, sector_t offset)
2249+{
2250+ int type;
2251+ struct swap_info_struct *si;
2252+
2253+ spin_lock(&swap_lock);
2254+
2255+ type = __find_hibernation_swap_type(device, offset);
2256+ if (type < 0) {
2257+ spin_unlock(&swap_lock);
2258+ return type;
2259+ }
2260+
2261+ si = swap_type_to_info(type);
2262+ if (WARN_ON_ONCE(!si)) {
2263+ spin_unlock(&swap_lock);
2264+ return -ENODEV;
2265+ }
2266+
2267+ /*
2268+ * hibernate_acquire() prevents concurrent hibernation sessions.
2269+ * This check additionally guards against double-pinning within
2270+ * the same session.
2271+ */
2272+ if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) {
2273+ spin_unlock(&swap_lock);
2274+ return -EBUSY;
2275+ }
2276+
2277+ si->flags |= SWP_HIBERNATION;
2278+
2279+ spin_unlock(&swap_lock);
2280+ return type;
2281+}
2282+
2283+/**
2284+ * unpin_hibernation_swap_type - Unpin the swap device for hibernation
2285+ * @type: Swap type previously returned by pin_hibernation_swap_type()
2286+ *
2287+ * Clear the hibernation pin on the given swap device, allowing
2288+ * swapoff() to proceed normally.
2289+ *
2290+ * If @type does not refer to a valid swap device, this function
2291+ * does nothing.
2292+ */
2293+void unpin_hibernation_swap_type(int type)
2294+{
2295+ struct swap_info_struct *si;
2296+
2297+ spin_lock(&swap_lock);
2298+ si = swap_type_to_info(type);
2299+ if (!si) {
2300+ spin_unlock(&swap_lock);
2301+ return;
2302+ }
2303+ si->flags &= ~SWP_HIBERNATION;
2304+ spin_unlock(&swap_lock);
2305+}
2306+
2307+/**
2308+ * find_hibernation_swap_type - Find swap type for hibernation
2309+ * @device: Block device containing the resume image
2310+ * @offset: Offset within the device identifying the swap area
2311+ *
2312+ * Locate the swap device corresponding to @device and @offset.
2313+ *
2314+ * Unlike pin_hibernation_swap_type(), this function only performs a
2315+ * lookup and does not mark the swap device as pinned for hibernation.
2316+ *
2317+ * This is safe in the sysfs-based hibernation path where user space
2318+ * is already frozen and swapoff() cannot run concurrently.
2319+ *
2320+ * Return:
2321+ * A non-negative swap type on success.
2322+ * -EINVAL if @device is invalid.
2323+ * -ENODEV if no matching swap device is found.
2324+ */
2325+int find_hibernation_swap_type(dev_t device, sector_t offset)
2326+{
2327+ int type;
2328+
2329+ spin_lock(&swap_lock);
2330+ type = __find_hibernation_swap_type(device, offset);
2331+ spin_unlock(&swap_lock);
2332+
2333+ return type;
2334+}
2335+
21762336 int find_first_swap(dev_t *device)
21772337 {
21782338 int type;
@@ -2869,7 +3029,7 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info, · 690 unmodified lines
28693029 ci = cluster_info + i;
28703030 /* Cluster with bad marks count will have a remaining table */
28713031 spin_lock(&ci->lock);
2872− if (rcu_dereference_protected(ci->table, true)) {
3032+ if (cluster_table_is_alloced(ci)) {
28733033 swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
28743034 swap_cluster_free_table(ci);
28753035 }
@@ -2903,7 +3063,6 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si) · 27 unmodified lines
29033063 SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
29043064 {
29053065 struct swap_info_struct *p = NULL;
2906− unsigned long *zeromap;
29073066 struct swap_cluster_info *cluster_info;
29083067 struct file *swap_file, *victim;
29093068 struct address_space *mapping;
@@ -2936,6 +3095,14 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 26 unmodified lines
29363095 spin_unlock(&swap_lock);
29373096 goto out_dput;
29383097 }
3098+
3099+ /* Refuse swapoff while the device is pinned for hibernation */
3100+ if (p->flags & SWP_HIBERNATION) {
3101+ err = -EBUSY;
3102+ spin_unlock(&swap_lock);
3103+ goto out_dput;
3104+ }
3105+
29393106 if (!security_vm_enough_memory_mm(current->mm, p->pages))
29403107 vm_unacct_memory(p->pages);
29413108 else {
@@ -2991,8 +3158,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 49 unmodified lines
29913158
29923159 swap_file = p->swap_file;
29933160 p->swap_file = NULL;
2994− zeromap = p->zeromap;
2995− p->zeromap = NULL;
29963161 maxpages = p->max;
29973162 cluster_info = p->cluster_info;
29983163 p->max = 0;
@@ -3004,10 +3169,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · 5 unmodified lines
30043169 mutex_unlock(&swapon_mutex);
30053170 kfree(p->global_cluster);
30063171 p->global_cluster = NULL;
3007− kvfree(zeromap);
30083172 free_swap_cluster_info(cluster_info, maxpages);
3009− /* Destroy swap account information */
3010− swap_cgroup_swapoff(p->type);
30113173
30123174 inode = mapping->host;
30133175
@@ -3538,21 +3700,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · 524 unmodified lines
35383700 if (error)
35393701 goto bad_swap_unlock_inode;
35403702
3541− error = swap_cgroup_swapon(si->type, maxpages);
3542− if (error)
3543− goto bad_swap_unlock_inode;
3544−
3545− /*
3546− * Use kvmalloc_array instead of bitmap_zalloc as the allocation order might
3547− * be above MAX_PAGE_ORDER incase of a large swap file.
3548− */
3549− si->zeromap = kvmalloc_array(BITS_TO_LONGS(maxpages), sizeof(long),
3550− GFP_KERNEL | __GFP_ZERO);
3551− if (!si->zeromap) {
3552− error = -ENOMEM;
3553− goto bad_swap_unlock_inode;
3554− }
3555−
35563703 if (si->bdev && bdev_stable_writes(si->bdev))
35573704 si->flags |= SWP_STABLE_WRITES;
35583705
@@ -3652,11 +3799,8 @@ bad_swap: · 93 unmodified lines
36523799 si->global_cluster = NULL;
36533800 inode = NULL;
36543801 destroy_swap_extents(si, swap_file);
3655− swap_cgroup_swapoff(si->type);
36563802 free_swap_cluster_info(si->cluster_info, si->max);
36573803 si->cluster_info = NULL;
3658− kvfree(si->zeromap);
3659− si->zeromap = NULL;
36603804 /*
36613805 * Clear the SWP_USED flag after all resources are freed so
36623806 * alloc_swap_info can reuse this si safely.
mm/userfaultfd.c+2247−12
@@ -2,7 +2,12 @@ · 1 unmodified line
22 /*
33 * mm/userfaultfd.c
44 *
5+ * Copyright (C) 2007 Davide Libenzi <davidel@xmailserver.org>
6+ * Copyright (C) 2008-2009 Red Hat, Inc.
57 * Copyright (C) 2015 Red Hat, Inc.
8+ *
9+ * Some part derived from fs/eventfd.c (anon inode setup) and
10+ * mm/ksm.c (mm hashing).
611 */
712
813 #include <linux/mm.h>
@@ -14,6 +19,17 @@ · 5 unmodified lines
1419 #include <linux/userfaultfd_k.h>
1520 #include <linux/mmu_notifier.h>
1621 #include <linux/hugetlb.h>
22+#include <linux/list.h>
23+#include <linux/sched/mm.h>
24+#include <linux/mm_inline.h>
25+#include <linux/poll.h>
26+#include <linux/slab.h>
27+#include <linux/seq_file.h>
28+#include <linux/bug.h>
29+#include <linux/anon_inodes.h>
30+#include <linux/syscalls.h>
31+#include <linux/miscdevice.h>
32+#include <linux/uio.h>
1733 #include <linux/file.h>
1834 #include <linux/cleanup.h>
1935 #include <asm/tlbflush.h>
@@ -1017,7 +1033,7 @@ out: · 997 unmodified lines
10171033 return copied ? copied : err;
10181034 }
10191035
1020−ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
1036+static ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
10211037 unsigned long src_start, unsigned long len,
10221038 uffd_flags_t flags)
10231039 {
@@ -1025,7 +1041,7 @@ ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start, · 1 unmodified line
10251041 uffd_flags_set_mode(flags, MFILL_ATOMIC_COPY));
10261042 }
10271043
1028−ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
1044+static ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
10291045 unsigned long start,
10301046 unsigned long len)
10311047 {
@@ -1033,7 +1049,7 @@ ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx, · 1 unmodified line
10331049 uffd_flags_set_mode(0, MFILL_ATOMIC_ZEROPAGE));
10341050 }
10351051
1036−ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
1052+static ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
10371053 unsigned long len, uffd_flags_t flags)
10381054 {
10391055
@@ -1049,7 +1065,7 @@ ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start, · 9 unmodified lines
10491065 uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE));
10501066 }
10511067
1052−ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
1068+static ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
10531069 unsigned long len, uffd_flags_t flags)
10541070 {
10551071 return mfill_atomic(ctx, start, 0, len,
@@ -1085,7 +1101,7 @@ long uffd_wp_range(struct vm_area_struct *dst_vma, · 29 unmodified lines
10851101 return ret;
10861102 }
10871103
1088−int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
1104+static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
10891105 unsigned long len, bool enable_wp)
10901106 {
10911107 struct mm_struct *dst_mm = ctx->mm;
@@ -1915,7 +1931,7 @@ static void uffd_move_unlock(struct vm_area_struct *dst_vma, · 823 unmodified lines
19151931 * in the regions or not, but preventing the risk of having to split
19161932 * the hugepmd during the remap.
19171933 */
1918−ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
1934+static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
19191935 unsigned long src_start, unsigned long len, __u64 mode)
19201936 {
19211937 struct mm_struct *mm = ctx->mm;
@@ -2090,7 +2106,7 @@ out: · 168 unmodified lines
20902106 return moved ? moved : err;
20912107 }
20922108
2093−bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
2109+static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
20942110 bool wp_async)
20952111 {
20962112 const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
@@ -2147,12 +2163,12 @@ static void userfaultfd_set_ctx(struct vm_area_struct *vma, · 50 unmodified lines
21472163 (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags);
21482164 }
21492165
2150−void userfaultfd_reset_ctx(struct vm_area_struct *vma)
2166+static void userfaultfd_reset_ctx(struct vm_area_struct *vma)
21512167 {
21522168 userfaultfd_set_ctx(vma, NULL, 0);
21532169 }
21542170
2155−struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
2171+static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
21562172 struct vm_area_struct *prev,
21572173 struct vm_area_struct *vma,
21582174 unsigned long start,
@@ -2191,7 +2207,7 @@ struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi, · 32 unmodified lines
21912207 }
21922208
21932209 /* Assumes mmap write lock taken, and mm_struct pinned. */
2194−int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
2210+static int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
21952211 struct vm_area_struct *vma,
21962212 vm_flags_t vm_flags,
21972213 unsigned long start, unsigned long end,
@@ -2255,7 +2271,7 @@ skip: · 57 unmodified lines
22552271 return 0;
22562272 }
22572273
2258−void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
2274+static void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
22592275 {
22602276 struct mm_struct *mm = ctx->mm;
22612277 struct vm_area_struct *vma;
@@ -2270,7 +2286,7 @@ void userfaultfd_release_new(struct userfaultfd_ctx *ctx) · 8 unmodified lines
22702286 mmap_write_unlock(mm);
22712287 }
22722288
2273−void userfaultfd_release_all(struct mm_struct *mm,
2289+static void userfaultfd_release_all(struct mm_struct *mm,
22742290 struct userfaultfd_ctx *ctx)
22752291 {
22762292 struct vm_area_struct *vma, *prev;
@@ -2305,3 +2321,2222 @@ void userfaultfd_release_all(struct mm_struct *mm, · 28 unmodified lines
23052321 mmap_write_unlock(mm);
23062322 mmput(mm);
23072323 }
2324+
2325+static int sysctl_unprivileged_userfaultfd __read_mostly;
2326+
2327+#ifdef CONFIG_SYSCTL
2328+static const struct ctl_table vm_userfaultfd_table[] = {
2329+ {
2330+ .procname = "unprivileged_userfaultfd",
2331+ .data = &sysctl_unprivileged_userfaultfd,
2332+ .maxlen = sizeof(sysctl_unprivileged_userfaultfd),
2333+ .mode = 0644,
2334+ .proc_handler = proc_dointvec_minmax,
2335+ .extra1 = SYSCTL_ZERO,
2336+ .extra2 = SYSCTL_ONE,
2337+ },
2338+};
2339+#endif
2340+
2341+static struct kmem_cache *userfaultfd_ctx_cachep __ro_after_init;
2342+
2343+struct userfaultfd_fork_ctx {
2344+ struct userfaultfd_ctx *orig;
2345+ struct userfaultfd_ctx *new;
2346+ struct list_head list;
2347+};
2348+
2349+struct userfaultfd_unmap_ctx {
2350+ struct userfaultfd_ctx *ctx;
2351+ unsigned long start;
2352+ unsigned long end;
2353+ struct list_head list;
2354+};
2355+
2356+struct userfaultfd_wait_queue {
2357+ struct uffd_msg msg;
2358+ wait_queue_entry_t wq;
2359+ struct userfaultfd_ctx *ctx;
2360+ bool waken;
2361+};
2362+
2363+struct userfaultfd_wake_range {
2364+ unsigned long start;
2365+ unsigned long len;
2366+};
2367+
2368+/* internal indication that UFFD_API ioctl was successfully executed */
2369+#define UFFD_FEATURE_INITIALIZED (1u << 31)
2370+
2371+static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
2372+{
2373+ return ctx->features & UFFD_FEATURE_INITIALIZED;
2374+}
2375+
2376+static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
2377+{
2378+ return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
2379+}
2380+
2381+/*
2382+ * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
2383+ * meaningful when userfaultfd_wp()==true on the vma and when it's
2384+ * anonymous.
2385+ */
2386+bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
2387+{
2388+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
2389+
2390+ if (!ctx)
2391+ return false;
2392+
2393+ return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
2394+}
2395+
2396+static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
2397+ int wake_flags, void *key)
2398+{
2399+ struct userfaultfd_wake_range *range = key;
2400+ int ret;
2401+ struct userfaultfd_wait_queue *uwq;
2402+ unsigned long start, len;
2403+
2404+ uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
2405+ ret = 0;
2406+ /* len == 0 means wake all */
2407+ start = range->start;
2408+ len = range->len;
2409+ if (len && (start > uwq->msg.arg.pagefault.address ||
2410+ start + len <= uwq->msg.arg.pagefault.address))
2411+ goto out;
2412+ WRITE_ONCE(uwq->waken, true);
2413+ /*
2414+ * The Program-Order guarantees provided by the scheduler
2415+ * ensure uwq->waken is visible before the task is woken.
2416+ */
2417+ ret = wake_up_state(wq->private, mode);
2418+ if (ret) {
2419+ /*
2420+ * Wake only once, autoremove behavior.
2421+ *
2422+ * After the effect of list_del_init is visible to the other
2423+ * CPUs, the waitqueue may disappear from under us, see the
2424+ * !list_empty_careful() in handle_userfault().
2425+ *
2426+ * try_to_wake_up() has an implicit smp_mb(), and the
2427+ * wq->private is read before calling the extern function
2428+ * "wake_up_state" (which in turns calls try_to_wake_up).
2429+ */
2430+ list_del_init(&wq->entry);
2431+ }
2432+out:
2433+ return ret;
2434+}
2435+
2436+/**
2437+ * userfaultfd_ctx_get - Acquires a reference to the internal userfaultfd
2438+ * context.
2439+ * @ctx: [in] Pointer to the userfaultfd context.
2440+ */
2441+static void userfaultfd_ctx_get(struct userfaultfd_ctx *ctx)
2442+{
2443+ refcount_inc(&ctx->refcount);
2444+}
2445+
2446+/**
2447+ * userfaultfd_ctx_put - Releases a reference to the internal userfaultfd
2448+ * context.
2449+ * @ctx: [in] Pointer to userfaultfd context.
2450+ *
2451+ * The userfaultfd context reference must have been previously acquired either
2452+ * with userfaultfd_ctx_get() or userfaultfd_ctx_fdget().
2453+ */
2454+static void userfaultfd_ctx_put(struct userfaultfd_ctx *ctx)
2455+{
2456+ if (refcount_dec_and_test(&ctx->refcount)) {
2457+ VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_pending_wqh.lock));
2458+ VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_pending_wqh));
2459+ VM_WARN_ON_ONCE(spin_is_locked(&ctx->fault_wqh.lock));
2460+ VM_WARN_ON_ONCE(waitqueue_active(&ctx->fault_wqh));
2461+ VM_WARN_ON_ONCE(spin_is_locked(&ctx->event_wqh.lock));
2462+ VM_WARN_ON_ONCE(waitqueue_active(&ctx->event_wqh));
2463+ VM_WARN_ON_ONCE(spin_is_locked(&ctx->fd_wqh.lock));
2464+ VM_WARN_ON_ONCE(waitqueue_active(&ctx->fd_wqh));
2465+ mmdrop(ctx->mm);
2466+ kmem_cache_free(userfaultfd_ctx_cachep, ctx);
2467+ }
2468+}
2469+
2470+static inline void msg_init(struct uffd_msg *msg)
2471+{
2472+ BUILD_BUG_ON(sizeof(struct uffd_msg) != 32);
2473+ /*
2474+ * Must use memset to zero out the paddings or kernel data is
2475+ * leaked to userland.
2476+ */
2477+ memset(msg, 0, sizeof(struct uffd_msg));
2478+}
2479+
2480+static inline struct uffd_msg userfault_msg(unsigned long address,
2481+ unsigned long real_address,
2482+ unsigned int flags,
2483+ unsigned long reason,
2484+ unsigned int features)
2485+{
2486+ struct uffd_msg msg;
2487+
2488+ msg_init(&msg);
2489+ msg.event = UFFD_EVENT_PAGEFAULT;
2490+
2491+ msg.arg.pagefault.address = (features & UFFD_FEATURE_EXACT_ADDRESS) ?
2492+ real_address : address;
2493+
2494+ /*
2495+ * These flags indicate why the userfault occurred:
2496+ * - UFFD_PAGEFAULT_FLAG_WP indicates a write protect fault.
2497+ * - UFFD_PAGEFAULT_FLAG_MINOR indicates a minor fault.
2498+ * - Neither of these flags being set indicates a MISSING fault.
2499+ *
2500+ * Separately, UFFD_PAGEFAULT_FLAG_WRITE indicates it was a write
2501+ * fault. Otherwise, it was a read fault.
2502+ */
2503+ if (flags & FAULT_FLAG_WRITE)
2504+ msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;
2505+ if (reason & VM_UFFD_WP)
2506+ msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;
2507+ if (reason & VM_UFFD_MINOR)
2508+ msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;
2509+ if (features & UFFD_FEATURE_THREAD_ID)
2510+ msg.arg.pagefault.feat.ptid = task_pid_vnr(current);
2511+ return msg;
2512+}
2513+
2514+#ifdef CONFIG_HUGETLB_PAGE
2515+/*
2516+ * Same functionality as userfaultfd_must_wait below with modifications for
2517+ * hugepmd ranges.
2518+ */
2519+static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
2520+ struct vm_fault *vmf,
2521+ unsigned long reason)
2522+{
2523+ struct vm_area_struct *vma = vmf->vma;
2524+ pte_t *ptep, pte;
2525+
2526+ assert_fault_locked(vmf);
2527+
2528+ ptep = hugetlb_walk(vma, vmf->address, vma_mmu_pagesize(vma));
2529+ if (!ptep)
2530+ return true;
2531+
2532+ pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
2533+
2534+ /*
2535+ * Lockless access: we're in a wait_event so it's ok if it
2536+ * changes under us.
2537+ */
2538+
2539+ /* Entry is still missing, wait for userspace to resolve the fault. */
2540+ if (huge_pte_none(pte))
2541+ return true;
2542+ /* UFFD PTE markers require userspace to resolve the fault. */
2543+ if (pte_is_uffd_marker(pte))
2544+ return true;
2545+ /*
2546+ * Concurrent migration may have replaced the present PTE with a
2547+ * non-marker swap entry between fault delivery and this lockless
2548+ * re-check. huge_pte_write() on a swap entry decodes random offset
2549+ * bits, so gate it on pte_present(). The migration completion path
2550+ * will re-deliver the fault if it still needs userspace.
2551+ */
2552+ if (!pte_present(pte))
2553+ return false;
2554+ /*
2555+ * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
2556+ * resolve the fault.
2557+ */
2558+ if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
2559+ return true;
2560+
2561+ return false;
2562+}
2563+#else
2564+static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
2565+ struct vm_fault *vmf,
2566+ unsigned long reason)
2567+{
2568+ /* Should never get here. */
2569+ VM_WARN_ON_ONCE(1);
2570+ return false;
2571+}
2572+#endif /* CONFIG_HUGETLB_PAGE */
2573+
2574+/*
2575+ * Verify the pagetables are still not ok after having registered into
2576+ * the fault_pending_wqh to avoid userland having to UFFDIO_WAKE any
2577+ * userfault that has already been resolved, if userfaultfd_read_iter and
2578+ * UFFDIO_COPY|ZEROPAGE are being run simultaneously on two different
2579+ * threads.
2580+ */
2581+static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
2582+ struct vm_fault *vmf,
2583+ unsigned long reason)
2584+{
2585+ struct mm_struct *mm = ctx->mm;
2586+ unsigned long address = vmf->address;
2587+ pgd_t *pgd;
2588+ p4d_t *p4d;
2589+ pud_t *pud;
2590+ pmd_t *pmd, _pmd;
2591+ pte_t *pte;
2592+ pte_t ptent;
2593+ bool ret;
2594+
2595+ assert_fault_locked(vmf);
2596+
2597+ pgd = pgd_offset(mm, address);
2598+ if (!pgd_present(*pgd))
2599+ return true;
2600+ p4d = p4d_offset(pgd, address);
2601+ if (!p4d_present(*p4d))
2602+ return true;
2603+ pud = pud_offset(p4d, address);
2604+ if (!pud_present(*pud))
2605+ return true;
2606+ pmd = pmd_offset(pud, address);
2607+again:
2608+ _pmd = pmdp_get_lockless(pmd);
2609+ if (pmd_none(_pmd))
2610+ return true;
2611+
2612+ /*
2613+ * A race could arise which would result in a softleaf entry such as
2614+ * migration entry unexpectedly being present in the PMD, so explicitly
2615+ * check for this and bail out if so.
2616+ */
2617+ if (!pmd_present(_pmd))
2618+ return false;
2619+
2620+ if (pmd_trans_huge(_pmd))
2621+ return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
2622+
2623+ pte = pte_offset_map(pmd, address);
2624+ if (!pte)
2625+ goto again;
2626+
2627+ /*
2628+ * Lockless access: we're in a wait_event so it's ok if it
2629+ * changes under us.
2630+ */
2631+ ptent = ptep_get(pte);
2632+
2633+ ret = true;
2634+ /* Entry is still missing, wait for userspace to resolve the fault. */
2635+ if (pte_none(ptent))
2636+ goto out;
2637+ /* UFFD PTE markers require userspace to resolve the fault. */
2638+ if (pte_is_uffd_marker(ptent))
2639+ goto out;
2640+ /*
2641+ * Concurrent swap-out / migration may have replaced the present PTE
2642+ * with a non-marker swap entry between fault delivery and this
2643+ * lockless re-check. pte_write() on a swap entry decodes random
2644+ * offset bits, so gate it on pte_present(). The page-in path will
2645+ * re-deliver the fault if it still needs userspace.
2646+ */
2647+ if (!pte_present(ptent)) {
2648+ ret = false;
2649+ goto out;
2650+ }
2651+ /*
2652+ * If VMA has UFFD WP faults enabled and WP fault, wait for userspace to
2653+ * resolve the fault.
2654+ */
2655+ if (!pte_write(ptent) && (reason & VM_UFFD_WP))
2656+ goto out;
2657+
2658+ ret = false;
2659+out:
2660+ pte_unmap(pte);
2661+ return ret;
2662+}
2663+
2664+static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags)
2665+{
2666+ if (flags & FAULT_FLAG_INTERRUPTIBLE)
2667+ return TASK_INTERRUPTIBLE;
2668+
2669+ if (flags & FAULT_FLAG_KILLABLE)
2670+ return TASK_KILLABLE;
2671+
2672+ return TASK_UNINTERRUPTIBLE;
2673+}
2674+
2675+/*
2676+ * The locking rules involved in returning VM_FAULT_RETRY depending on
2677+ * FAULT_FLAG_ALLOW_RETRY, FAULT_FLAG_RETRY_NOWAIT and
2678+ * FAULT_FLAG_KILLABLE are not straightforward. The "Caution"
2679+ * recommendation in __lock_page_or_retry is not an understatement.
2680+ *
2681+ * If FAULT_FLAG_ALLOW_RETRY is set, the mmap_lock must be released
2682+ * before returning VM_FAULT_RETRY only if FAULT_FLAG_RETRY_NOWAIT is
2683+ * not set.
2684+ *
2685+ * If FAULT_FLAG_ALLOW_RETRY is set but FAULT_FLAG_KILLABLE is not
2686+ * set, VM_FAULT_RETRY can still be returned if and only if there are
2687+ * fatal_signal_pending()s, and the mmap_lock must be released before
2688+ * returning it.
2689+ */
2690+vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
2691+{
2692+ struct vm_area_struct *vma = vmf->vma;
2693+ struct mm_struct *mm = vma->vm_mm;
2694+ struct userfaultfd_ctx *ctx;
2695+ struct userfaultfd_wait_queue uwq;
2696+ vm_fault_t ret = VM_FAULT_SIGBUS;
2697+ bool must_wait;
2698+ unsigned int blocking_state;
2699+
2700+ /*
2701+ * We don't do userfault handling for the final child pid update
2702+ * and when coredumping (faults triggered by get_dump_page()).
2703+ */
2704+ if (current->flags & (PF_EXITING|PF_DUMPCORE))
2705+ goto out;
2706+
2707+ assert_fault_locked(vmf);
2708+
2709+ ctx = vma->vm_userfaultfd_ctx.ctx;
2710+ if (!ctx)
2711+ goto out;
2712+
2713+ VM_WARN_ON_ONCE(ctx->mm != mm);
2714+
2715+ /* Any unrecognized flag is a bug. */
2716+ VM_WARN_ON_ONCE(reason & ~__VM_UFFD_FLAGS);
2717+ /* 0 or > 1 flags set is a bug; we expect exactly 1. */
2718+ VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
2719+
2720+ if (ctx->features & UFFD_FEATURE_SIGBUS)
2721+ goto out;
2722+ if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
2723+ goto out;
2724+
2725+ /*
2726+ * Check that we can return VM_FAULT_RETRY.
2727+ *
2728+ * NOTE: it should become possible to return VM_FAULT_RETRY
2729+ * even if FAULT_FLAG_TRIED is set without leading to gup()
2730+ * -EBUSY failures, if the userfaultfd is to be extended for
2731+ * VM_UFFD_WP tracking and we intend to arm the userfault
2732+ * without first stopping userland access to the memory. For
2733+ * VM_UFFD_MISSING userfaults this is enough for now.
2734+ */
2735+ if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
2736+ /*
2737+ * Validate the invariant that nowait must allow retry
2738+ * to be sure not to return SIGBUS erroneously on
2739+ * nowait invocations.
2740+ */
2741+ VM_WARN_ON_ONCE(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
2742+#ifdef CONFIG_DEBUG_VM
2743+ if (printk_ratelimit()) {
2744+ pr_warn("FAULT_FLAG_ALLOW_RETRY missing %x\n",
2745+ vmf->flags);
2746+ dump_stack();
2747+ }
2748+#endif
2749+ goto out;
2750+ }
2751+
2752+ /*
2753+ * Handle nowait, not much to do other than tell it to retry
2754+ * and wait.
2755+ */
2756+ ret = VM_FAULT_RETRY;
2757+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
2758+ goto out;
2759+
2760+ if (unlikely(READ_ONCE(ctx->released))) {
2761+ /*
2762+ * If a concurrent release is detected, do not return
2763+ * VM_FAULT_SIGBUS or VM_FAULT_NOPAGE, but instead always
2764+ * return VM_FAULT_RETRY with lock released proactively.
2765+ *
2766+ * If we were to return VM_FAULT_SIGBUS here, the non
2767+ * cooperative manager would be instead forced to
2768+ * always call UFFDIO_UNREGISTER before it can safely
2769+ * close the uffd, to avoid involuntary SIGBUS triggered.
2770+ *
2771+ * If we were to return VM_FAULT_NOPAGE, it would work for
2772+ * the fault path, in which the lock will be released
2773+ * later. However for GUP, faultin_page() does nothing
2774+ * special on NOPAGE, so GUP would spin retrying without
2775+ * releasing the mmap read lock, causing possible livelock.
2776+ *
2777+ * Here only VM_FAULT_RETRY would make sure the mmap lock
2778+ * be released immediately, so that the thread concurrently
2779+ * releasing the userfault would always make progress.
2780+ */
2781+ release_fault_lock(vmf);
2782+ goto out;
2783+ }
2784+
2785+ /* take the reference before dropping the mmap_lock */
2786+ userfaultfd_ctx_get(ctx);
2787+
2788+ init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
2789+ uwq.wq.private = current;
2790+ uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
2791+ reason, ctx->features);
2792+ uwq.ctx = ctx;
2793+ uwq.waken = false;
2794+
2795+ blocking_state = userfaultfd_get_blocking_state(vmf->flags);
2796+
2797+ /*
2798+ * Take the vma lock now, in order to safely call
2799+ * userfaultfd_huge_must_wait() later. Since acquiring the
2800+ * (sleepable) vma lock can modify the current task state, that
2801+ * must be before explicitly calling set_current_state().
2802+ */
2803+ if (is_vm_hugetlb_page(vma))
2804+ hugetlb_vma_lock_read(vma);
2805+
2806+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
2807+ /*
2808+ * After the __add_wait_queue the uwq is visible to userland
2809+ * through poll/read().
2810+ */
2811+ __add_wait_queue(&ctx->fault_pending_wqh, &uwq.wq);
2812+ /*
2813+ * The smp_mb() after __set_current_state prevents the reads
2814+ * following the spin_unlock to happen before the list_add in
2815+ * __add_wait_queue.
2816+ */
2817+ set_current_state(blocking_state);
2818+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2819+
2820+ if (is_vm_hugetlb_page(vma)) {
2821+ must_wait = userfaultfd_huge_must_wait(ctx, vmf, reason);
2822+ hugetlb_vma_unlock_read(vma);
2823+ } else {
2824+ must_wait = userfaultfd_must_wait(ctx, vmf, reason);
2825+ }
2826+
2827+ release_fault_lock(vmf);
2828+
2829+ if (likely(must_wait && !READ_ONCE(ctx->released))) {
2830+ wake_up_poll(&ctx->fd_wqh, EPOLLIN);
2831+ schedule();
2832+ }
2833+
2834+ __set_current_state(TASK_RUNNING);
2835+
2836+ /*
2837+ * Here we race with the list_del; list_add in
2838+ * userfaultfd_ctx_read(), however because we don't ever run
2839+ * list_del_init() to refile across the two lists, the prev
2840+ * and next pointers will never point to self. list_add also
2841+ * would never let any of the two pointers to point to
2842+ * self. So list_empty_careful won't risk to see both pointers
2843+ * pointing to self at any time during the list refile. The
2844+ * only case where list_del_init() is called is the full
2845+ * removal in the wake function and there we don't re-list_add
2846+ * and it's fine not to block on the spinlock. The uwq on this
2847+ * kernel stack can be released after the list_del_init.
2848+ */
2849+ if (!list_empty_careful(&uwq.wq.entry)) {
2850+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
2851+ /*
2852+ * No need of list_del_init(), the uwq on the stack
2853+ * will be freed shortly anyway.
2854+ */
2855+ list_del(&uwq.wq.entry);
2856+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
2857+ }
2858+
2859+ /*
2860+ * ctx may go away after this if the userfault pseudo fd is
2861+ * already released.
2862+ */
2863+ userfaultfd_ctx_put(ctx);
2864+
2865+out:
2866+ return ret;
2867+}
2868+
2869+static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
2870+ struct userfaultfd_wait_queue *ewq)
2871+{
2872+ struct userfaultfd_ctx *release_new_ctx;
2873+
2874+ if (WARN_ON_ONCE(current->flags & PF_EXITING))
2875+ goto out;
2876+
2877+ ewq->ctx = ctx;
2878+ init_waitqueue_entry(&ewq->wq, current);
2879+ release_new_ctx = NULL;
2880+
2881+ spin_lock_irq(&ctx->event_wqh.lock);
2882+ /*
2883+ * After the __add_wait_queue the uwq is visible to userland
2884+ * through poll/read().
2885+ */
2886+ __add_wait_queue(&ctx->event_wqh, &ewq->wq);
2887+ for (;;) {
2888+ set_current_state(TASK_KILLABLE);
2889+ if (ewq->msg.event == 0)
2890+ break;
2891+ if (READ_ONCE(ctx->released) ||
2892+ fatal_signal_pending(current)) {
2893+ /*
2894+ * &ewq->wq may be queued in fork_event, but
2895+ * __remove_wait_queue ignores the head
2896+ * parameter. It would be a problem if it
2897+ * didn't.
2898+ */
2899+ __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
2900+ if (ewq->msg.event == UFFD_EVENT_FORK) {
2901+ struct userfaultfd_ctx *new;
2902+
2903+ new = (struct userfaultfd_ctx *)
2904+ (unsigned long)
2905+ ewq->msg.arg.reserved.reserved1;
2906+ release_new_ctx = new;
2907+ }
2908+ break;
2909+ }
2910+
2911+ spin_unlock_irq(&ctx->event_wqh.lock);
2912+
2913+ wake_up_poll(&ctx->fd_wqh, EPOLLIN);
2914+ schedule();
2915+
2916+ spin_lock_irq(&ctx->event_wqh.lock);
2917+ }
2918+ __set_current_state(TASK_RUNNING);
2919+ spin_unlock_irq(&ctx->event_wqh.lock);
2920+
2921+ if (release_new_ctx) {
2922+ userfaultfd_release_new(release_new_ctx);
2923+ userfaultfd_ctx_put(release_new_ctx);
2924+ }
2925+
2926+ /*
2927+ * ctx may go away after this if the userfault pseudo fd is
2928+ * already released.
2929+ */
2930+out:
2931+ atomic_dec(&ctx->mmap_changing);
2932+ VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
2933+ userfaultfd_ctx_put(ctx);
2934+}
2935+
2936+static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
2937+ struct userfaultfd_wait_queue *ewq)
2938+{
2939+ ewq->msg.event = 0;
2940+ wake_up_locked(&ctx->event_wqh);
2941+ __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
2942+}
2943+
2944+int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
2945+{
2946+ struct userfaultfd_ctx *ctx = NULL, *octx;
2947+ struct userfaultfd_fork_ctx *fctx;
2948+
2949+ octx = vma->vm_userfaultfd_ctx.ctx;
2950+ if (!octx)
2951+ return 0;
2952+
2953+ if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
2954+ userfaultfd_reset_ctx(vma);
2955+ return 0;
2956+ }
2957+
2958+ list_for_each_entry(fctx, fcs, list)
2959+ if (fctx->orig == octx) {
2960+ ctx = fctx->new;
2961+ break;
2962+ }
2963+
2964+ if (!ctx) {
2965+ fctx = kmalloc_obj(*fctx);
2966+ if (!fctx)
2967+ return -ENOMEM;
2968+
2969+ ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
2970+ if (!ctx) {
2971+ kfree(fctx);
2972+ return -ENOMEM;
2973+ }
2974+
2975+ refcount_set(&ctx->refcount, 1);
2976+ ctx->flags = octx->flags;
2977+ ctx->features = octx->features;
2978+ ctx->released = false;
2979+ init_rwsem(&ctx->map_changing_lock);
2980+ atomic_set(&ctx->mmap_changing, 0);
2981+ ctx->mm = vma->vm_mm;
2982+ mmgrab(ctx->mm);
2983+
2984+ userfaultfd_ctx_get(octx);
2985+ down_write(&octx->map_changing_lock);
2986+ atomic_inc(&octx->mmap_changing);
2987+ up_write(&octx->map_changing_lock);
2988+ fctx->orig = octx;
2989+ fctx->new = ctx;
2990+ list_add_tail(&fctx->list, fcs);
2991+ }
2992+
2993+ vma->vm_userfaultfd_ctx.ctx = ctx;
2994+ return 0;
2995+}
2996+
2997+static void dup_fctx(struct userfaultfd_fork_ctx *fctx)
2998+{
2999+ struct userfaultfd_ctx *ctx = fctx->orig;
3000+ struct userfaultfd_wait_queue ewq;
3001+
3002+ msg_init(&ewq.msg);
3003+
3004+ ewq.msg.event = UFFD_EVENT_FORK;
3005+ ewq.msg.arg.reserved.reserved1 = (unsigned long)fctx->new;
3006+
3007+ userfaultfd_event_wait_completion(ctx, &ewq);
3008+}
3009+
3010+void dup_userfaultfd_complete(struct list_head *fcs)
3011+{
3012+ struct userfaultfd_fork_ctx *fctx, *n;
3013+
3014+ list_for_each_entry_safe(fctx, n, fcs, list) {
3015+ dup_fctx(fctx);
3016+ list_del(&fctx->list);
3017+ kfree(fctx);
3018+ }
3019+}
3020+
3021+void dup_userfaultfd_fail(struct list_head *fcs)
3022+{
3023+ struct userfaultfd_fork_ctx *fctx, *n;
3024+
3025+ /*
3026+ * An error has occurred on fork, we will tear memory down, but have
3027+ * allocated memory for fctx's and raised reference counts for both the
3028+ * original and child contexts (and on the mm for each as a result).
3029+ *
3030+ * These would ordinarily be taken care of by a user handling the event,
3031+ * but we are no longer doing so, so manually clean up here.
3032+ *
3033+ * mm tear down will take care of cleaning up VMA contexts.
3034+ */
3035+ list_for_each_entry_safe(fctx, n, fcs, list) {
3036+ struct userfaultfd_ctx *octx = fctx->orig;
3037+ struct userfaultfd_ctx *ctx = fctx->new;
3038+
3039+ atomic_dec(&octx->mmap_changing);
3040+ VM_WARN_ON_ONCE(atomic_read(&octx->mmap_changing) < 0);
3041+ userfaultfd_ctx_put(octx);
3042+ userfaultfd_ctx_put(ctx);
3043+
3044+ list_del(&fctx->list);
3045+ kfree(fctx);
3046+ }
3047+}
3048+
3049+void mremap_userfaultfd_prep(struct vm_area_struct *vma,
3050+ struct vm_userfaultfd_ctx *vm_ctx)
3051+{
3052+ struct userfaultfd_ctx *ctx;
3053+
3054+ ctx = vma->vm_userfaultfd_ctx.ctx;
3055+
3056+ if (!ctx)
3057+ return;
3058+
3059+ if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
3060+ vm_ctx->ctx = ctx;
3061+ userfaultfd_ctx_get(ctx);
3062+ down_write(&ctx->map_changing_lock);
3063+ atomic_inc(&ctx->mmap_changing);
3064+ up_write(&ctx->map_changing_lock);
3065+ } else {
3066+ /* Drop uffd context if remap feature not enabled */
3067+ userfaultfd_reset_ctx(vma);
3068+ }
3069+}
3070+
3071+void mremap_userfaultfd_complete(struct vm_userfaultfd_ctx *vm_ctx,
3072+ unsigned long from, unsigned long to,
3073+ unsigned long len)
3074+{
3075+ struct userfaultfd_ctx *ctx = vm_ctx->ctx;
3076+ struct userfaultfd_wait_queue ewq;
3077+
3078+ if (!ctx)
3079+ return;
3080+
3081+ msg_init(&ewq.msg);
3082+
3083+ ewq.msg.event = UFFD_EVENT_REMAP;
3084+ ewq.msg.arg.remap.from = from;
3085+ ewq.msg.arg.remap.to = to;
3086+ ewq.msg.arg.remap.len = len;
3087+
3088+ userfaultfd_event_wait_completion(ctx, &ewq);
3089+}
3090+
3091+void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
3092+{
3093+ struct userfaultfd_ctx *ctx = vm_ctx->ctx;
3094+
3095+ if (!ctx)
3096+ return;
3097+
3098+ atomic_dec(&ctx->mmap_changing);
3099+ VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);
3100+ userfaultfd_ctx_put(ctx);
3101+}
3102+
3103+bool userfaultfd_remove(struct vm_area_struct *vma,
3104+ unsigned long start, unsigned long end)
3105+{
3106+ struct mm_struct *mm = vma->vm_mm;
3107+ struct userfaultfd_ctx *ctx;
3108+ struct userfaultfd_wait_queue ewq;
3109+
3110+ ctx = vma->vm_userfaultfd_ctx.ctx;
3111+ if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
3112+ return true;
3113+
3114+ userfaultfd_ctx_get(ctx);
3115+ down_write(&ctx->map_changing_lock);
3116+ atomic_inc(&ctx->mmap_changing);
3117+ up_write(&ctx->map_changing_lock);
3118+ mmap_read_unlock(mm);
3119+
3120+ msg_init(&ewq.msg);
3121+
3122+ ewq.msg.event = UFFD_EVENT_REMOVE;
3123+ ewq.msg.arg.remove.start = start;
3124+ ewq.msg.arg.remove.end = end;
3125+
3126+ userfaultfd_event_wait_completion(ctx, &ewq);
3127+
3128+ return false;
3129+}
3130+
3131+static bool has_unmap_ctx(struct userfaultfd_ctx *ctx, struct list_head *unmaps,
3132+ unsigned long start, unsigned long end)
3133+{
3134+ struct userfaultfd_unmap_ctx *unmap_ctx;
3135+
3136+ list_for_each_entry(unmap_ctx, unmaps, list)
3137+ if (unmap_ctx->ctx == ctx && unmap_ctx->start == start &&
3138+ unmap_ctx->end == end)
3139+ return true;
3140+
3141+ return false;
3142+}
3143+
3144+int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
3145+ unsigned long end, struct list_head *unmaps)
3146+{
3147+ struct userfaultfd_unmap_ctx *unmap_ctx;
3148+ struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
3149+
3150+ if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
3151+ has_unmap_ctx(ctx, unmaps, start, end))
3152+ return 0;
3153+
3154+ unmap_ctx = kzalloc_obj(*unmap_ctx);
3155+ if (!unmap_ctx)
3156+ return -ENOMEM;
3157+
3158+ userfaultfd_ctx_get(ctx);
3159+ down_write(&ctx->map_changing_lock);
3160+ atomic_inc(&ctx->mmap_changing);
3161+ up_write(&ctx->map_changing_lock);
3162+ unmap_ctx->ctx = ctx;
3163+ unmap_ctx->start = start;
3164+ unmap_ctx->end = end;
3165+ list_add_tail(&unmap_ctx->list, unmaps);
3166+
3167+ return 0;
3168+}
3169+
3170+void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf)
3171+{
3172+ struct userfaultfd_unmap_ctx *ctx, *n;
3173+ struct userfaultfd_wait_queue ewq;
3174+
3175+ list_for_each_entry_safe(ctx, n, uf, list) {
3176+ msg_init(&ewq.msg);
3177+
3178+ ewq.msg.event = UFFD_EVENT_UNMAP;
3179+ ewq.msg.arg.remove.start = ctx->start;
3180+ ewq.msg.arg.remove.end = ctx->end;
3181+
3182+ userfaultfd_event_wait_completion(ctx->ctx, &ewq);
3183+
3184+ list_del(&ctx->list);
3185+ kfree(ctx);
3186+ }
3187+}
3188+
3189+static int userfaultfd_release(struct inode *inode, struct file *file)
3190+{
3191+ struct userfaultfd_ctx *ctx = file->private_data;
3192+ struct mm_struct *mm = ctx->mm;
3193+ /* len == 0 means wake all */
3194+ struct userfaultfd_wake_range range = { .len = 0, };
3195+
3196+ WRITE_ONCE(ctx->released, true);
3197+
3198+ userfaultfd_release_all(mm, ctx);
3199+
3200+ /*
3201+ * After no new page faults can wait on this fault_*wqh, flush
3202+ * the last page faults that may have been already waiting on
3203+ * the fault_*wqh.
3204+ */
3205+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
3206+ __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &range);
3207+ __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
3208+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
3209+
3210+ /* Flush pending events that may still wait on event_wqh */
3211+ wake_up_all(&ctx->event_wqh);
3212+
3213+ wake_up_poll(&ctx->fd_wqh, EPOLLHUP);
3214+ userfaultfd_ctx_put(ctx);
3215+ return 0;
3216+}
3217+
3218+/* fault_pending_wqh.lock must be hold by the caller */
3219+static inline struct userfaultfd_wait_queue *find_userfault_in(
3220+ wait_queue_head_t *wqh)
3221+{
3222+ wait_queue_entry_t *wq;
3223+ struct userfaultfd_wait_queue *uwq;
3224+
3225+ lockdep_assert_held(&wqh->lock);
3226+
3227+ uwq = NULL;
3228+ if (!waitqueue_active(wqh))
3229+ goto out;
3230+ /* walk in reverse to provide FIFO behavior to read userfaults */
3231+ wq = list_last_entry(&wqh->head, typeof(*wq), entry);
3232+ uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
3233+out:
3234+ return uwq;
3235+}
3236+
3237+static inline struct userfaultfd_wait_queue *find_userfault(
3238+ struct userfaultfd_ctx *ctx)
3239+{
3240+ return find_userfault_in(&ctx->fault_pending_wqh);
3241+}
3242+
3243+static inline struct userfaultfd_wait_queue *find_userfault_evt(
3244+ struct userfaultfd_ctx *ctx)
3245+{
3246+ return find_userfault_in(&ctx->event_wqh);
3247+}
3248+
3249+static __poll_t userfaultfd_poll(struct file *file, poll_table *wait)
3250+{
3251+ struct userfaultfd_ctx *ctx = file->private_data;
3252+ __poll_t ret;
3253+
3254+ poll_wait(file, &ctx->fd_wqh, wait);
3255+
3256+ if (!userfaultfd_is_initialized(ctx))
3257+ return EPOLLERR;
3258+
3259+ /*
3260+ * poll() never guarantees that read won't block.
3261+ * userfaults can be waken before they're read().
3262+ */
3263+ if (unlikely(!(file->f_flags & O_NONBLOCK)))
3264+ return EPOLLERR;
3265+ /*
3266+ * lockless access to see if there are pending faults
3267+ * __pollwait last action is the add_wait_queue but
3268+ * the spin_unlock would allow the waitqueue_active to
3269+ * pass above the actual list_add inside
3270+ * add_wait_queue critical section. So use a full
3271+ * memory barrier to serialize the list_add write of
3272+ * add_wait_queue() with the waitqueue_active read
3273+ * below.
3274+ */
3275+ ret = 0;
3276+ smp_mb();
3277+ if (waitqueue_active(&ctx->fault_pending_wqh))
3278+ ret = EPOLLIN;
3279+ else if (waitqueue_active(&ctx->event_wqh))
3280+ ret = EPOLLIN;
3281+
3282+ return ret;
3283+}
3284+
3285+static const struct file_operations userfaultfd_fops;
3286+
3287+static int resolve_userfault_fork(struct userfaultfd_ctx *new,
3288+ struct inode *inode,
3289+ struct uffd_msg *msg)
3290+{
3291+ int fd;
3292+
3293+ fd = anon_inode_create_getfd("[userfaultfd]", &userfaultfd_fops, new,
3294+ O_RDONLY | (new->flags & UFFD_SHARED_FCNTL_FLAGS), inode);
3295+ if (fd < 0)
3296+ return fd;
3297+
3298+ msg->arg.reserved.reserved1 = 0;
3299+ msg->arg.fork.ufd = fd;
3300+ return 0;
3301+}
3302+
3303+static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
3304+ struct uffd_msg *msg, struct inode *inode)
3305+{
3306+ ssize_t ret;
3307+ DECLARE_WAITQUEUE(wait, current);
3308+ struct userfaultfd_wait_queue *uwq;
3309+ /*
3310+ * Handling fork event requires sleeping operations, so
3311+ * we drop the event_wqh lock, then do these ops, then
3312+ * lock it back and wake up the waiter. While the lock is
3313+ * dropped the ewq may go away so we keep track of it
3314+ * carefully.
3315+ */
3316+ LIST_HEAD(fork_event);
3317+ struct userfaultfd_ctx *fork_nctx = NULL;
3318+
3319+ /* always take the fd_wqh lock before the fault_pending_wqh lock */
3320+ spin_lock_irq(&ctx->fd_wqh.lock);
3321+ __add_wait_queue(&ctx->fd_wqh, &wait);
3322+ for (;;) {
3323+ set_current_state(TASK_INTERRUPTIBLE);
3324+ spin_lock(&ctx->fault_pending_wqh.lock);
3325+ uwq = find_userfault(ctx);
3326+ if (uwq) {
3327+ /*
3328+ * Use a seqcount to repeat the lockless check
3329+ * in wake_userfault() to avoid missing
3330+ * wakeups because during the refile both
3331+ * waitqueue could become empty if this is the
3332+ * only userfault.
3333+ */
3334+ write_seqcount_begin(&ctx->refile_seq);
3335+
3336+ /*
3337+ * The fault_pending_wqh.lock prevents the uwq
3338+ * to disappear from under us.
3339+ *
3340+ * Refile this userfault from
3341+ * fault_pending_wqh to fault_wqh, it's not
3342+ * pending anymore after we read it.
3343+ *
3344+ * Use list_del() by hand (as
3345+ * userfaultfd_wake_function also uses
3346+ * list_del_init() by hand) to be sure nobody
3347+ * changes __remove_wait_queue() to use
3348+ * list_del_init() in turn breaking the
3349+ * !list_empty_careful() check in
3350+ * handle_userfault(). The uwq->wq.head list
3351+ * must never be empty at any time during the
3352+ * refile, or the waitqueue could disappear
3353+ * from under us. The "wait_queue_head_t"
3354+ * parameter of __remove_wait_queue() is unused
3355+ * anyway.
3356+ */
3357+ list_del(&uwq->wq.entry);
3358+ add_wait_queue(&ctx->fault_wqh, &uwq->wq);
3359+
3360+ write_seqcount_end(&ctx->refile_seq);
3361+
3362+ /* careful to always initialize msg if ret == 0 */
3363+ *msg = uwq->msg;
3364+ spin_unlock(&ctx->fault_pending_wqh.lock);
3365+ ret = 0;
3366+ break;
3367+ }
3368+ spin_unlock(&ctx->fault_pending_wqh.lock);
3369+
3370+ spin_lock(&ctx->event_wqh.lock);
3371+ uwq = find_userfault_evt(ctx);
3372+ if (uwq) {
3373+ *msg = uwq->msg;
3374+
3375+ if (uwq->msg.event == UFFD_EVENT_FORK) {
3376+ fork_nctx = (struct userfaultfd_ctx *)
3377+ (unsigned long)
3378+ uwq->msg.arg.reserved.reserved1;
3379+ list_move(&uwq->wq.entry, &fork_event);
3380+ /*
3381+ * fork_nctx can be freed as soon as
3382+ * we drop the lock, unless we take a
3383+ * reference on it.
3384+ */
3385+ userfaultfd_ctx_get(fork_nctx);
3386+ spin_unlock(&ctx->event_wqh.lock);
3387+ ret = 0;
3388+ break;
3389+ }
3390+
3391+ userfaultfd_event_complete(ctx, uwq);
3392+ spin_unlock(&ctx->event_wqh.lock);
3393+ ret = 0;
3394+ break;
3395+ }
3396+ spin_unlock(&ctx->event_wqh.lock);
3397+
3398+ if (signal_pending(current)) {
3399+ ret = -ERESTARTSYS;
3400+ break;
3401+ }
3402+ if (no_wait) {
3403+ ret = -EAGAIN;
3404+ break;
3405+ }
3406+ spin_unlock_irq(&ctx->fd_wqh.lock);
3407+ schedule();
3408+ spin_lock_irq(&ctx->fd_wqh.lock);
3409+ }
3410+ __remove_wait_queue(&ctx->fd_wqh, &wait);
3411+ __set_current_state(TASK_RUNNING);
3412+ spin_unlock_irq(&ctx->fd_wqh.lock);
3413+
3414+ if (!ret && msg->event == UFFD_EVENT_FORK) {
3415+ ret = resolve_userfault_fork(fork_nctx, inode, msg);
3416+ spin_lock_irq(&ctx->event_wqh.lock);
3417+ if (!list_empty(&fork_event)) {
3418+ /*
3419+ * The fork thread didn't abort, so we can
3420+ * drop the temporary refcount.
3421+ */
3422+ userfaultfd_ctx_put(fork_nctx);
3423+
3424+ uwq = list_first_entry(&fork_event,
3425+ typeof(*uwq),
3426+ wq.entry);
3427+ /*
3428+ * If fork_event list wasn't empty and in turn
3429+ * the event wasn't already released by fork
3430+ * (the event is allocated on fork kernel
3431+ * stack), put the event back to its place in
3432+ * the event_wq. fork_event head will be freed
3433+ * as soon as we return so the event cannot
3434+ * stay queued there no matter the current
3435+ * "ret" value.
3436+ */
3437+ list_del(&uwq->wq.entry);
3438+ __add_wait_queue(&ctx->event_wqh, &uwq->wq);
3439+
3440+ /*
3441+ * Leave the event in the waitqueue and report
3442+ * error to userland if we failed to resolve
3443+ * the userfault fork.
3444+ */
3445+ if (likely(!ret))
3446+ userfaultfd_event_complete(ctx, uwq);
3447+ } else {
3448+ /*
3449+ * Here the fork thread aborted and the
3450+ * refcount from the fork thread on fork_nctx
3451+ * has already been released. We still hold
3452+ * the reference we took before releasing the
3453+ * lock above. If resolve_userfault_fork
3454+ * failed we've to drop it because the
3455+ * fork_nctx has to be freed in such case. If
3456+ * it succeeded we'll hold it because the new
3457+ * uffd references it.
3458+ */
3459+ if (ret)
3460+ userfaultfd_ctx_put(fork_nctx);
3461+ }
3462+ spin_unlock_irq(&ctx->event_wqh.lock);
3463+ }
3464+
3465+ return ret;
3466+}
3467+
3468+static ssize_t userfaultfd_read_iter(struct kiocb *iocb, struct iov_iter *to)
3469+{
3470+ struct file *file = iocb->ki_filp;
3471+ struct userfaultfd_ctx *ctx = file->private_data;
3472+ ssize_t _ret, ret = 0;
3473+ struct uffd_msg msg;
3474+ struct inode *inode = file_inode(file);
3475+ bool no_wait;
3476+
3477+ if (!userfaultfd_is_initialized(ctx))
3478+ return -EINVAL;
3479+
3480+ no_wait = file->f_flags & O_NONBLOCK || iocb->ki_flags & IOCB_NOWAIT;
3481+ for (;;) {
3482+ if (iov_iter_count(to) < sizeof(msg))
3483+ return ret ? ret : -EINVAL;
3484+ _ret = userfaultfd_ctx_read(ctx, no_wait, &msg, inode);
3485+ if (_ret < 0)
3486+ return ret ? ret : _ret;
3487+ _ret = !copy_to_iter_full(&msg, sizeof(msg), to);
3488+ if (_ret)
3489+ return ret ? ret : -EFAULT;
3490+ ret += sizeof(msg);
3491+ /*
3492+ * Allow to read more than one fault at time but only
3493+ * block if waiting for the very first one.
3494+ */
3495+ no_wait = true;
3496+ }
3497+}
3498+
3499+static void __wake_userfault(struct userfaultfd_ctx *ctx,
3500+ struct userfaultfd_wake_range *range)
3501+{
3502+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
3503+ /* wake all in the range and autoremove */
3504+ if (waitqueue_active(&ctx->fault_pending_wqh))
3505+ __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
3506+ range);
3507+ if (waitqueue_active(&ctx->fault_wqh))
3508+ __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, range);
3509+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
3510+}
3511+
3512+static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
3513+ struct userfaultfd_wake_range *range)
3514+{
3515+ unsigned seq;
3516+ bool need_wakeup;
3517+
3518+ /*
3519+ * To be sure waitqueue_active() is not reordered by the CPU
3520+ * before the pagetable update, use an explicit SMP memory
3521+ * barrier here. PT lock release or mmap_read_unlock(mm) still
3522+ * have release semantics that can allow the
3523+ * waitqueue_active() to be reordered before the pte update.
3524+ */
3525+ smp_mb();
3526+
3527+ /*
3528+ * Use waitqueue_active because it's very frequent to
3529+ * change the address space atomically even if there are no
3530+ * userfaults yet. So we take the spinlock only when we're
3531+ * sure we've userfaults to wake.
3532+ */
3533+ do {
3534+ seq = read_seqcount_begin(&ctx->refile_seq);
3535+ need_wakeup = waitqueue_active(&ctx->fault_pending_wqh) ||
3536+ waitqueue_active(&ctx->fault_wqh);
3537+ cond_resched();
3538+ } while (read_seqcount_retry(&ctx->refile_seq, seq));
3539+ if (need_wakeup)
3540+ __wake_userfault(ctx, range);
3541+}
3542+
3543+static __always_inline int validate_unaligned_range(
3544+ struct mm_struct *mm, __u64 start, __u64 len)
3545+{
3546+ __u64 task_size = mm->task_size;
3547+
3548+ if (len & ~PAGE_MASK)
3549+ return -EINVAL;
3550+ if (!len)
3551+ return -EINVAL;
3552+ if (start >= task_size)
3553+ return -EINVAL;
3554+ if (len > task_size - start)
3555+ return -EINVAL;
3556+ if (start + len <= start)
3557+ return -EINVAL;
3558+ return 0;
3559+}
3560+
3561+static __always_inline int validate_range(struct mm_struct *mm,
3562+ __u64 start, __u64 len)
3563+{
3564+ if (start & ~PAGE_MASK)
3565+ return -EINVAL;
3566+
3567+ return validate_unaligned_range(mm, start, len);
3568+}
3569+
3570+static int userfaultfd_register(struct userfaultfd_ctx *ctx,
3571+ unsigned long arg)
3572+{
3573+ struct mm_struct *mm = ctx->mm;
3574+ struct vm_area_struct *vma, *cur;
3575+ int ret;
3576+ struct uffdio_register uffdio_register;
3577+ struct uffdio_register __user *user_uffdio_register;
3578+ vm_flags_t vm_flags;
3579+ bool found;
3580+ bool basic_ioctls;
3581+ unsigned long start, end;
3582+ struct vma_iterator vmi;
3583+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
3584+
3585+ user_uffdio_register = (struct uffdio_register __user *) arg;
3586+
3587+ ret = -EFAULT;
3588+ if (copy_from_user(&uffdio_register, user_uffdio_register,
3589+ sizeof(uffdio_register)-sizeof(__u64)))
3590+ goto out;
3591+
3592+ ret = -EINVAL;
3593+ if (!uffdio_register.mode)
3594+ goto out;
3595+ if (uffdio_register.mode & ~UFFD_API_REGISTER_MODES)
3596+ goto out;
3597+ vm_flags = 0;
3598+ if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
3599+ vm_flags |= VM_UFFD_MISSING;
3600+ if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
3601+ if (!pgtable_supports_uffd_wp())
3602+ goto out;
3603+
3604+ vm_flags |= VM_UFFD_WP;
3605+ }
3606+ if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
3607+#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
3608+ goto out;
3609+#endif
3610+ vm_flags |= VM_UFFD_MINOR;
3611+ }
3612+
3613+ ret = validate_range(mm, uffdio_register.range.start,
3614+ uffdio_register.range.len);
3615+ if (ret)
3616+ goto out;
3617+
3618+ start = uffdio_register.range.start;
3619+ end = start + uffdio_register.range.len;
3620+
3621+ ret = -ENOMEM;
3622+ if (!mmget_not_zero(mm))
3623+ goto out;
3624+
3625+ ret = -EINVAL;
3626+ mmap_write_lock(mm);
3627+ vma_iter_init(&vmi, mm, start);
3628+ vma = vma_find(&vmi, end);
3629+ if (!vma)
3630+ goto out_unlock;
3631+
3632+ /*
3633+ * If the first vma contains huge pages, make sure start address
3634+ * is aligned to huge page size.
3635+ */
3636+ if (is_vm_hugetlb_page(vma)) {
3637+ unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
3638+
3639+ if (start & (vma_hpagesize - 1))
3640+ goto out_unlock;
3641+ }
3642+
3643+ /*
3644+ * Search for not compatible vmas.
3645+ */
3646+ found = false;
3647+ basic_ioctls = false;
3648+ cur = vma;
3649+ do {
3650+ cond_resched();
3651+
3652+ VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
3653+ !!(cur->vm_flags & __VM_UFFD_FLAGS));
3654+
3655+ /* check not compatible vmas */
3656+ ret = -EINVAL;
3657+ if (!vma_can_userfault(cur, vm_flags, wp_async))
3658+ goto out_unlock;
3659+
3660+ /*
3661+ * UFFDIO_COPY will fill file holes even without
3662+ * PROT_WRITE. This check enforces that if this is a
3663+ * MAP_SHARED, the process has write permission to the backing
3664+ * file. If VM_MAYWRITE is set it also enforces that on a
3665+ * MAP_SHARED vma: there is no F_WRITE_SEAL and no further
3666+ * F_WRITE_SEAL can be taken until the vma is destroyed.
3667+ */
3668+ ret = -EPERM;
3669+ if (unlikely(!(cur->vm_flags & VM_MAYWRITE)))
3670+ goto out_unlock;
3671+
3672+ /*
3673+ * If this vma contains ending address, and huge pages
3674+ * check alignment.
3675+ */
3676+ if (is_vm_hugetlb_page(cur) && end <= cur->vm_end &&
3677+ end > cur->vm_start) {
3678+ unsigned long vma_hpagesize = vma_kernel_pagesize(cur);
3679+
3680+ ret = -EINVAL;
3681+
3682+ if (end & (vma_hpagesize - 1))
3683+ goto out_unlock;
3684+ }
3685+ if ((vm_flags & VM_UFFD_WP) && !(cur->vm_flags & VM_MAYWRITE))
3686+ goto out_unlock;
3687+
3688+ /*
3689+ * Check that this vma isn't already owned by a
3690+ * different userfaultfd. We can't allow more than one
3691+ * userfaultfd to own a single vma simultaneously or we
3692+ * wouldn't know which one to deliver the userfaults to.
3693+ */
3694+ ret = -EBUSY;
3695+ if (cur->vm_userfaultfd_ctx.ctx &&
3696+ cur->vm_userfaultfd_ctx.ctx != ctx)
3697+ goto out_unlock;
3698+
3699+ /*
3700+ * Note vmas containing huge pages
3701+ */
3702+ if (is_vm_hugetlb_page(cur))
3703+ basic_ioctls = true;
3704+
3705+ found = true;
3706+ } for_each_vma_range(vmi, cur, end);
3707+ VM_WARN_ON_ONCE(!found);
3708+
3709+ ret = userfaultfd_register_range(ctx, vma, vm_flags, start, end,
3710+ wp_async);
3711+
3712+out_unlock:
3713+ mmap_write_unlock(mm);
3714+ mmput(mm);
3715+ if (!ret) {
3716+ __u64 ioctls_out;
3717+
3718+ ioctls_out = basic_ioctls ? UFFD_API_RANGE_IOCTLS_BASIC :
3719+ UFFD_API_RANGE_IOCTLS;
3720+
3721+ /*
3722+ * Declare the WP ioctl only if the WP mode is
3723+ * specified and all checks passed with the range
3724+ */
3725+ if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_WP))
3726+ ioctls_out &= ~((__u64)1 << _UFFDIO_WRITEPROTECT);
3727+
3728+ /* CONTINUE ioctl is only supported for MINOR ranges. */
3729+ if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
3730+ ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
3731+
3732+ /*
3733+ * Now that we scanned all vmas we can already tell
3734+ * userland which ioctls methods are guaranteed to
3735+ * succeed on this range.
3736+ */
3737+ if (put_user(ioctls_out, &user_uffdio_register->ioctls))
3738+ ret = -EFAULT;
3739+ }
3740+out:
3741+ return ret;
3742+}
3743+
3744+static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
3745+ unsigned long arg)
3746+{
3747+ struct mm_struct *mm = ctx->mm;
3748+ struct vm_area_struct *vma, *prev, *cur;
3749+ int ret;
3750+ struct uffdio_range uffdio_unregister;
3751+ bool found;
3752+ unsigned long start, end, vma_end;
3753+ const void __user *buf = (void __user *)arg;
3754+ struct vma_iterator vmi;
3755+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
3756+
3757+ ret = -EFAULT;
3758+ if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
3759+ goto out;
3760+
3761+ ret = validate_range(mm, uffdio_unregister.start,
3762+ uffdio_unregister.len);
3763+ if (ret)
3764+ goto out;
3765+
3766+ start = uffdio_unregister.start;
3767+ end = start + uffdio_unregister.len;
3768+
3769+ ret = -ENOMEM;
3770+ if (!mmget_not_zero(mm))
3771+ goto out;
3772+
3773+ mmap_write_lock(mm);
3774+ ret = -EINVAL;
3775+ vma_iter_init(&vmi, mm, start);
3776+ vma = vma_find(&vmi, end);
3777+ if (!vma)
3778+ goto out_unlock;
3779+
3780+ /*
3781+ * If the first vma contains huge pages, make sure start address
3782+ * is aligned to huge page size.
3783+ */
3784+ if (is_vm_hugetlb_page(vma)) {
3785+ unsigned long vma_hpagesize = vma_kernel_pagesize(vma);
3786+
3787+ if (start & (vma_hpagesize - 1))
3788+ goto out_unlock;
3789+ }
3790+
3791+ /*
3792+ * Search for not compatible vmas.
3793+ */
3794+ found = false;
3795+ cur = vma;
3796+ do {
3797+ cond_resched();
3798+
3799+ VM_WARN_ON_ONCE(!!cur->vm_userfaultfd_ctx.ctx ^
3800+ !!(cur->vm_flags & __VM_UFFD_FLAGS));
3801+
3802+ /*
3803+ * Prevent unregistering through a different userfaultfd than
3804+ * the one used for registration.
3805+ */
3806+ if (cur->vm_userfaultfd_ctx.ctx &&
3807+ cur->vm_userfaultfd_ctx.ctx != ctx)
3808+ goto out_unlock;
3809+
3810+ /*
3811+ * Check not compatible vmas, not strictly required
3812+ * here as not compatible vmas cannot have an
3813+ * userfaultfd_ctx registered on them, but this
3814+ * provides for more strict behavior to notice
3815+ * unregistration errors.
3816+ */
3817+ if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
3818+ goto out_unlock;
3819+
3820+ found = true;
3821+ } for_each_vma_range(vmi, cur, end);
3822+ VM_WARN_ON_ONCE(!found);
3823+
3824+ vma_iter_set(&vmi, start);
3825+ prev = vma_prev(&vmi);
3826+ if (vma->vm_start < start)
3827+ prev = vma;
3828+
3829+ ret = 0;
3830+ for_each_vma_range(vmi, vma, end) {
3831+ cond_resched();
3832+
3833+ /* VMA not registered with userfaultfd. */
3834+ if (!vma->vm_userfaultfd_ctx.ctx)
3835+ goto skip;
3836+
3837+ VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx != ctx);
3838+ VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, wp_async));
3839+ VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
3840+
3841+ if (vma->vm_start > start)
3842+ start = vma->vm_start;
3843+ vma_end = min(end, vma->vm_end);
3844+
3845+ if (userfaultfd_missing(vma)) {
3846+ /*
3847+ * Wake any concurrent pending userfault while
3848+ * we unregister, so they will not hang
3849+ * permanently and it avoids userland to call
3850+ * UFFDIO_WAKE explicitly.
3851+ */
3852+ struct userfaultfd_wake_range range;
3853+ range.start = start;
3854+ range.len = vma_end - start;
3855+ wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range);
3856+ }
3857+
3858+ vma = userfaultfd_clear_vma(&vmi, prev, vma,
3859+ start, vma_end);
3860+ if (IS_ERR(vma)) {
3861+ ret = PTR_ERR(vma);
3862+ break;
3863+ }
3864+
3865+skip:
3866+ prev = vma;
3867+ start = vma->vm_end;
3868+ }
3869+
3870+out_unlock:
3871+ mmap_write_unlock(mm);
3872+ mmput(mm);
3873+out:
3874+ return ret;
3875+}
3876+
3877+/*
3878+ * userfaultfd_wake may be used in combination with the
3879+ * UFFDIO_*_MODE_DONTWAKE to wakeup userfaults in batches.
3880+ */
3881+static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
3882+ unsigned long arg)
3883+{
3884+ int ret;
3885+ struct uffdio_range uffdio_wake;
3886+ struct userfaultfd_wake_range range;
3887+ const void __user *buf = (void __user *)arg;
3888+
3889+ ret = -EFAULT;
3890+ if (copy_from_user(&uffdio_wake, buf, sizeof(uffdio_wake)))
3891+ goto out;
3892+
3893+ ret = validate_range(ctx->mm, uffdio_wake.start, uffdio_wake.len);
3894+ if (ret)
3895+ goto out;
3896+
3897+ range.start = uffdio_wake.start;
3898+ range.len = uffdio_wake.len;
3899+
3900+ /*
3901+ * len == 0 means wake all and we don't want to wake all here,
3902+ * so check it again to be sure.
3903+ */
3904+ VM_WARN_ON_ONCE(!range.len);
3905+
3906+ wake_userfault(ctx, &range);
3907+ ret = 0;
3908+
3909+out:
3910+ return ret;
3911+}
3912+
3913+static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
3914+ unsigned long arg)
3915+{
3916+ __s64 ret;
3917+ struct uffdio_copy uffdio_copy;
3918+ struct uffdio_copy __user *user_uffdio_copy;
3919+ struct userfaultfd_wake_range range;
3920+ uffd_flags_t flags = 0;
3921+
3922+ user_uffdio_copy = (struct uffdio_copy __user *) arg;
3923+
3924+ ret = -EAGAIN;
3925+ if (unlikely(atomic_read(&ctx->mmap_changing))) {
3926+ if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
3927+ return -EFAULT;
3928+ goto out;
3929+ }
3930+
3931+ ret = -EFAULT;
3932+ if (copy_from_user(&uffdio_copy, user_uffdio_copy,
3933+ /* don't copy "copy" last field */
3934+ sizeof(uffdio_copy)-sizeof(__s64)))
3935+ goto out;
3936+
3937+ ret = validate_unaligned_range(ctx->mm, uffdio_copy.src,
3938+ uffdio_copy.len);
3939+ if (ret)
3940+ goto out;
3941+ ret = validate_range(ctx->mm, uffdio_copy.dst, uffdio_copy.len);
3942+ if (ret)
3943+ goto out;
3944+
3945+ ret = -EINVAL;
3946+ if (uffdio_copy.mode & ~(UFFDIO_COPY_MODE_DONTWAKE|UFFDIO_COPY_MODE_WP))
3947+ goto out;
3948+ if (uffdio_copy.mode & UFFDIO_COPY_MODE_WP)
3949+ flags |= MFILL_ATOMIC_WP;
3950+ if (mmget_not_zero(ctx->mm)) {
3951+ ret = mfill_atomic_copy(ctx, uffdio_copy.dst, uffdio_copy.src,
3952+ uffdio_copy.len, flags);
3953+ mmput(ctx->mm);
3954+ } else {
3955+ return -ESRCH;
3956+ }
3957+ if (unlikely(put_user(ret, &user_uffdio_copy->copy)))
3958+ return -EFAULT;
3959+ if (ret < 0)
3960+ goto out;
3961+ VM_WARN_ON_ONCE(!ret);
3962+ /* len == 0 would wake all */
3963+ range.len = ret;
3964+ if (!(uffdio_copy.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
3965+ range.start = uffdio_copy.dst;
3966+ wake_userfault(ctx, &range);
3967+ }
3968+ ret = range.len == uffdio_copy.len ? 0 : -EAGAIN;
3969+out:
3970+ return ret;
3971+}
3972+
3973+static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
3974+ unsigned long arg)
3975+{
3976+ __s64 ret;
3977+ struct uffdio_zeropage uffdio_zeropage;
3978+ struct uffdio_zeropage __user *user_uffdio_zeropage;
3979+ struct userfaultfd_wake_range range;
3980+
3981+ user_uffdio_zeropage = (struct uffdio_zeropage __user *) arg;
3982+
3983+ ret = -EAGAIN;
3984+ if (unlikely(atomic_read(&ctx->mmap_changing))) {
3985+ if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
3986+ return -EFAULT;
3987+ goto out;
3988+ }
3989+
3990+ ret = -EFAULT;
3991+ if (copy_from_user(&uffdio_zeropage, user_uffdio_zeropage,
3992+ /* don't copy "zeropage" last field */
3993+ sizeof(uffdio_zeropage)-sizeof(__s64)))
3994+ goto out;
3995+
3996+ ret = validate_range(ctx->mm, uffdio_zeropage.range.start,
3997+ uffdio_zeropage.range.len);
3998+ if (ret)
3999+ goto out;
4000+ ret = -EINVAL;
4001+ if (uffdio_zeropage.mode & ~UFFDIO_ZEROPAGE_MODE_DONTWAKE)
4002+ goto out;
4003+
4004+ if (mmget_not_zero(ctx->mm)) {
4005+ ret = mfill_atomic_zeropage(ctx, uffdio_zeropage.range.start,
4006+ uffdio_zeropage.range.len);
4007+ mmput(ctx->mm);
4008+ } else {
4009+ return -ESRCH;
4010+ }
4011+ if (unlikely(put_user(ret, &user_uffdio_zeropage->zeropage)))
4012+ return -EFAULT;
4013+ if (ret < 0)
4014+ goto out;
4015+ /* len == 0 would wake all */
4016+ VM_WARN_ON_ONCE(!ret);
4017+ range.len = ret;
4018+ if (!(uffdio_zeropage.mode & UFFDIO_ZEROPAGE_MODE_DONTWAKE)) {
4019+ range.start = uffdio_zeropage.range.start;
4020+ wake_userfault(ctx, &range);
4021+ }
4022+ ret = range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
4023+out:
4024+ return ret;
4025+}
4026+
4027+static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
4028+ unsigned long arg)
4029+{
4030+ int ret;
4031+ struct uffdio_writeprotect uffdio_wp;
4032+ struct uffdio_writeprotect __user *user_uffdio_wp;
4033+ struct userfaultfd_wake_range range;
4034+ bool mode_wp, mode_dontwake;
4035+
4036+ if (atomic_read(&ctx->mmap_changing))
4037+ return -EAGAIN;
4038+
4039+ user_uffdio_wp = (struct uffdio_writeprotect __user *) arg;
4040+
4041+ if (copy_from_user(&uffdio_wp, user_uffdio_wp,
4042+ sizeof(struct uffdio_writeprotect)))
4043+ return -EFAULT;
4044+
4045+ ret = validate_range(ctx->mm, uffdio_wp.range.start,
4046+ uffdio_wp.range.len);
4047+ if (ret)
4048+ return ret;
4049+
4050+ if (uffdio_wp.mode & ~(UFFDIO_WRITEPROTECT_MODE_DONTWAKE |
4051+ UFFDIO_WRITEPROTECT_MODE_WP))
4052+ return -EINVAL;
4053+
4054+ mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP;
4055+ mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE;
4056+
4057+ if (mode_wp && mode_dontwake)
4058+ return -EINVAL;
4059+
4060+ if (mmget_not_zero(ctx->mm)) {
4061+ ret = mwriteprotect_range(ctx, uffdio_wp.range.start,
4062+ uffdio_wp.range.len, mode_wp);
4063+ mmput(ctx->mm);
4064+ } else {
4065+ return -ESRCH;
4066+ }
4067+
4068+ if (ret)
4069+ return ret;
4070+
4071+ if (!mode_wp && !mode_dontwake) {
4072+ range.start = uffdio_wp.range.start;
4073+ range.len = uffdio_wp.range.len;
4074+ wake_userfault(ctx, &range);
4075+ }
4076+ return ret;
4077+}
4078+
4079+static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
4080+{
4081+ __s64 ret;
4082+ struct uffdio_continue uffdio_continue;
4083+ struct uffdio_continue __user *user_uffdio_continue;
4084+ struct userfaultfd_wake_range range;
4085+ uffd_flags_t flags = 0;
4086+
4087+ user_uffdio_continue = (struct uffdio_continue __user *)arg;
4088+
4089+ ret = -EAGAIN;
4090+ if (unlikely(atomic_read(&ctx->mmap_changing))) {
4091+ if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
4092+ return -EFAULT;
4093+ goto out;
4094+ }
4095+
4096+ ret = -EFAULT;
4097+ if (copy_from_user(&uffdio_continue, user_uffdio_continue,
4098+ /* don't copy the output fields */
4099+ sizeof(uffdio_continue) - (sizeof(__s64))))
4100+ goto out;
4101+
4102+ ret = validate_range(ctx->mm, uffdio_continue.range.start,
4103+ uffdio_continue.range.len);
4104+ if (ret)
4105+ goto out;
4106+
4107+ ret = -EINVAL;
4108+ if (uffdio_continue.mode & ~(UFFDIO_CONTINUE_MODE_DONTWAKE |
4109+ UFFDIO_CONTINUE_MODE_WP))
4110+ goto out;
4111+ if (uffdio_continue.mode & UFFDIO_CONTINUE_MODE_WP)
4112+ flags |= MFILL_ATOMIC_WP;
4113+
4114+ if (mmget_not_zero(ctx->mm)) {
4115+ ret = mfill_atomic_continue(ctx, uffdio_continue.range.start,
4116+ uffdio_continue.range.len, flags);
4117+ mmput(ctx->mm);
4118+ } else {
4119+ return -ESRCH;
4120+ }
4121+
4122+ if (unlikely(put_user(ret, &user_uffdio_continue->mapped)))
4123+ return -EFAULT;
4124+ if (ret < 0)
4125+ goto out;
4126+
4127+ /* len == 0 would wake all */
4128+ VM_WARN_ON_ONCE(!ret);
4129+ range.len = ret;
4130+ if (!(uffdio_continue.mode & UFFDIO_CONTINUE_MODE_DONTWAKE)) {
4131+ range.start = uffdio_continue.range.start;
4132+ wake_userfault(ctx, &range);
4133+ }
4134+ ret = range.len == uffdio_continue.range.len ? 0 : -EAGAIN;
4135+
4136+out:
4137+ return ret;
4138+}
4139+
4140+static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long arg)
4141+{
4142+ __s64 ret;
4143+ struct uffdio_poison uffdio_poison;
4144+ struct uffdio_poison __user *user_uffdio_poison;
4145+ struct userfaultfd_wake_range range;
4146+
4147+ user_uffdio_poison = (struct uffdio_poison __user *)arg;
4148+
4149+ ret = -EAGAIN;
4150+ if (unlikely(atomic_read(&ctx->mmap_changing))) {
4151+ if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
4152+ return -EFAULT;
4153+ goto out;
4154+ }
4155+
4156+ ret = -EFAULT;
4157+ if (copy_from_user(&uffdio_poison, user_uffdio_poison,
4158+ /* don't copy the output fields */
4159+ sizeof(uffdio_poison) - (sizeof(__s64))))
4160+ goto out;
4161+
4162+ ret = validate_range(ctx->mm, uffdio_poison.range.start,
4163+ uffdio_poison.range.len);
4164+ if (ret)
4165+ goto out;
4166+
4167+ ret = -EINVAL;
4168+ if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE)
4169+ goto out;
4170+
4171+ if (mmget_not_zero(ctx->mm)) {
4172+ ret = mfill_atomic_poison(ctx, uffdio_poison.range.start,
4173+ uffdio_poison.range.len, 0);
4174+ mmput(ctx->mm);
4175+ } else {
4176+ return -ESRCH;
4177+ }
4178+
4179+ if (unlikely(put_user(ret, &user_uffdio_poison->updated)))
4180+ return -EFAULT;
4181+ if (ret < 0)
4182+ goto out;
4183+
4184+ /* len == 0 would wake all */
4185+ VM_WARN_ON_ONCE(!ret);
4186+ range.len = ret;
4187+ if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) {
4188+ range.start = uffdio_poison.range.start;
4189+ wake_userfault(ctx, &range);
4190+ }
4191+ ret = range.len == uffdio_poison.range.len ? 0 : -EAGAIN;
4192+
4193+out:
4194+ return ret;
4195+}
4196+
4197+bool userfaultfd_wp_async(struct vm_area_struct *vma)
4198+{
4199+ return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
4200+}
4201+
4202+static inline unsigned int uffd_ctx_features(__u64 user_features)
4203+{
4204+ /*
4205+ * For the current set of features the bits just coincide. Set
4206+ * UFFD_FEATURE_INITIALIZED to mark the features as enabled.
4207+ */
4208+ return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
4209+}
4210+
4211+static int userfaultfd_move(struct userfaultfd_ctx *ctx,
4212+ unsigned long arg)
4213+{
4214+ __s64 ret;
4215+ struct uffdio_move uffdio_move;
4216+ struct uffdio_move __user *user_uffdio_move;
4217+ struct userfaultfd_wake_range range;
4218+ struct mm_struct *mm = ctx->mm;
4219+
4220+ user_uffdio_move = (struct uffdio_move __user *) arg;
4221+
4222+ ret = -EAGAIN;
4223+ if (unlikely(atomic_read(&ctx->mmap_changing))) {
4224+ if (unlikely(put_user(ret, &user_uffdio_move->move)))
4225+ return -EFAULT;
4226+ goto out;
4227+ }
4228+
4229+ if (copy_from_user(&uffdio_move, user_uffdio_move,
4230+ /* don't copy "move" last field */
4231+ sizeof(uffdio_move)-sizeof(__s64)))
4232+ return -EFAULT;
4233+
4234+ /* Do not allow cross-mm moves. */
4235+ if (mm != current->mm)
4236+ return -EINVAL;
4237+
4238+ ret = validate_range(mm, uffdio_move.dst, uffdio_move.len);
4239+ if (ret)
4240+ return ret;
4241+
4242+ ret = validate_range(mm, uffdio_move.src, uffdio_move.len);
4243+ if (ret)
4244+ return ret;
4245+
4246+ if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES|
4247+ UFFDIO_MOVE_MODE_DONTWAKE))
4248+ return -EINVAL;
4249+
4250+ if (mmget_not_zero(mm)) {
4251+ ret = move_pages(ctx, uffdio_move.dst, uffdio_move.src,
4252+ uffdio_move.len, uffdio_move.mode);
4253+ mmput(mm);
4254+ } else {
4255+ return -ESRCH;
4256+ }
4257+
4258+ if (unlikely(put_user(ret, &user_uffdio_move->move)))
4259+ return -EFAULT;
4260+ if (ret < 0)
4261+ goto out;
4262+
4263+ /* len == 0 would wake all */
4264+ VM_WARN_ON(!ret);
4265+ range.len = ret;
4266+ if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) {
4267+ range.start = uffdio_move.dst;
4268+ wake_userfault(ctx, &range);
4269+ }
4270+ ret = range.len == uffdio_move.len ? 0 : -EAGAIN;
4271+
4272+out:
4273+ return ret;
4274+}
4275+
4276+/*
4277+ * userland asks for a certain API version and we return which bits
4278+ * and ioctl commands are implemented in this kernel for such API
4279+ * version or -EINVAL if unknown.
4280+ */
4281+static int userfaultfd_api(struct userfaultfd_ctx *ctx,
4282+ unsigned long arg)
4283+{
4284+ struct uffdio_api uffdio_api;
4285+ void __user *buf = (void __user *)arg;
4286+ unsigned int ctx_features;
4287+ int ret;
4288+ __u64 features;
4289+
4290+ ret = -EFAULT;
4291+ if (copy_from_user(&uffdio_api, buf, sizeof(uffdio_api)))
4292+ goto out;
4293+ features = uffdio_api.features;
4294+ ret = -EINVAL;
4295+ if (uffdio_api.api != UFFD_API)
4296+ goto err_out;
4297+ ret = -EPERM;
4298+ if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
4299+ goto err_out;
4300+
4301+ /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
4302+ if (features & UFFD_FEATURE_WP_ASYNC)
4303+ features |= UFFD_FEATURE_WP_UNPOPULATED;
4304+
4305+ /* report all available features and ioctls to userland */
4306+ uffdio_api.features = UFFD_API_FEATURES;
4307+#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
4308+ uffdio_api.features &=
4309+ ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
4310+#endif
4311+ if (!pgtable_supports_uffd_wp())
4312+ uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
4313+
4314+ if (!uffd_supports_wp_marker()) {
4315+ uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
4316+ uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
4317+ uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
4318+ }
4319+
4320+ ret = -EINVAL;
4321+ if (features & ~uffdio_api.features)
4322+ goto err_out;
4323+
4324+ uffdio_api.ioctls = UFFD_API_IOCTLS;
4325+ ret = -EFAULT;
4326+ if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
4327+ goto out;
4328+
4329+ /* only enable the requested features for this uffd context */
4330+ ctx_features = uffd_ctx_features(features);
4331+ ret = -EINVAL;
4332+ if (cmpxchg(&ctx->features, 0, ctx_features) != 0)
4333+ goto err_out;
4334+
4335+ ret = 0;
4336+out:
4337+ return ret;
4338+err_out:
4339+ memset(&uffdio_api, 0, sizeof(uffdio_api));
4340+ if (copy_to_user(buf, &uffdio_api, sizeof(uffdio_api)))
4341+ ret = -EFAULT;
4342+ goto out;
4343+}
4344+
4345+static long userfaultfd_ioctl(struct file *file, unsigned cmd,
4346+ unsigned long arg)
4347+{
4348+ int ret = -EINVAL;
4349+ struct userfaultfd_ctx *ctx = file->private_data;
4350+
4351+ if (cmd != UFFDIO_API && !userfaultfd_is_initialized(ctx))
4352+ return -EINVAL;
4353+
4354+ switch (cmd) {
4355+ case UFFDIO_API:
4356+ ret = userfaultfd_api(ctx, arg);
4357+ break;
4358+ case UFFDIO_REGISTER:
4359+ ret = userfaultfd_register(ctx, arg);
4360+ break;
4361+ case UFFDIO_UNREGISTER:
4362+ ret = userfaultfd_unregister(ctx, arg);
4363+ break;
4364+ case UFFDIO_WAKE:
4365+ ret = userfaultfd_wake(ctx, arg);
4366+ break;
4367+ case UFFDIO_COPY:
4368+ ret = userfaultfd_copy(ctx, arg);
4369+ break;
4370+ case UFFDIO_ZEROPAGE:
4371+ ret = userfaultfd_zeropage(ctx, arg);
4372+ break;
4373+ case UFFDIO_MOVE:
4374+ ret = userfaultfd_move(ctx, arg);
4375+ break;
4376+ case UFFDIO_WRITEPROTECT:
4377+ ret = userfaultfd_writeprotect(ctx, arg);
4378+ break;
4379+ case UFFDIO_CONTINUE:
4380+ ret = userfaultfd_continue(ctx, arg);
4381+ break;
4382+ case UFFDIO_POISON:
4383+ ret = userfaultfd_poison(ctx, arg);
4384+ break;
4385+ }
4386+ return ret;
4387+}
4388+
4389+#ifdef CONFIG_PROC_FS
4390+static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
4391+{
4392+ struct userfaultfd_ctx *ctx = f->private_data;
4393+ wait_queue_entry_t *wq;
4394+ unsigned long pending = 0, total = 0;
4395+
4396+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
4397+ list_for_each_entry(wq, &ctx->fault_pending_wqh.head, entry) {
4398+ pending++;
4399+ total++;
4400+ }
4401+ list_for_each_entry(wq, &ctx->fault_wqh.head, entry) {
4402+ total++;
4403+ }
4404+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
4405+
4406+ /*
4407+ * If more protocols will be added, there will be all shown
4408+ * separated by a space. Like this:
4409+ * protocols: aa:... bb:...
4410+ */
4411+ seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
4412+ pending, total, UFFD_API, ctx->features,
4413+ UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
4414+}
4415+#endif
4416+
4417+static const struct file_operations userfaultfd_fops = {
4418+#ifdef CONFIG_PROC_FS
4419+ .show_fdinfo = userfaultfd_show_fdinfo,
4420+#endif
4421+ .release = userfaultfd_release,
4422+ .poll = userfaultfd_poll,
4423+ .read_iter = userfaultfd_read_iter,
4424+ .unlocked_ioctl = userfaultfd_ioctl,
4425+ .compat_ioctl = compat_ptr_ioctl,
4426+ .llseek = noop_llseek,
4427+};
4428+
4429+static void init_once_userfaultfd_ctx(void *mem)
4430+{
4431+ struct userfaultfd_ctx *ctx = (struct userfaultfd_ctx *) mem;
4432+
4433+ init_waitqueue_head(&ctx->fault_pending_wqh);
4434+ init_waitqueue_head(&ctx->fault_wqh);
4435+ init_waitqueue_head(&ctx->event_wqh);
4436+ init_waitqueue_head(&ctx->fd_wqh);
4437+ seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock);
4438+}
4439+
4440+static int new_userfaultfd(int flags)
4441+{
4442+ struct userfaultfd_ctx *ctx __free(kfree) = NULL;
4443+
4444+ VM_WARN_ON_ONCE(!current->mm);
4445+
4446+ /* Check the UFFD_* constants for consistency. */
4447+ BUILD_BUG_ON(UFFD_USER_MODE_ONLY & UFFD_SHARED_FCNTL_FLAGS);
4448+
4449+ if (flags & ~(UFFD_SHARED_FCNTL_FLAGS | UFFD_USER_MODE_ONLY))
4450+ return -EINVAL;
4451+
4452+ ctx = kmem_cache_alloc(userfaultfd_ctx_cachep, GFP_KERNEL);
4453+ if (!ctx)
4454+ return -ENOMEM;
4455+
4456+ refcount_set(&ctx->refcount, 1);
4457+ ctx->flags = flags;
4458+ ctx->features = 0;
4459+ ctx->released = false;
4460+ init_rwsem(&ctx->map_changing_lock);
4461+ atomic_set(&ctx->mmap_changing, 0);
4462+ ctx->mm = current->mm;
4463+
4464+ FD_PREPARE(fdf, flags & UFFD_SHARED_FCNTL_FLAGS,
4465+ anon_inode_create_getfile("[userfaultfd]", &userfaultfd_fops, ctx,
4466+ O_RDONLY | (flags & UFFD_SHARED_FCNTL_FLAGS),
4467+ NULL));
4468+ if (fdf.err)
4469+ return fdf.err;
4470+
4471+ /* prevent the mm struct to be freed */
4472+ mmgrab(ctx->mm);
4473+ fd_prepare_file(fdf)->f_mode |= FMODE_NOWAIT;
4474+ retain_and_null_ptr(ctx);
4475+ return fd_publish(fdf);
4476+}
4477+
4478+static inline bool userfaultfd_syscall_allowed(int flags)
4479+{
4480+ /* Userspace-only page faults are always allowed */
4481+ if (flags & UFFD_USER_MODE_ONLY)
4482+ return true;
4483+
4484+ /*
4485+ * The user is requesting a userfaultfd which can handle kernel faults.
4486+ * Privileged users are always allowed to do this.
4487+ */
4488+ if (capable(CAP_SYS_PTRACE))
4489+ return true;
4490+
4491+ /* Otherwise, access to kernel fault handling is sysctl controlled. */
4492+ return sysctl_unprivileged_userfaultfd;
4493+}
4494+
4495+SYSCALL_DEFINE1(userfaultfd, int, flags)
4496+{
4497+ if (!userfaultfd_syscall_allowed(flags))
4498+ return -EPERM;
4499+
4500+ return new_userfaultfd(flags);
4501+}
4502+
4503+static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags)
4504+{
4505+ if (cmd != USERFAULTFD_IOC_NEW)
4506+ return -EINVAL;
4507+
4508+ return new_userfaultfd(flags);
4509+}
4510+
4511+static const struct file_operations userfaultfd_dev_fops = {
4512+ .unlocked_ioctl = userfaultfd_dev_ioctl,
4513+ .compat_ioctl = userfaultfd_dev_ioctl,
4514+ .owner = THIS_MODULE,
4515+ .llseek = noop_llseek,
4516+};
4517+
4518+static struct miscdevice userfaultfd_misc = {
4519+ .minor = MISC_DYNAMIC_MINOR,
4520+ .name = "userfaultfd",
4521+ .fops = &userfaultfd_dev_fops
4522+};
4523+
4524+static int __init userfaultfd_init(void)
4525+{
4526+ int ret;
4527+
4528+ ret = misc_register(&userfaultfd_misc);
4529+ if (ret)
4530+ return ret;
4531+
4532+ userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache",
4533+ sizeof(struct userfaultfd_ctx),
4534+ 0,
4535+ SLAB_HWCACHE_ALIGN|SLAB_PANIC,
4536+ init_once_userfaultfd_ctx);
4537+#ifdef CONFIG_SYSCTL
4538+ register_sysctl_init("vm", vm_userfaultfd_table);
4539+#endif
4540+ return 0;
4541+}
4542+__initcall(userfaultfd_init);
mm/util.c+22−10
@@ -1192,6 +1192,7 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, · 1191 unmodified lines
11921192 desc->vm_file = vma->vm_file;
11931193 desc->vma_flags = vma->flags;
11941194 desc->page_prot = vma->vm_page_prot;
1195+ desc->vm_ops = vma->vm_ops;
11951196
11961197 /* Default. */
11971198 desc->action.type = MMAP_NOTHING;
@@ -1396,8 +1397,6 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 198 unmodified lines
13961397
13971398 if (!err)
13981399 err = call_vma_mapped(vma);
1399− if (!err && action->success_hook)
1400− err = action->success_hook(vma);
14011400
14021401 /* do_munmap() might take rmap lock, so release if held. */
14031402 maybe_rmap_unlock_action(vma, action);
@@ -1415,16 +1414,22 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 11 unmodified lines
14151414 */
14161415 len = vma_pages(vma) << PAGE_SHIFT;
14171416 do_munmap(current->mm, vma->vm_start, len, NULL);
1418− if (action->error_hook) {
1419− /* We may want to filter the error. */
1420− err = action->error_hook(err);
1421− /* The caller should not clear the error. */
1422− VM_WARN_ON_ONCE(!err);
1423− }
1424− return err;
1417+
1418+ return action->error_override ?: err;
14251419 }
14261420
14271421 #ifdef CONFIG_MMU
1422+
1423+static int check_mmap_action(struct mmap_action *action)
1424+{
1425+ const unsigned long override = action->error_override;
1426+
1427+ if (WARN_ON_ONCE(override && !IS_ERR_VALUE(override)))
1428+ return -EINVAL;
1429+
1430+ return 0;
1431+}
1432+
14281433 /**
14291434 * mmap_action_prepare - Perform preparatory setup for an VMA descriptor
14301435 * action which need to be performed.
@@ -1434,7 +1439,14 @@ static int mmap_action_finish(struct vm_area_struct *vma, · 3 unmodified lines
14341439 */
14351440 int mmap_action_prepare(struct vm_area_desc *desc)
14361441 {
1437− switch (desc->action.type) {
1442+ struct mmap_action *action = &desc->action;
1443+ int err;
1444+
1445+ err = check_mmap_action(action);
1446+ if (err)
1447+ return err;
1448+
1449+ switch (action->type) {
14381450 case MMAP_NOTHING:
14391451 return 0;
14401452 case MMAP_REMAP_PFN:
mm/vma.c+3−0
@@ -2697,6 +2697,8 @@ static void set_vma_user_defined_fields(struct vm_area_struct *vma, · 2696 unmodified lines
26972697 {
26982698 if (map->vm_ops)
26992699 vma->vm_ops = map->vm_ops;
2700+ else /* Only /dev/zero should do this. */
2701+ vma_set_anonymous(vma);
27002702 vma->vm_private_data = map->vm_private_data;
27012703 }
27022704
@@ -2744,6 +2746,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, · 41 unmodified lines
27442746 .action = {
27452747 .type = MMAP_NOTHING, /* Default to no further action. */
27462748 },
2749+ .vm_ops = &vma_dummy_vm_ops,
27472750 };
27482751 bool allocated_new = false;
27492752 int error;
mm/vmalloc.c+107−23
@@ -3416,6 +3416,32 @@ void vfree_atomic(const void *addr) · 3415 unmodified lines
34163416 schedule_work(&p->wq);
34173417 }
34183418
3419+/*
3420+ * vm_area_free_pages - free a range of pages from a vmalloc allocation
3421+ * @vm: the vm_struct containing the pages
3422+ * @start_idx: first page index to free (inclusive)
3423+ * @end_idx: last page index to free (exclusive)
3424+ *
3425+ * Free pages [start_idx, end_idx) updating NR_VMALLOC stat accounting.
3426+ * Freed vm->pages[] entries are set to NULL.
3427+ * Caller is responsible for unmapping (vunmap_range) and KASAN
3428+ * poisoning before calling this.
3429+ */
3430+static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx,
3431+ unsigned int end_idx)
3432+{
3433+ unsigned int i;
3434+
3435+ if (!(vm->flags & VM_MAP_PUT_PAGES)) {
3436+ for (i = start_idx; i < end_idx; i++)
3437+ mod_lruvec_page_state(vm->pages[i], NR_VMALLOC, -1);
3438+ }
3439+ free_pages_bulk(vm->pages + start_idx, end_idx - start_idx);
3440+
3441+ for (i = start_idx; i < end_idx; i++)
3442+ vm->pages[i] = NULL;
3443+}
3444+
34193445 /**
34203446 * vfree - Release memory allocated by vmalloc()
34213447 * @addr: Memory base address
@@ -3436,7 +3462,6 @@ void vfree_atomic(const void *addr) · 14 unmodified lines
34363462 void vfree(const void *addr)
34373463 {
34383464 struct vm_struct *vm;
3439− int i;
34403465
34413466 if (unlikely(in_interrupt())) {
34423467 vfree_atomic(addr);
@@ -3459,19 +3484,8 @@ void vfree(const void *addr) · 16 unmodified lines
34593484
34603485 if (unlikely(vm->flags & VM_FLUSH_RESET_PERMS))
34613486 vm_reset_perms(vm);
3462− for (i = 0; i < vm->nr_pages; i++) {
3463− struct page *page = vm->pages[i];
34643487
3465− BUG_ON(!page);
3466− /*
3467− * High-order allocs for huge vmallocs are split, so
3468− * can be freed as an array of order-0 allocations
3469− */
3470− if (!(vm->flags & VM_MAP_PUT_PAGES))
3471− mod_lruvec_page_state(page, NR_VMALLOC, -1);
3472− __free_page(page);
3473− cond_resched();
3474− }
3488+ vm_area_free_pages(vm, 0, vm->nr_pages);
34753489 kvfree(vm->pages);
34763490 kfree(vm);
34773491 }
@@ -3939,7 +3953,7 @@ fail: · 461 unmodified lines
39393953 __GFP_NOFAIL | __GFP_ZERO |\
39403954 __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\
39413955 GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
3942− GFP_USER | __GFP_NOLOCKDEP)
3956+ GFP_USER | __GFP_NOLOCKDEP | __GFP_SKIP_KASAN)
39433957
39443958 static gfp_t vmalloc_fix_flags(gfp_t flags)
39453959 {
@@ -3980,6 +3994,9 @@ static gfp_t vmalloc_fix_flags(gfp_t flags) · 34 unmodified lines
39803994 *
39813995 * %__GFP_NOWARN can be used to suppress failure messages.
39823996 *
3997+ * %__GFP_SKIP_KASAN can be used to skip unpoisoning of mapped pages
3998+ * (when prot=%PAGE_KERNEL).
3999+ *
39834000 * Can not be called from interrupt nor NMI contexts.
39844001 * Return: the address of the area or %NULL on failure
39854002 */
@@ -3993,6 +4010,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · 7 unmodified lines
39934010 kasan_vmalloc_flags_t kasan_flags = KASAN_VMALLOC_NONE;
39944011 unsigned long original_align = align;
39954012 unsigned int shift = PAGE_SHIFT;
4013+ bool skip_vmalloc_kasan = kasan_hw_tags_enabled() && (gfp_mask & __GFP_SKIP_KASAN);
39964014
39974015 if (WARN_ON_ONCE(!size))
39984016 return NULL;
@@ -4023,7 +4041,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, · 24 unmodified lines
40234041 again:
40244042 area = __get_vm_area_node(size, align, shift, VM_ALLOC |
40254043 VM_UNINITIALIZED | vm_flags, start, end, node,
4026− gfp_mask, caller);
4044+ gfp_mask & ~__GFP_SKIP_KASAN, caller);
40274045 if (!area) {
40284046 bool nofail = gfp_mask & __GFP_NOFAIL;
40294047 warn_alloc(gfp_mask, NULL,
@@ -4041,7 +4059,7 @@ again: · 11 unmodified lines
40414059 * kasan_unpoison_vmalloc().
40424060 */
40434061 if (pgprot_val(prot) == pgprot_val(PAGE_KERNEL)) {
4044− if (kasan_hw_tags_enabled()) {
4062+ if (kasan_hw_tags_enabled() && !skip_vmalloc_kasan) {
40454063 /*
40464064 * Modify protection bits to allow tagging.
40474065 * This must be done before mapping.
@@ -4078,7 +4096,8 @@ again: · 30 unmodified lines
40784096 (gfp_mask & __GFP_SKIP_ZERO))
40794097 kasan_flags |= KASAN_VMALLOC_INIT;
40804098 /* KASAN_VMALLOC_PROT_NORMAL already set if required. */
4081− area->addr = kasan_unpoison_vmalloc(area->addr, size, kasan_flags);
4099+ if (!skip_vmalloc_kasan)
4100+ area->addr = kasan_unpoison_vmalloc(area->addr, size, kasan_flags);
40824101
40834102 /*
40844103 * In this function, newly allocated vm_struct has VM_UNINITIALIZED
@@ -4324,16 +4343,70 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 239 unmodified lines
43244343 if (unlikely(flags & __GFP_THISNODE) && nid != NUMA_NO_NODE &&
43254344 nid != page_to_nid(vmalloc_to_page(p)))
43264345 goto need_realloc;
4346+ } else {
4347+ /*
4348+ * If p is NULL, vrealloc behaves exactly like vmalloc.
4349+ * Skip the shrink and in-place grow paths.
4350+ */
4351+ goto need_realloc;
43274352 }
43284353
4329− /*
4330− * TODO: Shrink the vm_area, i.e. unmap and free unused pages. What
4331− * would be a good heuristic for when to shrink the vm_area?
4332− */
43334354 if (size <= old_size) {
4355+ unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
4356+
43344357 /* Zero out "freed" memory, potentially for future realloc. */
43354358 if (want_init_on_free() || want_init_on_alloc(flags))
43364359 memset((void *)p + size, 0, old_size - size);
4360+
4361+ /*
4362+ * Free tail pages when shrink crosses a page boundary.
4363+ *
4364+ * Skip huge page allocations (page_order > 0) as partial
4365+ * freeing would require splitting.
4366+ *
4367+ * Skip VM_FLUSH_RESET_PERMS, as direct-map permissions must
4368+ * be reset before pages are returned to the allocator.
4369+ *
4370+ * Skip VM_USERMAP, as remap_vmalloc_range_partial() validates
4371+ * mapping requests against the unchanged vm->size; freeing
4372+ * tail pages would cause vmalloc_to_page() to return NULL for
4373+ * the unmapped range.
4374+ *
4375+ * Skip if either GFP_NOFS or GFP_NOIO are used.
4376+ * kmemleak_free_part() internally allocates with
4377+ * GFP_KERNEL, which could trigger a recursive deadlock
4378+ * if we are under filesystem or I/O reclaim.
4379+ */
4380+ if (new_nr_pages < vm->nr_pages && !vm_area_page_order(vm) &&
4381+ !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) &&
4382+ gfp_has_io_fs(flags)) {
4383+ unsigned long addr = (unsigned long)kasan_reset_tag(p);
4384+ unsigned int old_nr_pages = vm->nr_pages;
4385+
4386+ /*
4387+ * Use the node lock to synchronize with concurrent
4388+ * readers (vmalloc_info_show).
4389+ */
4390+ struct vmap_node *vn = addr_to_node(addr);
4391+
4392+ spin_lock(&vn->busy.lock);
4393+ vm->nr_pages = new_nr_pages;
4394+ spin_unlock(&vn->busy.lock);
4395+
4396+ /* Notify kmemleak of the reduced allocation size before unmapping. */
4397+ kmemleak_free_part(
4398+ (void *)addr + ((unsigned long)new_nr_pages
4399+ << PAGE_SHIFT),
4400+ (unsigned long)(old_nr_pages - new_nr_pages)
4401+ << PAGE_SHIFT);
4402+
4403+ vunmap_range(addr + ((unsigned long)new_nr_pages
4404+ << PAGE_SHIFT),
4405+ addr + ((unsigned long)old_nr_pages
4406+ << PAGE_SHIFT));
4407+
4408+ vm_area_free_pages(vm, new_nr_pages, old_nr_pages);
4409+ }
43374410 vm->requested_size = size;
43384411 kasan_vrealloc(p, old_size, size);
43394412 return (void *)p;
@@ -4342,7 +4415,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 2 unmodified lines
43424415 /*
43434416 * We already have the bytes available in the allocation; use them.
43444417 */
4345− if (size <= alloced_size) {
4418+ if (size <= vm->nr_pages << PAGE_SHIFT) {
43464419 /*
43474420 * No need to zero memory here, as unused memory will have
43484421 * already been zeroed at initial allocation time or during
@@ -4641,7 +4714,18 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · 292 unmodified lines
46414714 smp_rmb();
46424715
46434716 vaddr = (char *) va->va_start;
4644− size = vm ? get_vm_area_size(vm) : va_size(va);
4717+ if (vm)
4718+ /*
4719+ * For VM_ALLOC areas, use nr_pages rather than
4720+ * get_vm_area_size() because vrealloc() may shrink
4721+ * the mapping without updating area->size. Other
4722+ * mapping types (vmap, ioremap) don't set nr_pages.
4723+ */
4724+ size = (vm->flags & VM_ALLOC && vm->nr_pages) ?
4725+ (vm->nr_pages << PAGE_SHIFT) :
4726+ get_vm_area_size(vm);
4727+ else
4728+ size = va_size(va);
46454729
46464730 if (addr >= vaddr + size)
46474731 goto next_va;
mm/vmpressure.c+12−3
@@ -218,6 +218,7 @@ static void vmpressure_work_fn(struct work_struct *work) · 217 unmodified lines
218218 /**
219219 * vmpressure() - Account memory pressure through scanned/reclaimed ratio
220220 * @gfp: reclaimer's gfp mask
221+ * @order: allocation order being reclaimed for
221222 * @memcg: cgroup memory controller handle
222223 * @tree: legacy subtree mode
223224 * @scanned: number of pages scanned
@@ -236,7 +237,7 @@ static void vmpressure_work_fn(struct work_struct *work) · 12 unmodified lines
236237 *
237238 * This function does not return any value.
238239 */
239−void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,
240+void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
240241 unsigned long scanned, unsigned long reclaimed)
241242 {
242243 struct vmpressure *vmpr;
@@ -307,7 +308,15 @@ void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree, · 64 unmodified lines
307308
308309 level = vmpressure_calc_level(scanned, reclaimed);
309310
310− if (level > VMPRESSURE_LOW) {
311+ /*
312+ * Once we go above COSTLY_ORDER, reclaim relies heavily on
313+ * compaction to make progress. Reclaim efficiency was never a
314+ * great proxy for pressure to begin with, but it's outright
315+ * misleading with these high orders. Don't throttle sockets
316+ * because somebody is attempting something crazy like an order-7
317+ * and predictably struggling.
318+ */
319+ if (level > VMPRESSURE_LOW && order <= PAGE_ALLOC_COSTLY_ORDER) {
311320 /*
312321 * Let the socket buffer allocator know that
313322 * we are having trouble reclaiming LRU pages.
@@ -348,7 +357,7 @@ void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio) · 34 unmodified lines
348357 * to the vmpressure() basically means that we signal 'critical'
349358 * level.
350359 */
351− vmpressure(gfp, memcg, true, vmpressure_win, 0);
360+ vmpressure(gfp, 0, memcg, true, vmpressure_win, 0);
352361 }
353362
354363 #define MAX_VMPRESSURE_ARGS_LEN (strlen("critical") + strlen("hierarchy") + 2)
mm/vmscan.c+168−202
@@ -109,7 +109,7 @@ struct scan_control { · 108 unmodified lines
109109 /* zone_reclaim_mode */
110110 unsigned int may_unmap:1;
111111
112− /* zome_reclaim_mode, boost reclaim, cgroup restrictions */
112+ /* zone_reclaim_mode, boost reclaim, cgroup restrictions */
113113 unsigned int may_swap:1;
114114
115115 /* Not allow cache_trim_mode to be turned on as part of reclaim? */
@@ -169,11 +169,9 @@ struct scan_control { · 53 unmodified lines
169169
170170 struct {
171171 unsigned int dirty;
172− unsigned int unqueued_dirty;
173172 unsigned int congested;
174173 unsigned int writeback;
175174 unsigned int immediate;
176− unsigned int file_taken;
177175 unsigned int taken;
178176 } nr;
179177
@@ -739,7 +737,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, · 559 unmodified lines
739737
740738 if (reclaimed && !mapping_exiting(mapping))
741739 shadow = workingset_eviction(folio, target_memcg);
742− memcg1_swapout(folio, swap);
740+ __memcg1_swapout(folio, ci);
743741 __swap_cache_del_folio(ci, folio, swap, shadow);
744742 swap_cluster_unlock_irq(ci);
745743 } else {
@@ -850,7 +848,11 @@ static bool lru_gen_set_refs(struct folio *folio) · 104 unmodified lines
850848 return false;
851849 }
852850
853− set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
851+ /* Promote on second access */
852+ if (folio_lru_refs(folio) > 1)
853+ set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
854+ else
855+ folio_mark_accessed(folio);
854856 return true;
855857 }
856858 #else
@@ -1944,6 +1946,44 @@ static int current_may_throttle(void) · 1087 unmodified lines
19441946 return !(current->flags & PF_LOCAL_THROTTLE);
19451947 }
19461948
1949+static void handle_reclaim_writeback(unsigned long nr_taken,
1950+ struct pglist_data *pgdat,
1951+ struct scan_control *sc,
1952+ struct reclaim_stat *stat)
1953+{
1954+ /*
1955+ * If dirty folios are scanned that are not queued for IO, it
1956+ * implies that flushers are not doing their job. This can
1957+ * happen when memory pressure pushes dirty folios to the end of
1958+ * the LRU before the dirty limits are breached and the dirty
1959+ * data has expired. It can also happen when the proportion of
1960+ * dirty folios grows not through writes but through memory
1961+ * pressure reclaiming all the clean cache. And in some cases,
1962+ * the flushers simply cannot keep up with the allocation
1963+ * rate. Nudge the flusher threads in case they are asleep.
1964+ */
1965+ if (stat->nr_unqueued_dirty == nr_taken) {
1966+ wakeup_flusher_threads(WB_REASON_VMSCAN);
1967+ /*
1968+ * For cgroupv1 dirty throttling is achieved by waking up
1969+ * the kernel flusher here and later waiting on folios
1970+ * which are in writeback to finish (see shrink_folio_list()).
1971+ *
1972+ * Flusher may not be able to issue writeback quickly
1973+ * enough for cgroupv1 writeback throttling to work
1974+ * on a large system.
1975+ */
1976+ if (!writeback_throttling_sane(sc))
1977+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
1978+ }
1979+
1980+ sc->nr.dirty += stat->nr_dirty;
1981+ sc->nr.congested += stat->nr_congested;
1982+ sc->nr.writeback += stat->nr_writeback;
1983+ sc->nr.immediate += stat->nr_immediate;
1984+ sc->nr.taken += nr_taken;
1985+}
1986+
19471987 /*
19481988 * shrink_inactive_list() is a helper for shrink_node(). It returns the number
19491989 * of reclaimed pages
@@ -2007,42 +2047,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, · 57 unmodified lines
20072047 lruvec_lock_irq(lruvec);
20082048 lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
20092049 nr_scanned - nr_reclaimed);
2010−
2011− /*
2012− * If dirty folios are scanned that are not queued for IO, it
2013− * implies that flushers are not doing their job. This can
2014− * happen when memory pressure pushes dirty folios to the end of
2015− * the LRU before the dirty limits are breached and the dirty
2016− * data has expired. It can also happen when the proportion of
2017− * dirty folios grows not through writes but through memory
2018− * pressure reclaiming all the clean cache. And in some cases,
2019− * the flushers simply cannot keep up with the allocation
2020− * rate. Nudge the flusher threads in case they are asleep.
2021− */
2022− if (stat.nr_unqueued_dirty == nr_taken) {
2023− wakeup_flusher_threads(WB_REASON_VMSCAN);
2024− /*
2025− * For cgroupv1 dirty throttling is achieved by waking up
2026− * the kernel flusher here and later waiting on folios
2027− * which are in writeback to finish (see shrink_folio_list()).
2028− *
2029− * Flusher may not be able to issue writeback quickly
2030− * enough for cgroupv1 writeback throttling to work
2031− * on a large system.
2032− */
2033− if (!writeback_throttling_sane(sc))
2034− reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
2035− }
2036−
2037− sc->nr.dirty += stat.nr_dirty;
2038− sc->nr.congested += stat.nr_congested;
2039− sc->nr.unqueued_dirty += stat.nr_unqueued_dirty;
2040− sc->nr.writeback += stat.nr_writeback;
2041− sc->nr.immediate += stat.nr_immediate;
2042− sc->nr.taken += nr_taken;
2043− if (file)
2044− sc->nr.file_taken += nr_taken;
2045−
2050+ handle_reclaim_writeback(nr_taken, pgdat, sc, &stat);
20462051 trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
20472052 nr_scanned, nr_reclaimed, &stat, sc->priority, file);
20482053 return nr_reclaimed;
@@ -3220,7 +3225,7 @@ static int folio_update_gen(struct folio *folio, int gen) · 1171 unmodified lines
32203225 }
32213226
32223227 /* protect pages accessed multiple times through file descriptors */
3223−static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
3228+static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
32243229 {
32253230 int type = folio_is_file_lru(folio);
32263231 struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -3239,9 +3244,6 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai · 12 unmodified lines
32393244
32403245 new_flags = old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS);
32413246 new_flags |= (new_gen + 1UL) << LRU_GEN_PGOFF;
3242− /* for folio_end_writeback() */
3243− if (reclaiming)
3244− new_flags |= BIT(PG_reclaim);
32453247 } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
32463248
32473249 lru_gen_update_size(lruvec, folio, old_gen, new_gen);
@@ -3855,7 +3857,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) · 607 unmodified lines
38553857 VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio);
38563858 VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio);
38573859
3858− new_gen = folio_inc_gen(lruvec, folio, false);
3860+ new_gen = folio_inc_gen(lruvec, folio);
38593861 list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]);
38603862
38613863 /* don't count the workingset being lazily promoted */
@@ -3878,10 +3880,9 @@ done: · 16 unmodified lines
38783880 return true;
38793881 }
38803882
3881−static bool try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
3883+static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
38823884 {
38833885 int gen, type, zone;
3884− bool success = false;
38853886 bool seq_inc_flag = false;
38863887 struct lru_gen_folio *lrugen = &lruvec->lrugen;
38873888 DEFINE_MIN_SEQ(lruvec);
@@ -3907,11 +3908,10 @@ next: · 19 unmodified lines
39073908
39083909 /*
39093910 * If min_seq[type] of both anonymous and file is not increased,
3910− * we can directly return false to avoid unnecessary checking
3911− * overhead later.
3911+ * return here to avoid unnecessary checking overhead later.
39123912 */
39133913 if (!seq_inc_flag)
3914− return success;
3914+ return;
39153915
39163916 /* see the comment on lru_gen_folio */
39173917 if (swappiness && swappiness <= MAX_SWAPPINESS) {
@@ -3929,10 +3929,7 @@ next: · 11 unmodified lines
39293929
39303930 reset_ctrl_pos(lruvec, type, true);
39313931 WRITE_ONCE(lrugen->min_seq[type], min_seq[type]);
3932− success = true;
39333932 }
3934−
3935− return success;
39363933 }
39373934
39383935 static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness)
@@ -4084,27 +4081,33 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control · 145 unmodified lines
40844081 sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
40854082 }
40864083
4087−static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
4084+static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
40884085 {
40894086 int gen, type, zone;
4090− unsigned long total = 0;
4091− int swappiness = get_swappiness(lruvec, sc);
4087+ unsigned long seq, total = 0;
40924088 struct lru_gen_folio *lrugen = &lruvec->lrugen;
4093− struct mem_cgroup *memcg = lruvec_memcg(lruvec);
40944089 DEFINE_MAX_SEQ(lruvec);
40954090 DEFINE_MIN_SEQ(lruvec);
40964091
40974092 for_each_evictable_type(type, swappiness) {
4098− unsigned long seq;
4099−
41004093 for (seq = min_seq[type]; seq <= max_seq; seq++) {
41014094 gen = lru_gen_from_seq(seq);
4102−
41034095 for (zone = 0; zone < MAX_NR_ZONES; zone++)
41044096 total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
41054097 }
41064098 }
41074099
4100+ return total;
4101+}
4102+
4103+static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
4104+{
4105+ unsigned long total;
4106+ int swappiness = get_swappiness(lruvec, sc);
4107+ struct mem_cgroup *memcg = lruvec_memcg(lruvec);
4108+
4109+ total = lruvec_evictable_size(lruvec, swappiness);
4110+
41084111 /* whether the size is big enough to be helpful */
41094112 return mem_cgroup_online(memcg) ? (total >> sc->priority) : total;
41104113 }
@@ -4577,7 +4580,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 466 unmodified lines
45774580 int tier_idx)
45784581 {
45794582 bool success;
4580− bool dirty, writeback;
45814583 int gen = folio_lru_gen(folio);
45824584 int type = folio_is_file_lru(folio);
45834585 int zone = folio_zonenum(folio);
@@ -4607,7 +4609,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 23 unmodified lines
46074609
46084610 /* protected */
46094611 if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {
4610− gen = folio_inc_gen(lruvec, folio, false);
4612+ gen = folio_inc_gen(lruvec, folio);
46114613 list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
46124614
46134615 /* don't count the workingset being lazily promoted */
@@ -4622,26 +4624,11 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 8 unmodified lines
46224624
46234625 /* ineligible */
46244626 if (zone > sc->reclaim_idx) {
4625− gen = folio_inc_gen(lruvec, folio, false);
4627+ gen = folio_inc_gen(lruvec, folio);
46264628 list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
46274629 return true;
46284630 }
46294631
4630− dirty = folio_test_dirty(folio);
4631− writeback = folio_test_writeback(folio);
4632− if (type == LRU_GEN_FILE && dirty) {
4633− sc->nr.file_taken += delta;
4634− if (!writeback)
4635− sc->nr.unqueued_dirty += delta;
4636− }
4637−
4638− /* waiting for writeback */
4639− if (writeback || (type == LRU_GEN_FILE && dirty)) {
4640− gen = folio_inc_gen(lruvec, folio, true);
4641− list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
4642− return true;
4643− }
4644−
46454632 return false;
46464633 }
46474634
@@ -4649,12 +4636,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 1 unmodified line
46494636 {
46504637 bool success;
46514638
4652− /* swap constrained */
4653− if (!(sc->gfp_mask & __GFP_IO) &&
4654− (folio_test_dirty(folio) ||
4655− (folio_test_anon(folio) && !folio_test_swapcache(folio))))
4656− return false;
4657−
46584639 /* raced with release_pages() */
46594640 if (!folio_try_get(folio))
46604641 return false;
@@ -4669,9 +4650,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 8 unmodified lines
46694650 if (!folio_test_referenced(folio))
46704651 set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0);
46714652
4672− /* for shrink_folio_list() */
4673− folio_clear_reclaim(folio);
4674−
46754653 success = lru_gen_del_folio(lruvec, folio, true);
46764654 VM_WARN_ON_ONCE_FOLIO(!success, folio);
46774655
@@ -4680,7 +4658,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca · 2 unmodified lines
46804658
46814659 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
46824660 struct scan_control *sc, int type, int tier,
4683− struct list_head *list)
4661+ struct list_head *list, int *isolatedp)
46844662 {
46854663 int i;
46864664 int gen;
@@ -4689,10 +4667,10 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 2 unmodified lines
46894667 int scanned = 0;
46904668 int isolated = 0;
46914669 int skipped = 0;
4692− int scan_batch = min(nr_to_scan, MAX_LRU_BATCH);
4693− int remaining = scan_batch;
4670+ unsigned long remaining = nr_to_scan;
46944671 struct lru_gen_folio *lrugen = &lruvec->lrugen;
46954672
4673+ VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH);
46964674 VM_WARN_ON_ONCE(!list_empty(list));
46974675
46984676 if (get_nr_gens(lruvec, type) == MIN_NR_GENS)
@@ -4745,16 +4723,12 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 46 unmodified lines
47454723 mod_lruvec_state(lruvec, item, isolated);
47464724 mod_lruvec_state(lruvec, PGREFILL, sorted);
47474725 mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated);
4748− trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, scan_batch,
4726+ trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan,
47494727 scanned, skipped, isolated,
47504728 type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
4751− if (type == LRU_GEN_FILE)
4752− sc->nr.file_taken += isolated;
4753− /*
4754− * There might not be eligible folios due to reclaim_idx. Check the
4755− * remaining to prevent livelock if it's not making progress.
4756− */
4757− return isolated || !remaining ? scanned : 0;
4729+
4730+ *isolatedp = isolated;
4731+ return scanned;
47584732 }
47594733
47604734 static int get_tier_idx(struct lruvec *lruvec, int type)
@@ -4798,33 +4772,41 @@ static int get_type_to_scan(struct lruvec *lruvec, int swappiness) · 37 unmodified lines
47984772
47994773 static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
48004774 struct scan_control *sc, int swappiness,
4801− int *type_scanned, struct list_head *list)
4775+ struct list_head *list, int *isolated,
4776+ int *isolate_type, int *isolate_scanned)
48024777 {
48034778 int i;
4779+ int total_scanned = 0;
48044780 int type = get_type_to_scan(lruvec, swappiness);
48054781
48064782 for_each_evictable_type(i, swappiness) {
48074783 int scanned;
48084784 int tier = get_tier_idx(lruvec, type);
48094785
4810− *type_scanned = type;
4811−
4812− scanned = scan_folios(nr_to_scan, lruvec, sc, type, tier, list);
4813− if (scanned)
4814− return scanned;
4786+ scanned = scan_folios(nr_to_scan, lruvec, sc,
4787+ type, tier, list, isolated);
48154788
4816− type = !type;
4789+ total_scanned += scanned;
4790+ if (*isolated) {
4791+ *isolate_type = type;
4792+ *isolate_scanned = scanned;
4793+ break;
4794+ }
4795+ /*
4796+ * If scanned > 0 and isolated == 0, avoid falling back to the
4797+ * other type, as this type remains sufficient. Falling back
4798+ * too readily can disrupt the positive_ctrl_err() bias.
4799+ */
4800+ if (!scanned)
4801+ type = !type;
48174802 }
48184803
4819− return 0;
4804+ return total_scanned;
48204805 }
48214806
48224807 static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
48234808 struct scan_control *sc, int swappiness)
48244809 {
4825− int type;
4826− int scanned;
4827− int reclaimed;
48284810 LIST_HEAD(list);
48294811 LIST_HEAD(clean);
48304812 struct folio *folio;

332 more lines aren’t shown here.

mm/workingset.c+6−4

This diff is large: its 18 lines aren’t shown here.

mm/zswap.c+10−15

This diff is large: its 50 lines aren’t shown here.

net/rds/tcp_recv.c+1−1

This diff is large: its 9 lines aren’t shown here.

rust/kernel/page.rs+1−0

This diff is large: its 8 lines aren’t shown here.

samples/damon/mtier.c+6−4

This diff is large: its 29 lines aren’t shown here.

tools/mm/page-types.c+4−4

This diff is large: its 31 lines aren’t shown here.

tools/testing/selftests/cgroup/lib/cgroup_util.c+9−9

This diff is large: its 81 lines aren’t shown here.

tools/testing/selftests/cgroup/lib/include/cgroup_util.h+2−2

This diff is large: its 11 lines aren’t shown here.

tools/testing/selftests/cgroup/test_core.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/cgroup/test_freezer.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/cgroup/test_kmem.c+7−6

This diff is large: its 37 lines aren’t shown here.

tools/testing/selftests/cgroup/test_memcontrol.c+9−4

This diff is large: its 55 lines aren’t shown here.

tools/testing/selftests/cgroup/test_zswap.c+127−54

This diff is large: its 371 lines aren’t shown here.

tools/testing/selftests/damon/_damon_sysfs.py+28−3

This diff is large: its 82 lines aren’t shown here.

tools/testing/selftests/damon/drgn_dump_damon_status.py+3−0

This diff is large: its 17 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.py+55−5

This diff is large: its 120 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.sh+62−0

This diff is large: its 90 lines aren’t shown here.

tools/testing/selftests/memfd/fuse_test.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/memfd/memfd_test.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+7−3

This diff is large: its 38 lines aren’t shown here.

tools/testing/selftests/mm/check_config.sh+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/droppable.c+8−1

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/mm/hmm-tests.c+33−3

This diff is large: its 73 lines aren’t shown here.

tools/testing/selftests/mm/khugepaged.c+15−3

This diff is large: its 33 lines aren’t shown here.

tools/testing/selftests/mm/ksft_kmemleak_dedup.sh+222−0

This diff is large: its 223 lines aren’t shown here.

tools/testing/selftests/mm/ksm_functional_tests.c+11−8

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/mm/mlock2-tests.c+75−9

This diff is large: its 136 lines aren’t shown here.

tools/testing/selftests/mm/mremap_test.c+10−99

This diff is large: its 138 lines aren’t shown here.

tools/testing/selftests/mm/process_madv.c+28−0

This diff is large: its 35 lines aren’t shown here.

tools/testing/selftests/mm/run_vmtests.sh+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/split_huge_page_test.c+8−4

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/proc/proc-maps-race.c+231−62

This diff is large: its 421 lines aren’t shown here.

tools/testing/vma/include/dup.h+4−16

This diff is large: its 35 lines aren’t shown here.