History

Merge tag 'mm-stable-2026-08-26-15-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull more MM updates from Andrew Morton: - "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff" (Lorenzo Stoakes) Index MAP_PRIVATE file-backed folios by their anonymous page offset to resolve confusion around reverse mapping for zeroed and CoW'd file-backed memory. Use this new VMA anonymous page offset tracking to eliminate index conflicts and lay the foundation for scalable CoW performance improvements. - "promote mapped executable folios after first usage for MGLRU" (Baolin Wang) Make MGLRU's protection of mapped executable file folios more reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance. - "mm: vmscan: fix node reclaim ignoring swappiness parameter" (Ridong Chen) Fix per-node proactive reclaim interface's ignoring the swappiness parameter when CONFIG_MEMCG is disabled by consolidating sc_swappiness() into a single function that checks proactive_swappiness regardless of kernel configuration. - "mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost" (Usama Arif) Reduce lru_lock contention in the reclaim path by deriving scan-balance costs from vmstat counters rather than lock-acquired producer updates. Read and decay these cost signals on the reclaim side under a dedicated per-lruvec lock, reducing total LRU lock wait time by over 60% without impacting scan throughput. - "zram: fix zram issues reported by sashiko" (Sergey Senozhatsky) Fix two low-risk zram bugs which Sashiko spotted in drive-by review. - "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg" (Zi Yan) Fix xas_split_alloc() by enabling target folio memcg charging during splits and adding the missing __GFP_ACCOUNT flag for proper XArray node memory accounting. - "selftests/mm: use pattern matching in .gitignore" (Pratyush Mallick) Replace hardcoded binary names in selftests/mm/.gitignore with a generic pattern-matching rule to automatically ignore generated test files and avoid manual updates when adding new tests. - "mm/page_ext: remove pgdat_page_ext_init()" (Sang-Heon Jeon) Make the incompatibility between FLATMEM and NUMA explicit in mm/Kconfig and remove the unused pgdat_page_ext_init() function. - "zram: fix zstd error paths and add parameter validation" (Haoqin Huang) Clean up zram compression backends by removing redundant error cleanup, adding parameter and dictionary validation, auto-prefixing algorithm error logs, and resetting parameters prior to reinitialization. - "zram: fix stale scan bounds after reinitialization" (Longlong Xia) Prevent out-of-bounds slot accesses during concurrent zram resets by moving table scan bound calculations under dev_lock in writeback_store() and read_block_state(). - "add anon mTHP collapse test cases" (Baolin Wang) Extend selftests helper functions to support arbitrary page orders and add new test cases and options for mTHP collapse in khugepaged. - "selftests/mm: Handle unsupported and transient test conditions" (Muhammad Usama Anjum) Update MM selftests to report a SKIP status instead of a failure when required kernel or filesystem features are unsupported, while adding retry logic for transient page migration errors. - "mm/zswap: Fixes and improves the zswap shrink" (Hao Jia) Fix the missing zswap global shrinker when CONFIG_MEMCG is disabled and extend shrink_memcg() to support batch writeback for improved writeback efficiency. - "alloc_tag: introduce IOCTL-based filtering for MAP" (Suren Baghdasaryan) Introduce an IOCTL-based binary interface for memory allocation profiling that enables kernel-side filtering before per-CPU counter aggregation. This eliminates the text-parsing overhead of /proc/allocinfo and provides up to a 20x speedup by transferring only filtered allocation data to userspace. - "better block swap batching and a different take on swap_ops v5" (Christoph Hellwig) Refactor block swap I/O to use swap_iocb for batching instead of single-bio requests and rebase the swap_ops interface, achieving faster swap throughput during kernel builds. - "mm: kmemleak: reduce transient false positives by confirming leaks" (Catalin Marinas) Reduce false-positive kmemleak reports by combining two kmemleak enhancements that add a second confirmation scan and a configurable minimum unreferenced scan count module parameter. - "mm: kmemleak: default min_unref_scans to 2 for verbose kernels" (Breno Leitao) Auto-scanning kernels can generate false-positive memory leak reports on single scans, so this patch defaults min_unref_scans to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled to require a second confirming scan. - "swap_ops updates" (Christoph Hellwig) Batching I/O for synchronous swap devices causes performance regressions and filesystem-based swap suffers from double-indirection overhead. This series resolves both issues by reintroducing per-folio writes for synchronous swap and allowing filesystems to directly export their own swap_ops. - "mm/khugepaged: several cleanups" (Nico Pache) khugepaged accumulated redundant state-checking patterns and outdated comments following mTHP integration. Introduce dedicated helpers for PTE validation and event counting while refreshing the internal documentation. - "maple_tree: lock checking and clean ups" (Liam Howlett) Syzbot reports incorrectly blame memory management exit paths for locking bugs, maple tree erase operations risk allocation failures without gfp flags and internal documentation lacks clarity. Improve lock error detection, update docs, fix race and allocation edge cases and optimize erase allocations using a fallback to GFP_KERNEL | GFP_NOFAIL. * tag 'mm-stable-2026-08-26-15-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (172 commits) selftests/proc: make proc-maps-race work with READ_IMPLIES_EXEC memcg: move LRU size accounting on reparenting instead of copying it mm/vmscan: fix comment logic in balance_pgdat maple_tree: add helper mas_make_walkable() maple_tree: avoid extra gap calculation maple_tree: fix argument name in header maple_tree: change two GFP flags in tests maple_tree: document erase and allocations better maple_tree: avoid mas_erase() and mtree_erase() failures maple_tree: document that erase may use GFP_KERNEL for allocations maple_tree: catch race in mas_alloc_cyclic() maple_tree: add bulk parent set helper maple_tree: micro optimisation of mas_wr_store_type() maple_tree: optimise mas_wr_node_store() when not in rcu mode maple_tree: use prefetched value in mas_wr_store_type() maple_tree: clarify comments on mas_nomem() maple_tree: drop MAPLE_ALLOC_SLOTS maple_tree: drop dead code from mas_extend_spanning_null() maple_tree: documentation fix maple_tree: add write lock checking with lockdep sequence numbers ...

Linus Torvalds committed 2 months ago

Commit 18fbf512 parents 5e6ff28 0685630168 files · +4561 −1770Browse files

Unified · Split

168 files
  1. Documentation/admin-guide/blockdev/zram.rst +33 −1
  2. Documentation/admin-guide/kernel-parameters.txt +14 −2
  3. Documentation/admin-guide/mm/transhuge.rst +4 −2
  4. Documentation/core-api/maple_tree.rst +20 −4
  5. Documentation/dev-tools/kmemleak.rst +10 −0
  6. Documentation/filesystems/fscrypt.rst +4 −0
  7. Documentation/filesystems/locking.rst +2 −6
  8. Documentation/filesystems/vfs.rst +2 −6
  9. Documentation/mm/allocation-profiling.rst +16 −0
  10. Documentation/mm/ksm.rst +2 −2
  11. Documentation/mm/page_migration.rst +3 −3
  12. Documentation/translations/zh_CN/mm/ksm.rst +2 −2
  13. Documentation/userspace-api/ioctl/ioctl-number.rst +2 −0
  14. MAINTAINERS +6 −0
  15. arch/s390/mm/gmap_helpers.c +1 −1
  16. drivers/base/Kconfig +0 −7
  17. drivers/base/Makefile +0 −1
  18. drivers/block/zram/backend_842.c +10 −0
  19. drivers/block/zram/backend_deflate.c +23 −2
  20. drivers/block/zram/backend_lz4.c +9 −1
  21. drivers/block/zram/backend_lz4hc.c +15 −1
  22. drivers/block/zram/backend_lzo.c +10 −0
  23. drivers/block/zram/backend_lzorle.c +10 −0
  24. drivers/block/zram/backend_zstd.c +8 −3
  25. drivers/block/zram/zram_drv.c +109 −79
  26. drivers/block/zram/zram_drv.h +14 −0
  27. drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +2 −2
  28. drivers/gpu/drm/drm_gem_shmem_helper.c +1 −1
  29. drivers/gpu/drm/i915/gem/i915_gem_shmem.c +1 −1
  30. drivers/gpu/drm/panthor/panthor_gem.c +1 −1
  31. drivers/gpu/drm/ttm/ttm_backup.c +1 −1
  32. drivers/gpu/drm/ttm/ttm_bo_vm.c +1 −1
  33. drivers/gpu/drm/xe/xe_device.c +1 −1
  34. fs/nfs/direct.c +0 −20
  35. fs/nfs/file.c +38 −6
  36. fs/proc/task_mmu.c +1 −1
  37. fs/ramfs/file-nommu.c +3 −0
  38. fs/smb/client/file.c +39 −26
  39. include/linux/codetag.h +2 −0
  40. include/linux/execmem.h +1 −1
  41. include/linux/fs.h +0 −1
  42. include/linux/lockdep.h +3 −0
  43. include/linux/lockdep_types.h +2 −1
  44. include/linux/maple_tree.h +9 −7
  45. include/linux/memcontrol.h +17 −4
  46. include/linux/mm.h +125 −7
  47. include/linux/mm_types.h +26 −14
  48. include/linux/mmzone.h +13 −3
  49. include/linux/nfs_fs.h +0 −1
  50. include/linux/page_ext.h +0 −5
  51. include/linux/pagemap.h +37 −3
  52. include/linux/percpu-defs.h +1 −6
  53. include/linux/rmap.h +6 −5
  54. include/linux/sched.h +1 −0
  55. include/linux/shmem_fs.h +1 −4
  56. include/linux/swap.h +9 −21
  57. include/linux/swap_ops.h +44 −0
  58. include/linux/swapops.h +4 −3
  59. include/linux/vm_event_item.h +4 −0
  60. include/linux/vmalloc.h +1 −1
  61. include/linux/vmstat.h +16 −1
  62. include/rv/da_monitor.h +1 −1
  63. include/uapi/linux/alloc_tag.h +99 −0
  64. kernel/events/uprobes.c +1 −1
  65. kernel/locking/lockdep.c +51 −7
  66. lib/Kconfig.debug +0 −15
  67. lib/codetag.c +18 −0
  68. lib/maple_tree.c +280 −121
  69. lib/test_hmm.c +2 −3
  70. lib/xarray.c +3 −0
  71. mm/Kconfig +11 −11
  72. mm/Kconfig.debug +0 −1
  73. mm/Makefile +1 −0
  74. mm/alloc_tag.c +349 −2
  75. mm/arch_numa.c +12 −0
  76. mm/cma.c +1 −1
  77. mm/cma.h +1 −0
  78. mm/debug_page_alloc.c +4 −4
  79. mm/folio-compat.c +1 −0
  80. mm/folio.c +55 −69
  81. mm/gup.c +9 −14
  82. mm/gup_test.c +62 −32
  83. mm/hmm.c +4 −1
  84. mm/huge_memory.c +28 −18
  85. mm/hugetlb.c +7 −3
  86. mm/hugetlb_cma.c +134 −8
  87. mm/hugetlb_cma.h +2 −0
  88. mm/internal.h +54 −23
  89. mm/interval_tree.c +2 −2
  90. mm/kasan/quarantine.c +11 −1
  91. mm/khugepaged.c +65 −65
  92. mm/kmemleak.c +92 −13
  93. mm/ksm.c +7 −9
  94. mm/madvise.c +9 −8
  95. mm/memcontrol-v1.c +2 −2
  96. mm/memcontrol-v1.h +1 −0
  97. mm/memcontrol.c +66 −3
  98. mm/memory-failure.c +2 −2
  99. mm/memory.c +101 −39
  100. mm/mempolicy.c +7 −16
  101. mm/migrate.c +10 −11
  102. mm/migrate_device.c +14 −1
  103. mm/mm_init.c +0 −1
  104. mm/mmzone.c +1 −0
  105. mm/mremap.c +5 −3
  106. mm/nommu.c +4 −0
  107. mm/page-writeback.c +5 −0
  108. mm/page_alloc.c +20 −8
  109. mm/page_ext.c +0 −9
  110. mm/page_io.c +281 −300
  111. mm/page_reporting.c +18 −10
  112. mm/page_vma_mapped.c +5 −1
  113. mm/pgalloc-track.h +3 −0
  114. mm/rmap.c +25 −18
  115. mm/shmem.c +24 −5
  116. mm/show_mem.c +6 −6
  117. mm/shuffle.h +2 −0
  118. mm/sparse.c +4 −14
  119. mm/sparse.h +6 −0
  120. mm/swap.h +31 −34
  121. mm/swap_state.c +32 −22
  122. mm/swapfile.c +16 −8
  123. mm/userfaultfd.c +2 −2
  124. mm/vma.c +79 −34
  125. mm/vma.h +59 −30
  126. mm/vma_exec.c +1 −1
  127. mm/vma_init.c +1 −0
  128. mm/vmalloc.c +34 −37
  129. mm/vmalloc.h +2 −0
  130. mm/vmscan.c +151 −79
  131. mm/vmstat.c +18 −1
  132. mm/workingset.c +0 −5
  133. mm/zsmalloc.c +7 −4
  134. mm/zswap.c +29 −17
  135. tools/testing/radix-tree/maple.c +2 −2
  136. tools/testing/selftests/Makefile +1 −0
  137. tools/testing/selftests/alloc_tag/Makefile +8 −0
  138. tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c +548 −0
  139. tools/testing/selftests/cgroup/lib/include/cgroup_util.h +1 −0
  140. tools/testing/selftests/cgroup/test_zswap.c +11 −2
  141. tools/testing/selftests/mm/.gitignore +9 −67
  142. tools/testing/selftests/mm/Makefile +4 −3
  143. tools/testing/selftests/mm/check_config.sh +1 −1
  144. tools/testing/selftests/mm/compaction_test.c +21 −17
  145. tools/testing/selftests/mm/cow.c +8 −3
  146. tools/testing/selftests/mm/folio_split_race_test.c +1 −1
  147. tools/testing/selftests/mm/guard-regions.c +7 −3
  148. tools/testing/selftests/mm/gup_longterm.c +1 −1
  149. tools/testing/selftests/mm/khugepaged.c +155 −49
  150. tools/testing/selftests/mm/ksft_kmemleak_confirm.sh +130 −0
  151. tools/testing/selftests/mm/memory-failure.c +12 −2
  152. tools/testing/selftests/mm/merge.c +57 −0
  153. tools/testing/selftests/mm/migration.c +20 −17
  154. tools/testing/selftests/mm/mseal_test.c +1 −2
  155. tools/testing/selftests/mm/pagemap_ioctl.c +1 −7
  156. tools/testing/selftests/mm/prctl_thp_disable.c +1 −1
  157. tools/testing/selftests/mm/run_vmtests.sh +2 −0
  158. tools/testing/selftests/mm/soft-dirty.c +3 −2
  159. tools/testing/selftests/mm/split_huge_page_test.c +8 −131
  160. tools/testing/selftests/mm/uffd-common.c +3 −1
  161. tools/testing/selftests/mm/vm_util.c +197 −10
  162. tools/testing/selftests/mm/vm_util.h +5 −3
  163. tools/testing/selftests/proc/proc-maps-race.c +2 −1
  164. tools/testing/vma/include/dup.h +62 −2
  165. tools/testing/vma/shared.c +2 −1
  166. tools/testing/vma/tests/merge.c +39 −10
  167. tools/testing/vma/tests/vma.c +47 −3
  168. tools/testing/vma/vma_internal.h +1 −0
Documentation/admin-guide/blockdev/zram.rst+33−1
@@ -109,14 +109,41 @@ path to the `dict` along with other parameters:: · 108 unmodified lines
109109 #pass path to pre-trained zstd dictionary
110110 echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/algorithm_params
111111
112+ #pass path to pre-trained zstd dictionary and compression level
113+ echo "algo=zstd level=8 dict=/etc/dictionary" > \
114+ /sys/block/zram0/algorithm_params
115+
112116 #same, but using algorithm priority
117+ echo "algo=zstd priority=1" > /sys/block/zram0/recomp_algorithm
113118 echo "priority=1 dict=/etc/dictionary" > \
114119 /sys/block/zram0/algorithm_params
115120
116− #pass path to pre-trained zstd dictionary and compression level
121+Each write to `algorithm_params` replaces the entire set of parameters of
122+the corresponding algorithm, parameters that are not listed in the write
123+are reset to their default values. Configure all of the parameters of an
124+algorithm in one write::
125+
126+ #WRONG: the second write resets level back to its default value
127+ echo "algo=zstd level=8" > /sys/block/zram0/algorithm_params
128+ echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/algorithm_params
129+
130+ #RIGHT
117131 echo "algo=zstd level=8 dict=/etc/dictionary" > \
118132 /sys/block/zram0/algorithm_params
119133
134+Select the compression algorithm before configuring its parameters. The
135+parameters of one algorithm are not necessarily valid for another one, so
136+changing the algorithm of a particular priority resets that priority's
137+parameters::
138+
139+ #WRONG: comp_algorithm write resets the previously configured level
140+ echo "level=8" > /sys/block/zram0/algorithm_params
141+ echo zstd > /sys/block/zram0/comp_algorithm
142+
143+ #RIGHT
144+ echo zstd > /sys/block/zram0/comp_algorithm
145+ echo "algo=zstd level=8" > /sys/block/zram0/algorithm_params
146+
120147 Parameters are algorithm specific: not all algorithms support pre-trained
121148 dictionaries, not all algorithms support `level`. Furthermore, for certain
122149 algorithms `level` controls the compression level (the higher the value the
@@ -124,6 +151,11 @@ better the compression ratio, it even can take negatives values for some · 1 unmodified line
124151 algorithms), for other algorithms `level` is acceleration level (the higher
125152 the value the lower the compression ratio).
126153
154+Parameters are handed over to the compression algorithm when the device is
155+initialised, hence invalid parameters (or parameters that the selected
156+algorithm does not support) are reported by the `disksize` write, and not
157+by the `algorithm_params` write that has configured them.
158+
127159 Set Disksize
128160 ============
129161
Documentation/admin-guide/kernel-parameters.txt+14−2
@@ -2077,8 +2077,14 @@ Kernel parameters · 2076 unmodified lines
20772077 hugetlb_cma= [HW,CMA,EARLY] The size of a CMA area used for allocation
20782078 of gigantic hugepages. Or using node format, the size
20792079 of a CMA area per node can be specified.
2080− Format: nn[KMGTPE] or (node format)
2081− <node>:nn[KMGTPE][,<node>:nn[KMGTPE]]
2080+ The size can be an absolute value (e.g., 2G) or a
2081+ percentage of the total memory or node memory (e.g., 20%).
2082+ Percentage-derived sizes are rounded down to a multiple of
2083+ the architecture's gigantic hugepage size and may become
2084+ zero.
2085+ Format: nn[KMGTPE] or nn% or (node format)
2086+ <node>:nn[KMGTPE][,<node>:nn[KMGTPE]] or
2087+ <node>:nn%[,<node>:nn%]
20822088
20832089 The size must be a multiple of the gigantic page size.
20842090 When using node format, this applies to each per-node size.
@@ -4854,6 +4860,12 @@ Kernel parameters · 2769 unmodified lines
48544860 Adjust the minimal page reporting order. The page
48554861 reporting is disabled when it exceeds MAX_PAGE_ORDER.
48564862
4863+ page_reporting.page_reporting_delay_ms=
4864+ [KNL] Free page reporting delay in milliseconds
4865+ Format: <unsigned integer>
4866+ Adjust the delay in milliseconds between free page
4867+ reporting intervals. Default is 2000 (2 seconds).
4868+
48574869 panic= [KNL] Kernel behaviour on panic: delay <timeout>
48584870 timeout > 0: seconds before rebooting
48594871 timeout = 0: wait forever
Documentation/admin-guide/mm/transhuge.rst+4−2
@@ -224,7 +224,7 @@ khugepaged will be automatically started when any THP size is enabled · 223 unmodified lines
224224 (either of the per-size anon control or the top-level control are set
225225 to "always" or "madvise"), and it'll be automatically shutdown when
226226 all THP sizes are disabled (when both the per-size anon control and the
227−top-level control are "never")
227+top-level control are "never").
228228
229229 process THP controls
230230 --------------------
@@ -301,7 +301,9 @@ being replaced by a PMD mapping, or (2) physical pages replaced by one · 70 unmodified lines
301301 hugepage of various sizes (PMD-sized or mTHP). Each may happen independently,
302302 or together, depending on the type of memory and the failures that occur.
303303 As such, this value should be interpreted roughly as a sign of progress,
304−and counters in /proc/vmstat consulted for more accurate accounting)::
304+and counters in /proc/vmstat consulted for more accurate accounting.
305+Per-order mTHP collapse statistics are also available under
306+/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/stats/)::
305307
306308 /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
307309
Documentation/core-api/maple_tree.rst+20−4
@@ -17,7 +17,8 @@ supports iterating over a range of entries and going to the previous or next · 16 unmodified lines
1717 entry in a cache-efficient manner. The tree can also be put into an RCU-safe
1818 mode of operation which allows reading and writing concurrently. Writers must
1919 synchronize on a lock, which can be the default spinlock, or the user can set
20−the lock to an external lock of a different type.
20+the lock to an external lock of a different type. Note that external locks may
21+interfere with allocations in a low memory situation.
2122
2223 The Maple Tree maintains a small memory footprint and was designed to use
2324 modern processor cache efficiently. The majority of the users will be able to
@@ -42,6 +43,15 @@ successful store operation within a given · 18 unmodified lines
4243 code segment when allocating cannot be done. Allocations of nodes are
4344 relatively small at around 256 bytes.
4445
46+Since the maple tree uses internal nodes that are allocated and has rules on
47+data density, erasing an entry may cause allocations to occur. That is,
48+erasing an entry may consume memory. Users must take care to ensure that they
49+do not violate the larger system constraints on when and how memory is
50+allocated. Most situations are fine to allocate, but the pre-allocation
51+support is provided as a mechanism to avoid trickier situations. There is also
52+the possibility of using special entries and clean up the tree later, in
53+extreme circumstances.
54+
4555 .. _maple-tree-normal-api:
4656
4757 Normal API
@@ -63,7 +73,10 @@ success or an error code otherwise. mtree_store_range() works in the same way · 15 unmodified lines
6373 but takes a range. mtree_load() is used to retrieve the entry stored at a
6474 given index. You can use mtree_erase() to erase an entire range by only
6575 knowing one value within that range, or mtree_store() call with an entry of
66−NULL may be used to partially erase a range or many ranges at once.
76+NULL may be used to partially erase a range or many ranges at once. Note that
77+mtree_erase() may use GFP_KERNEL | __GFP_NOFAIL for allocations and cannot
78+fail. mtree_erase() can sleep, so it must not be called from an atomic
79+context.
6780
6881 If you want to only store a new entry to a range (or index) if that range is
6982 currently ``NULL``, you can use mtree_insert_range() or mtree_insert() which
@@ -163,7 +176,10 @@ You can use mas_erase() to erase an entire range by setting index and · 93 unmodified lines
163176 last of the maple state to the desired range to erase. This will erase
164177 the first range that is found in that range, set the maple state index
165178 and last as the range that was erased and return the entry that existed
166−at that location.
179+at that location. Note that mas_erase() may allocate with the GFP_KERNEL
180+__GFP_NOFAIL and cannot fail, but may sleep. If this is not okay, consider
181+using mas_store_gfp() and pass it a ``NULL``,
182+after setting up the correct range by walking to the entry.
167183
168184 You can walk each entry within a range by using mas_for_each(). If you want
169185 to walk each element of the tree then ``0`` and ``ULONG_MAX`` may be used as
@@ -211,7 +227,7 @@ Advanced Locking · 41 unmodified lines
211227
212228 The maple tree uses a spinlock by default, but external locks can be used for
213229 tree updates as well. To use an external lock, the tree must be initialized
214−with the ``MT_FLAGS_LOCK_EXTERN flag``, this is usually done with the
230+with the ``MT_FLAGS_LOCK_EXTERN`` flag, this is usually done with the
215231 MTREE_INIT_EXT() #define, which takes an external lock as an argument.
216232
217233 Functions and structures
Documentation/dev-tools/kmemleak.rst+10−0
@@ -198,6 +198,16 @@ systems, because of pointers temporarily stored in CPU registers or · 197 unmodified lines
198198 stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing
199199 the minimum age of an object to be reported as a memory leak.
200200
201+The ``min_unref_scans`` module parameter requires an object to be seen
202+unreferenced in that many consecutive scans before it is reported. It
203+defaults to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled, where the
204+periodic scan thread confirms a leak on its own, and to 1 otherwise. A
205+value of 1 preserves the historical behaviour; higher values filter the
206+transient false positives described above, at the cost of delaying genuine
207+reports by up to that many scans. It can be set at boot with
208+``kmemleak.min_unref_scans=<n>`` or at run-time via
209+``/sys/module/kmemleak/parameters/min_unref_scans``.
210+
201211 Limitations and Drawbacks
202212 -------------------------
203213
Documentation/filesystems/fscrypt.rst+4−0
@@ -1237,6 +1237,10 @@ astute users may notice some differences in behavior: · 1236 unmodified lines
12371237
12381238 - DAX (Direct Access) is not supported on encrypted files.
12391239
1240+- Encrypted files cannot be used directly as swap files. To swap to
1241+ an encrypted file, set up a loopback device on top of it.
1242+ Alternatively, encrypted swap can use a dm-crypt device.
1243+
12401244 - The maximum length of an encrypted symlink is 2 bytes shorter than
12411245 the maximum length of an unencrypted symlink. For example, on an
12421246 EXT4 filesystem with a 4K block size, unencrypted symlinks can be up
Documentation/filesystems/locking.rst+2−6
@@ -266,7 +266,6 @@ prototypes:: · 265 unmodified lines
266266 int (*error_remove_folio)(struct address_space *, struct folio *);
267267 int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span)
268268 int (*swap_deactivate)(struct file *);
269− int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
270269
271270 locking rules:
272271 All except dirty_folio and free_folio may block
@@ -291,7 +290,6 @@ is_partially_uptodate: yes · 18 unmodified lines
291290 error_remove_folio: yes
292291 swap_activate: no
293292 swap_deactivate: no
294−swap_rw: yes, unlocks
295293 ====================== ======================== ========= ===============
296294
297295 ->write_begin(), ->write_end() and ->read_folio() may be called from
@@ -355,14 +353,12 @@ should perform any validation and preparation necessary to ensure that · 57 unmodified lines
355353 writes can be performed with minimal memory allocation. It should call
356354 add_swap_extent(), or the helper iomap_swapfile_activate(), and return
357355 the number of extents added. If IO should be submitted through
358−->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted
359−directly to the block device ``sis->bdev``.
356+the file system it should call swap_fs_activate, otherwise IO will be
357+submitted directly to the block device ``sis->bdev``.
360358
361359 ->swap_deactivate() will be called in the sys_swapoff()
362360 path after ->swap_activate() returned success.
363361
364−->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate().
365−
366362 file_lock_operations
367363 ====================
368364
Documentation/filesystems/vfs.rst+2−6
@@ -776,7 +776,6 @@ cache in your filesystem. The following members are defined: · 775 unmodified lines
776776 int (*error_remove_folio)(struct mapping *mapping, struct folio *);
777777 int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span)
778778 int (*swap_deactivate)(struct file *);
779− int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
780779 };
781780
782781 ``read_folio``
@@ -977,16 +976,13 @@ cache in your filesystem. The following members are defined: · 194 unmodified lines
977976 can be performed with minimal memory allocation. It should call
978977 add_swap_extent(), or the helper iomap_swapfile_activate(), and
979978 return the number of extents added. If IO should be submitted
980− through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will
981− be submitted directly to the block device ``sis->bdev``.
979+ through the file system it should call swap_fs_activate, otherwise IO
980+ will be submitted directly to the block device ``sis->bdev``.
982981
983982 ``swap_deactivate``
984983 Called during swapoff on files where swap_activate was
985984 successful.
986985
987−``swap_rw``
988− Called to read or write swap pages when SWP_FS_OPS is set.
989−
990986 The File Object
991987 ===============
992988
Documentation/mm/allocation-profiling.rst+16−0
@@ -43,9 +43,25 @@ sysctl: · 42 unmodified lines
4343 warnings produced by allocations made while profiling is disabled and freed
4444 when it's enabled.
4545
46+ /proc/sys/vm/mem_profiling_compressed
47+
48+ 1: Page alloc tag compression is enabled.
49+
50+ 0: Page alloc tag compression is disabled.
51+
52+ This reflects a static boot-time configuration of how page allocation tags are
53+ stored (in page flags when compression is enabled and in page_ext when disabled).
54+ Toggling ``mem_profiling`` at runtime does not change the state of
55+ ``mem_profiling_compressed``.
56+
4657 Runtime info:
4758 /proc/allocinfo
4859
60+ Profiling data can be retrieved either by reading `/proc/allocinfo` directly as
61+ text or programmatically via `ioctl()` calls defined in `<uapi/linux/alloc_tag.h>`.
62+ The ioctl interface supports structured binary data extraction as well as filtering
63+ by module name, function, file, line number, accuracy, or allocation size limits.
64+
4965 Example output::
5066
5167 root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec
Documentation/mm/ksm.rst+2−2
@@ -24,13 +24,13 @@ tree. · 23 unmodified lines
2424
2525 If a KSM page is shared between less than ``max_page_sharing`` VMAs,
2626 the node of the stable tree that represents such KSM page points to a
27−list of struct ksm_rmap_item and the ``page->mapping`` of the
27+list of struct ksm_rmap_item and the ``folio->mapping`` of the
2828 KSM page points to the stable tree node.
2929
3030 When the sharing passes this threshold, KSM adds a second dimension to
3131 the stable tree. The tree node becomes a "chain" that links one or
3232 more "dups". Each "dup" keeps reverse mapping information for a KSM
33−page with ``page->mapping`` pointing to that "dup".
33+page with ``folio->mapping`` pointing to that "dup".
3434
3535 Every "chain" and all "dups" linked into a "chain" enforce the
3636 invariant that they represent the same write protected memory content,
Documentation/mm/page_migration.rst+3−3
@@ -110,13 +110,13 @@ Steps: · 109 unmodified lines
110110 6. The refcount of the page is examined and we back out if references remain.
111111 Otherwise, we know that we are the only one referencing this page.
112112
113−7. The radix tree is checked and if it does not contain the pointer to this
114− page then we back out because someone else modified the radix tree.
113+7. The page cache tree is checked and if it does not contain the pointer to this
114+ page then we back out because someone else modified the page cache tree.
115115
116116 8. The new page is prepped with some settings from the old page so that
117117 accesses to the new page will discover a page with the correct settings.
118118
119−9. The radix tree is changed to point to the new page.
119+9. The page cache tree is changed to point to the new page.
120120
121121 10. The reference count of the old page is dropped because the address space
122122 reference is gone. A reference to the new page is established because
Documentation/translations/zh_CN/mm/ksm.rst+2−2
@@ -31,10 +31,10 @@ KSM维护着稳定树中的KSM页的逆映射信息。 · 30 unmodified lines
3131
3232 当KSM页面的共享数小于 ``max_page_sharing`` 的虚拟内存区域(VMAs)时，则代表了
3333 KSM页的稳定树其中的节点指向了一个ksm_rmap_item结构体类型的列表。同时，这个KSM页
34−的 ``page->mapping`` 指向了该稳定树节点。
34+的 ``folio->mapping`` 指向了该稳定树节点。
3535
3636 如果共享数超过了阈值，KSM将给稳定树添加第二个维度。稳定树就变成链接一个或多
37−个稳定树"副本"的"链"。每个副本都保留KSM页的逆映射信息，其中 ``page->mapping``
37+个稳定树"副本"的"链"。每个副本都保留KSM页的逆映射信息，其中 ``folio->mapping``
3838 指向该"副本"。
3939
4040 每个链以及链接到该链中的所有"副本"强制不变的是，它们代表了相同的写保护内存
Documentation/userspace-api/ioctl/ioctl-number.rst+2−0
@@ -346,6 +346,8 @@ Code Seq# Include File Comments · 345 unmodified lines
346346 <mailto:luzmaximilian@gmail.com>
347347 0xA5 20-2F linux/surface_aggregator/dtx.h Microsoft Surface DTX driver
348348 <mailto:luzmaximilian@gmail.com>
349+0xA6 00-0F uapi/linux/alloc_tag.h Memory allocation profiling
350+ <mailto:surenb@google.com>
349351 0xAA 00-3F linux/uapi/linux/userfaultfd.h
350352 0xAB 00-1F linux/nbd.h
351353 0xAC 00-1F linux/raw.h
MAINTAINERS+6−0
@@ -17022,6 +17022,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git fixes · 17021 unmodified lines
1702217022 F: Documentation/core-api/boot-time-mm.rst
1702317023 F: include/linux/kho/abi/memblock.h
1702417024 F: include/linux/memblock.h
17025+F: mm/arch_numa.c
1702517026 F: mm/memblock.c
1702617027 F: mm/memtest.c
1702717028 F: mm/mm_init.c
@@ -17081,7 +17082,9 @@ S: Maintained · 53 unmodified lines
1708117082 F: Documentation/mm/allocation-profiling.rst
1708217083 F: include/linux/alloc_tag.h
1708317084 F: include/linux/pgalloc_tag.h
17085+F: include/uapi/linux/alloc_tag.h
1708417086 F: mm/alloc_tag.c
17087+F: tools/testing/selftests/alloc_tag/
1708517088
1708617089 MEMORY MANAGEMENT - BALLOON
1708717090 M: Andrew Morton <akpm@linux-foundation.org>
@@ -17254,6 +17257,7 @@ F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers · 166 unmodified lines
1725417257 F: Documentation/ABI/testing/sysfs-kernel-mm-numa
1725517258 F: Documentation/admin-guide/mm/
1725617259 F: Documentation/mm/
17260+F: drivers/char/mem.c
1725717261 F: include/linux/cma.h
1725817262 F: include/linux/dmapool.h
1725917263 F: include/linux/ioremap.h
@@ -17381,6 +17385,7 @@ S: Maintained · 121 unmodified lines
1738117385 F: Documentation/ABI/testing/sysfs-kernel-mm-swap
1738217386 F: Documentation/mm/swap-table.rst
1738317387 F: include/linux/swap.h
17388+F: include/linux/swap_ops.h
1738417389 F: include/linux/swapfile.h
1738517390 F: include/linux/swapops.h
1738617391 F: mm/page_io.c
@@ -17457,6 +17462,7 @@ L: linux-mm@kvack.org · 70 unmodified lines
1745717462 S: Maintained
1745817463 W: http://www.linux-mm.org
1745917464 T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
17465+F: drivers/char/mem.c
1746017466 F: include/trace/events/mmap.h
1746117467 F: fs/proc/task_mmu.c
1746217468 F: fs/proc/task_nommu.c
arch/s390/mm/gmap_helpers.c+1−1
@@ -203,7 +203,7 @@ static int find_zeropage_pte_entry(pte_t *pte, unsigned long addr, · 202 unmodified lines
203203 * currently only works in COW mappings, which is also where
204204 * mm_forbids_zeropage() is checked.
205205 */
206− if (!is_cow_mapping(walk->vma->vm_flags))
206+ if (!vma_is_cow_mapping(walk->vma))
207207 return -EFAULT;
208208
209209 *found_addr = addr;
drivers/base/Kconfig+0−7
@@ -230,13 +230,6 @@ config GENERIC_ARCH_TOPOLOGY · 229 unmodified lines
230230 appropriate scaling, sysfs interface for reading capacity values at
231231 runtime.
232232
233−config GENERIC_ARCH_NUMA
234− bool
235− select NUMA_MEMBLKS
236− help
237− Enable support for generic NUMA implementation. Currently, RISC-V
238− and ARM64 use it.
239−
240233 config FW_DEVLINK_SYNC_STATE_TIMEOUT
241234 bool "sync_state() behavior defaults to timeout instead of strict"
242235 help
drivers/base/Makefile+0−1
@@ -25,7 +25,6 @@ obj-$(CONFIG_PINCTRL) += pinctrl.o · 24 unmodified lines
2525 obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
2626 obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o
2727 obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
28−obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o
2928 obj-$(CONFIG_ACPI) += physical_location.o
3029
3130 obj-y += test/
drivers/block/zram/backend_842.c+10−0
@@ -1,5 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0-or-later
22
3+#define pr_fmt(fmt) "842: " fmt
4+
35 #include <linux/kernel.h>
46 #include <linux/slab.h>
57 #include <linux/sw842.h>
@@ -13,6 +15,14 @@ static void release_params_842(struct zcomp_params *params) · 7 unmodified lines
1315
1416 static int setup_params_842(struct zcomp_params *params)
1517 {
18+ if (params->dict_sz) {
19+ pr_err("dictionary is not supported\n");
20+ return -EOPNOTSUPP;
21+ }
22+ if (params->level != ZCOMP_PARAM_NOT_SET) {
23+ pr_err("compression level is not supported\n");
24+ return -EOPNOTSUPP;
25+ }
1626 return 0;
1727 }
1828
drivers/block/zram/backend_deflate.c+23−2
@@ -1,5 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0-or-later
22
3+#define pr_fmt(fmt) "deflate: " fmt
4+
35 #include <linux/kernel.h>
46 #include <linux/slab.h>
57 #include <linux/vmalloc.h>
@@ -22,10 +24,29 @@ static void deflate_release_params(struct zcomp_params *params) · 16 unmodified lines
2224
2325 static int deflate_setup_params(struct zcomp_params *params)
2426 {
25− if (params->level == ZCOMP_PARAM_NOT_SET)
27+ if (params->dict_sz) {
28+ pr_err("dictionary is not supported\n");
29+ return -EOPNOTSUPP;
30+ }
31+
32+ if (params->level == ZCOMP_PARAM_NOT_SET) {
2633 params->level = Z_DEFAULT_COMPRESSION;
27− if (params->deflate.winbits == ZCOMP_PARAM_NOT_SET)
34+ } else if (params->level < Z_DEFAULT_COMPRESSION ||
35+ params->level > Z_BEST_COMPRESSION) {
36+ pr_err("invalid compression level %d\n", params->level);
37+ return -EINVAL;
38+ }
39+
40+ if (params->deflate.winbits == ZCOMP_PARAM_NOT_SET) {
2841 params->deflate.winbits = DEFLATE_DEF_WINBITS;
42+ } else {
43+ s32 wb = params->deflate.winbits;
44+
45+ if ((wb < -15 || wb > -9) && (wb < 9 || wb > 15)) {
46+ pr_err("invalid winbits %d\n", wb);
47+ return -EINVAL;
48+ }
49+ }
2950
3051 return 0;
3152 }
drivers/block/zram/backend_lz4.c+9−1
@@ -1,3 +1,7 @@
1+// SPDX-License-Identifier: GPL-2.0-or-later
2+
3+#define pr_fmt(fmt) "lz4: " fmt
4+
15 #include <linux/kernel.h>
26 #include <linux/lz4.h>
37 #include <linux/slab.h>
@@ -28,8 +32,12 @@ static int lz4_setup_params(struct zcomp_params *params) · 24 unmodified lines
2832 LZ4_stream_t *dict_stream;
2933 int ret;
3034
31− if (params->level == ZCOMP_PARAM_NOT_SET)
35+ if (params->level == ZCOMP_PARAM_NOT_SET) {
3236 params->level = LZ4_ACCELERATION_DEFAULT;
37+ } else if (params->level < LZ4_ACCELERATION_DEFAULT) {
38+ pr_err("invalid compression level %d\n", params->level);
39+ return -EINVAL;
40+ }
3341
3442 if (!params->dict || !params->dict_sz)
3543 return 0;
drivers/block/zram/backend_lz4hc.c+15−1
@@ -1,3 +1,7 @@
1+// SPDX-License-Identifier: GPL-2.0-or-later
2+
3+#define pr_fmt(fmt) "lz4hc: " fmt
4+
15 #include <linux/kernel.h>
26 #include <linux/lz4.h>
37 #include <linux/slab.h>
@@ -18,8 +22,18 @@ static void lz4hc_release_params(struct zcomp_params *params) · 14 unmodified lines
1822
1923 static int lz4hc_setup_params(struct zcomp_params *params)
2024 {
21− if (params->level == ZCOMP_PARAM_NOT_SET)
25+ if (params->level == ZCOMP_PARAM_NOT_SET) {
2226 params->level = LZ4HC_DEFAULT_CLEVEL;
27+ } else if (params->level < 1 || params->level > LZ4HC_MAX_CLEVEL) {
28+ /*
29+ * Use < 1 rather than < LZ4HC_MIN_CLEVEL here because
30+ * LZ4HC_compress_generic() only clamps levels below 1
31+ * (levels 1 and 2 are valid). LZ4HC_MIN_CLEVEL (3) is
32+ * advisory and not enforced by the library.
33+ */
34+ pr_err("invalid compression level %d\n", params->level);
35+ return -EINVAL;
36+ }
2337
2438 return 0;
2539 }
drivers/block/zram/backend_lzo.c+10−0
@@ -1,5 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0-or-later
22
3+#define pr_fmt(fmt) "lzo: " fmt
4+
35 #include <linux/kernel.h>
46 #include <linux/slab.h>
57 #include <linux/lzo.h>
@@ -12,6 +14,14 @@ static void lzo_release_params(struct zcomp_params *params) · 6 unmodified lines
1214
1315 static int lzo_setup_params(struct zcomp_params *params)
1416 {
17+ if (params->dict_sz) {
18+ pr_err("dictionary is not supported\n");
19+ return -EOPNOTSUPP;
20+ }
21+ if (params->level != ZCOMP_PARAM_NOT_SET) {
22+ pr_err("compression level is not supported\n");
23+ return -EOPNOTSUPP;
24+ }
1525 return 0;
1626 }
1727
drivers/block/zram/backend_lzorle.c+10−0
@@ -1,5 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0-or-later
22
3+#define pr_fmt(fmt) "lzo-rle: " fmt
4+
35 #include <linux/kernel.h>
46 #include <linux/slab.h>
57 #include <linux/lzo.h>
@@ -12,6 +14,14 @@ static void lzorle_release_params(struct zcomp_params *params) · 6 unmodified lines
1214
1315 static int lzorle_setup_params(struct zcomp_params *params)
1416 {
17+ if (params->dict_sz) {
18+ pr_err("dictionary is not supported\n");
19+ return -EOPNOTSUPP;
20+ }
21+ if (params->level != ZCOMP_PARAM_NOT_SET) {
22+ pr_err("compression level is not supported\n");
23+ return -EOPNOTSUPP;
24+ }
1525 return 0;
1626 }
1727
drivers/block/zram/backend_zstd.c+8−3
@@ -1,5 +1,7 @@
11 // SPDX-License-Identifier: GPL-2.0-or-later
22
3+#define pr_fmt(fmt) "zstd: " fmt
4+
35 #include <linux/kernel.h>
46 #include <linux/slab.h>
57 #include <linux/vmalloc.h>
@@ -58,8 +60,13 @@ static int zstd_setup_params(struct zcomp_params *params) · 52 unmodified lines
5860 return -ENOMEM;
5961
6062 params->drv_data = zp;
61− if (params->level == ZCOMP_PARAM_NOT_SET)
63+ if (params->level == ZCOMP_PARAM_NOT_SET) {
6264 params->level = zstd_default_clevel();
65+ } else if (params->level < zstd_min_clevel() ||
66+ params->level > zstd_max_clevel()) {
67+ pr_err("invalid compression level %d\n", params->level);
68+ goto error;
69+ }
6370
6471 zp->cprm = zstd_get_params(params->level, PAGE_SIZE);
6572
@@ -85,7 +92,6 @@ static int zstd_setup_params(struct zcomp_params *params) · 19 unmodified lines
8592 return 0;
8693
8794 error:
88− zstd_release_params(params);
8995 return -EINVAL;
9096 }
9197
@@ -161,7 +167,6 @@ static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx) · 69 unmodified lines
161167 return 0;
162168
163169 error:
164− zstd_release_params(params);
165170 zstd_destroy(ctx);
166171 return -EINVAL;
167172 }
drivers/block/zram/zram_drv.c+109−79
@@ -56,7 +56,7 @@ static size_t huge_class_size; · 55 unmodified lines
5656
5757 static const struct block_device_operations zram_devops;
5858
59−static void slot_free(struct zram *zram, u32 index);
59+static void slot_free(struct zram *zram, unsigned long index);
6060
6161 /*
6262 * entry locking rules:
@@ -70,11 +70,11 @@ static void slot_free(struct zram *zram, u32 index); · 7 unmodified lines
7070 * 4) Use TRY lock variant when in atomic context
7171 * - must check return value and handle locking failers
7272 */
73−static __must_check bool slot_trylock(struct zram *zram, u32 index)
73+static __must_check bool slot_trylock(struct zram *zram, unsigned long index)
7474 {
7575 unsigned long *lock = &zram->table[index].__lock;
7676
77− if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) {
77+ if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK_BIT, lock)) {
7878 mutex_acquire(&zram->table_lock_map, 0, 1, _RET_IP_);
7979 lock_acquired(&zram->table_lock_map, _RET_IP_);
8080 return true;
@@ -83,21 +83,21 @@ static __must_check bool slot_trylock(struct zram *zram, u32 index) · 2 unmodified lines
8383 return false;
8484 }
8585
86−static void slot_lock(struct zram *zram, u32 index)
86+static void slot_lock(struct zram *zram, unsigned long index)
8787 {
8888 unsigned long *lock = &zram->table[index].__lock;
8989
9090 mutex_acquire(&zram->table_lock_map, 0, 0, _RET_IP_);
91− wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE);
91+ wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK_BIT, TASK_UNINTERRUPTIBLE);
9292 lock_acquired(&zram->table_lock_map, _RET_IP_);
9393 }
9494
95−static void slot_unlock(struct zram *zram, u32 index)
95+static void slot_unlock(struct zram *zram, unsigned long index)
9696 {
9797 unsigned long *lock = &zram->table[index].__lock;
9898
9999 mutex_release(&zram->table_lock_map, _RET_IP_);
100− clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock);
100+ clear_and_wake_up_bit(ZRAM_ENTRY_LOCK_BIT, lock);
101101 }
102102
103103 static inline bool init_done(struct zram *zram)
@@ -110,55 +110,56 @@ static inline struct zram *dev_to_zram(struct device *dev) · 6 unmodified lines
110110 return (struct zram *)dev_to_disk(dev)->private_data;
111111 }
112112
113−static unsigned long get_slot_handle(struct zram *zram, u32 index)
113+static unsigned long get_slot_handle(struct zram *zram, unsigned long index)
114114 {
115115 return zram->table[index].handle;
116116 }
117117
118−static void set_slot_handle(struct zram *zram, u32 index, unsigned long handle)
118+static void set_slot_handle(struct zram *zram, unsigned long index,
119+ unsigned long handle)
119120 {
120121 zram->table[index].handle = handle;
121122 }
122123
123−static bool test_slot_flag(struct zram *zram, u32 index,
124+static bool test_slot_flag(struct zram *zram, unsigned long index,
124125 enum zram_pageflags flag)
125126 {
126127 return zram->table[index].attr.flags & BIT(flag);
127128 }
128129
129−static void set_slot_flag(struct zram *zram, u32 index,
130+static void set_slot_flag(struct zram *zram, unsigned long index,
130131 enum zram_pageflags flag)
131132 {
132133 zram->table[index].attr.flags |= BIT(flag);
133134 }
134135
135−static void clear_slot_flag(struct zram *zram, u32 index,
136+static void clear_slot_flag(struct zram *zram, unsigned long index,
136137 enum zram_pageflags flag)
137138 {
138139 zram->table[index].attr.flags &= ~BIT(flag);
139140 }
140141
141−static size_t get_slot_size(struct zram *zram, u32 index)
142+static size_t get_slot_size(struct zram *zram, unsigned long index)
142143 {
143144 return zram->table[index].attr.flags & (BIT(ZRAM_FLAG_SHIFT) - 1);
144145 }
145146
146−static void set_slot_size(struct zram *zram, u32 index, size_t size)
147+static void set_slot_size(struct zram *zram, unsigned long index, size_t size)
147148 {
148149 unsigned long flags = zram->table[index].attr.flags >> ZRAM_FLAG_SHIFT;
149150
150151 zram->table[index].attr.flags = (flags << ZRAM_FLAG_SHIFT) | size;
151152 }
152153
153−static inline bool slot_allocated(struct zram *zram, u32 index)
154+static inline bool slot_allocated(struct zram *zram, unsigned long index)
154155 {
155156 return get_slot_size(zram, index) ||
156157 test_slot_flag(zram, index, ZRAM_SAME) ||
157158 test_slot_flag(zram, index, ZRAM_WB);
158159 }
159160
160−static inline void set_slot_comp_priority(struct zram *zram, u32 index,
161− u32 prio)
161+static inline void set_slot_comp_priority(struct zram *zram,
162+ unsigned long index, u32 prio)
162163 {
163164 prio &= ZRAM_COMP_PRIORITY_MASK;
164165 /*
@@ -170,14 +171,14 @@ static inline void set_slot_comp_priority(struct zram *zram, u32 index, · 5 unmodified lines
170171 zram->table[index].attr.flags |= (prio << ZRAM_COMP_PRIORITY_BIT1);
171172 }
172173
173−static inline u32 get_slot_comp_priority(struct zram *zram, u32 index)
174+static inline u32 get_slot_comp_priority(struct zram *zram, unsigned long index)
174175 {
175176 u32 prio = zram->table[index].attr.flags >> ZRAM_COMP_PRIORITY_BIT1;
176177
177178 return prio & ZRAM_COMP_PRIORITY_MASK;
178179 }
179180
180−static void mark_slot_accessed(struct zram *zram, u32 index)
181+static void mark_slot_accessed(struct zram *zram, unsigned long index)
181182 {
182183 clear_slot_flag(zram, index, ZRAM_IDLE);
183184 clear_slot_flag(zram, index, ZRAM_PP_SLOT);
@@ -284,7 +285,7 @@ static void release_pp_ctl(struct zram *zram, struct zram_pp_ctl *ctl) · 100 unmodified lines
284285 }
285286
286287 static bool place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl,
287− u32 index)
288+ unsigned long index)
288289 {
289290 struct zram_pp_slot *pps;
290291 u32 bid;
@@ -418,7 +419,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) · 127 unmodified lines
418419 {
419420 int is_idle = 1;
420421 unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
421− int index;
422+ unsigned long index;
422423
423424 for (index = 0; index < nr_pages; index++) {
424425 /*
@@ -485,8 +486,9 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr, · 60 unmodified lines
485486 #define INVALID_BDEV_BLOCK (~0UL)
486487
487488 static int read_from_zspool_raw(struct zram *zram, struct page *page,
488− u32 index);
489−static int read_from_zspool(struct zram *zram, struct page *page, u32 index);
489+ unsigned long index);
490+static int read_from_zspool(struct zram *zram, struct page *page,
491+ unsigned long index);
490492
491493 struct zram_wb_ctl {
492494 /* idle list is accessed only by the writeback task, no concurency */
@@ -522,7 +524,7 @@ struct zram_rb_req { · 29 unmodified lines
522524 /* error status (sync read) */
523525 int error;
524526 };
525− u32 index;
527+ unsigned long index;
526528 };
527529
528530 #define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12)))
@@ -910,7 +912,7 @@ static void zram_account_writeback_submit(struct zram *zram) · 381 unmodified lines
910912
911913 static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req)
912914 {
913− u32 index = req->pps->index;
915+ unsigned long index = req->pps->index;
914916 int err;
915917
916918 err = blk_status_to_errno(req->bio.bi_status);
@@ -1032,7 +1034,7 @@ static int zram_writeback_slots(struct zram *zram, · 115 unmodified lines
10321034 struct zram_wb_req *req = NULL;
10331035 struct zram_pp_slot *pps;
10341036 int ret = 0, err = 0;
1035− u32 index = 0;
1037+ unsigned long index = 0;
10361038
10371039 while ((pps = select_pp_slot(ctl))) {
10381040 if (zram->wb_limit_enable && !zram->bd_wb_limit) {
@@ -1198,7 +1200,7 @@ static void scan_slots_for_writeback(struct zram *zram, u32 mode, · 159 unmodified lines
11981200 unsigned long lo, unsigned long hi,
11991201 struct zram_pp_ctl *ctl)
12001202 {
1201− u32 index = lo;
1203+ unsigned long index = lo;
12021204
12031205 while (index < hi) {
12041206 bool ok = true;
@@ -1235,8 +1237,8 @@ static ssize_t writeback_store(struct device *dev, · 30 unmodified lines
12351237 const char *buf, size_t len)
12361238 {
12371239 struct zram *zram = dev_to_zram(dev);
1238− u64 nr_pages = zram->disksize >> PAGE_SHIFT;
1239− unsigned long lo = 0, hi = nr_pages;
1240+ unsigned long nr_pages;
1241+ unsigned long lo = 0, hi;
12401242 struct zram_pp_ctl *pp_ctl = NULL;
12411243 struct zram_wb_ctl *wb_ctl = NULL;
12421244 char *args, *param, *val;
@@ -1250,6 +1252,9 @@ static ssize_t writeback_store(struct device *dev, · 7 unmodified lines
12501252 if (!zram->backing_dev)
12511253 return -ENODEV;
12521254
1255+ nr_pages = zram->disksize >> PAGE_SHIFT;
1256+ hi = nr_pages;
1257+
12531258 pp_ctl = init_pp_ctl();
12541259 if (!pp_ctl)
12551260 return -ENOMEM;
@@ -1333,7 +1338,8 @@ out: · 77 unmodified lines
13331338 return ret;
13341339 }
13351340
1336−static int decompress_bdev_page(struct zram *zram, struct page *page, u32 index)
1341+static int decompress_bdev_page(struct zram *zram, struct page *page,
1342+ unsigned long index)
13371343 {
13381344 struct zcomp_strm *zstrm;
13391345 unsigned int size;
@@ -1375,7 +1381,7 @@ static void zram_deferred_decompress(struct work_struct *w) · 35 unmodified lines
13751381 struct zram_rb_req *req = container_of(w, struct zram_rb_req, work);
13761382 struct page *page = bio_first_page_all(req->bio);
13771383 struct zram *zram = req->zram;
1378− u32 index = req->index;
1384+ unsigned long index = req->index;
13791385 int ret;
13801386
13811387 ret = decompress_bdev_page(zram, page, index);
@@ -1426,7 +1432,7 @@ static void zram_async_read_endio(struct bio *bio) · 44 unmodified lines
14261432 }
14271433
14281434 static int read_from_bdev_async(struct zram *zram, struct page *page,
1429− u32 index, unsigned long blk_idx,
1435+ unsigned long index, unsigned long blk_idx,
14301436 struct bio *parent)
14311437 {
14321438 struct zram_rb_req *req;
@@ -1476,8 +1482,8 @@ static void zram_sync_read(struct work_struct *w) · 43 unmodified lines
14761482 * chained IO with parent IO in same context, it's a deadlock. To avoid that,
14771483 * use a worker thread context.
14781484 */
1479−static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index,
1480− unsigned long blk_idx)
1485+static int read_from_bdev_sync(struct zram *zram, struct page *page,
1486+ unsigned long index, unsigned long blk_idx)
14811487 {
14821488 struct zram_rb_req req;
14831489
@@ -1496,8 +1502,9 @@ static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, · 12 unmodified lines
14961502 return decompress_bdev_page(zram, page, index);
14971503 }
14981504
1499−static int read_from_bdev(struct zram *zram, struct page *page, u32 index,
1500− unsigned long blk_idx, struct bio *parent)
1505+static int read_from_bdev(struct zram *zram, struct page *page,
1506+ unsigned long index, unsigned long blk_idx,
1507+ struct bio *parent)
15011508 {
15021509 atomic64_inc(&zram->stats.bd_reads);
15031510 if (!parent) {
@@ -1509,8 +1516,9 @@ static int read_from_bdev(struct zram *zram, struct page *page, u32 index, · 5 unmodified lines
15091516 }
15101517 #else
15111518 static inline void reset_bdev(struct zram *zram) {};
1512−static int read_from_bdev(struct zram *zram, struct page *page, u32 index,
1513− unsigned long blk_idx, struct bio *parent)
1519+static int read_from_bdev(struct zram *zram, struct page *page,
1520+ unsigned long index, unsigned long blk_idx,
1521+ struct bio *parent)
15141522 {
15151523 return -EIO;
15161524 }
@@ -1538,9 +1546,10 @@ static ssize_t read_block_state(struct file *file, char __user *buf, · 21 unmodified lines
15381546 size_t count, loff_t *ppos)
15391547 {
15401548 char *kbuf;
1541− ssize_t index, written = 0;
1549+ unsigned long index;
1550+ ssize_t written = 0;
15421551 struct zram *zram = file->private_data;
1543− unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
1552+ unsigned long nr_pages;
15441553
15451554 kbuf = kvmalloc(count, GFP_KERNEL);
15461555 if (!kbuf)
@@ -1552,6 +1561,8 @@ static ssize_t read_block_state(struct file *file, char __user *buf, · 5 unmodified lines
15521561 return -EINVAL;
15531562 }
15541563
1564+ nr_pages = zram->disksize >> PAGE_SHIFT;
1565+
15551566 for (index = *ppos; index < nr_pages; index++) {
15561567 int copied;
15571568
@@ -1560,7 +1571,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, · 2 unmodified lines
15601571 goto next;
15611572
15621573 copied = snprintf(kbuf + written, count,
1563− "%12zd %12u.%06d %c%c%c%c%c%c\n",
1574+ "%12lu %12u.%06d %c%c%c%c%c%c\n",
15641575 index, zram->table[index].attr.ac_time, 0,
15651576 test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.',
15661577 test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.',
@@ -1652,6 +1663,17 @@ static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) · 85 unmodified lines
16521663 zram->comp_algs[prio] = alg;
16531664 }
16541665
1666+static void comp_params_reset(struct zram *zram, u32 prio)
1667+{
1668+ struct zcomp_params *params = &zram->params[prio];
1669+
1670+ vfree(params->dict);
1671+ params->level = ZCOMP_PARAM_NOT_SET;
1672+ params->deflate.winbits = ZCOMP_PARAM_NOT_SET;
1673+ params->dict_sz = 0;
1674+ params->dict = NULL;
1675+}
1676+
16551677 static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf)
16561678 {
16571679 const char *alg;
@@ -1672,20 +1694,10 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) · 14 unmodified lines
16721694 }
16731695
16741696 comp_algorithm_set(zram, prio, alg);
1697+ comp_params_reset(zram, prio);
16751698 return 0;
16761699 }
16771700
1678−static void comp_params_reset(struct zram *zram, u32 prio)
1679−{
1680− struct zcomp_params *params = &zram->params[prio];
1681−
1682− vfree(params->dict);
1683− params->level = ZCOMP_PARAM_NOT_SET;
1684− params->deflate.winbits = ZCOMP_PARAM_NOT_SET;
1685− params->dict_sz = 0;
1686− params->dict = NULL;
1687−}
1688−
16891701 static int comp_params_store(struct zram *zram, u32 prio, s32 level,
16901702 const char *dict_path,
16911703 struct deflate_params *deflate_params)
@@ -1700,8 +1712,16 @@ static int comp_params_store(struct zram *zram, u32 prio, s32 level, · 8 unmodified lines
17001712 INT_MAX,
17011713 NULL,
17021714 READING_POLICY);
1703− if (sz < 0)
1715+ if (sz < 0) {
1716+ pr_err("failed to load dictionary %s (err=%zd)\n",
1717+ dict_path, sz);
1718+ return sz;
1719+ }
1720+ if (sz == 0) {
1721+ pr_err("failed to load dictionary %s (empty file)\n",
1722+ dict_path);
17041723 return -EINVAL;
1724+ }
17051725 }
17061726
17071727 zram->params[prio].dict_sz = sz;
@@ -1958,8 +1978,8 @@ static ssize_t debug_stat_show(struct device *dev, · 250 unmodified lines
19581978
19591979 static void zram_meta_free(struct zram *zram, u64 disksize)
19601980 {
1961− size_t num_pages = disksize >> PAGE_SHIFT;
1962− size_t index;
1981+ unsigned long num_pages = disksize >> PAGE_SHIFT;
1982+ unsigned long index;
19631983
19641984 if (!zram->table)
19651985 return;
@@ -1976,7 +1996,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) · 10 unmodified lines
19761996
19771997 static bool zram_meta_alloc(struct zram *zram, u64 disksize)
19781998 {
1979− size_t num_pages;
1999+ unsigned long num_pages;
19802000
19812001 num_pages = disksize >> PAGE_SHIFT;
19822002 zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table)));
@@ -1999,7 +2019,7 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) · 16 unmodified lines
19992019 return true;
20002020 }
20012021
2002−static void slot_free(struct zram *zram, u32 index)
2022+static void slot_free(struct zram *zram, unsigned long index)
20032023 {
20042024 unsigned long handle;
20052025
@@ -2053,7 +2073,7 @@ out: · 47 unmodified lines
20532073 }
20542074
20552075 static int read_same_filled_page(struct zram *zram, struct page *page,
2056− u32 index)
2076+ unsigned long index)
20572077 {
20582078 void *mem;
20592079
@@ -2064,7 +2084,7 @@ static int read_same_filled_page(struct zram *zram, struct page *page, · 4 unmodified lines
20642084 }
20652085
20662086 static int read_incompressible_page(struct zram *zram, struct page *page,
2067− u32 index)
2087+ unsigned long index)
20682088 {
20692089 unsigned long handle;
20702090 void *src, *dst;
@@ -2079,7 +2099,8 @@ static int read_incompressible_page(struct zram *zram, struct page *page, · 8 unmodified lines
20792099 return 0;
20802100 }
20812101
2082−static int read_compressed_page(struct zram *zram, struct page *page, u32 index)
2102+static int read_compressed_page(struct zram *zram, struct page *page,
2103+ unsigned long index)
20832104 {
20842105 struct zcomp_strm *zstrm;
20852106 unsigned long handle;
@@ -2104,7 +2125,8 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) · 18 unmodified lines
21042125 }
21052126
21062127 #if defined CONFIG_ZRAM_WRITEBACK
2107−static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index)
2128+static int read_from_zspool_raw(struct zram *zram, struct page *page,
2129+ unsigned long index)
21082130 {
21092131 struct zcomp_strm *zstrm;
21102132 unsigned long handle;
@@ -2136,7 +2158,8 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) · 25 unmodified lines
21362158 * Reads (decompresses if needed) a page from zspool (zsmalloc).
21372159 * Corresponding ZRAM slot should be locked.
21382160 */
2139−static int read_from_zspool(struct zram *zram, struct page *page, u32 index)
2161+static int read_from_zspool(struct zram *zram, struct page *page,
2162+ unsigned long index)
21402163 {
21412164 if (test_slot_flag(zram, index, ZRAM_SAME) ||
21422165 !get_slot_handle(zram, index))
@@ -2148,8 +2171,8 @@ static int read_from_zspool(struct zram *zram, struct page *page, u32 index) · 5 unmodified lines
21482171 return read_incompressible_page(zram, page, index);
21492172 }
21502173
2151−static int zram_read_page(struct zram *zram, struct page *page, u32 index,
2152− struct bio *parent)
2174+static int zram_read_page(struct zram *zram, struct page *page,
2175+ unsigned long index, struct bio *parent)
21532176 {
21542177 int ret;
21552178
@@ -2171,7 +2194,7 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, · 15 unmodified lines
21712194
21722195 /* Should NEVER happen. Return bio error if it does. */
21732196 if (WARN_ON(ret < 0))
2174− pr_err("Decompression failed! err=%d, page=%u\n", ret, index);
2197+ pr_err("Decompression failed! err=%d, page=%lu\n", ret, index);
21752198
21762199 return ret;
21772200 }
@@ -2181,7 +2204,7 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, · 3 unmodified lines
21812204 * always expects a full page for the output.
21822205 */
21832206 static int zram_bvec_read_partial(struct zram *zram, struct bio_vec *bvec,
2184− u32 index, int offset)
2207+ unsigned long index, int offset)
21852208 {
21862209 struct page *page = alloc_page(GFP_NOIO);
21872210 int ret;
@@ -2196,7 +2219,7 @@ static int zram_bvec_read_partial(struct zram *zram, struct bio_vec *bvec, · 8 unmodified lines
21962219 }
21972220
21982221 static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
2199− u32 index, int offset, struct bio *bio)
2222+ unsigned long index, int offset, struct bio *bio)
22002223 {
22012224 if (is_partial_io(bvec))
22022225 return zram_bvec_read_partial(zram, bvec, index, offset);
@@ -2204,7 +2227,7 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, · 1 unmodified line
22042227 }
22052228
22062229 static int write_same_filled_page(struct zram *zram, unsigned long fill,
2207− u32 index)
2230+ unsigned long index)
22082231 {
22092232 slot_lock(zram, index);
22102233 slot_free(zram, index);
@@ -2219,7 +2242,7 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill, · 8 unmodified lines
22192242 }
22202243
22212244 static int write_incompressible_page(struct zram *zram, struct page *page,
2222− u32 index)
2245+ unsigned long index)
22232246 {
22242247 unsigned long handle;
22252248 void *src;
@@ -2259,7 +2282,8 @@ static int write_incompressible_page(struct zram *zram, struct page *page, · 33 unmodified lines
22592282 return 0;
22602283 }
22612284
2262−static int zram_write_page(struct zram *zram, struct page *page, u32 index)
2285+static int zram_write_page(struct zram *zram, struct page *page,
2286+ unsigned long index)
22632287 {
22642288 int ret = 0;
22652289 unsigned long handle;
@@ -2326,7 +2350,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) · 60 unmodified lines
23262350 * This is a partial IO. Read the full page before writing the changes.
23272351 */
23282352 static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,
2329− u32 index, int offset)
2353+ unsigned long index, int offset)
23302354 {
23312355 struct page *page = alloc_page(GFP_NOIO);
23322356 int ret;
@@ -2344,7 +2368,7 @@ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, · 11 unmodified lines
23442368 }
23452369
23462370 static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
2347− u32 index, int offset)
2371+ unsigned long index, int offset)
23482372 {
23492373 if (is_partial_io(bvec))
23502374 return zram_bvec_write_partial(zram, bvec, index, offset);
@@ -2412,8 +2436,9 @@ next: · 61 unmodified lines
24122436 *
24132437 * Corresponding ZRAM slot should be locked.
24142438 */
2415−static int recompress_slot(struct zram *zram, u32 index, struct page *page,
2416− u64 *num_recomp_pages, u32 threshold, u32 prio)
2439+static int recompress_slot(struct zram *zram, unsigned long index,
2440+ struct page *page, u64 *num_recomp_pages,
2441+ u32 threshold, u32 prio)
24172442 {
24182443 struct zcomp_strm *zstrm = NULL;
24192444 unsigned long handle_old;
@@ -2665,7 +2690,7 @@ out: · 245 unmodified lines
26652690 static void zram_bio_discard(struct zram *zram, struct bio *bio)
26662691 {
26672692 size_t n = bio->bi_iter.bi_size;
2668− u32 index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
2693+ unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
26692694 u32 offset = (bio->bi_iter.bi_sector & (SECTORS_PER_PAGE - 1)) <<
26702695 SECTOR_SHIFT;
26712696
@@ -2706,7 +2731,7 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) · 34 unmodified lines
27062731 struct bvec_iter iter = bio->bi_iter;
27072732
27082733 do {
2709− u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
2734+ unsigned long index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
27102735 u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) <<
27112736 SECTOR_SHIFT;
27122737 struct bio_vec bv = bio_iter_iovec(bio, iter);
@@ -2737,7 +2762,7 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) · 24 unmodified lines
27372762 struct bvec_iter iter = bio->bi_iter;
27382763
27392764 do {
2740− u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
2765+ unsigned long index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
27412766 u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) <<
27422767 SECTOR_SHIFT;
27432768 struct bio_vec bv = bio_iter_iovec(bio, iter);
@@ -2828,6 +2853,7 @@ static void zram_destroy_comps(struct zram *zram) · 84 unmodified lines
28282853 zram->comp_algs[prio] = NULL;
28292854
28302855 zram_comp_params_reset(zram);
2856+ comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);
28312857 }
28322858
28332859 static void zram_reset_device(struct zram *zram)
@@ -2845,13 +2871,12 @@ static void zram_reset_device(struct zram *zram) · 11 unmodified lines
28452871 zram_destroy_comps(zram);
28462872 memset(&zram->stats, 0, sizeof(zram->stats));
28472873 reset_bdev(zram);
2848−
2849− comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);
28502874 }
28512875
28522876 static ssize_t disksize_store(struct device *dev, struct device_attribute *attr,
28532877 const char *buf, size_t len)
28542878 {
2879+ unsigned long num_pages;
28552880 u64 disksize;
28562881 struct zcomp *comp;
28572882 struct zram *zram = dev_to_zram(dev);
@@ -2869,6 +2894,11 @@ static ssize_t disksize_store(struct device *dev, struct device_attribute *attr, · 11 unmodified lines
28692894 }
28702895
28712896 disksize = PAGE_ALIGN(disksize);
2897+ num_pages = disksize >> PAGE_SHIFT;
2898+ /* Slots are addressed by an unsigned long index */
2899+ if (!num_pages || ((u64)num_pages << PAGE_SHIFT) != disksize)
2900+ return -EINVAL;
2901+
28722902 if (!zram_meta_alloc(zram, disksize))
28732903 return -ENOMEM;
28742904
drivers/block/zram/zram_drv.h+14−0
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515 #ifndef _ZRAM_DRV_H_
1616 #define _ZRAM_DRV_H_
1717
18+#include <asm/byteorder.h>
1819 #include <linux/rwsem.h>
1920 #include <linux/zsmalloc.h>
2021
@@ -57,6 +58,19 @@ enum zram_pageflags { · 36 unmodified lines
5758 __NR_ZRAM_PAGEFLAGS,
5859 };
5960
61+/*
62+ * The slot lock is a bit-wait lock on the whole __lock word, while
63+ * flags and ac_time alias that word as two u32s. The lock bit must
64+ * land in the slot that ZRAM_ENTRY_LOCK reserves in attr.flags; on
65+ * 64-bit big-endian the flags word maps to the upper half of __lock,
66+ * so the bit position has to be shifted up.
67+ */
68+#if defined(CONFIG_64BIT) && defined(__BIG_ENDIAN)
69+#define ZRAM_ENTRY_LOCK_BIT (ZRAM_ENTRY_LOCK + 32)
70+#else
71+#define ZRAM_ENTRY_LOCK_BIT ZRAM_ENTRY_LOCK
72+#endif
73+
6074 /*
6175 * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit
6276 * of flags) to save memory. There can be plenty of entries and standard
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c+2−2
@@ -377,9 +377,9 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str · 376 unmodified lines
377377 /* Workaround for Thunk bug creating PROT_NONE,MAP_PRIVATE mappings
378378 * for debugger access to invisible VRAM. Should have used MAP_SHARED
379379 * instead. Clearing VM_MAYWRITE prevents the mapping from ever
380− * becoming writable and makes is_cow_mapping(vm_flags) false.
380+ * becoming writable and makes vma_is_cow_mapping(vma) false.
381381 */
382− if (is_cow_mapping(vma->vm_flags) &&
382+ if (vma_is_cow_mapping(vma) &&
383383 !(vma->vm_flags & VM_ACCESS_FLAGS))
384384 vm_flags_clear(vma, VM_MAYWRITE);
385385
drivers/gpu/drm/drm_gem_shmem_helper.c+1−1
@@ -771,7 +771,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct · 770 unmodified lines
771771 return ret;
772772 }
773773
774− if (is_cow_mapping(vma->vm_flags))
774+ if (vma_is_cow_mapping(vma))
775775 return -EINVAL;
776776
777777 dma_resv_lock(shmem->base.resv, NULL);
drivers/gpu/drm/i915/gem/i915_gem_shmem.c+1−1
@@ -325,7 +325,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping) · 324 unmodified lines
325325 if (folio_mapped(folio))
326326 folio_redirty_for_writepage(&wbc, folio);
327327 else
328− error = shmem_writeout(folio, NULL, NULL);
328+ error = shmem_write_folio(folio);
329329 }
330330 }
331331
drivers/gpu/drm/panthor/panthor_gem.c+1−1
@@ -761,7 +761,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v · 760 unmodified lines
761761 return ret;
762762 }
763763
764− if (is_cow_mapping(vma->vm_flags))
764+ if (vma_is_cow_mapping(vma))
765765 return -EINVAL;
766766
767767 if (!refcount_inc_not_zero(&bo->cmap.mmap_count)) {
drivers/gpu/drm/ttm/ttm_backup.c+1−1
@@ -160,7 +160,7 @@ ttm_backup_backup_folio(struct file *backup, struct folio *folio, · 159 unmodified lines
160160 if (writeback && !folio_mapped(to_folio) &&
161161 folio_clear_dirty_for_io(to_folio)) {
162162 folio_set_reclaim(to_folio);
163− ret = shmem_writeout(to_folio, NULL, NULL);
163+ ret = shmem_write_folio(to_folio);
164164 if (!folio_test_writeback(to_folio))
165165 folio_clear_reclaim(to_folio);
166166 if (ret == AOP_WRITEPAGE_ACTIVATE)
drivers/gpu/drm/ttm/ttm_bo_vm.c+1−1
@@ -489,7 +489,7 @@ static const struct vm_operations_struct ttm_bo_vm_ops = { · 488 unmodified lines
489489 int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo)
490490 {
491491 /* Enforce no COW since would have really strange behavior with it. */
492− if (is_cow_mapping(vma->vm_flags))
492+ if (vma_is_cow_mapping(vma))
493493 return -EINVAL;
494494
495495 drm_gem_object_get(&bo->base);
drivers/gpu/drm/xe/xe_device.c+1−1
@@ -331,7 +331,7 @@ static int xe_pci_barrier_mmap(struct file *filp, · 330 unmodified lines
331331 if (vma->vm_end - vma->vm_start > SZ_4K)
332332 return -EINVAL;
333333
334− if (is_cow_mapping(vma->vm_flags))
334+ if (vma_is_cow_mapping(vma))
335335 return -EINVAL;
336336
337337 if (vma->vm_flags & (VM_READ | VM_EXEC))
fs/nfs/direct.c+0−20
@@ -145,26 +145,6 @@ static void nfs_direct_file_adjust_size_locked(struct inode *inode, · 144 unmodified lines
145145 }
146146 }
147147
148−/**
149− * nfs_swap_rw - NFS address space operation for swap I/O
150− * @iocb: target I/O control block
151− * @iter: I/O buffer
152− *
153− * Perform IO to the swap-file. This is much like direct IO.
154− */
155−int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter)
156−{
157− ssize_t ret;
158−
159− if (iov_iter_rw(iter) == READ)
160− ret = nfs_file_direct_read(iocb, iter, true);
161− else
162− ret = nfs_file_direct_write(iocb, iter, true);
163− if (ret < 0)
164− return ret;
165− return 0;
166−}
167−
168148 static void nfs_direct_release_pages(struct page **pages, unsigned int npages)
169149 {
170150 unsigned int i;
fs/nfs/file.c+38−6
@@ -29,9 +29,8 @@ · 28 unmodified lines
2929 #include <linux/pagemap.h>
3030 #include <linux/gfp.h>
3131 #include <linux/rmap.h>
32−#include <linux/swap.h>
3332 #include <linux/compaction.h>
34−
33+#include <linux/swap_ops.h>
3534 #include <linux/uaccess.h>
3635 #include <linux/filelock.h>
3736
@@ -575,6 +574,38 @@ static int nfs_launder_folio(struct folio *folio) · 537 unmodified lines
575574 return ret;
576575 }
577576
577+#ifdef CONFIG_SWAP
578+static void nfs_swap_submit_write(struct swap_io_ctx *ctx)
579+{
580+ struct swap_iocb *sio = ctx->sio;
581+ struct iov_iter iter;
582+ int ret;
583+
584+ swap_fs_prepare_rw(ctx, WRITE, &iter);
585+ ret = nfs_file_direct_write(&sio->iocb, &iter, true);
586+ if (ret != -EIOCBQUEUED)
587+ sio->iocb.ki_complete(&sio->iocb, ret);
588+}
589+
590+static void nfs_swap_submit_read(struct swap_io_ctx *ctx)
591+{
592+ struct swap_iocb *sio = ctx->sio;
593+ struct iov_iter iter;
594+ int ret;
595+
596+ swap_fs_prepare_rw(ctx, READ, &iter);
597+ ret = nfs_file_direct_read(&sio->iocb, &iter, true);
598+ if (ret != -EIOCBQUEUED)
599+ sio->iocb.ki_complete(&sio->iocb, ret);
600+}
601+
602+static const struct swap_ops nfs_swap_ops = {
603+ .flags = SWAP_OPS_F_REQUIRE_NOFS,
604+ .submit_write = nfs_swap_submit_write,
605+ .submit_read = nfs_swap_submit_read,
606+ .can_merge = swap_fs_can_merge,
607+};
608+
578609 static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file,
579610 sector_t *span)
580611 {
@@ -597,7 +628,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, · 16 unmodified lines
597628 ret = rpc_clnt_swap_activate(clnt);
598629 if (ret)
599630 return ret;
600− ret = add_swap_extent(sis, 0, sis->max, 0);
631+ ret = swap_fs_activate(sis, &nfs_swap_ops);
601632 if (ret < 0) {
602633 rpc_clnt_swap_deactivate(clnt);
603634 return ret;
@@ -607,8 +638,6 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, · 3 unmodified lines
607638
608639 if (cl->rpc_ops->enable_swap)
609640 cl->rpc_ops->enable_swap(inode);
610−
611− sis->flags |= SWP_FS_OPS;
612641 return ret;
613642 }
614643
@@ -622,6 +651,10 @@ static void nfs_swap_deactivate(struct file *file) · 7 unmodified lines
622651 if (cl->rpc_ops->disable_swap)
623652 cl->rpc_ops->disable_swap(file_inode(file));
624653 }
654+#else
655+#define nfs_swap_activate NULL
656+#define nfs_swap_deactivate NULL
657+#endif /* CONFIG_SWAP */
625658
626659 const struct address_space_operations nfs_file_aops = {
627660 .read_folio = nfs_read_folio,
@@ -638,7 +671,6 @@ const struct address_space_operations nfs_file_aops = { · 10 unmodified lines
638671 .error_remove_folio = generic_error_remove_folio,
639672 .swap_activate = nfs_swap_activate,
640673 .swap_deactivate = nfs_swap_deactivate,
641− .swap_rw = nfs_swap_rw,
642674 };
643675
644676 /*
fs/proc/task_mmu.c+1−1
@@ -1693,7 +1693,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, · 1692 unmodified lines
16931693
16941694 if (!pte_write(pte))
16951695 return false;
1696− if (!is_cow_mapping(vma->vm_flags))
1696+ if (!vma_is_cow_mapping(vma))
16971697 return false;
16981698 if (likely(!mm_flags_test(MMF_HAS_PINNED, vma->vm_mm)))
16991699 return false;
fs/ramfs/file-nommu.c+3−0
@@ -69,6 +69,9 @@ int ramfs_nommu_expand_for_mapping(struct inode *inode, size_t newsize) · 68 unmodified lines
6969 gfp_t gfp = mapping_gfp_mask(inode->i_mapping);
7070
7171 /* make various checks */
72+ if (!newsize)
73+ return 0;
74+
7275 order = get_order(newsize);
7376 if (unlikely(order > MAX_PAGE_ORDER))
7477 return -EFBIG;
fs/smb/client/file.c+39−26
@@ -20,7 +20,7 @@ · 19 unmodified lines
2020 #include <linux/delay.h>
2121 #include <linux/mount.h>
2222 #include <linux/slab.h>
23−#include <linux/swap.h>
23+#include <linux/swap_ops.h>
2424 #include <linux/mm.h>
2525 #include <asm/div64.h>
2626 #include "cifsfs.h"
@@ -3406,6 +3406,38 @@ out: · 3379 unmodified lines
34063406 cifs_done_oplock_break(cinode);
34073407 }
34083408
3409+#ifdef CONFIG_SWAP
3410+static void cifs_swap_submit_write(struct swap_io_ctx *ctx)
3411+{
3412+ struct swap_iocb *sio = ctx->sio;
3413+ struct iov_iter iter;
3414+ int ret;
3415+
3416+ swap_fs_prepare_rw(ctx, WRITE, &iter);
3417+ ret = netfs_unbuffered_write_iter_locked(&sio->iocb, &iter, NULL);
3418+ if (ret != -EIOCBQUEUED)
3419+ sio->iocb.ki_complete(&sio->iocb, ret);
3420+}
3421+
3422+static void cifs_swap_submit_read(struct swap_io_ctx *ctx)
3423+{
3424+ struct swap_iocb *sio = ctx->sio;
3425+ struct iov_iter iter;
3426+ int ret;
3427+
3428+ swap_fs_prepare_rw(ctx, READ, &iter);
3429+ ret = netfs_unbuffered_read_iter_locked(&sio->iocb, &iter);
3430+ if (ret != -EIOCBQUEUED)
3431+ sio->iocb.ki_complete(&sio->iocb, ret);
3432+}
3433+
3434+static const struct swap_ops cifs_swap_ops = {
3435+ .flags = SWAP_OPS_F_REQUIRE_NOFS,
3436+ .submit_write = cifs_swap_submit_write,
3437+ .submit_read = cifs_swap_submit_read,
3438+ .can_merge = swap_fs_can_merge,
3439+};
3440+
34093441 static int cifs_swap_activate(struct swap_info_struct *sis,
34103442 struct file *swap_file, sector_t *span)
34113443 {
@@ -3416,7 +3448,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, · 4 unmodified lines
34163448
34173449 cifs_dbg(FYI, "swap activate\n");
34183450
3419− if (!swap_file->f_mapping->a_ops->swap_rw)
3451+ if (swap_file->f_mapping->a_ops != &cifs_addr_ops)
34203452 /* Cannot support swap */
34213453 return -EINVAL;
34223454
@@ -3447,9 +3479,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, · 24 unmodified lines
34473479 * but we could add call to grab a byte range lock to prevent others
34483480 * from reading or writing the file
34493481 */
3450−
3451− sis->flags |= SWP_FS_OPS;
3452− return add_swap_extent(sis, 0, sis->max, 0);
3482+ return swap_fs_activate(sis, &cifs_swap_ops);
34533483 }
34543484
34553485 static void cifs_swap_deactivate(struct file *file)
@@ -3465,26 +3495,10 @@ static void cifs_swap_deactivate(struct file *file) · 9 unmodified lines
34653495
34663496 /* do we need to unpin (or unlock) the file */
34673497 }
3468−
3469−/**
3470− * cifs_swap_rw - SMB3 address space operation for swap I/O
3471− * @iocb: target I/O control block
3472− * @iter: I/O buffer
3473− *
3474− * Perform IO to the swap-file. This is much like direct IO.
3475− */
3476−static int cifs_swap_rw(struct kiocb *iocb, struct iov_iter *iter)
3477−{
3478− ssize_t ret;
3479−
3480− if (iov_iter_rw(iter) == READ)
3481− ret = netfs_unbuffered_read_iter_locked(iocb, iter);
3482− else
3483− ret = netfs_unbuffered_write_iter_locked(iocb, iter, NULL);
3484− if (ret < 0)
3485− return ret;
3486− return 0;
3487−}
3498+#else
3499+#define cifs_swap_activate NULL
3500+#define cifs_swap_deactivate NULL
3501+#endif /* CONFIG_SWAP */
34883502
34893503 const struct address_space_operations cifs_addr_ops = {
34903504 .read_folio = netfs_read_folio,
@@ -3501,7 +3515,6 @@ const struct address_space_operations cifs_addr_ops = { · 10 unmodified lines
35013515 */
35023516 .swap_activate = cifs_swap_activate,
35033517 .swap_deactivate = cifs_swap_deactivate,
3504− .swap_rw = cifs_swap_rw,
35053518 };
35063519
35073520 /*
include/linux/codetag.h+2−0
@@ -77,6 +77,8 @@ struct codetag_iterator { · 76 unmodified lines
7777 void codetag_lock_module_list(struct codetag_type *cttype);
7878 bool codetag_trylock_module_list(struct codetag_type *cttype);
7979 void codetag_unlock_module_list(struct codetag_type *cttype);
80+unsigned long codetag_get_content_id(struct codetag_type *cttype);
81+unsigned int codetag_get_count(struct codetag_type *cttype);
8082 struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype);
8183 struct codetag *codetag_next_ct(struct codetag_iterator *iter);
8284
include/linux/execmem.h+1−1
@@ -89,7 +89,7 @@ static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; } · 88 unmodified lines
8989 * @end: address space end (inclusive)
9090 * @fallback_start: start of the secondary address space range for fallback
9191 * allocations on architectures that require it
92− * @fallback_end: start of the secondary address space (inclusive)
92+ * @fallback_end: end of the secondary address space (inclusive)
9393 * @pgprot: permissions for memory in this address space
9494 * @alignment: alignment required for text allocations
9595 * @flags: options for memory allocations for this range
include/linux/fs.h+0−1
@@ -438,7 +438,6 @@ struct address_space_operations { · 437 unmodified lines
438438 int (*swap_activate)(struct swap_info_struct *sis, struct file *file,
439439 sector_t *span);
440440 void (*swap_deactivate)(struct file *file);
441− int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter);
442441 };
443442
444443 extern const struct address_space_operations empty_aops;
include/linux/lockdep.h+3−0
@@ -273,6 +273,9 @@ extern struct pin_cookie lock_pin_lock(struct lockdep_map *lock); · 272 unmodified lines
273273 extern void lock_repin_lock(struct lockdep_map *lock, struct pin_cookie);
274274 extern void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie);
275275
276+extern u32 lock_sequence(struct lockdep_map *lock);
277+#define lockdep_sequence(lock) lock_sequence(&(lock)->dep_map)
278+
276279 #define lockdep_depth(tsk) (debug_locks ? (tsk)->lockdep_depth : 0)
277280
278281 #define lockdep_assert(cond) \
include/linux/lockdep_types.h+2−1
@@ -253,7 +253,8 @@ struct held_lock { · 252 unmodified lines
253253 unsigned int hardirqs_off:1;
254254 unsigned int sync:1;
255255 unsigned int references:11; /* 32 bits */
256− unsigned int pin_count;
256+ unsigned int pin_count:24;
257+ unsigned int seq_count:8;
257258 };
258259
259260 #else /* !CONFIG_LOCKDEP */
include/linux/maple_tree.h+9−7
@@ -11,7 +11,6 @@ · 10 unmodified lines
1111 #include <linux/kernel.h>
1212 #include <linux/rcupdate.h>
1313 #include <linux/spinlock.h>
14−/* #define CONFIG_MAPLE_RCU_DISABLED */
1514
1615 /*
1716 * Allocated nodes are mutable until they have been inserted into the tree,
@@ -29,13 +28,11 @@ · 11 unmodified lines
2928 #define MAPLE_NODE_SLOTS 31 /* 256 bytes including ->parent */
3029 #define MAPLE_RANGE64_SLOTS 16 /* 256 bytes */
3130 #define MAPLE_ARANGE64_SLOTS 10 /* 240 bytes */
32−#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 1)
3331 #else
3432 /* 32bit sizes */
3533 #define MAPLE_NODE_SLOTS 63 /* 256 bytes including ->parent */
3634 #define MAPLE_RANGE64_SLOTS 32 /* 256 bytes */
3735 #define MAPLE_ARANGE64_SLOTS 21 /* 240 bytes */
38−#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 2)
3936 #endif /* defined(CONFIG_64BIT) || defined(BUILD_VDSO32_64) */
4037
4138 #define MAPLE_NODE_MASK 255UL
@@ -485,6 +482,12 @@ struct ma_state { · 443 unmodified lines
485482 unsigned char mas_flags;
486483 unsigned char end; /* The end of the node */
487484 enum store_type store_type; /* The type of store needed for this operation */
485+#ifdef CONFIG_LOCKDEP
486+ u32 ld_seq;
487+#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD
488+ unsigned long rcu_gp;
489+#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */
490+#endif /* CONFIG_LOCKDEP */
488491 };
489492
490493 struct ma_wr_state {
@@ -567,12 +570,14 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, · 76 unmodified lines
567570 unsigned long *next, gfp_t gfp);
568571
569572 bool mas_nomem(struct ma_state *mas, gfp_t gfp);
573+bool mas_nomem_nofail(struct ma_state *mas, unsigned long index,
574+ unsigned long last);
570575 void mas_pause(struct ma_state *mas);
571576 void maple_tree_init(void);
572577 void mas_destroy(struct ma_state *mas);
573578
574579 void *mas_prev(struct ma_state *mas, unsigned long min);
575−void *mas_prev_range(struct ma_state *mas, unsigned long max);
580+void *mas_prev_range(struct ma_state *mas, unsigned long min);
576581 void *mas_next(struct ma_state *mas, unsigned long max);
577582 void *mas_next_range(struct ma_state *mas, unsigned long max);
578583
@@ -864,9 +869,6 @@ static inline void mt_init(struct maple_tree *mt) · 285 unmodified lines
864869
865870 static inline bool mt_in_rcu(struct maple_tree *mt)
866871 {
867−#ifdef CONFIG_MAPLE_RCU_DISABLED
868− return false;
869−#endif
870872 return mt->ma_flags & MT_FLAGS_USE_RCU;
871873 }
872874
include/linux/memcontrol.h+17−4
@@ -239,8 +239,6 @@ struct mem_cgroup { · 238 unmodified lines
239239 */
240240 bool oom_group;
241241
242− int swappiness;
243−
244242 /* memory.events and memory.events.local */
245243 struct cgroup_file events_file;
246244 struct cgroup_file events_local_file;
@@ -270,10 +268,15 @@ struct mem_cgroup { · 23 unmodified lines
270268 #endif
271269 int kmemcg_id;
272270
273− struct memcg_vmstats_percpu __percpu *vmstats_percpu;
274−
275271 #ifdef CONFIG_CGROUP_WRITEBACK
276272 struct list_head cgwb_list;
273+#endif
274+
275+ /* Keep the hot per-CPU stats pointer away from memory event counters. */
276+ struct memcg_vmstats_percpu __percpu *vmstats_percpu
277+ ____cacheline_aligned_in_smp;
278+
279+#ifdef CONFIG_CGROUP_WRITEBACK
277280 struct wb_domain cgwb_domain;
278281 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
279282 #endif
@@ -318,6 +321,8 @@ struct mem_cgroup { · 38 unmodified lines
318321 /* List of events which userspace want to receive */
319322 struct list_head event_list;
320323 spinlock_t event_list_lock;
324+
325+ int swappiness;
321326 #endif /* CONFIG_MEMCG_V1 */
322327
323328 struct mem_cgroup_per_node *nodeinfo[];
@@ -947,6 +952,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item); · 623 unmodified lines
947952 bool memcg_stat_item_valid(int idx);
948953 bool memcg_vm_event_item_valid(enum vm_event_item idx);
949954 unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx);
955+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
956+ enum node_stat_item idx);
950957 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
951958 enum node_stat_item idx);
952959
@@ -1399,6 +1406,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, · 446 unmodified lines
13991406 return node_page_state(lruvec_pgdat(lruvec), idx);
14001407 }
14011408
1409+static inline unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
1410+ enum node_stat_item idx)
1411+{
1412+ return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
1413+}
1414+
14021415 static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,
14031416 enum node_stat_item idx)
14041417 {
include/linux/mm.h+125−7
@@ -1556,7 +1556,7 @@ static inline void vma_desc_set_anonymous(struct vm_area_desc *desc) · 1555 unmodified lines
15561556 desc->vm_ops = NULL;
15571557 }
15581558
1559−static inline bool vma_is_anonymous(struct vm_area_struct *vma)
1559+static inline bool vma_is_anonymous(const struct vm_area_struct *vma)
15601560 {
15611561 return !vma->vm_ops;
15621562 }
@@ -2271,17 +2271,76 @@ void unpin_user_pages(struct page **pages, unsigned long npages); · 708 unmodified lines
22712271 void unpin_user_folio(struct folio *folio, unsigned long npages);
22722272 void unpin_folios(struct folio **folios, unsigned long nfolios);
22732273
2274−static inline bool is_cow_mapping(vm_flags_t flags)
2274+/**
2275+ * vma_flags_is_cow_mapping() - Do these VMA flags imply a CoW mapping?
2276+ * @flags: The VMA flags to check.
2277+ *
2278+ * Mappings which could be CoW'd (subject to Copy-On-Write faults) are
2279+ * described as CoW mappings.
2280+ *
2281+ * All mappings backed by anonymous folios (all anonymous mappings and most
2282+ * MAP_PRIVATE-file backed ranges) are CoW mappings.
2283+ *
2284+ * All other mappings (including all MAP_SHARED mappings) are non-CoW.
2285+ *
2286+ * The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT.
2287+ *
2288+ * VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both
2289+ * future mprotect() calls which can render a read-only mapping writable, and
2290+ * GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping.
2291+ *
2292+ * - No anonymous mapping can ever clear VMA_MAYWRITE_BIT.
2293+ *
2294+ * - Writes to anonymous mappings do not immediately result in CoW faults but
2295+ * may do so after the process is forked or if a read is followed by a
2296+ * write.
2297+ *
2298+ * - Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may
2299+ * do so again after fork.
2300+ *
2301+ * - MAP_SHARED mappings of a file opened read-only are transformed into
2302+ * VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain
2303+ * non-CoW.
2304+ *
2305+ * - Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot
2306+ * mark themselves anonymous. Having cleared this flag it is not valid for
2307+ * them to leave the VMA_WRITE_BIT flag set.
2308+ *
2309+ * As a consequence, the anonymous reverse mapping only tracks CoW mappings.
2310+ *
2311+ * Returns: true if the flags indicate a CoW mapping, otherwise false.
2312+ */
2313+static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags)
22752314 {
2276− return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
2315+ return vma_flags_test(flags, VMA_MAYWRITE_BIT) &&
2316+ !vma_flags_test(flags, VMA_SHARED_BIT);
22772317 }
22782318
2279−static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc)
2319+/**
2320+ * vma_is_cow_mapping() - Is this VMA a CoW mapping?
2321+ * @vma: The VMA to check.
2322+ *
2323+ * See vma_flags_is_cow_mapping() for details.
2324+ *
2325+ * Returns: true if the VMA is a CoW mapping, otherwise false.
2326+ */
2327+static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma)
22802328 {
2281− const vma_flags_t *flags = &desc->vma_flags;
2329+ return vma_flags_is_cow_mapping(&vma->flags);
2330+}
22822331
2283− return vma_flags_test(flags, VMA_MAYWRITE_BIT) &&
2284− !vma_flags_test(flags, VMA_SHARED_BIT);
2332+/**
2333+ * vma_desc_is_cow_mapping() - Is this VMA descriptor a CoW mapping?
2334+ * @desc: The VMA descriptor to check.
2335+ *
2336+ * See vma_flags_is_cow_mapping() for details.
2337+ *
2338+ * Returns: true if the VMA descriptor describes a CoW mapping, otherwise
2339+ * false.
2340+ */
2341+static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc)
2342+{
2343+ return vma_flags_is_cow_mapping(&desc->vma_flags);
22852344 }
22862345
22872346 #ifndef CONFIG_MMU
@@ -4393,6 +4452,65 @@ static inline pgoff_t vma_last_pgoff(const struct vm_area_struct *vma) · 2105 unmodified lines
43934452 return vma_end_pgoff(vma) - 1;
43944453 }
43954454
4455+/**
4456+ * vma_start_anon_pgoff() - Get the anonymous page offset of the start of @vma
4457+ * @vma: The VMA whose anonymous page offset is required.
4458+ *
4459+ * If unfaulted, then this is vma->vm_start >> PAGE_SHIFT, if faulted then the
4460+ * anonymous page offset at the time of first fault.
4461+ *
4462+ * If the VMA is anonymous, this returns the same value as vma_start_pgoff().
4463+ *
4464+ * This value is used for tracking MAP_PRIVATE file-backed mappings by their
4465+ * anonymous page offset.
4466+ *
4467+ * Returns: The anonymous page offset of the start of @vma.
4468+ */
4469+static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma)
4470+{
4471+ pgoff_t pgoff = 0;
4472+
4473+#ifdef CONFIG_64BIT
4474+ pgoff += vma->__vm_anon_pgoff_hi;
4475+ pgoff <<= 32;
4476+#endif
4477+ pgoff += vma->__vm_anon_pgoff_lo;
4478+ return pgoff;
4479+}
4480+
4481+/**
4482+ * vma_end_anon_pgoff() - Get the anonymous page offset of the exclusive end of
4483+ * @vma.
4484+ * @vma: The VMA whose end anonymous page offset is required.
4485+ *
4486+ * This returns the anonymous exclusive end page offset of @vma, which is useful
4487+ * for expressing page offset ranges.
4488+ *
4489+ * See the description of vma_start_anon_pgoff() for a description of VMA
4490+ * anonymous page offsets.
4491+ *
4492+ * Returns: The exclusive end anonymous page offset of @vma.
4493+ */
4494+static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma)
4495+{
4496+ return vma_start_anon_pgoff(vma) + vma_pages(vma);
4497+}
4498+
4499+/**
4500+ * vma_last_anon_pgoff() - Get the anonymous page offset of the last page in
4501+ * @vma.
4502+ * @vma: The VMA whose last anonymous page offset is required.
4503+ *
4504+ * See the description of vma_start_anon_pgoff() for a description of VMA
4505+ * anonymous page offsets.
4506+ *
4507+ * Returns: The last anonymous page offset of @vma.
4508+ */
4509+static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma)
4510+{
4511+ return vma_end_anon_pgoff(vma) - 1;
4512+}
4513+
43964514 static inline unsigned long vma_desc_size(const struct vm_area_desc *desc)
43974515 {
43984516 return desc->end - desc->start;
include/linux/mm_types.h+26−14
@@ -967,6 +967,11 @@ struct vm_area_struct { · 966 unmodified lines
967967 */
968968 unsigned int vm_lock_seq;
969969 #endif
970+ /*
971+ * Low 32-bits of anonymous page offset.
972+ * See vma_start_anon_pgoff() comment for details.
973+ */
974+ unsigned int __vm_anon_pgoff_lo;
970975 /*
971976 * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma
972977 * list, after a COW of one of the file pages. A MAP_SHARED vma
@@ -1041,6 +1046,13 @@ struct vm_area_struct { · 68 unmodified lines
10411046 #ifdef CONFIG_DEBUG_LOCK_ALLOC
10421047 struct lockdep_map vmlock_dep_map;
10431048 #endif
1049+#endif
1050+#ifdef CONFIG_64BIT
1051+ /*
1052+ * High 32-bits of anonymous page offset.
1053+ * See vma_start_anon_pgoff() comment for details.
1054+ */
1055+ unsigned int __vm_anon_pgoff_hi;
10441056 #endif
10451057 /*
10461058 * For areas with an address space and backing store,
@@ -1706,20 +1718,20 @@ enum vm_fault_reason { · 659 unmodified lines
17061718 VM_FAULT_SIGSEGV | VM_FAULT_HWPOISON | \
17071719 VM_FAULT_HWPOISON_LARGE | VM_FAULT_FALLBACK)
17081720
1709−#define VM_FAULT_RESULT_TRACE \
1710− { VM_FAULT_OOM, "OOM" }, \
1711− { VM_FAULT_SIGBUS, "SIGBUS" }, \
1712− { VM_FAULT_MAJOR, "MAJOR" }, \
1713− { VM_FAULT_HWPOISON, "HWPOISON" }, \
1714− { VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \
1715− { VM_FAULT_SIGSEGV, "SIGSEGV" }, \
1716− { VM_FAULT_NOPAGE, "NOPAGE" }, \
1717− { VM_FAULT_LOCKED, "LOCKED" }, \
1718− { VM_FAULT_RETRY, "RETRY" }, \
1719− { VM_FAULT_FALLBACK, "FALLBACK" }, \
1720− { VM_FAULT_DONE_COW, "DONE_COW" }, \
1721− { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \
1722− { VM_FAULT_COMPLETED, "COMPLETED" }
1721+#define VM_FAULT_RESULT_TRACE \
1722+ { (__force u32)VM_FAULT_OOM, "OOM" }, \
1723+ { (__force u32)VM_FAULT_SIGBUS, "SIGBUS" }, \
1724+ { (__force u32)VM_FAULT_MAJOR, "MAJOR" }, \
1725+ { (__force u32)VM_FAULT_HWPOISON, "HWPOISON" }, \
1726+ { (__force u32)VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \
1727+ { (__force u32)VM_FAULT_SIGSEGV, "SIGSEGV" }, \
1728+ { (__force u32)VM_FAULT_NOPAGE, "NOPAGE" }, \
1729+ { (__force u32)VM_FAULT_LOCKED, "LOCKED" }, \
1730+ { (__force u32)VM_FAULT_RETRY, "RETRY" }, \
1731+ { (__force u32)VM_FAULT_FALLBACK, "FALLBACK" }, \
1732+ { (__force u32)VM_FAULT_DONE_COW, "DONE_COW" }, \
1733+ { (__force u32)VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \
1734+ { (__force u32)VM_FAULT_COMPLETED, "COMPLETED" }
17231735
17241736 struct vm_special_mapping {
17251737 const char *name; /* The name, e.g. "[vdso]". */
include/linux/mmzone.h+13−3
@@ -323,6 +323,8 @@ enum node_stat_item { · 322 unmodified lines
323323 PGSCAN_PROACTIVE,
324324 PGSCAN_ANON,
325325 PGSCAN_FILE,
326+ PGROTATE_ANON,
327+ PGROTATE_FILE,
326328 PGREFILL,
327329 #ifdef CONFIG_HUGETLB_PAGE
328330 NR_HUGETLB,
@@ -755,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, · 426 unmodified lines
755757
756758 #endif /* CONFIG_LRU_GEN */
757759
760+struct lru_cost {
761+ unsigned long count;
762+ unsigned long last_rotated;
763+ unsigned long last_io;
764+};
765+
758766 struct lruvec {
759767 struct list_head lists[NR_LRU_LISTS];
760768 /* per lruvec lru_lock for memcg */
@@ -763,9 +771,12 @@ struct lruvec { · 2 unmodified lines
763771 * These track the cost of reclaiming one LRU - file or anon -
764772 * over the other. As the observed cost of reclaiming one LRU
765773 * increases, the reclaim scan balance tips toward the other.
774+ * Updated and decayed at prepare_scan_control() time; cost_lock
775+ * serialises that update.
766776 */
767− unsigned long anon_cost;
768− unsigned long file_cost;
777+ struct lru_cost cost[ANON_AND_FILE];
778+ /* Protects cost[]. */
779+ spinlock_t cost_lock;
769780 /* Non-resident age, driven by LRU movement */
770781 atomic_long_t nonresident_age;
771782 /* Refaults at the time of last reclaim cycle */
@@ -2058,7 +2069,6 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) · 1286 unmodified lines
20582069 #endif
20592070 return &mem_section[root][nr & SECTION_ROOT_MASK];
20602071 }
2061−extern size_t mem_section_usage_size(void);
20622072
20632073 /*
20642074 * We use the lower bits of the mem_map pointer to store a little bit of
include/linux/nfs_fs.h+0−1
@@ -552,7 +552,6 @@ static inline const struct cred *nfs_file_cred(struct file *file) · 551 unmodified lines
552552 /*
553553 * linux/fs/nfs/direct.c
554554 */
555−int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter);
556555 ssize_t nfs_file_direct_read(struct kiocb *iocb,
557556 struct iov_iter *iter, bool swap);
558557 ssize_t nfs_file_direct_write(struct kiocb *iocb,
include/linux/page_ext.h+0−5
@@ -55,7 +55,6 @@ struct page_ext { · 54 unmodified lines
5555
5656 extern bool early_page_ext;
5757 extern unsigned long page_ext_size;
58−extern void pgdat_page_ext_init(struct pglist_data *pgdat);
5958
6059 static inline bool early_page_ext_enabled(void)
6160 {
@@ -202,10 +201,6 @@ static inline bool early_page_ext_enabled(void) · 140 unmodified lines
202201 return false;
203202 }
204203
205−static inline void pgdat_page_ext_init(struct pglist_data *pgdat)
206−{
207−}
208−
209204 static inline void page_ext_init(void)
210205 {
211206 }
include/linux/pagemap.h+37−3
@@ -1094,10 +1094,44 @@ static inline pgoff_t linear_page_delta(const struct vm_area_struct *vma, · 1093 unmodified lines
10941094 static inline pgoff_t linear_page_index(const struct vm_area_struct *vma,
10951095 const unsigned long address)
10961096 {
1097− pgoff_t pgoff;
1097+ return linear_page_delta(vma, address) + vma_start_pgoff(vma);
1098+}
1099+
1100+static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
1101+ const unsigned long address)
1102+{
1103+ return linear_page_delta(vma, address) + vma_start_anon_pgoff(vma);
1104+}
1105+
1106+/**
1107+ * linear_anon_page_index() - Determine the absolute anonymous page offset of
1108+ * @address within @vma.
1109+ * @vma: An anonymous or MAP_PRIVATE file-backed VMA in which @address resides.
1110+ * @address: The address whose absolute page offset is required.
1111+ *
1112+ * This returns the anonymous page offset of @address, which is the page offset
1113+ * the address possessed at the time the VMA was first faulted.
1114+ *
1115+ * For anonymous mappings, this returns the same value as linear_page_index().
1116+ *
1117+ * For MAP_PRIVATE file-backed mappings, this returns the anonymous page offset
1118+ * of @address, which is the page offset the address possessed at the time the
1119+ * VMA was first faulted.
1120+ *
1121+ * It is not valid to call this function for shared file-backed mappings.
1122+ *
1123+ * Returns: The absolute anonymous page offset of @address within @vma.
1124+ */
1125+static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
1126+ const unsigned long address)
1127+{
1128+ const pgoff_t pgoff = __linear_anon_page_index(vma, address);
1129+
1130+ VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
1131+ /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */
1132+ if (vma_is_anonymous(vma) && !vma->vm_file)
1133+ VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
10981134
1099− pgoff = linear_page_delta(vma, address);
1100− pgoff += vma_start_pgoff(vma);
11011135 return pgoff;
11021136 }
11031137
include/linux/percpu-defs.h+1−6
@@ -65,13 +65,8 @@ · 64 unmodified lines
6565 *
6666 * Archs which need weak percpu definitions should set
6767 * CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU when necessary.
68− *
69− * To ensure that the generic code observes the above two
70− * restrictions, if CONFIG_DEBUG_FORCE_WEAK_PER_CPU is set weak
71− * definition is used for all cases.
7268 */
73−#if (defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE)) || \
74− defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
69+#if defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE)
7570 /*
7671 * __pcpu_scope_* dummy variable is used to enforce scope. It
7772 * receives the static modifier when it's used in front of
include/linux/rmap.h+6−5
@@ -843,7 +843,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, · 842 unmodified lines
843843 * Called from mm/vmscan.c to handle paging out
844844 */
845845 int folio_referenced(struct folio *, int is_locked,
846− struct mem_cgroup *memcg, vm_flags_t *vm_flags);
846+ struct mem_cgroup *memcg, vma_flags_t *vma_flags);
847847
848848 void try_to_migrate(struct folio *folio, enum ttu_flags flags);
849849 void try_to_unmap(struct folio *, enum ttu_flags flags);
@@ -864,13 +864,14 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, · 14 unmodified lines
864864 struct page_vma_mapped_walk {
865865 unsigned long pfn;
866866 unsigned long nr_pages;
867− pgoff_t pgoff;
867+ pgoff_t pgoff; /* Only meaningful if nr_pages > 1 and not a KSM walk */
868868 struct vm_area_struct *vma;
869869 unsigned long address;
870870 pmd_t *pmd;
871871 pte_t *pte;
872872 spinlock_t *ptl;
873873 unsigned int flags;
874+ bool pgoff_is_anon : 1;
874875 };
875876
876877 #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \
@@ -881,6 +882,7 @@ struct page_vma_mapped_walk { · 4 unmodified lines
881882 .vma = _vma, \
882883 .address = _address, \
883884 .flags = _flags, \
885+ .pgoff_is_anon = folio_test_anon(_folio), \
884886 }
885887
886888 static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw)
@@ -975,10 +977,9 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, · 88 unmodified lines
975977 #define anon_vma_prepare(vma) (0)
976978
977979 static inline int folio_referenced(struct folio *folio, int is_locked,
978− struct mem_cgroup *memcg,
979− vm_flags_t *vm_flags)
980+ struct mem_cgroup *memcg, vma_flags_t *vma_flags)
980981 {
981− *vm_flags = 0;
982+ vma_flags_clear_all(vma_flags);
982983 return 0;
983984 }
984985
include/linux/sched.h+1−0
@@ -1298,6 +1298,7 @@ struct task_struct { · 1297 unmodified lines
12981298 u64 curr_chain_key;
12991299 int lockdep_depth;
13001300 unsigned int lockdep_recursion;
1301+ unsigned int lockdep_seq;
13011302 struct held_lock held_locks[MAX_LOCK_DEPTH];
13021303 #endif
13031304
include/linux/shmem_fs.h+1−4
@@ -12,8 +12,6 @@ · 11 unmodified lines
1212 #include <linux/userfaultfd_k.h>
1313 #include <linux/bits.h>
1414
15−struct swap_iocb;
16−
1715 /* inode in-kernel data */
1816
1917 #ifdef CONFIG_TMPFS_QUOTA
@@ -123,8 +121,7 @@ static inline bool shmem_mapping(const struct address_space *mapping) · 103 unmodified lines
123121 void shmem_unlock_mapping(struct address_space *mapping);
124122 struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,
125123 pgoff_t index, gfp_t gfp_mask);
126−int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
127− struct list_head *folio_list);
124+int shmem_write_folio(struct folio *folio);
128125 void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end);
129126 int shmem_unuse(unsigned int type);
130127
include/linux/swap.h+9−21
@@ -202,7 +202,6 @@ enum { · 201 unmodified lines
202202 SWP_SOLIDSTATE = (1 << 4), /* blkdev seeks are cheap */
203203 SWP_BLKDEV = (1 << 6), /* its a block device */
204204 SWP_ACTIVATED = (1 << 7), /* set after swap_activate success */
205− SWP_FS_OPS = (1 << 8), /* swapfile operations go through fs */
206205 SWP_AREA_DISCARD = (1 << 9), /* single-time swap area discards */
207206 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
208207 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
@@ -276,6 +275,7 @@ struct swap_info_struct { · 67 unmodified lines
276275 struct work_struct reclaim_work; /* reclaim worker */
277276 struct list_head discard_clusters; /* discard clusters list */
278277 struct plist_node avail_list; /* entry in swap_avail_head */
278+ const struct swap_ops *ops;
279279 };
280280
281281 static inline swp_entry_t page_swap_entry(struct page *page)
@@ -298,6 +298,14 @@ void folio_add_lru(struct folio *folio); · 16 unmodified lines
298298 void folio_mark_accessed(struct folio *folio);
299299 void lru_add_drain_all(void);
300300
301+enum lru_cache_drained {
302+ LRU_CACHE_NOT_DRAINED,
303+ LRU_CACHE_DRAINED,
304+ LRU_CACHE_DRAINED_ALL,
305+};
306+void lru_cache_drain_for_folio(const struct folio *folio,
307+ unsigned int extra_refs, enum lru_cache_drained *drained);
308+
301309 /* linux/mm/folio-compat.c */
302310 void mark_page_accessed(struct page *page);
303311
@@ -309,7 +317,6 @@ static inline bool lru_cache_disabled(void) · 5 unmodified lines
309317 }
310318
311319 extern unsigned long shrink_all_memory(unsigned long nr_pages);
312−extern int vm_swappiness;
313320 long remove_mapping(struct address_space *mapping, struct folio *folio);
314321
315322 #if defined(CONFIG_SYSFS) && defined(CONFIG_NUMA)
@@ -334,7 +341,6 @@ extern void __meminit kswapd_run(int nid); · 18 unmodified lines
334341 extern void __meminit kswapd_stop(int nid);
335342
336343 #ifdef CONFIG_SWAP
337−
338344 int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page,
339345 unsigned long nr_pages, sector_t start_block);
340346 int generic_swapfile_activate(struct swap_info_struct *, struct file *,
@@ -468,25 +474,7 @@ static inline int add_swap_extent(struct swap_info_struct *sis, · 127 unmodified lines
468474 }
469475 #endif /* CONFIG_SWAP */
470476 #ifdef CONFIG_MEMCG
471−static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
472−{
473− /* Cgroup2 doesn't have per-cgroup swappiness */
474− if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
475− return READ_ONCE(vm_swappiness);
476−
477− /* root ? */
478− if (mem_cgroup_disabled() || mem_cgroup_is_root(memcg))
479− return READ_ONCE(vm_swappiness);
480−
481− return READ_ONCE(memcg->swappiness);
482−}
483−
484477 void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int nid);
485−#else
486−static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
487−{
488− return READ_ONCE(vm_swappiness);
489−}
490478 #endif
491479
492480 #if defined(CONFIG_SWAP) && defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
include/linux/swap_ops.h+44−0
@@ -0,0 +1,44 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef _MM_SWAP_OPS_H
3+#define _MM_SWAP_OPS_H
4+
5+#include <linux/swap.h> /* for SWAP_CLUSTER_MAX */
6+
7+struct swap_iocb {
8+ union {
9+ struct kiocb iocb;
10+ struct bio bio;
11+ };
12+ struct bio_vec bvecs[SWAP_CLUSTER_MAX];
13+ int nr_bvecs;
14+ int len;
15+};
16+
17+struct swap_io_ctx {
18+ struct swap_iocb *sio;
19+ struct swap_info_struct *sis;
20+};
21+
22+/*
23+ * SWAP_OPS_F_REQUIRE_NOFS:
24+ * When set, all reclaim operations must operated as GFS_NOFS and not
25+ * just GFP_NOIO, as GFP_NOIO allocations could recourse into the
26+ * file system backing this swap file.
27+ */
28+#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0)
29+
30+struct swap_ops {
31+ unsigned int flags;
32+
33+ bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
34+ size_t prev_folio_size, int rw);
35+ void (*submit_write)(struct swap_io_ctx *ctx);
36+ void (*submit_read)(struct swap_io_ctx *ctx);
37+};
38+
39+void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter);
40+bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
41+ size_t prev_folio_size, int rw);
42+int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops);
43+
44+#endif /* _MM_SWAP_OPS_H */
include/linux/swapops.h+4−3
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 #include <linux/radix-tree.h>
66 #include <linux/bug.h>
77 #include <linux/mm_types.h>
8+#include <linux/swap.h>
89
910 #ifdef CONFIG_MMU
1011
@@ -325,8 +326,8 @@ struct page_vma_mapped_walk; · 314 unmodified lines
325326 extern int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
326327 struct page *page);
327328
328−extern void remove_migration_pmd(struct page_vma_mapped_walk *pvmw,
329− struct page *new);
329+void remove_migration_pmd(struct page_vma_mapped_walk *pvmw,
330+ struct folio *folio);
330331
331332 extern void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd);
332333
@@ -346,7 +347,7 @@ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · 13 unmodified lines
346347 }
347348
348349 static inline void remove_migration_pmd(struct page_vma_mapped_walk *pvmw,
349− struct page *new)
350+ struct folio *folio)
350351 {
351352 BUILD_BUG();
352353 }
include/linux/vm_event_item.h+4−0
@@ -175,6 +175,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, · 174 unmodified lines
175175 KSTACK_REST,
176176 #endif
177177 #endif /* CONFIG_DEBUG_STACK_USAGE */
178+#ifdef CONFIG_SWAP
179+ NRSWPIN,
180+ NRSWPOUT,
181+#endif /* CONFIG_SWAP */
178182 NR_VM_EVENT_ITEMS
179183 };
180184
include/linux/vmalloc.h+1−1
@@ -62,7 +62,7 @@ struct vm_struct { · 61 unmodified lines
6262 #ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
6363 unsigned int page_order;
6464 #endif
65− unsigned int nr_pages;
65+ unsigned long nr_pages;
6666 phys_addr_t phys_addr;
6767 const void *caller;
6868 unsigned long requested_size;
include/linux/vmstat.h+16−1
@@ -20,7 +20,6 @@ struct reclaim_stat { · 19 unmodified lines
2020 unsigned nr_congested;
2121 unsigned nr_writeback;
2222 unsigned nr_immediate;
23− unsigned nr_pageout;
2423 unsigned nr_activate[ANON_AND_FILE];
2524 unsigned nr_ref_keep;
2625 unsigned nr_unmap_fail;
@@ -194,6 +193,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item) · 167 unmodified lines
194193 return x;
195194 }
196195
196+/*
197+ * Non-clamping variant of global_node_page_state() intended for callers that
198+ * snapshot a monotonically-incremented counter and subtract two samples.
199+ * Returns the raw wrapping value so that unsigned modular subtraction stays
200+ * correct across a signed-long overflow (a real hazard on 32-bit) that the
201+ * clamp in global_node_page_state() would otherwise turn into a huge spurious
202+ * delta. Do NOT use for non-monotonic page-count reads.
203+ */
204+static inline unsigned long global_node_page_state_monotonic(enum node_stat_item item)
205+{
206+ return (unsigned long)atomic_long_read(&vm_node_stat[item]);
207+}
208+
197209 static inline unsigned long global_node_page_state(enum node_stat_item item)
198210 {
199211 VM_WARN_ON_ONCE(vmstat_item_in_bytes(item));
@@ -259,11 +271,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat, · 59 unmodified lines
259271 enum node_stat_item item);
260272 extern unsigned long node_page_state_pages(struct pglist_data *pgdat,
261273 enum node_stat_item item);
274+extern unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
275+ enum node_stat_item item);
262276 extern void fold_vm_numa_events(void);
263277 #else
264278 #define sum_zone_node_page_state(node, item) global_zone_page_state(item)
265279 #define node_page_state(node, item) global_node_page_state(item)
266280 #define node_page_state_pages(node, item) global_node_page_state_pages(item)
281+#define node_page_state_monotonic(node, item) global_node_page_state_monotonic(item)
267282 static inline void fold_vm_numa_events(void)
268283 {
269284 }
include/rv/da_monitor.h+1−1
@@ -25,7 +25,7 @@ · 24 unmodified lines
2525
2626 /*
2727 * Per-cpu variables require a unique name although static in some
28− * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules).
28+ * configurations (e.g. alpha modules).
2929 */
3030 #define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME)
3131
include/uapi/linux/alloc_tag.h+99−0
@@ -0,0 +1,99 @@
1+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
2+/*
3+ * alloc_tag IOCTL API definition
4+ *
5+ * Copyright (C) 2026 Google, LLC. All rights reserved.
6+ *
7+ * This program is free software; you can redistribute it and/or modify
8+ * it under the terms of the GNU General Public License version 2 as
9+ * published by the Free Software Foundation.
10+ */
11+
12+#ifndef _UAPI_ALLOC_TAG_H
13+#define _UAPI_ALLOC_TAG_H
14+
15+#include <linux/types.h>
16+
17+/*
18+ * Function, file and module names often have the same prefixes, therefore
19+ * when filtering by these criteria, we compare the last 64 characters to
20+ * minimize the chances of name collisions
21+ */
22+#define ALLOCINFO_STR_SIZE 64
23+
24+struct allocinfo_content_id {
25+ __u64 id;
26+};
27+
28+struct allocinfo_tag {
29+ /* Longer names are trimmed */
30+ char modname[ALLOCINFO_STR_SIZE];
31+ char function[ALLOCINFO_STR_SIZE];
32+ char filename[ALLOCINFO_STR_SIZE];
33+ __u64 lineno;
34+};
35+
36+/* The alignment ensures 32-bit compatible interfaces are not broken */
37+struct allocinfo_counter {
38+ __u64 bytes;
39+ __u64 calls;
40+ __u8 accurate;
41+} __attribute__((aligned(8)));
42+
43+struct allocinfo_tag_data {
44+ struct allocinfo_tag tag;
45+ struct allocinfo_counter counter;
46+};
47+
48+enum {
49+ ALLOCINFO_FILTER_MODNAME,
50+ ALLOCINFO_FILTER_FUNCTION,
51+ ALLOCINFO_FILTER_FILENAME,
52+ ALLOCINFO_FILTER_LINENO,
53+ ALLOCINFO_FILTER_INACCURATE,
54+ ALLOCINFO_FILTER_MIN_SIZE,
55+ ALLOCINFO_FILTER_MAX_SIZE,
56+ __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_MAX_SIZE
57+};
58+
59+#define ALLOCINFO_FILTER_MASK_MODNAME (1 << ALLOCINFO_FILTER_MODNAME)
60+#define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION)
61+#define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME)
62+#define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO)
63+#define ALLOCINFO_FILTER_MASK_INACCURATE (1 << ALLOCINFO_FILTER_INACCURATE)
64+#define ALLOCINFO_FILTER_MASK_MIN_SIZE (1 << ALLOCINFO_FILTER_MIN_SIZE)
65+#define ALLOCINFO_FILTER_MASK_MAX_SIZE (1 << ALLOCINFO_FILTER_MAX_SIZE)
66+
67+#define ALLOCINFO_FILTER_MASKS \
68+ ((1 << (__ALLOCINFO_FILTER_LAST + 1)) - 1)
69+
70+struct allocinfo_filter {
71+ __u64 mask; /* bitmask of the filter fields used */
72+ struct allocinfo_tag fields;
73+ __u64 min_size;
74+ __u64 max_size;
75+ /* filter criteria only; see allocinfo_counter.accurate for actual accuracy */
76+ __u64 inaccurate;
77+};
78+
79+struct allocinfo_get_at {
80+ /* inputs */
81+ __u64 pos;
82+ struct allocinfo_filter filter;
83+ /* output */
84+ struct allocinfo_tag_data data;
85+};
86+
87+#define _ALLOCINFO_IOC_CONTENT_ID 0
88+#define _ALLOCINFO_IOC_GET_AT 1
89+#define _ALLOCINFO_IOC_GET_NEXT 2
90+
91+#define ALLOCINFO_IOC_BASE 0xA6
92+#define ALLOCINFO_IOC_CONTENT_ID _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_CONTENT_ID, \
93+ struct allocinfo_content_id)
94+#define ALLOCINFO_IOC_GET_AT _IOWR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_AT, \
95+ struct allocinfo_get_at)
96+#define ALLOCINFO_IOC_GET_NEXT _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_NEXT, \
97+ struct allocinfo_tag_data)
98+
99+#endif /* _UAPI_ALLOC_TAG_H */
kernel/events/uprobes.c+1−1
@@ -513,7 +513,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, · 512 unmodified lines
513513
514514 uprobe = container_of(auprobe, struct uprobe, arch);
515515
516− if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags)))
516+ if (WARN_ON_ONCE(!vma_is_cow_mapping(vma)))
517517 return -EINVAL;
518518
519519 /*
kernel/locking/lockdep.c+51−7
@@ -5093,7 +5093,7 @@ static int __lock_is_held(const struct lockdep_map *lock, int read); · 5092 unmodified lines
50935093 static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
50945094 int trylock, int read, int check, int hardirqs_off,
50955095 struct lockdep_map *nest_lock, unsigned long ip,
5096− int references, int pin_count, int sync)
5096+ int references, int pin_count, int sync, int seq)
50975097 {
50985098 struct task_struct *curr = current;
50995099 struct lock_class *class = NULL;
@@ -5199,6 +5199,7 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, · 99 unmodified lines
51995199 hlock->holdtime_stamp = lockstat_clock();
52005200 #endif
52015201 hlock->pin_count = pin_count;
5202+ hlock->seq_count = seq;
52025203
52035204 if (check_wait_context(curr, hlock))
52045205 return 0;
@@ -5404,7 +5405,7 @@ static int reacquire_held_locks(struct task_struct *curr, unsigned int depth, · 199 unmodified lines
54045405 hlock->read, hlock->check,
54055406 hlock->hardirqs_off,
54065407 hlock->nest_lock, hlock->acquire_ip,
5407− hlock->references, hlock->pin_count, 0)) {
5408+ hlock->references, hlock->pin_count, 0, hlock->seq_count)) {
54085409 case 0:
54095410 return 1;
54105411 case 1:
@@ -5687,14 +5688,17 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie · 276 unmodified lines
56875688 struct held_lock *hlock = curr->held_locks + i;
56885689
56895690 if (match_held_lock(hlock, lock)) {
5691+ int pin_count;
5692+
56905693 if (WARN(!hlock->pin_count, "unpinning an unpinned lock\n"))
56915694 return;
56925695
5693− hlock->pin_count -= cookie.val;
5696+ pin_count = hlock->pin_count - cookie.val;
56945697
5695− if (WARN((int)hlock->pin_count < 0, "pin count corrupted\n"))
5696− hlock->pin_count = 0;
5698+ if (WARN(pin_count < 0, "pin count corrupted\n"))
5699+ pin_count = 0;
56975700
5701+ hlock->pin_count = pin_count;
56985702 return;
56995703 }
57005704 }
@@ -5702,6 +5706,24 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie · 1 unmodified line
57025706 WARN(1, "unpinning an unheld lock\n");
57035707 }
57045708
5709+static u32 __lock_sequence(struct lockdep_map *lock)
5710+{
5711+ struct task_struct *curr = current;
5712+ int i;
5713+
5714+ if (unlikely(!debug_locks))
5715+ return ~0;
5716+
5717+ for (i = 0; i < curr->lockdep_depth; i++) {
5718+ struct held_lock *hlock = curr->held_locks + i;
5719+
5720+ if (match_held_lock(hlock, lock))
5721+ return hlock->seq_count;
5722+ }
5723+
5724+ return ~0;
5725+}
5726+
57055727 /*
57065728 * Check whether we follow the irq-flags state precisely:
57075729 */
@@ -5884,7 +5906,8 @@ void lock_acquire(struct lockdep_map *lock, unsigned int subclass, · 176 unmodified lines
58845906
58855907 lockdep_recursion_inc();
58865908 __lock_acquire(lock, subclass, trylock, read, check,
5887− irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0);
5909+ irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0,
5910+ ++current->lockdep_seq);
58885911 lockdep_recursion_finish();
58895912 raw_local_irq_restore(flags);
58905913 }
@@ -5932,7 +5955,8 @@ void lock_sync(struct lockdep_map *lock, unsigned subclass, int read, · 41 unmodified lines
59325955
59335956 lockdep_recursion_inc();
59345957 __lock_acquire(lock, subclass, 0, read, check,
5935− irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1);
5958+ irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1,
5959+ ++current->lockdep_seq);
59365960 check_chain_key(current);
59375961 lockdep_recursion_finish();
59385962 raw_local_irq_restore(flags);
@@ -6018,6 +6042,26 @@ void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie) · 79 unmodified lines
60186042 }
60196043 EXPORT_SYMBOL_GPL(lock_unpin_lock);
60206044
6045+u32 lock_sequence(struct lockdep_map *lock)
6046+{
6047+ unsigned long flags;
6048+ u32 seq = ~0;
6049+
6050+ if (unlikely(!lockdep_enabled()))
6051+ return seq;
6052+
6053+ raw_local_irq_save(flags);
6054+ check_flags(flags);
6055+
6056+ lockdep_recursion_inc();
6057+ seq = __lock_sequence(lock);
6058+ lockdep_recursion_finish();
6059+ raw_local_irq_restore(flags);
6060+
6061+ return seq;
6062+}
6063+EXPORT_SYMBOL_GPL(lock_sequence);
6064+
60216065 #ifdef CONFIG_LOCK_STAT
60226066 static void print_lock_contention_bug(struct task_struct *curr,
60236067 struct lockdep_map *lock,
lib/Kconfig.debug+0−15
@@ -613,21 +613,6 @@ config BUILTIN_MODULE_RANGES · 612 unmodified lines
613613 It also records an anchor symbol to determine the load address of the
614614 section.
615615
616−config DEBUG_FORCE_WEAK_PER_CPU
617− bool "Force weak per-cpu definitions"
618− depends on DEBUG_KERNEL
619− help
620− s390 and alpha require percpu variables in modules to be
621− defined weak to work around addressing range issue which
622− puts the following two restrictions on percpu variable
623− definitions.
624−
625− 1. percpu symbols must be unique whether static or not
626− 2. percpu variables can't be defined inside a function
627−
628− To ensure that generic code follows the above rules, this
629− option forces all percpu variables to be defined as weak.
630−
631616 config WARN_CONTEXT_ANALYSIS
632617 bool "Compiler context-analysis warnings"
633618 depends on CC_IS_CLANG && CLANG_VERSION >= 230000
lib/codetag.c+18−0
@@ -19,6 +19,8 @@ struct codetag_type { · 18 unmodified lines
1919 struct codetag_type_desc desc;
2020 /* generates unique sequence number for module load */
2121 unsigned long next_mod_seq;
22+ /* bumped on every module load and unload */
23+ unsigned long content_id;
2224 };
2325
2426 struct codetag_range {
@@ -50,6 +52,20 @@ void codetag_unlock_module_list(struct codetag_type *cttype) · 25 unmodified lines
5052 up_read(&cttype->mod_lock);
5153 }
5254
55+unsigned long codetag_get_content_id(struct codetag_type *cttype)
56+{
57+ lockdep_assert_held(&cttype->mod_lock);
58+
59+ return cttype->content_id;
60+}
61+
62+unsigned int codetag_get_count(struct codetag_type *cttype)
63+{
64+ lockdep_assert_held(&cttype->mod_lock);
65+
66+ return cttype->count;
67+}
68+
5369 struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype)
5470 {
5571 struct codetag_iterator iter = {
@@ -204,6 +220,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) · 148 unmodified lines
204220
205221 down_write(&cttype->mod_lock);
206222 cmod->mod_seq = ++cttype->next_mod_seq;
223+ ++cttype->content_id;
207224 mod_id = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL);
208225 if (mod_id >= 0) {
209226 if (cttype->desc.module_load) {
@@ -368,6 +385,7 @@ void codetag_unload_module(struct module *mod) · 158 unmodified lines
368385 cttype->count -= range_size(cttype, &cmod->range);
369386 idr_remove(&cttype->mod_idr, mod_id);
370387 kfree(cmod);
388+ ++cttype->content_id;
371389 }
372390 up_write(&cttype->mod_lock);
373391 if (found && cttype->desc.free_section_mem)
lib/maple_tree.c+280−121
@@ -261,6 +261,12 @@ static inline bool mas_is_underflow(struct ma_state *mas) · 260 unmodified lines
261261 return mas->status == ma_underflow;
262262 }
263263
264+static inline void mas_make_walkable(struct ma_state *mas)
265+{
266+ if (!mas_is_active(mas) && !mas_is_start(mas))
267+ mas->status = ma_start;
268+}
269+
264270 static __always_inline struct maple_node *mte_to_node(
265271 const struct maple_enode *entry)
266272 {
@@ -356,11 +362,6 @@ static __always_inline bool mte_is_root(const struct maple_enode *node) · 89 unmodified lines
356362 return ma_is_root(mte_to_node(node));
357363 }
358364
359−static inline bool mas_is_root_limits(const struct ma_state *mas)
360−{
361− return !mas->min && mas->max == ULONG_MAX;
362−}
363−
364365 static __always_inline bool mt_is_alloc(struct maple_tree *mt)
365366 {
366367 return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE);
@@ -455,46 +456,6 @@ enum maple_type mas_parent_type(struct ma_state *mas, struct maple_enode *enode) · 88 unmodified lines
455456 return 0;
456457 }
457458
458−/*
459− * mas_set_parent() - Set the parent node and encode the slot
460− * @mas: The maple state
461− * @enode: The encoded maple node.
462− * @parent: The encoded maple node that is the parent of @enode.
463− * @slot: The slot that @enode resides in @parent.
464− *
465− * Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the
466− * parent type.
467− */
468−static inline
469−void mas_set_parent(struct ma_state *mas, struct maple_enode *enode,
470− const struct maple_enode *parent, unsigned char slot)
471−{
472− unsigned long val = (unsigned long)parent;
473− unsigned long shift;
474− unsigned long type;
475− enum maple_type p_type = mte_node_type(parent);
476−
477− MAS_BUG_ON(mas, p_type == maple_dense);
478− MAS_BUG_ON(mas, p_type == maple_leaf_64);
479−
480− switch (p_type) {
481− case maple_range_64:
482− case maple_arange_64:
483− shift = MAPLE_PARENT_SLOT_SHIFT;
484− type = MAPLE_PARENT_RANGE64;
485− break;
486− default:
487− case maple_dense:
488− case maple_leaf_64:
489− shift = type = 0;
490− break;
491− }
492−
493− val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */
494− val |= (slot << shift) | type;
495− mte_to_node(enode)->parent = ma_parent_ptr(val);
496−}
497−
498459 /*
499460 * mte_parent_slot() - get the parent slot of @enode.
500461 * @enode: The encoded maple node.
@@ -876,6 +837,42 @@ static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt, · 375 unmodified lines
876837 meta->gap = offset;
877838 }
878839
840+/*
841+ * mas_set_parent_slots() - Bulk operation to set many slot parent pointers
842+ * @mas: The maple state
843+ * @parent: The encoded maple node that is the parent of @enode.
844+ * @slot: The slot that of the @enode.
845+ * @start_slot: The offset into @slot
846+ * @count: The number of slots to set (eg: exclusive)
847+ */
848+static inline
849+void mas_set_parent_slots(struct ma_state *mas, struct maple_enode *parent,
850+ void __rcu **slots, unsigned char start_slot, unsigned char count)
851+{
852+ unsigned long val;
853+ unsigned long shift;
854+ unsigned long type;
855+ enum maple_type p_type = mte_node_type(parent);
856+ unsigned char i;
857+
858+ MAS_BUG_ON(mas, p_type != maple_range_64 &&
859+ p_type != maple_arange_64);
860+
861+ shift = MAPLE_PARENT_SLOT_SHIFT;
862+ type = MAPLE_PARENT_RANGE64;
863+
864+ val = (unsigned long)parent;
865+ val &= ~MAPLE_NODE_MASK;
866+
867+ for (i = 0; i < count; i++) {
868+ unsigned long pval = val | ((start_slot + i) << shift) | type;
869+ struct maple_enode *child;
870+
871+ child = mt_slot_locked(mas->tree, slots, i);
872+ mte_to_node(child)->parent = ma_parent_ptr(pval);
873+ }
874+}
875+
879876 /*
880877 * mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes.
881878 * @mat: the ma_topiary, a linked list of dead nodes.
@@ -1153,6 +1150,79 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) · 271 unmodified lines
11531150 ma_free_rcu(mte_to_node(used));
11541151 }
11551152
1153+
1154+#ifdef CONFIG_LOCKDEP
1155+static struct lockdep_map *mas_lockdep_map(struct ma_state *mas)
1156+{
1157+ struct maple_tree *mt = mas->tree;
1158+
1159+ if (mt_external_lock(mt))
1160+ return mt->ma_external_lock;
1161+
1162+ return &(mt->ma_lock).dep_map;
1163+}
1164+
1165+#endif
1166+
1167+static void mas_lock_check(struct ma_state *mas)
1168+{
1169+#ifdef CONFIG_LOCKDEP
1170+ struct lockdep_map *map;
1171+ u32 seq;
1172+
1173+ if (!mas_is_active(mas))
1174+ return;
1175+
1176+#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD
1177+ if (!mt_locked(mas->tree)) {
1178+ if (mt_in_rcu(mas->tree))
1179+ WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp));
1180+ }
1181+#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */
1182+
1183+ map = mas_lockdep_map(mas);
1184+ if (!map)
1185+ return;
1186+
1187+ seq = lock_sequence(map);
1188+ if (seq != UINT_MAX && mas->ld_seq != UINT_MAX)
1189+ WARN_ON_ONCE(mas->ld_seq != seq);
1190+#endif /* CONFIG_LOCKDEP */
1191+
1192+}
1193+
1194+static void mas_init_lock_check(struct ma_state *mas)
1195+{
1196+#ifdef CONFIG_LOCKDEP
1197+ struct lockdep_map *map;
1198+#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD
1199+ if (!mt_locked(mas->tree)) {
1200+ if (mt_in_rcu(mas->tree))
1201+ mas->rcu_gp = get_state_synchronize_rcu();
1202+ return;
1203+ }
1204+#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */
1205+
1206+ map = mas_lockdep_map(mas);
1207+ if (map) /* Update regardless of lock state */
1208+ mas->ld_seq = lock_sequence(map);
1209+#endif /* CONFIG_LOCKDEP */
1210+
1211+}
1212+
1213+static void mas_may_init_lock_check(struct ma_state *mas)
1214+{
1215+#ifdef CONFIG_LOCKDEP
1216+#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD
1217+ if (mas_is_start(mas) || mas_is_paused(mas)) {
1218+ mas_init_lock_check(mas);
1219+ return;
1220+ }
1221+#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */
1222+ mas_lock_check(mas);
1223+#endif /* CONFIG_LOCKDEP */
1224+}
1225+
11561226 /*
11571227 * mas_start() - Sets up maple state for operations.
11581228 * @mas: The maple state.
@@ -1171,6 +1241,7 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) · 12 unmodified lines
11711241 if (likely(mas_is_start(mas))) {
11721242 struct maple_enode *root;
11731243
1244+ mas_init_lock_check(mas);
11741245 mas->min = 0;
11751246 mas->max = ULONG_MAX;
11761247
@@ -1501,14 +1572,26 @@ ascend: · 324 unmodified lines
15011572 goto ascend;
15021573 }
15031574
1575+static __always_inline void mas_update_gap_known(struct ma_state *mas,
1576+ unsigned long gap)
1577+{
1578+ unsigned char pslot;
1579+ unsigned long p_gap;
1580+
1581+ pslot = mte_parent_slot(mas->node);
1582+ p_gap = ma_gaps(mte_parent(mas->node),
1583+ mas_parent_type(mas, mas->node))[pslot];
1584+
1585+ if (p_gap != gap)
1586+ mas_parent_gap(mas, pslot, gap);
1587+}
1588+
15041589 /*
15051590 * mas_update_gap() - Update a nodes gaps and propagate up if necessary.
15061591 * @mas: the maple state.
15071592 */
15081593 static inline void mas_update_gap(struct ma_state *mas)
15091594 {
1510− unsigned char pslot;
1511− unsigned long p_gap;
15121595 unsigned long max_gap;
15131596
15141597 if (!mt_is_alloc(mas->tree))
@@ -1518,13 +1601,7 @@ static inline void mas_update_gap(struct ma_state *mas) · 3 unmodified lines
15181601 return;
15191602
15201603 max_gap = mas_max_gap(mas);
1521−
1522− pslot = mte_parent_slot(mas->node);
1523− p_gap = ma_gaps(mte_parent(mas->node),
1524− mas_parent_type(mas, mas->node))[pslot];
1525−
1526− if (p_gap != max_gap)
1527− mas_parent_gap(mas, pslot, max_gap);
1604+ mas_update_gap_known(mas, max_gap);
15281605 }
15291606
15301607 /*
@@ -1540,14 +1617,10 @@ static inline void mas_adopt_children(struct ma_state *mas, · 9 unmodified lines
15401617 struct maple_node *node = mte_to_node(parent);
15411618 void __rcu **slots = ma_slots(node, type);
15421619 unsigned long *pivots = ma_pivots(node, type);
1543− struct maple_enode *child;
1544− unsigned char offset;
1620+ unsigned char end;
15451621
1546− offset = ma_data_end(node, type, pivots, mas->max);
1547− do {
1548− child = mas_slot_locked(mas, slots, offset);
1549− mas_set_parent(mas, child, parent, offset);
1550− } while (offset--);
1622+ end = ma_data_end(node, type, pivots, mas->max);
1623+ mas_set_parent_slots(mas, parent, slots, 0, end + 1);
15511624 }
15521625
15531626 /*
@@ -1929,15 +2002,10 @@ unsigned long node_copy(struct ma_state *mas, struct maple_node *src, · 375 unmodified lines
19292002 s_slots = ma_slots(src, s_mt) + start;
19302003 s_pivots = ma_pivots(src, s_mt) + start;
19312004 memcpy(d_slots, s_slots, size * sizeof(void __rcu *));
1932− if (!ma_is_leaf(d_mt) && s_mt == maple_copy) {
1933− struct maple_enode *edst = mt_mk_node(dst, d_mt);
1934−
19352005
1936− for (int i = 0; i < size; i++)
1937− mas_set_parent(mas,
1938− mt_slot_locked(mas->tree, d_slots, i),
1939− edst, d_start + i);
1940− }
2006+ if (!ma_is_leaf(d_mt) && s_mt == maple_copy)
2007+ mas_set_parent_slots(mas, mt_mk_node(dst, d_mt),
2008+ d_slots, d_start, size);
19412009
19422010 d_gaps = ma_gaps(dst, d_mt);
19432011 if (d_gaps) {
@@ -2081,8 +2149,8 @@ static inline void mas_wmb_replace(struct ma_state *mas, struct maple_copy *cp) · 137 unmodified lines
20812149 mas->node = mt_slot_locked(mas->tree, cp->slot, 0);
20822150 /* Insert the new data in the tree */
20832151 mas_topiary_replace(mas, old_enode, cp->height);
2084− if (!mte_is_leaf(mas->node))
2085− mas_update_gap(mas);
2152+ if (mt_is_alloc(mas->tree) && !mte_is_root(mas->node))
2153+ mas_update_gap_known(mas, cp->gap[0]);
20862154
20872155 mtree_range_walk(mas);
20882156 }
@@ -2928,13 +2996,6 @@ static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas, · 839 unmodified lines
29282996 if (r_mas->last < r_wr_mas->r_max)
29292997 r_mas->last = r_wr_mas->r_max;
29302998 r_mas->offset++;
2931− } else if ((r_mas->last == r_wr_mas->r_max) &&
2932− (r_mas->last < r_mas->max) &&
2933− !mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) {
2934− r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots,
2935− r_wr_mas->type, r_mas->offset + 1);
2936− r_mas->offset++;
2937− r_wr_mas->r_max = r_mas->last;
29382999 }
29393000 }
29403001
@@ -3125,7 +3186,7 @@ static void mas_wr_spanning_store(struct ma_wr_state *wr_mas) · 184 unmodified lines
31253186 static inline void mas_wr_node_store(struct ma_wr_state *wr_mas)
31263187 {
31273188 unsigned char dst_offset, offset_end;
3128− unsigned char copy_size, node_pivots;
3189+ unsigned char copy_size, node_pivots, node_slots;
31293190 struct maple_node reuse, *newnode;
31303191 unsigned long *dst_pivots;
31313192 void __rcu **dst_slots;
@@ -3138,6 +3199,7 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) · 6 unmodified lines
31383199 in_rcu = mt_in_rcu(mas->tree);
31393200 offset_end = wr_mas->offset_end;
31403201 node_pivots = mt_pivots[wr_mas->type];
3202+ node_slots = mt_slots[wr_mas->type];
31413203 /* Assume last adds an entry */
31423204 new_end = mas->end + 1 - offset_end + mas->offset;
31433205 if (mas->last == wr_mas->end_piv) {
@@ -3149,7 +3211,6 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) · 5 unmodified lines
31493211 if (in_rcu) {
31503212 newnode = mas_pop_node(mas);
31513213 } else {
3152− memset(&reuse, 0, sizeof(struct maple_node));
31533214 newnode = &reuse;
31543215 }
31553216
@@ -3193,7 +3254,21 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) · 37 unmodified lines
31933254 dst_pivots[new_end] = mas->max;
31943255
31953256 done:
3196− mas_leaf_set_meta(newnode, maple_leaf_64, new_end);
3257+ if (!in_rcu && new_end + 2 < node_slots) {
3258+ unsigned char clear_from = new_end + 1;
3259+
3260+ /*
3261+ * Note that the last slot is never cleared, since the metadata
3262+ * will be stored there or it has a value.
3263+ */
3264+ memset(dst_slots + clear_from, 0,
3265+ sizeof(void __rcu *) * (node_slots - clear_from));
3266+ if (clear_from < node_pivots)
3267+ memset(dst_pivots + clear_from, 0,
3268+ sizeof(unsigned long) * (node_pivots - clear_from));
3269+ }
3270+
3271+ mas_leaf_set_meta(newnode, wr_mas->type, new_end);
31973272 if (in_rcu) {
31983273 struct maple_enode *old_enode = mas->node;
31993274
@@ -3218,7 +3293,7 @@ static inline void mas_wr_slot_store(struct ma_wr_state *wr_mas) · 18 unmodified lines
32183293 void __rcu **slots = wr_mas->slots;
32193294 bool gap = false;
32203295
3221− gap |= !mt_slot_locked(mas->tree, slots, offset);
3296+ gap |= !wr_mas->content;
32223297 gap |= !mt_slot_locked(mas->tree, slots, offset + 1);
32233298
32243299 if (wr_mas->offset_end - offset == 1) {
@@ -3595,7 +3670,7 @@ set_content: · 370 unmodified lines
35953670
35963671 /**
35973672 * mas_prealloc_calc() - Calculate number of nodes needed for a
3598− * given store oepration
3673+ * given store operation
35993674 * @wr_mas: The maple write state
36003675 * @entry: The entry to store into the tree
36013676 *
@@ -3661,6 +3736,9 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) · 59 unmodified lines
36613736 {
36623737 struct ma_state *mas = wr_mas->mas;
36633738 unsigned char new_end;
3739+ bool appending;
3740+ bool one_slot;
3741+ bool in_rcu;
36643742
36653743 if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
36663744 return wr_store_root;
@@ -3680,21 +3758,30 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) · 13 unmodified lines
36803758 return wr_new_root;
36813759
36823760 new_end = mas_wr_new_end(wr_mas);
3761+ in_rcu = mt_in_rcu(mas->tree);
3762+ appending = mas->offset == mas->end;
3763+ one_slot = wr_mas->offset_end - mas->offset == 1;
3764+
36833765 /* Potential spanning rebalance collapsing a node */
36843766 if (new_end < mt_min_slots[wr_mas->type]) {
36853767 if (!mte_is_root(mas->node))
36863768 return wr_rebalance;
3769+ if (!in_rcu) {
3770+ if (appending)
3771+ return wr_append;
3772+ else if (mas->end == new_end && one_slot)
3773+ return wr_slot_store;
3774+ }
36873775 return wr_node_store;
36883776 }
36893777
36903778 if (new_end >= mt_slots[wr_mas->type])
36913779 return wr_split_store;
36923780
3693− if (!mt_in_rcu(mas->tree) && (mas->offset == mas->end))
3781+ if (!in_rcu && appending)
36943782 return wr_append;
36953783
3696− if ((new_end == mas->end) && (!mt_in_rcu(mas->tree) ||
3697− (wr_mas->offset_end - mas->offset == 1)))
3784+ if (new_end == mas->end && (!in_rcu || one_slot))
36983785 return wr_slot_store;
36993786
37003787 return wr_node_store;
@@ -3793,35 +3880,40 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, · 92 unmodified lines
37933880 void *entry, unsigned long range_lo, unsigned long range_hi,
37943881 unsigned long *next, gfp_t gfp)
37953882 {
3796− unsigned long min = range_lo;
3797− int ret = 0;
3798−
3799− range_lo = max(min, *next);
3800− ret = mas_empty_area(mas, range_lo, range_hi, 1);
3801− if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) {
3802− mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED;
3803− ret = 1;
3804− }
3805− if (ret < 0 && range_lo > min) {
3806− mas_reset(mas);
3807− ret = mas_empty_area(mas, min, range_hi, 1);
3808− if (ret == 0)
3809− ret = 1;
3810− }
3811− if (ret < 0)
3812− return ret;
3883+ int ret;
3884+ unsigned long min;
38133885
3886+ min = range_lo;
38143887 do {
3888+ range_lo = max(min, *next);
3889+ ret = mas_empty_area(mas, range_lo, range_hi, 1);
3890+ if (ret < 0 && range_lo > min) {
3891+ mas_reset(mas);
3892+ ret = mas_empty_area(mas, min, range_hi, 1);
3893+ if (ret == 0)
3894+ ret = 1;
3895+ }
3896+ if (ret < 0)
3897+ goto out;
3898+
38153899 mas_insert(mas, entry);
38163900 } while (mas_nomem(mas, gfp));
3817− if (mas_is_err(mas))
3818− return xa_err(mas->node);
38193901
3902+ if (mas_is_err(mas)) {
3903+ ret = xa_err(mas->node);
3904+ goto out;
3905+ }
3906+
3907+ if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) {
3908+ mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED;
3909+ ret = 1;
3910+ }
38203911 *startp = mas->index;
38213912 *next = *startp + 1;
38223913 if (*next == 0)
38233914 mas->tree->ma_flags |= MT_FLAGS_ALLOC_WRAPPED;
38243915
3916+out:
38253917 mas_destroy(mas);
38263918 return ret;
38273919 }
@@ -4360,8 +4452,8 @@ void *mas_walk(struct ma_state *mas) · 532 unmodified lines
43604452 {
43614453 void *entry;
43624454
4363− if (!mas_is_active(mas) && !mas_is_start(mas))
4364− mas->status = ma_start;
4455+ mas_may_init_lock_check(mas);
4456+ mas_make_walkable(mas);
43654457 retry:
43664458 entry = mas_state_walk(mas);
43674459 if (mas_is_start(mas)) {
@@ -4831,6 +4923,7 @@ void *mas_store(struct ma_state *mas, void *entry) · 463 unmodified lines
48314923 {
48324924 MA_WR_STATE(wr_mas, mas, entry);
48334925
4926+ mas_may_init_lock_check(mas);
48344927 trace_ma_write(TP_FCT, mas, 0, entry);
48354928 #ifdef CONFIG_DEBUG_MAPLE_TREE
48364929 if (MAS_WARN_ON(mas, mas->index > mas->last))
@@ -4889,6 +4982,7 @@ int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp) · 52 unmodified lines
48894982 MA_WR_STATE(wr_mas, mas, entry);
48904983 int ret = 0;
48914984
4985+ mas_may_init_lock_check(mas);
48924986 retry:
48934987 mas_wr_preallocate(&wr_mas, entry);
48944988 if (unlikely(mas_nomem(mas, gfp))) {
@@ -4919,6 +5013,7 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) · 24 unmodified lines
49195013 {
49205014 MA_WR_STATE(wr_mas, mas, entry);
49215015
5016+ mas_lock_check(mas);
49225017 if (mas->store_type == wr_store_root) {
49235018 mas_wr_prealloc_setup(&wr_mas);
49245019 goto store;
@@ -4951,6 +5046,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) · 26 unmodified lines
49515046 {
49525047 MA_WR_STATE(wr_mas, mas, entry);
49535048
5049+ mas_may_init_lock_check(mas);
49545050 mas_wr_prealloc_setup(&wr_mas);
49555051 mas->store_type = mas_wr_store_type(&wr_mas);
49565052 mas_prealloc_calc(&wr_mas, entry);
@@ -4997,6 +5093,7 @@ static void mas_may_activate(struct ma_state *mas) · 40 unmodified lines
49975093 mas->status = ma_start;
49985094 } else {
49995095 mas->status = ma_active;
5096+ mas_lock_check(mas);
50005097 }
50015098 }
50025099
@@ -5074,6 +5171,7 @@ void *mas_next(struct ma_state *mas, unsigned long max) · 71 unmodified lines
50745171 {
50755172 void *entry = NULL;
50765173
5174+ mas_may_init_lock_check(mas);
50775175 if (mas_next_setup(mas, max, &entry))
50785176 return entry;
50795177
@@ -5097,6 +5195,7 @@ void *mas_next_range(struct ma_state *mas, unsigned long max) · 17 unmodified lines
50975195 {
50985196 void *entry = NULL;
50995197
5198+ mas_may_init_lock_check(mas);
51005199 if (mas_next_setup(mas, max, &entry))
51015200 return entry;
51025201
@@ -5205,6 +5304,7 @@ void *mas_prev(struct ma_state *mas, unsigned long min) · 102 unmodified lines
52055304 {
52065305 void *entry = NULL;
52075306
5307+ mas_may_init_lock_check(mas);
52085308 if (mas_prev_setup(mas, min, &entry))
52095309 return entry;
52105310
@@ -5228,6 +5328,7 @@ void *mas_prev_range(struct ma_state *mas, unsigned long min) · 17 unmodified lines
52285328 {
52295329 void *entry = NULL;
52305330
5331+ mas_may_init_lock_check(mas);
52315332 if (mas_prev_setup(mas, min, &entry))
52325333 return entry;
52335334
@@ -5274,6 +5375,7 @@ EXPORT_SYMBOL_GPL(mt_prev); · 40 unmodified lines
52745375 */
52755376 void mas_pause(struct ma_state *mas)
52765377 {
5378+ mas_lock_check(mas);
52775379 mas->status = ma_pause;
52785380 mas->node = NULL;
52795381 }
@@ -5382,6 +5484,7 @@ void *mas_find(struct ma_state *mas, unsigned long max) · 102 unmodified lines
53825484 {
53835485 void *entry = NULL;
53845486
5487+ mas_may_init_lock_check(mas);
53855488 if (mas_find_setup(mas, max, &entry))
53865489 return entry;
53875490
@@ -5409,6 +5512,7 @@ void *mas_find_range(struct ma_state *mas, unsigned long max) · 21 unmodified lines
54095512 {
54105513 void *entry = NULL;
54115514
5515+ mas_may_init_lock_check(mas);
54125516 if (mas_find_setup(mas, max, &entry))
54135517 return entry;
54145518
@@ -5428,7 +5532,6 @@ EXPORT_SYMBOL_GPL(mas_find_range); · 13 unmodified lines
54285532 static bool mas_find_rev_setup(struct ma_state *mas, unsigned long min,
54295533 void **entry)
54305534 {
5431−
54325535 switch (mas->status) {
54335536 case ma_active:
54345537 goto active;
@@ -5521,6 +5624,7 @@ void *mas_find_rev(struct ma_state *mas, unsigned long min) · 86 unmodified lines
55215624 {
55225625 void *entry = NULL;
55235626
5627+ mas_may_init_lock_check(mas);
55245628 if (mas_find_rev_setup(mas, min, &entry))
55255629 return entry;
55265630
@@ -5547,6 +5651,7 @@ void *mas_find_range_rev(struct ma_state *mas, unsigned long min) · 20 unmodified lines
55475651 {
55485652 void *entry = NULL;
55495653
5654+ mas_may_init_lock_check(mas);
55505655 if (mas_find_rev_setup(mas, min, &entry))
55515656 return entry;
55525657
@@ -5564,6 +5669,11 @@ EXPORT_SYMBOL_GPL(mas_find_range_rev); · 11 unmodified lines
55645669 * Searches for @mas->index, sets @mas->index and @mas->last to the range and
55655670 * erases that range.
55665671 *
5672+ * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL
5673+ * to do so if necessary. If the allocation fails, the internal lock will be
5674+ * dropped to retry. An externally locked tree must be protected by a lock that
5675+ * allows blocking for this API.
5676+ *
55675677 * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated.
55685678 */
55695679 void *mas_erase(struct ma_state *mas)
@@ -5572,25 +5682,25 @@ void *mas_erase(struct ma_state *mas) · 2 unmodified lines
55725682 unsigned long index = mas->index;
55735683 MA_WR_STATE(wr_mas, mas, NULL);
55745684
5575− if (!mas_is_active(mas) || !mas_is_start(mas))
5576− mas->status = ma_start;
5685+ /*
5686+ * In low memory situations, the allocation is retried with the gfp flag
5687+ * GFP_KERNEL | __GFP_NOFAIL. The internal spinlock is dropped in
5688+ * mas_nomem_nofail(), however the external lock is not dropped.
5689+ */
5690+ if (mt_external_lock(mas->tree))
5691+ might_alloc(GFP_KERNEL);
55775692
5693+ mas_make_walkable(mas);
55785694 write_retry:
55795695 entry = mas_state_walk(mas);
55805696 if (!entry)
5581− return NULL;
5697+ goto out;
55825698
55835699 /* Must reset to ensure spanning writes of last slot are detected */
55845700 mas_reset(mas);
55855701 mas_wr_preallocate(&wr_mas, NULL);
5586− if (mas_nomem(mas, GFP_KERNEL)) {
5587− /* in case the range of entry changed when unlocked */
5588− mas->index = mas->last = index;
5702+ if (mas_nomem_nofail(mas, index, index))
55895703 goto write_retry;
5590− }
5591−
5592− if (mas_is_err(mas))
5593− goto out;
55945704
55955705 mas_wr_store_entry(&wr_mas);
55965706 out:
@@ -5601,10 +5711,11 @@ EXPORT_SYMBOL_GPL(mas_erase); · 4 unmodified lines
56015711
56025712 /**
56035713 * mas_nomem() - Check if there was an error allocating and do the allocation
5604− * if necessary If there are allocations, then free them.
5714+ * if necessary.
5715+ *
56055716 * @mas: The maple state
56065717 * @gfp: The GFP_FLAGS to use for allocations
5607− * Return: true on allocation, false otherwise.
5718+ * Return: False on no memory. True otherwise (partial success as well)
56085719 */
56095720 bool mas_nomem(struct ma_state *mas, gfp_t gfp)
56105721 __must_hold(mas->tree->ma_lock)
@@ -5612,6 +5723,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) · 1 unmodified line
56125723 if (likely(mas->node != MA_ERROR(-ENOMEM)))
56135724 return false;
56145725
5726+ /* Allocations can fail, don't do this. */
5727+ WARN_ON_ONCE(!gfpflags_allow_blocking(gfp) &&
5728+ mt_external_lock(mas->tree));
5729+
56155730 if (gfpflags_allow_blocking(gfp) && !mt_external_lock(mas->tree)) {
56165731 mtree_unlock(mas->tree);
56175732 mas_alloc_nodes(mas, gfp);
@@ -5620,10 +5735,49 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) · 2 unmodified lines
56205735 mas_alloc_nodes(mas, gfp);
56215736 }
56225737
5738+ /*
5739+ * Return false on zero forward progress. Partial allocations are kept
5740+ * so the retry path will attempt to get the rest. The failure should
5741+ * not happen as we try our best to reclaim. The user would need an
5742+ * external lock with a non-blocking gfp in a low memory situation.
5743+ */
56235744 if (!mas->sheaf && !mas->alloc)
56245745 return false;
56255746
5626− mas->status = ma_start;
5747+ mas_reset(mas);
5748+ return true;
5749+}
5750+
5751+/**
5752+ * mas_nomem_nofail() - Retry allocations with __GFP_NOFAIL, if the maple state
5753+ * has stored the -ENOMEM error.
5754+ * @mas: The maple state
5755+ * @index: The start of the range for the @mas reset
5756+ * @last: The end of the range for the @mas reset
5757+ *
5758+ * Return: false if @mas isn't in an -ENOMEM state. True if the allocation
5759+ * happens, the state is reset. The internal lock will be dropped and external
5760+ * locks must allow blocking.
5761+ */
5762+bool mas_nomem_nofail(struct ma_state *mas, unsigned long index,
5763+ unsigned long last)
5764+ __must_hold(mas->tree->ma_lock)
5765+{
5766+ gfp_t gfp;
5767+
5768+ if (likely(mas->node != MA_ERROR(-ENOMEM)))
5769+ return false;
5770+
5771+ gfp = GFP_KERNEL | __GFP_NOFAIL;
5772+ if (!mt_external_lock(mas->tree)) {
5773+ mtree_unlock(mas->tree);
5774+ mas_alloc_nodes(mas, gfp);
5775+ mtree_lock(mas->tree);
5776+ } else {
5777+ mas_alloc_nodes(mas, gfp);
5778+ }
5779+
5780+ mas_set_range(mas, index, last);
56275781 return true;
56285782 }
56295783
@@ -5913,6 +6067,10 @@ EXPORT_SYMBOL(mtree_alloc_rrange); · 283 unmodified lines
59136067 * Erasing is the same as a walk to an entry then a store of a NULL to that
59146068 * ENTIRE range. In fact, it is implemented as such using the advanced API.
59156069 *
6070+ * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL
6071+ * to do so if necessary. If the allocation fails, the internal lock will be
6072+ * dropped to retry.
6073+ *
59166074 * Return: The entry stored at the @index or %NULL
59176075 */
59186076 void *mtree_erase(struct maple_tree *mt, unsigned long index)
@@ -5922,6 +6080,7 @@ void *mtree_erase(struct maple_tree *mt, unsigned long index) · 3 unmodified lines
59226080 MA_STATE(mas, mt, index, index);
59236081 trace_ma_op(TP_FCT, &mas);
59246082
6083+ might_alloc(GFP_KERNEL);
59256084 mtree_lock(mt);
59266085 entry = mas_erase(&mas);
59276086 mtree_unlock(mt);
lib/test_hmm.c+2−3
@@ -1151,10 +1151,9 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, · 1150 unmodified lines
11511151 if (!dpage && !order)
11521152 return VM_FAULT_OOM;
11531153
1154− pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n",
1155− page_to_pfn(spage), page_to_pfn(dpage));
1156−
11571154 if (dpage) {
1155+ pr_debug("migrating from dev to sys pfn src: 0x%lx pfn dst: 0x%lx\n",
1156+ page_to_pfn(spage), page_to_pfn(dpage));
11581157 lock_page(dpage);
11591158 *dst |= migrate_pfn(page_to_pfn(dpage));
11601159 }
lib/xarray.c+3−0
@@ -1053,6 +1053,9 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, · 1052 unmodified lines
10531053 if (xas->xa_shift + XA_CHUNK_SHIFT > order)
10541054 return;
10551055
1056+ if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT)
1057+ gfp |= __GFP_ACCOUNT;
1058+
10561059 do {
10571060 struct xa_node *node;
10581061
mm/Kconfig+11−11
@@ -125,8 +125,6 @@ config ZSWAP_COMPRESSOR_DEFAULT · 124 unmodified lines
125125 config ZSMALLOC
126126 tristate
127127
128−if ZSMALLOC
129−
130128 menu "Zsmalloc allocator options"
131129 depends on ZSMALLOC
132130
@@ -161,8 +159,6 @@ config ZSMALLOC_CHAIN_SIZE · 28 unmodified lines
161159
162160 endmenu
163161
164−endif
165−
166162 menu "Slab allocator options"
167163
168164 config SLUB
@@ -394,6 +390,7 @@ choice · 225 unmodified lines
394390
395391 config FLATMEM_MANUAL
396392 bool "Flat Memory"
393+ depends on !NUMA
397394 depends on !ARCH_SPARSEMEM_ENABLE || ARCH_FLATMEM_ENABLE
398395 help
399396 This option is best suited for non-NUMA systems with
@@ -428,6 +425,7 @@ config SPARSEMEM · 28 unmodified lines
428425
429426 config FLATMEM
430427 def_bool y
428+ depends on !NUMA
431429 depends on !SPARSEMEM || FLATMEM_MANUAL
432430
433431 #
@@ -583,12 +581,10 @@ endchoice · 149 unmodified lines
583581
584582 config MEMORY_HOTREMOVE
585583 bool "Allow for memory hot remove"
586− depends on MEMORY_HOTPLUG
587584 select MIGRATION
588585
589586 config MHP_MEMMAP_ON_MEMORY
590587 def_bool y
591− depends on MEMORY_HOTPLUG && SPARSEMEM_VMEMMAP
592588 depends on ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE
593589
594590 endif # MEMORY_HOTPLUG
@@ -771,6 +767,7 @@ config MEMORY_FAILURE · 176 unmodified lines
771767 depends on ARCH_SUPPORTS_MEMORY_FAILURE
772768 bool "Enable recovery from hardware memory errors"
773769 select INTERVAL_TREE
770+ select MIGRATION
774771 help
775772 Enables code to recover from some memory failures on systems
776773 with MCA recovery. This allows a system to continue running
@@ -1227,9 +1224,7 @@ config ZONE_DMA32 · 450 unmodified lines
12271224
12281225 config ZONE_DEVICE
12291226 bool "Device memory (pmem, HMM, etc...) hotplug support"
1230− depends on MEMORY_HOTPLUG
12311227 depends on MEMORY_HOTREMOVE
1232− depends on SPARSEMEM_VMEMMAP
12331228 select XARRAY_MULTI
12341229
12351230 help
@@ -1395,17 +1390,15 @@ menuconfig USERFAULTFD · 159 unmodified lines
13951390 Enable the userfaultfd() system call that allows to intercept and
13961391 handle page faults in userland.
13971392
1398−if USERFAULTFD
13991393 config PTE_MARKER_UFFD_WP
14001394 bool "Userfaultfd write protection support for shmem/hugetlbfs"
14011395 default y
1402− depends on HAVE_ARCH_USERFAULTFD_WP
1396+ depends on USERFAULTFD && HAVE_ARCH_USERFAULTFD_WP
14031397
14041398 help
14051399 Allows to create marker PTEs for userfaultfd write protection
14061400 purposes. It is required to enable userfaultfd write protection on
14071401 file-backed memory types like shmem and hugetlbfs.
1408−endif # USERFAULTFD
14091402
14101403 # multi-gen LRU {
14111404 config LRU_GEN
@@ -1471,6 +1464,13 @@ config NUMA_EMU · 59 unmodified lines
14711464 into virtual nodes when booted with "numa=fake=N", where N is the
14721465 number of nodes. This is only useful for debugging.
14731466
1467+config GENERIC_ARCH_NUMA
1468+ bool
1469+ select NUMA_MEMBLKS
1470+ help
1471+ Enable support for generic NUMA implementation. Currently, RISC-V
1472+ and ARM64 use it.
1473+
14741474 config ARCH_HAS_USER_SHADOW_STACK
14751475 bool
14761476 help
mm/Kconfig.debug+0−1
@@ -326,7 +326,6 @@ config MEM_ALLOC_PROFILING · 325 unmodified lines
326326 default n
327327 depends on MMU
328328 depends on PROC_FS
329− depends on !DEBUG_FORCE_WEAK_PER_CPU
330329 select CODE_TAGGING
331330 select PAGE_EXTENSION
332331 select SLAB_OBJ_EXT
mm/Makefile+1−0
@@ -122,6 +122,7 @@ obj-$(CONFIG_CMA) += cma.o · 121 unmodified lines
122122 obj-$(CONFIG_NUMA) += numa.o
123123 obj-$(CONFIG_NUMA_MEMBLKS) += numa_memblks.o
124124 obj-$(CONFIG_NUMA_EMU) += numa_emulation.o
125+obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o
125126 obj-$(CONFIG_BALLOON) += balloon.o
126127 obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o
127128 obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o
mm/alloc_tag.c+349−2
@@ -5,6 +5,8 @@ · 4 unmodified lines
55 #include <linux/gfp.h>
66 #include <linux/kallsyms.h>
77 #include <linux/module.h>
8+#include <linux/mutex.h>
9+#include <linux/compat.h>
810 #include <linux/page_ext.h>
911 #include <linux/pgalloc_tag.h>
1012 #include <linux/proc_fs.h>
@@ -14,6 +16,7 @@ · 3 unmodified lines
1416 #include <linux/string_choices.h>
1517 #include <linux/vmalloc.h>
1618 #include <linux/kmemleak.h>
19+#include <uapi/linux/alloc_tag.h>
1720
1821 #include "internal.h"
1922 #include "page_alloc.h"
@@ -59,6 +62,11 @@ struct allocinfo_private { · 39 unmodified lines
5962 struct codetag_iterator iter;
6063 struct codetag_iterator reported_iter;
6164 bool print_header;
65+ struct allocinfo_filter filter;
66+ /* ioctl uses a separate iterator not to interfere with reads */
67+ struct codetag_iterator ioctl_iter;
68+ bool positioned; /* seq_open_private() sets to 0 */
69+ struct mutex ioctl_lock;
6270 };
6371
6472 static void *allocinfo_start(struct seq_file *m, loff_t *pos)
@@ -142,6 +150,340 @@ static const struct seq_operations allocinfo_seq_op = { · 77 unmodified lines
142150 .show = allocinfo_show,
143151 };
144152
153+/*
154+ * Initializes seq_file operations and allocates private state when opening
155+ * the /proc/allocinfo procfs entry.
156+ */
157+static int allocinfo_open(struct inode *inode, struct file *file)
158+{
159+ int ret;
160+
161+ ret = seq_open_private(file, &allocinfo_seq_op,
162+ sizeof(struct allocinfo_private));
163+ if (!ret) {
164+ struct seq_file *m = file->private_data;
165+ struct allocinfo_private *priv = m->private;
166+
167+ mutex_init(&priv->ioctl_lock);
168+ }
169+ return ret;
170+}
171+
172+/*
173+ * Cleans up the seq_file state and frees up the private state allocated in
174+ * allocinfo_open() when closing the /proc/allocinfo file descriptor.
175+ */
176+static int allocinfo_release(struct inode *inode, struct file *file)
177+{
178+ struct seq_file *m = file->private_data;
179+ struct allocinfo_private *priv = m->private;
180+
181+ mutex_destroy(&priv->ioctl_lock);
182+ return seq_release_private(inode, file);
183+}
184+
185+/*
186+ * Returns a pointer to the suffix of a string so that its length fits within
187+ * ALLOCINFO_STR_SIZE, preserving the trailing characters.
188+ * Function, file and module names often have the same prefixes, therefore
189+ * when filtering by these criteria, we compare the last 64 characters to
190+ * minimize the chances of name collisions
191+ */
192+static const char *allocinfo_str(const char *str)
193+{
194+ size_t len = strlen(str);
195+
196+ /* Keep an extra space for the trailing NULL. */
197+ if (len >= ALLOCINFO_STR_SIZE)
198+ str += (len - ALLOCINFO_STR_SIZE) + 1;
199+ return str;
200+}
201+
202+/* Copy a string and trim from the beginning if it's too long */
203+static void allocinfo_copy_str(char *dest, const char *src)
204+{
205+ strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE);
206+}
207+
208+/* Compare two strings and only consider the trimmed suffix if s1 is too long */
209+static int allocinfo_cmp_str(const char *str, const char *template)
210+{
211+ return strncmp(allocinfo_str(str), template, ALLOCINFO_STR_SIZE);
212+}
213+
214+/* Fetch the per-CPU counters */
215+static inline struct alloc_tag_counters allocinfo_prefetch_counters(struct codetag *ct)
216+{
217+ return alloc_tag_read(ct_to_alloc_tag(ct));
218+}
219+
220+/*
221+ * Populates the UAPI allocinfo_tag_data structure with active runtime
222+ * profiling counters extracted from the given kernel codetag.
223+ */
224+static void allocinfo_to_params(struct codetag *ct,
225+ struct allocinfo_tag_data *data,
226+ struct alloc_tag_counters *counters)
227+{
228+ if (ct->modname)
229+ allocinfo_copy_str(data->tag.modname, ct->modname);
230+ else
231+ data->tag.modname[0] = '\0';
232+ allocinfo_copy_str(data->tag.function, ct->function);
233+ allocinfo_copy_str(data->tag.filename, ct->filename);
234+ data->tag.lineno = ct->lineno;
235+ data->counter.bytes = counters->bytes;
236+ data->counter.calls = counters->calls;
237+ data->counter.accurate = !alloc_tag_is_inaccurate(ct_to_alloc_tag(ct));
238+}
239+
240+/*
241+ * Retrieves the unique content ID representing the current allocation tag module
242+ * layout, allowing userspace to detect if modules were loaded / unloaded.
243+ */
244+static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg)
245+{
246+ struct allocinfo_content_id params;
247+
248+ codetag_lock_module_list(alloc_tag_cttype);
249+ params.id = codetag_get_content_id(alloc_tag_cttype);
250+ codetag_unlock_module_list(alloc_tag_cttype);
251+ if (copy_to_user(arg, &params, sizeof(params)))
252+ return -EFAULT;
253+
254+ return 0;
255+}
256+
257+/*
258+ * Verifies whether a given codetag satisfies the active filtering criteria by
259+ * matching its characteristics against the specified filter.
260+ */
261+static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter,
262+ struct alloc_tag_counters *counters,
263+ bool *fetched_counters)
264+{
265+ bool inaccurate;
266+
267+ if (!filter || !filter->mask)
268+ return true;
269+
270+ if (filter->mask & ALLOCINFO_FILTER_MASK_MODNAME) {
271+ /* user wants to filter by modname but ct->modname is NULL */
272+ if (!ct->modname) {
273+ /* validate if user was attempting to filter for built-in allocations */
274+ if (filter->fields.modname[0] != '\0')
275+ return false;
276+ } else if (allocinfo_cmp_str(ct->modname, filter->fields.modname))
277+ return false;
278+ }
279+
280+ if ((filter->mask & ALLOCINFO_FILTER_MASK_FUNCTION) &&
281+ ct->function && allocinfo_cmp_str(ct->function, filter->fields.function))
282+ return false;
283+
284+ if ((filter->mask & ALLOCINFO_FILTER_MASK_FILENAME) &&
285+ ct->filename && allocinfo_cmp_str(ct->filename, filter->fields.filename))
286+ return false;
287+
288+ if ((filter->mask & ALLOCINFO_FILTER_MASK_LINENO) &&
289+ ct->lineno != filter->fields.lineno)
290+ return false;
291+
292+ if (filter->mask & ALLOCINFO_FILTER_MASK_INACCURATE) {
293+ inaccurate = !!(ct->flags & CODETAG_FLAG_INACCURATE);
294+ if (inaccurate != !!(filter->inaccurate))
295+ return false;
296+ }
297+
298+ if (filter->mask & (ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE)) {
299+ if (!*fetched_counters) {
300+ *counters = allocinfo_prefetch_counters(ct);
301+ *fetched_counters = true;
302+ }
303+ if ((filter->mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) &&
304+ counters->bytes < filter->min_size)
305+ return false;
306+ if ((filter->mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) &&
307+ counters->bytes > filter->max_size)
308+ return false;
309+ }
310+
311+ return true;
312+}
313+
314+/*
315+ * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its
316+ * profiling data and returns it to userspace.
317+ */
318+static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg)
319+{
320+ struct allocinfo_private *priv;
321+ struct codetag *ct;
322+ struct allocinfo_get_at params = {0};
323+ __u64 skip_count;
324+ struct alloc_tag_counters counters;
325+ bool fetched_counters;
326+
327+ if (copy_from_user(&params, arg, sizeof(params)))
328+ return -EFAULT;
329+
330+ if (params.filter.mask & ~ALLOCINFO_FILTER_MASKS)
331+ return -EINVAL;
332+
333+ if ((params.filter.mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) &&
334+ (params.filter.mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) &&
335+ params.filter.min_size > params.filter.max_size)
336+ return -EINVAL;
337+
338+ priv = m->private;
339+
340+ mutex_lock(&priv->ioctl_lock);
341+ codetag_lock_module_list(alloc_tag_cttype);
342+
343+ if (params.pos >= codetag_get_count(alloc_tag_cttype)) {
344+ codetag_unlock_module_list(alloc_tag_cttype);
345+ mutex_unlock(&priv->ioctl_lock);
346+ return -ENOENT;
347+ }
348+
349+ skip_count = params.pos;
350+
351+ if (params.filter.mask)
352+ priv->filter = params.filter;
353+ else
354+ priv->filter.mask = 0;
355+
356+ /* Find the codetag */
357+ priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype);
358+ ct = codetag_next_ct(&priv->ioctl_iter);
359+
360+ while (ct) {
361+ fetched_counters = false;
362+ if (matches_filter(ct, &priv->filter, &counters, &fetched_counters)) {
363+ if (skip_count == 0)
364+ break;
365+ skip_count--;
366+ }
367+ ct = codetag_next_ct(&priv->ioctl_iter);
368+ }
369+
370+ if (ct) {
371+ if (!fetched_counters)
372+ counters = allocinfo_prefetch_counters(ct);
373+ allocinfo_to_params(ct, &params.data, &counters);
374+ priv->positioned = true;
375+ }
376+
377+ codetag_unlock_module_list(alloc_tag_cttype);
378+ mutex_unlock(&priv->ioctl_lock);
379+
380+ if (!ct)
381+ return -ENOENT;
382+
383+ if (copy_to_user(arg, &params, sizeof(params)))
384+ return -EFAULT;
385+
386+ return 0;
387+}
388+
389+/*
390+ * Advances the ioctl iterator to the next allocation tag in the sequence and
391+ * returns its profiling data to userspace.
392+ */
393+static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg)
394+{
395+ struct allocinfo_private *priv;
396+ struct codetag *ct;
397+ struct allocinfo_tag_data params;
398+ int ret = 0;
399+ struct alloc_tag_counters counters;
400+ bool fetched_counters;
401+
402+ memset(&params, 0, sizeof(params));
403+ priv = m->private;
404+
405+ mutex_lock(&priv->ioctl_lock);
406+ codetag_lock_module_list(alloc_tag_cttype);
407+
408+ if (!priv->positioned) {
409+ priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype);
410+ priv->positioned = true;
411+ }
412+
413+ ct = codetag_next_ct(&priv->ioctl_iter);
414+ while (ct) {
415+ fetched_counters = false;
416+ if (matches_filter(ct, &priv->filter, &counters, &fetched_counters))
417+ break;
418+ ct = codetag_next_ct(&priv->ioctl_iter);
419+ }
420+
421+ if (ct) {
422+ if (!fetched_counters)
423+ counters = allocinfo_prefetch_counters(ct);
424+ allocinfo_to_params(ct, &params, &counters);
425+ }
426+ if (!ct) {
427+ priv->positioned = false;
428+ ret = -ENOENT;
429+ }
430+ codetag_unlock_module_list(alloc_tag_cttype);
431+ mutex_unlock(&priv->ioctl_lock);
432+
433+ if (ret == 0) {
434+ if (copy_to_user(arg, &params, sizeof(params)))
435+ return -EFAULT;
436+ }
437+ return ret;
438+}
439+
440+/*
441+ * Entry point ioctl function for /proc/allocinfo routing requests to fetch the
442+ * layout content ID, seek to a specific tag, or read sequential tags.
443+ */
444+static long allocinfo_ioctl(struct file *file, unsigned int cmd,
445+ unsigned long __arg)
446+{
447+ void __user *arg = (void __user *)__arg;
448+ int ret;
449+
450+ switch (cmd) {
451+ case ALLOCINFO_IOC_CONTENT_ID:
452+ ret = allocinfo_ioctl_get_content_id(file->private_data, arg);
453+ break;
454+ case ALLOCINFO_IOC_GET_AT:
455+ ret = allocinfo_ioctl_get_at(file->private_data, arg);
456+ break;
457+ case ALLOCINFO_IOC_GET_NEXT:
458+ ret = allocinfo_ioctl_get_next(file->private_data, arg);
459+ break;
460+ default:
461+ ret = -ENOIOCTLCMD;
462+ break;
463+ }
464+
465+ return ret;
466+}
467+
468+#ifdef CONFIG_COMPAT
469+static long allocinfo_compat_ioctl(struct file *file, unsigned int cmd,
470+ unsigned long arg)
471+{
472+ return allocinfo_ioctl(file, cmd, (unsigned long)compat_ptr(arg));
473+}
474+#endif
475+
476+static const struct proc_ops allocinfo_proc_ops = {
477+ .proc_open = allocinfo_open,
478+ .proc_read_iter = seq_read_iter,
479+ .proc_lseek = seq_lseek,
480+ .proc_release = allocinfo_release,
481+ .proc_ioctl = allocinfo_ioctl,
482+#ifdef CONFIG_COMPAT
483+ .proc_compat_ioctl = allocinfo_compat_ioctl,
484+#endif
485+};
486+
145487 size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep)
146488 {
147489 struct codetag_iterator iter;
@@ -961,6 +1303,12 @@ static const struct ctl_table memory_allocation_profiling_sysctls[] = { · 813 unmodified lines
9611303 .mode = 0644,
9621304 .proc_handler = proc_mem_profiling_handler,
9631305 },
1306+ {
1307+ .procname = "mem_profiling_compressed",
1308+ .data = &mem_profiling_compressed,
1309+ .mode = 0444,
1310+ .proc_handler = proc_do_static_key,
1311+ },
9641312 };
9651313
9661314 static void __init sysctl_init(void)
@@ -993,8 +1341,7 @@ static int __init alloc_tag_init(void) · 26 unmodified lines
9931341 return 0;
9941342 }
9951343
996− if (!proc_create_seq_private(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op,
997− sizeof(struct allocinfo_private), NULL)) {
1344+ if (!proc_create(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_proc_ops)) {
9981345 pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME);
9991346 shutdown_mem_profiling(false);
10001347 return -ENOMEM;
mm/arch_numa.crenamed from drivers/base/arch_numa.c+12−0
@@ -105,6 +105,18 @@ static void __init setup_node_to_cpumask_map(void) · 104 unmodified lines
105105 if (nr_node_ids == MAX_NUMNODES)
106106 setup_nr_node_ids();
107107
108+ /*
109+ * This check should never be true but it makes it clear to compilers
110+ * that node_to_cpumask_map is bound by nr_node_ids, avoiding false
111+ * positive fortify warnings when accessing node_to_cpumask_map in the
112+ * for loop below.
113+ */
114+ if (unlikely(nr_node_ids > MAX_NUMNODES)) {
115+ pr_err("nr_node_ids (%u) is larger than MAX_NUMNODES (%u)\n",
116+ nr_node_ids, MAX_NUMNODES);
117+ return;
118+ }
119+
108120 /* allocate and clear the mapping */
109121 for (node = 0; node < nr_node_ids; node++) {
110122 alloc_bootmem_cpumask_var(&node_to_cpumask_map[node]);
mm/cma.c+1−1
@@ -242,7 +242,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size, · 241 unmodified lines
242242 if (name)
243243 strscpy(cma->name, name);
244244 else
245− snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count);
245+ snprintf(cma->name, CMA_MAX_NAME, "cma%d", cma_area_count);
246246
247247 cma->available_count = cma->count = size >> PAGE_SHIFT;
248248 cma->order_per_bit = order_per_bit;
mm/cma.h+1−0
@@ -2,6 +2,7 @@ · 1 unmodified line
22 #ifndef __MM_CMA_H__
33 #define __MM_CMA_H__
44
5+#include <linux/cma.h>
56 #include <linux/debugfs.h>
67 #include <linux/kobject.h>
78
mm/debug_page_alloc.c+4−4
@@ -20,14 +20,14 @@ early_param("debug_pagealloc", early_debug_pagealloc); · 19 unmodified lines
2020
2121 static int __init debug_guardpage_minorder_setup(char *buf)
2222 {
23− unsigned long res;
23+ unsigned int res;
2424
25− if (kstrtoul(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) {
26− pr_err("Bad debug_guardpage_minorder value: %s\n", buf);
25+ if (!buf || kstrtouint(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) {
26+ pr_err("Bad debug_guardpage_minorder value: %s\n", buf ?: "(missing)");
2727 return 0;
2828 }
2929 _debug_guardpage_minorder = res;
30− pr_info("Setting debug_guardpage_minorder to %lu\n", res);
30+ pr_info("Setting debug_guardpage_minorder to %u\n", res);
3131 return 0;
3232 }
3333 early_param("debug_guardpage_minorder", debug_guardpage_minorder_setup);
mm/folio-compat.c+1−0
@@ -41,6 +41,7 @@ void set_page_writeback(struct page *page) · 40 unmodified lines
4141 }
4242 EXPORT_SYMBOL(set_page_writeback);
4343
44+/* Read the comment above folio_mark_dirty() regarding required locks! */
4445 bool set_page_dirty(struct page *page)
4546 {
4647 return folio_mark_dirty(page_folio(page));
mm/folio.c+55−69
@@ -265,73 +265,6 @@ void folio_rotate_reclaimable(struct folio *folio) · 264 unmodified lines
265265 folio_batch_add_and_move(folio, lru_move_tail);
266266 }
267267
268−void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
269− unsigned int nr_io, unsigned int nr_rotated)
270− __releases(lruvec->lru_lock)
271− __releases(rcu)
272−{
273− unsigned long cost;
274−
275− /*
276− * Reflect the relative cost of incurring IO and spending CPU
277− * time on rotations. This doesn't attempt to make a precise
278− * comparison, it just says: if reloads are about comparable
279− * between the LRU lists, or rotations are overwhelmingly
280− * different between them, adjust scan balance for CPU work.
281− */
282− cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
283− if (!cost) {
284− spin_unlock_irq(&lruvec->lru_lock);
285− rcu_read_unlock();
286− return;
287− }
288−
289− for (;;) {
290− unsigned long lrusize;
291−
292− /* Record cost event */
293− if (file)
294− lruvec->file_cost += cost;
295− else
296− lruvec->anon_cost += cost;
297−
298− /*
299− * Decay previous events
300− *
301− * Because workloads change over time (and to avoid
302− * overflow) we keep these statistics as a floating
303− * average, which ends up weighing recent refaults
304− * more than old ones.
305− */
306− lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
307− lruvec_page_state(lruvec, NR_ACTIVE_ANON) +
308− lruvec_page_state(lruvec, NR_INACTIVE_FILE) +
309− lruvec_page_state(lruvec, NR_ACTIVE_FILE);
310−
311− if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) {
312− lruvec->file_cost /= 2;
313− lruvec->anon_cost /= 2;
314− }
315−
316− spin_unlock_irq(&lruvec->lru_lock);
317− lruvec = parent_lruvec(lruvec);
318− if (!lruvec) {
319− rcu_read_unlock();
320− break;
321− }
322− spin_lock_irq(&lruvec->lru_lock);
323− }
324−}
325−
326−void lru_note_cost_refault(struct folio *folio)
327−{
328− struct lruvec *lruvec;
329−
330− lruvec = folio_lruvec_lock_irq(folio);
331− lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio),
332− folio_nr_pages(folio), 0);
333−}
334−
335268 static void lru_activate(struct lruvec *lruvec, struct folio *folio)
336269 {
337270 long nr_pages = folio_nr_pages(folio);
@@ -948,6 +881,52 @@ void lru_add_drain_all(void) · 610 unmodified lines
948881 }
949882 #endif /* CONFIG_SMP */
950883
884+/**
885+ * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold
886+ * folio references
887+ * @folio: The folio.
888+ * @extra_refs: Extra folio references held by the caller.
889+ * @drained: Drain status for batch folio processing.
890+ *
891+ * Drain LRU caches if the caches might hold folio references. Start
892+ * with a local LRU cache drain, to then drain LRU caches on all CPUs if
893+ * local draining was insufficient.
894+ *
895+ * This function detects LRU cache references by comparing the folio refcount
896+ * with the sum of the expected folio refcount + extra references held by the
897+ * caller. Note that we cannot rely on PG_lru to reliably detect all LRU
898+ * cache references, and there are rare scenarios (concurrent folio (un)mapping)
899+ * where this function might miss detecting LRU cache references.
900+ *
901+ * If @drained is not NULL, the function will avoid re-draining LRU caches
902+ * when processing multiple folios in a row. In that case, the variable
903+ * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before
904+ * the first invocation by the caller.
905+ */
906+void lru_cache_drain_for_folio(const struct folio *folio,
907+ unsigned int extra_refs, enum lru_cache_drained *drained)
908+{
909+ if (!folio_may_be_lru_cached(folio))
910+ return;
911+
912+ if (!drained || *drained == LRU_CACHE_NOT_DRAINED) {
913+ if (folio_ref_count(folio) ==
914+ folio_expected_ref_count(folio) + extra_refs)
915+ return;
916+ lru_add_drain();
917+ if (drained)
918+ *drained = LRU_CACHE_DRAINED;
919+ }
920+ if (!drained || *drained == LRU_CACHE_DRAINED) {
921+ if (folio_ref_count(folio) ==
922+ folio_expected_ref_count(folio) + extra_refs)
923+ return;
924+ lru_add_drain_all();
925+ if (drained)
926+ *drained = LRU_CACHE_DRAINED_ALL;
927+ }
928+}
929+
951930 atomic_t lru_disable_count = ATOMIC_INIT(0);
952931
953932 /*
@@ -1151,7 +1130,16 @@ static void lruvec_reparent_lru(struct lruvec *child_lruvec, · 197 unmodified lines
11511130 for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
11521131 unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
11531132
1133+ if (!size)
1134+ continue;
1135+
1136+ /*
1137+ * The folios are accounted to the parent from now on, so the
1138+ * size has to be moved, not just copied. Leaving it behind
1139+ * makes the dying child describe folios it no longer owns.
1140+ */
11541141 mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
1142+ mem_cgroup_update_lru_size(child_lruvec, lru, zid, -(long)size);
11551143 }
11561144 }
11571145
@@ -1162,8 +1150,6 @@ void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int · 4 unmodified lines
11621150
11631151 child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
11641152 parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
1165− parent_lruvec->anon_cost += child_lruvec->anon_cost;
1166− parent_lruvec->file_cost += child_lruvec->file_cost;
11671153
11681154 for_each_lru(lru)
11691155 lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
mm/gup.c+9−14
@@ -1236,7 +1236,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) · 1235 unmodified lines
12361236 * Anon pages in shared mappings are surprising: now
12371237 * just reject it.
12381238 */
1239− if (!is_cow_mapping(vm_flags))
1239+ if (!vma_is_cow_mapping(vma))
12401240 return -EFAULT;
12411241 }
12421242 } else if (!(vm_flags & VM_READ)) {
@@ -2266,13 +2266,14 @@ static unsigned long collect_longterm_unpinnable_folios( · 1023 unmodified lines
22662266 struct list_head *movable_folio_list,
22672267 struct pages_or_folios *pofs)
22682268 {
2269+ enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED;
22692270 unsigned long collected = 0;
22702271 struct folio *folio;
2271− int drained = 0;
22722272 long i = 0;
22732273
22742274 for (folio = pofs_get_folio(pofs, i); folio;
22752275 folio = pofs_next_folio(folio, pofs, &i)) {
2276+ const int pin_refs = folio_has_pincount(folio) ? 1 : GUP_PIN_COUNTING_BIAS;
22762277
22772278 if (folio_is_longterm_pinnable(folio))
22782279 continue;
@@ -2287,18 +2288,12 @@ static unsigned long collect_longterm_unpinnable_folios( · 8 unmodified lines
22872288 continue;
22882289 }
22892290
2290− if (drained == 0 && folio_may_be_lru_cached(folio) &&
2291− folio_ref_count(folio) !=
2292− folio_expected_ref_count(folio) + 1) {
2293− lru_add_drain();
2294− drained = 1;
2295− }
2296− if (drained == 1 && folio_may_be_lru_cached(folio) &&
2297− folio_ref_count(folio) !=
2298− folio_expected_ref_count(folio) + 1) {
2299− lru_add_drain_all();
2300− drained = 2;
2301− }
2291+ /*
2292+ * We drain not only to make the folio_isolate_lru() succeed,
2293+ * but also to remove any other folio references from LRU
2294+ * caches.
2295+ */
2296+ lru_cache_drain_for_folio(folio, pin_refs, &drained);
23022297
23032298 if (!folio_isolate_lru(folio))
23042299 continue;
mm/gup_test.c+62−32
@@ -8,6 +8,12 @@ · 7 unmodified lines
88 #include <linux/highmem.h>
99 #include "gup_test.h"
1010
11+struct gup_test_data {
12+ struct mutex longterm_mutex;
13+ struct page **longterm_pages;
14+ unsigned long longterm_nr_pages;
15+};
16+
1117 static void put_back_pages(unsigned int cmd, struct page **pages,
1218 unsigned long nr_pages, unsigned int gup_test_flags)
1319 {
@@ -208,23 +214,20 @@ free_pages: · 194 unmodified lines
208214 return ret;
209215 }
210216
211−static DEFINE_MUTEX(pin_longterm_test_mutex);
212−static struct page **pin_longterm_test_pages;
213−static unsigned long pin_longterm_test_nr_pages;
214−
215−static inline void pin_longterm_test_stop(void)
217+static inline void pin_longterm_test_stop(struct gup_test_data *data)
216218 {
217− if (pin_longterm_test_pages) {
218− if (pin_longterm_test_nr_pages)
219− unpin_user_pages(pin_longterm_test_pages,
220− pin_longterm_test_nr_pages);
221− kvfree(pin_longterm_test_pages);
222− pin_longterm_test_pages = NULL;
223− pin_longterm_test_nr_pages = 0;
219+ if (data->longterm_pages) {
220+ if (data->longterm_nr_pages)
221+ unpin_user_pages(data->longterm_pages,
222+ data->longterm_nr_pages);
223+ kvfree(data->longterm_pages);
224+ data->longterm_pages = NULL;
225+ data->longterm_nr_pages = 0;
224226 }
225227 }
226228
227−static inline int pin_longterm_test_start(unsigned long arg)
229+static inline int pin_longterm_test_start(struct gup_test_data *data,
230+ unsigned long arg)
228231 {
229232 long nr_pages, cur_pages, addr, remaining_pages;
230233 int gup_flags = FOLL_LONGTERM;
@@ -233,7 +236,7 @@ static inline int pin_longterm_test_start(unsigned long arg) · 2 unmodified lines
233236 int ret = 0;
234237 bool fast;
235238
236− if (pin_longterm_test_pages)
239+ if (data->longterm_pages)
237240 return -EINVAL;
238241
239242 if (copy_from_user(&args, (void __user *)arg, sizeof(args)))
@@ -263,12 +266,12 @@ static inline int pin_longterm_test_start(unsigned long arg) · 23 unmodified lines
263266 return -EINTR;
264267 }
265268
266− pin_longterm_test_pages = pages;
267− pin_longterm_test_nr_pages = 0;
269+ data->longterm_pages = pages;
270+ data->longterm_nr_pages = 0;
268271
269− while (nr_pages - pin_longterm_test_nr_pages) {
270− remaining_pages = nr_pages - pin_longterm_test_nr_pages;
271− addr = args.addr + pin_longterm_test_nr_pages * PAGE_SIZE;
272+ while (nr_pages - data->longterm_nr_pages) {
273+ remaining_pages = nr_pages - data->longterm_nr_pages;
274+ addr = args.addr + data->longterm_nr_pages * PAGE_SIZE;
272275
273276 if (fast)
274277 cur_pages = pin_user_pages_fast(addr, remaining_pages,
@@ -277,11 +280,11 @@ static inline int pin_longterm_test_start(unsigned long arg) · 2 unmodified lines
277280 cur_pages = pin_user_pages(addr, remaining_pages,
278281 gup_flags, pages);
279282 if (cur_pages < 0) {
280− pin_longterm_test_stop();
283+ pin_longterm_test_stop(data);
281284 ret = cur_pages;
282285 break;
283286 }
284− pin_longterm_test_nr_pages += cur_pages;
287+ data->longterm_nr_pages += cur_pages;
285288 pages += cur_pages;
286289 }
287290
@@ -290,19 +293,20 @@ static inline int pin_longterm_test_start(unsigned long arg) · 2 unmodified lines
290293 return ret;
291294 }
292295
293−static inline int pin_longterm_test_read(unsigned long arg)
296+static inline int pin_longterm_test_read(struct gup_test_data *data,
297+ unsigned long arg)
294298 {
295299 __u64 user_addr;
296300 unsigned long i;
297301
298− if (!pin_longterm_test_pages)
302+ if (!data->longterm_pages)
299303 return -EINVAL;
300304
301305 if (copy_from_user(&user_addr, (void __user *)arg, sizeof(user_addr)))
302306 return -EFAULT;
303307
304− for (i = 0; i < pin_longterm_test_nr_pages; i++) {
305− void *addr = kmap_local_page(pin_longterm_test_pages[i]);
308+ for (i = 0; i < data->longterm_nr_pages; i++) {
309+ void *addr = kmap_local_page(data->longterm_pages[i]);
306310 unsigned long ret;
307311
308312 ret = copy_to_user((void __user *)(unsigned long)user_addr, addr,
@@ -318,25 +322,26 @@ static inline int pin_longterm_test_read(unsigned long arg) · 9 unmodified lines
318322 static long pin_longterm_test_ioctl(struct file *filep, unsigned int cmd,
319323 unsigned long arg)
320324 {
325+ struct gup_test_data *data = filep->private_data;
321326 int ret = -EINVAL;
322327
323− if (mutex_lock_killable(&pin_longterm_test_mutex))
328+ if (mutex_lock_killable(&data->longterm_mutex))
324329 return -EINTR;
325330
326331 switch (cmd) {
327332 case PIN_LONGTERM_TEST_START:
328− ret = pin_longterm_test_start(arg);
333+ ret = pin_longterm_test_start(data, arg);
329334 break;
330335 case PIN_LONGTERM_TEST_STOP:
331− pin_longterm_test_stop();
336+ pin_longterm_test_stop(data);
332337 ret = 0;
333338 break;
334339 case PIN_LONGTERM_TEST_READ:
335− ret = pin_longterm_test_read(arg);
340+ ret = pin_longterm_test_read(data, arg);
336341 break;
337342 }
338343
339− mutex_unlock(&pin_longterm_test_mutex);
344+ mutex_unlock(&data->longterm_mutex);
340345 return ret;
341346 }
342347
@@ -375,15 +380,40 @@ static long gup_test_ioctl(struct file *filep, unsigned int cmd, · 32 unmodified lines
375380 return 0;
376381 }
377382
383+static int gup_test_open(struct inode *inode, struct file *file)
384+{
385+ struct gup_test_data *data;
386+ int ret;
387+
388+ data = kzalloc_obj(*data);
389+ if (!data)
390+ return -ENOMEM;
391+
392+ ret = nonseekable_open(inode, file);
393+ if (ret) {
394+ kfree(data);
395+ return ret;
396+ }
397+
398+ mutex_init(&data->longterm_mutex);
399+ file->private_data = data;
400+ return 0;
401+}
402+
378403 static int gup_test_release(struct inode *inode, struct file *file)
379404 {
380− pin_longterm_test_stop();
405+ struct gup_test_data *data = file->private_data;
406+
407+ pin_longterm_test_stop(data);
408+ mutex_destroy(&data->longterm_mutex);
409+ kfree(data);
410+ file->private_data = NULL;
381411
382412 return 0;
383413 }
384414
385415 static const struct file_operations gup_test_fops = {
386− .open = nonseekable_open,
416+ .open = gup_test_open,
387417 .unlocked_ioctl = gup_test_ioctl,
388418 .compat_ioctl = compat_ptr_ioctl,
389419 .release = gup_test_release,
mm/hmm.c+4−1
@@ -670,7 +670,10 @@ static int hmm_do_fault(struct mm_struct *mm, · 669 unmodified lines
670670 ret = handle_mm_fault(vma, addr, fault_flags, NULL);
671671
672672 if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
673− *hmm_vma_walk->locked = false;
673+ if (hmm_vma_walk->locked) /* needed by sparse */
674+ *hmm_vma_walk->locked = false;
675+ else
676+ WARN_ON_ONCE(1); /* broken fault handler */
674677 return HMM_FAULT_UNLOCKED;
675678 }
676679
mm/huge_memory.c+28−18
@@ -1681,7 +1681,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn, · 1680 unmodified lines
16811681 BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
16821682 BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
16831683 (VM_PFNMAP|VM_MIXEDMAP));
1684− BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
1684+ BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
16851685
16861686 pfnmap_setup_cachemode_pfn(pfn, &pgprot);
16871687
@@ -1789,7 +1789,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn, · 101 unmodified lines
17891789 BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
17901790 BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
17911791 (VM_PFNMAP|VM_MIXEDMAP));
1792− BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
1792+ BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
17931793
17941794 pfnmap_setup_cachemode_pfn(pfn, &pgprot);
17951795
@@ -1931,7 +1931,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 135 unmodified lines
19311931 * applied special bit, or we made the PRIVATE mapping be
19321932 * able to wrongly write to the backend MMIO.
19331933 */
1934− VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd));
1934+ VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd));
19351935 goto set_pmd;
19361936 }
19371937
@@ -2052,7 +2052,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 114 unmodified lines
20522052 * TODO: once we support anonymous pages, use
20532053 * folio_try_dup_anon_rmap_*() and split if duplicating fails.
20542054 */
2055− if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) {
2055+ if (vma_is_cow_mapping(vma) && pud_write(pud)) {
20562056 pudp_set_wrprotect(src_mm, addr, src_pud);
20572057 pud = pud_wrprotect(pud);
20582058 }
@@ -2930,7 +2930,7 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · 871 unmodified lines
29302930 }
29312931
29322932 folio_move_anon_rmap(src_folio, dst_vma);
2933− src_folio->index = linear_page_index(dst_vma, dst_addr);
2933+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
29342934
29352935 _dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot);
29362936 /* Follow mremap() behavior and treat the entry dirty after the move */
@@ -4105,34 +4105,42 @@ static int __folio_split(struct folio *folio, unsigned int new_order, · 1168 unmodified lines
41054105 XA_STATE(xas, &folio->mapping->i_pages, folio->index);
41064106 struct folio *end_folio = folio_next(folio);
41074107 bool is_anon = folio_test_anon(folio);
4108+ struct mem_cgroup *memcg, *old_memcg;
41084109 struct address_space *mapping = NULL;
41094110 struct anon_vma *anon_vma = NULL;
41104111 int old_order = folio_order(folio);
41114112 struct folio *new_folio, *next;
41124113 int nr_shmem_dropped = 0;
41134114 enum ttu_flags ttu_flags = 0;
4114− int ret;
41154115 pgoff_t end = 0;
4116+ int ret;
41164117
41174118 VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
41184119 VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
41194120
41204121 if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
41214122 ret = -EINVAL;
4122− goto out;
4123+ goto out_no_memcg;
41234124 }
41244125
41254126 if (new_order >= old_order) {
41264127 ret = -EINVAL;
4127− goto out;
4128+ goto out_no_memcg;
41284129 }
41294130
41304131 ret = folio_check_splittable(folio, new_order, split_type);
41314132 if (ret) {
41324133 VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
4133− goto out;
4134+ goto out_no_memcg;
41344135 }
41354136
4137+ /*
4138+ * switch to folio's memcg as xarray node allocation can happen and
4139+ * needs to charge to it.
4140+ */
4141+ memcg = get_mem_cgroup_from_folio(folio);
4142+ old_memcg = set_active_memcg(memcg);
4143+
41364144 if (is_anon) {
41374145 /*
41384146 * The caller does not necessarily hold an mmap_lock that would
@@ -4275,6 +4283,10 @@ out_unlock: · 136 unmodified lines
42754283 if (mapping)
42764284 i_mmap_unlock_read(mapping);
42774285 out:
4286+ /* restore to caller's old_memcg */
4287+ set_active_memcg(old_memcg);
4288+ mem_cgroup_put(memcg);
4289+out_no_memcg:
42784290 xas_destroy(&xas);
42794291 if (is_pmd_order(old_order))
42804292 count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
@@ -5077,9 +5089,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, · 796 unmodified lines
50775089 return 0;
50785090 }
50795091
5080−void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)
5092+void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio)
50815093 {
5082− struct folio *folio = page_folio(new);
50835094 struct vm_area_struct *vma = pvmw->vma;
50845095 struct mm_struct *mm = vma->vm_mm;
50855096 unsigned long address = pvmw->address;
@@ -5115,11 +5126,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · 29 unmodified lines
51155126 swp_entry_t entry;
51165127
51175128 if (pmd_write(pmde))
5118− entry = make_writable_device_private_entry(
5119− page_to_pfn(new));
5129+ entry = make_writable_device_private_entry(folio_pfn(folio));
51205130 else
5121− entry = make_readable_device_private_entry(
5122− page_to_pfn(new));
5131+ entry = make_readable_device_private_entry(folio_pfn(folio));
51235132 pmde = softleaf_to_pmd(entry);
51245133
51255134 if (pmd_swp_soft_dirty(*pvmw->pmd))
@@ -5134,11 +5143,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) · 8 unmodified lines
51345143 if (!softleaf_is_migration_read(entry))
51355144 rmap_flags |= RMAP_EXCLUSIVE;
51365145
5137− folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags);
5146+ folio_add_anon_rmap_pmd(folio, &folio->page, vma, haddr, rmap_flags);
51385147 } else {
5139− folio_add_file_rmap_pmd(folio, new, vma);
5148+ folio_add_file_rmap_pmd(folio, &folio->page, vma);
51405149 }
5141− VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new));
5150+ VM_WARN_ON_ONCE(pmd_write(pmde) && folio_test_anon(folio) &&
5151+ !PageAnonExclusive(&folio->page));
51425152 set_pmd_at(mm, haddr, pvmw->pmd, pmde);
51435153
51445154 /* No need to invalidate - it was non-present before */
mm/hugetlb.c+7−3
@@ -4888,7 +4888,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, · 4887 unmodified lines
48884888 pte_t *src_pte, *dst_pte, entry;
48894889 struct folio *pte_folio;
48904890 unsigned long addr;
4891− bool cow = is_cow_mapping(src_vma->vm_flags);
4891+ bool cow = vma_is_cow_mapping(src_vma);
48924892 struct hstate *h = hstate_vma(src_vma);
48934893 unsigned long sz = huge_page_size(h);
48944894 unsigned long npages = pages_per_huge_page(h);
@@ -5208,6 +5208,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, · 313 unmodified lines
52085208 bool adjust_reservation;
52095209 unsigned long last_addr_mask;
52105210
5211+ i_mmap_assert_write_locked(vma->vm_file->f_mapping);
52115212 WARN_ON(!is_vm_hugetlb_page(vma));
52125213 BUG_ON(start & ~huge_page_mask(h));
52135214 BUG_ON(end & ~huge_page_mask(h));
@@ -5299,7 +5300,10 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, · 85 unmodified lines
52995300
53005301 /*
53015302 * Restore the reservation for anonymous page, otherwise the
5302− * backing page could be stolen by someone.
5303+ * backing page could be stolen by someone. Restore only on the
5304+ * last unmap, otherwise the owner could empty its resv map
5305+ * while the folio is still mapped by a child. Note that holding
5306+ * i_mmap_lock_write is needed to check the number of mappings.
53035307 * If there we are freeing a surplus, do not set the restore
53045308 * reservation bit.
53055309 */
@@ -5307,7 +5311,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, · 1 unmodified line
53075311
53085312 spin_lock_irq(&hugetlb_lock);
53095313 if (!h->surplus_huge_pages && __vma_private_lock(vma) &&
5310− folio_test_anon(folio)) {
5314+ !folio_mapped(folio) && folio_test_anon(folio)) {
53115315 folio_set_hugetlb_restore_reserve(folio);
53125316 /* Reservation to be adjusted after the spin lock */
53135317 adjust_reservation = true;
mm/hugetlb_cma.c+134−8
@@ -9,6 +9,9 @@ · 8 unmodified lines
99 #include <asm/setup.h>
1010
1111 #include <linux/hugetlb.h>
12+#include <linux/memblock.h>
13+#include <linux/math.h>
14+#include <linux/math64.h>
1215 #include "internal.h"
1316 #include "hugetlb_cma.h"
1417
@@ -18,6 +21,28 @@ static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; · 3 unmodified lines
1821 static bool hugetlb_cma_only __ro_after_init;
1922 static unsigned long hugetlb_cma_size __ro_after_init;
2023
24+static unsigned int hugetlb_cma_percent __initdata;
25+static unsigned int hugetlb_cma_percent_in_node[MAX_NUMNODES] __initdata;
26+
27+#ifdef CONFIG_NUMA
28+static phys_addr_t __init memblock_node_memory_size(int nid)
29+{
30+ struct memblock_region *reg;
31+ phys_addr_t size = 0;
32+
33+ for_each_mem_region(reg) {
34+ if (reg->nid == nid)
35+ size += reg->size;
36+ }
37+ return size;
38+}
39+#else
40+static phys_addr_t __init memblock_node_memory_size(int nid)
41+{
42+ return memblock_phys_mem_size();
43+}
44+#endif
45+
2146 void hugetlb_cma_free_frozen_folio(struct folio *folio)
2247 {
2348 WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)],
@@ -90,14 +115,31 @@ static int __init cmdline_parse_hugetlb_cma(char *p) · 66 unmodified lines
90115 break;
91116
92117 if (s[count] == ':') {
118+ char *next;
119+
93120 if (tmp >= MAX_NUMNODES)
94121 break;
95122 nid = array_index_nospec(tmp, MAX_NUMNODES);
96123
124+ hugetlb_cma_size = 0;
125+ hugetlb_cma_percent = 0;
126+
97127 s += count + 1;
98− tmp = memparse(s, &s);
99− hugetlb_cma_size_in_node[nid] = tmp;
100− hugetlb_cma_size += tmp;
128+ tmp = memparse(s, &next);
129+ if (*next == '%') {
130+ if (tmp > 100) {
131+ pr_warn("hugetlb_cma: invalid percentage %lu for node %d\n",
132+ tmp, nid);
133+ break;
134+ }
135+ hugetlb_cma_percent_in_node[nid] = tmp;
136+ hugetlb_cma_size_in_node[nid] = 0;
137+ s = next + 1;
138+ } else {
139+ hugetlb_cma_size_in_node[nid] = tmp;
140+ hugetlb_cma_percent_in_node[nid] = 0;
141+ s = next;
142+ }
101143
102144 /*
103145 * Skip the separator if have one, otherwise
@@ -108,7 +150,28 @@ static int __init cmdline_parse_hugetlb_cma(char *p) · 4 unmodified lines
108150 else
109151 break;
110152 } else {
111− hugetlb_cma_size = memparse(p, &p);
153+ char *next;
154+
155+ tmp = memparse(p, &next);
156+ if (*next == '%') {
157+ if (tmp > 100) {
158+ pr_warn("hugetlb_cma: invalid percentage %lu\n", tmp);
159+ } else {
160+ hugetlb_cma_percent = tmp;
161+ hugetlb_cma_size = 0;
162+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
163+ hugetlb_cma_size_in_node[nid] = 0;
164+ hugetlb_cma_percent_in_node[nid] = 0;
165+ }
166+ }
167+ } else {
168+ hugetlb_cma_size = tmp;
169+ hugetlb_cma_percent = 0;
170+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
171+ hugetlb_cma_size_in_node[nid] = 0;
172+ hugetlb_cma_percent_in_node[nid] = 0;
173+ }
174+ }
112175 break;
113176 }
114177 }
@@ -134,8 +197,36 @@ void __init hugetlb_cma_reserve(void) · 19 unmodified lines
134197 {
135198 unsigned long size, reserved, per_node, order, gigantic_page_size;
136199 bool node_specific_cma_alloc = false;
200+ bool has_node_specific_param = false;
137201 int nid;
138202
203+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
204+ if (hugetlb_cma_size_in_node[nid] || hugetlb_cma_percent_in_node[nid]) {
205+ has_node_specific_param = true;
206+ break;
207+ }
208+ }
209+
210+ if (has_node_specific_param) {
211+ hugetlb_cma_size = 0;
212+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
213+ if (hugetlb_cma_percent_in_node[nid]) {
214+ phys_addr_t node_gfp_mem = memblock_node_memory_size(nid);
215+ u64 s;
216+
217+ s = mul_u64_u32_div((u64)node_gfp_mem,
218+ hugetlb_cma_percent_in_node[nid],
219+ 100);
220+
221+ hugetlb_cma_size_in_node[nid] = s;
222+ }
223+ hugetlb_cma_size += hugetlb_cma_size_in_node[nid];
224+ }
225+ } else if (hugetlb_cma_percent) {
226+ hugetlb_cma_size = mul_u64_u32_div((u64)memblock_phys_mem_size(),
227+ hugetlb_cma_percent, 100);
228+ }
229+
139230 if (!hugetlb_cma_size)
140231 return;
141232
@@ -154,6 +245,32 @@ void __init hugetlb_cma_reserve(void) · 12 unmodified lines
154245 VM_WARN_ON(order <= MAX_PAGE_ORDER);
155246 gigantic_page_size = PAGE_SIZE << order;
156247
248+ if (hugetlb_cma_percent) {
249+ unsigned long orig_size = hugetlb_cma_size;
250+
251+ hugetlb_cma_size = ALIGN_DOWN(hugetlb_cma_size, PAGE_SIZE << order);
252+ if (orig_size && !hugetlb_cma_size)
253+ pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%)\n",
254+ orig_size / SZ_1M, hugetlb_cma_percent);
255+ } else if (has_node_specific_param) {
256+ hugetlb_cma_size = 0;
257+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
258+ if (hugetlb_cma_percent_in_node[nid]) {
259+ unsigned long orig_size = hugetlb_cma_size_in_node[nid];
260+
261+ hugetlb_cma_size_in_node[nid] =
262+ ALIGN_DOWN(hugetlb_cma_size_in_node[nid],
263+ PAGE_SIZE << order);
264+ if (orig_size && !hugetlb_cma_size_in_node[nid])
265+ pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%) on node %d\n",
266+ orig_size / SZ_1M,
267+ hugetlb_cma_percent_in_node[nid],
268+ nid);
269+ }
270+ hugetlb_cma_size += hugetlb_cma_size_in_node[nid];
271+ }
272+ }
273+
157274 hugetlb_bootmem_set_nodes();
158275
159276 for (nid = 0; nid < MAX_NUMNODES; nid++) {
@@ -194,8 +311,13 @@ void __init hugetlb_cma_reserve(void) · 34 unmodified lines
194311 per_node = DIV_ROUND_UP(hugetlb_cma_size,
195312 nodes_weight(hugetlb_bootmem_nodes));
196313 per_node = round_up(per_node, gigantic_page_size);
197− pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
198− hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
314+ if (hugetlb_cma_percent)
315+ pr_info("hugetlb_cma: reserve %lu MiB (%u%%), up to %lu MiB per node\n",
316+ hugetlb_cma_size / SZ_1M, hugetlb_cma_percent,
317+ per_node / SZ_1M);
318+ else
319+ pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
320+ hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
199321 }
200322
201323 reserved = 0;
@@ -230,8 +352,12 @@ void __init hugetlb_cma_reserve(void) · 28 unmodified lines
230352 }
231353
232354 reserved += size;
233− pr_info("hugetlb_cma: reserved %lu MiB on node %d\n",
234− size / SZ_1M, nid);
355+ if (hugetlb_cma_percent_in_node[nid])
356+ pr_info("hugetlb_cma: reserved %lu MiB (%u%%) on node %d\n",
357+ size / SZ_1M, hugetlb_cma_percent_in_node[nid], nid);
358+ else
359+ pr_info("hugetlb_cma: reserved %lu MiB on node %d\n",
360+ size / SZ_1M, nid);
235361
236362 if (reserved >= hugetlb_cma_size)
237363 break;
mm/hugetlb_cma.h+2−0
@@ -2,6 +2,8 @@ · 1 unmodified line
22 #ifndef _LINUX_HUGETLB_CMA_H
33 #define _LINUX_HUGETLB_CMA_H
44
5+#include <linux/hugetlb.h>
6+
57 #ifdef CONFIG_CMA
68 void hugetlb_cma_free_frozen_folio(struct folio *folio);
79 struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
mm/internal.h+54−23
@@ -41,12 +41,9 @@ void workingset_refault(struct folio *folio, void *shadow); · 40 unmodified lines
4141 void workingset_activation(struct folio *folio);
4242
4343 /* mm/folio.c */
44−void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
45− unsigned int nr_io, unsigned int nr_rotated);
46−void lru_note_cost_refault(struct folio *folio);
4744 void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma);
4845
49−static inline bool folio_may_be_lru_cached(struct folio *folio)
46+static inline bool folio_may_be_lru_cached(const struct folio *folio)
5047 {
5148 /*
5249 * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting.
@@ -945,7 +942,8 @@ folio_within_range(struct folio *folio, struct vm_area_struct *vma, · 892 unmodified lines
945942 return false;
946943
947944 pgoff_folio = folio_pgoff(folio);
948− pgoff_vma_start = vma_start_pgoff(vma);
945+ pgoff_vma_start = folio_test_anon(folio) ?
946+ vma_start_anon_pgoff(vma) : vma_start_pgoff(vma);
949947
950948 if (start < vma->vm_start)
951949 start = vma->vm_start;
@@ -1017,19 +1015,9 @@ void mlock_drain_remote(int cpu); · 65 unmodified lines
10171015
10181016 extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
10191017
1020−/**
1021− * vma_address - Find the virtual address a page range is mapped at
1022− * @vma: The vma which maps this object.
1023− * @pgoff: The page offset within its object.
1024− * @nr_pages: The number of pages to consider.
1025− *
1026− * If any page in this range is mapped by this VMA, return the first address
1027− * where any of these pages appear. Otherwise, return -EFAULT.
1028− */
1029−static inline unsigned long vma_address(const struct vm_area_struct *vma,
1030− pgoff_t pgoff, unsigned long nr_pages)
1018+static inline unsigned long __vma_address(const struct vm_area_struct *vma,
1019+ pgoff_t pgoff, pgoff_t pgoff_start, unsigned long nr_pages)
10311020 {
1032− const pgoff_t pgoff_start = vma_start_pgoff(vma);
10331021 unsigned long address;
10341022
10351023 if (pgoff >= pgoff_start) {
@@ -1047,23 +1035,66 @@ static inline unsigned long vma_address(const struct vm_area_struct *vma, · 11 unmodified lines
10471035 return address;
10481036 }
10491037
1038+/**
1039+ * vma_filebacked_address - Find the virtual address a file-backed page range is
1040+ * mapped at.
1041+ * @vma: The vma which maps this object.
1042+ * @pgoff: The page offset within its object.
1043+ * @nr_pages: The number of pages to consider.
1044+ *
1045+ * Returns: If any page in this range is mapped by this VMA, return the first
1046+ * address where any of these pages appear. Otherwise, return -EFAULT.
1047+ */
1048+static inline unsigned long vma_filebacked_address(const struct vm_area_struct *vma,
1049+ pgoff_t pgoff, unsigned long nr_pages)
1050+{
1051+ VM_WARN_ON_ONCE(vma_is_anonymous(vma));
1052+
1053+ return __vma_address(vma, pgoff, vma_start_pgoff(vma), nr_pages);
1054+}
1055+
1056+/**
1057+ * vma_anon_address - Find the virtual address an anonymous page range is mapped
1058+ * at.
1059+ * @vma: The vma which maps this object.
1060+ * @pgoff_anon: The anonymous page index belonging to the folio.
1061+ * @nr_pages: The number of pages to consider.
1062+ *
1063+ * This is only valid for anonymous or MAP_PRIVATE-mapped file-backed VMAs.
1064+ *
1065+ * Returns: If any page in this range is mapped by this VMA, return the first
1066+ * address where any of these pages appear. Otherwise, return -EFAULT.
1067+ */
1068+static inline unsigned long vma_anon_address(const struct vm_area_struct *vma,
1069+ pgoff_t pgoff_anon, unsigned long nr_pages)
1070+{
1071+ VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
1072+
1073+ return __vma_address(vma, pgoff_anon, vma_start_anon_pgoff(vma), nr_pages);
1074+}
1075+
10501076 /*
1051− * Then at what user virtual address will none of the range be found in vma?
1077+ * At what user virtual address will none of the range be found in vma?
10521078 * Assumes that vma_address() already returned a good starting address.
10531079 */
10541080 static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw)
10551081 {
1056− struct vm_area_struct *vma = pvmw->vma;
1057− pgoff_t pgoff;
1082+ const pgoff_t pgoff_end = pvmw->pgoff + pvmw->nr_pages;
1083+ const struct vm_area_struct *vma = pvmw->vma;
1084+ pgoff_t pgoff_vma_start;
10581085 unsigned long address;
10591086
10601087 /* Common case, plus ->pgoff is invalid for KSM */
10611088 if (pvmw->nr_pages == 1)
10621089 return pvmw->address + PAGE_SIZE;
10631090
1064− pgoff = pvmw->pgoff + pvmw->nr_pages;
1091+ if (pvmw->pgoff_is_anon)
1092+ pgoff_vma_start = vma_start_anon_pgoff(vma);
1093+ else
1094+ pgoff_vma_start = vma_start_pgoff(vma);
1095+
10651096 address = vma->vm_start +
1066− ((pgoff - vma_start_pgoff(vma)) << PAGE_SHIFT);
1097+ ((pgoff_end - pgoff_vma_start) << PAGE_SHIFT);
10671098 /* Check for address beyond vma (or wrapped through 0?) */
10681099 if (address < vma->vm_start || address > vma->vm_end)
10691100 address = vma->vm_end;
@@ -1353,7 +1384,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, · 283 unmodified lines
13531384 * ... because we only care about writable private ("COW")
13541385 * mappings where we have to break COW early.
13551386 */
1356− return is_cow_mapping(vma->vm_flags);
1387+ return vma_is_cow_mapping(vma);
13571388 }
13581389
13591390 /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */
mm/interval_tree.c+2−2
@@ -83,12 +83,12 @@ mapping_rmap_tree_iter_next(struct vm_area_struct *vma, · 82 unmodified lines
8383
8484 static pgoff_t avc_start_pgoff(struct anon_vma_chain *avc)
8585 {
86− return vma_start_pgoff(avc->vma);
86+ return vma_start_anon_pgoff(avc->vma);
8787 }
8888
8989 static pgoff_t avc_last_pgoff(struct anon_vma_chain *avc)
9090 {
91− return vma_last_pgoff(avc->vma);
91+ return vma_last_anon_pgoff(avc->vma);
9292 }
9393
9494 INTERVAL_TREE_DEFINE(struct anon_vma_chain, rb, pgoff_t, rb_subtree_last,
mm/kasan/quarantine.c+11−1
@@ -355,7 +355,12 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache) · 354 unmodified lines
355355 */
356356 on_each_cpu(per_cpu_remove_cache, cache, 1);
357357
358− for_each_online_cpu(cpu) {
358+ /*
359+ * A CPU can go offline after on_each_cpu() returns, leaving cache
360+ * objects on that CPU's shrink list. Scan all possible CPUs to
361+ * drain those lists.
362+ */
363+ for_each_possible_cpu(cpu) {
359364 sq = per_cpu_ptr(&shrink_qlist, cpu);
360365 raw_spin_lock_irqsave(&sq->lock, flags);
361366 qlist_move_cache(&sq->qlist, &to_free, cache);
@@ -365,9 +370,14 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache) · 3 unmodified lines
365370
366371 raw_spin_lock_irqsave(&quarantine_lock, flags);
367372 for (i = 0; i < QUARANTINE_BATCHES; i++) {
373+ size_t old_bytes;
374+
368375 if (qlist_empty(&global_quarantine[i]))
369376 continue;
377+ old_bytes = global_quarantine[i].bytes;
370378 qlist_move_cache(&global_quarantine[i], &to_free, cache);
379+ WRITE_ONCE(quarantine_size, quarantine_size -
380+ (old_bytes - global_quarantine[i].bytes));
371381 /* Scanning whole quarantine can take a while. */
372382 raw_spin_unlock_irqrestore(&quarantine_lock, flags);
373383 cond_resched();
mm/khugepaged.c+65−65
@@ -620,7 +620,7 @@ void __khugepaged_exit(struct mm_struct *mm) · 619 unmodified lines
620620 /*
621621 * This is required to serialize against
622622 * collapse_test_exit() (which is guaranteed to run
623− * under mmap sem read mode). Stop here (after we return all
623+ * under mmap_lock read mode). Stop here (after we return all
624624 * pagetables will be destroyed) until khugepaged has finished
625625 * working on the pagetables under the mmap_lock.
626626 */
@@ -629,6 +629,13 @@ void __khugepaged_exit(struct mm_struct *mm) · 2 unmodified lines
629629 }
630630 }
631631
632+static void collapse_control_init_scan(struct collapse_control *cc)
633+{
634+ memset(cc->node_load, 0, sizeof(cc->node_load));
635+ nodes_clear(cc->alloc_nmask);
636+ bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
637+}
638+
632639 static void release_pte_folio(struct folio *folio)
633640 {
634641 node_stat_mod_folio(folio,
@@ -665,6 +672,24 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, · 30 unmodified lines
665672 }
666673 }
667674
675+static bool folio_pte_referenced(struct folio *folio,
676+ struct vm_area_struct *vma, unsigned long addr, pte_t pteval)
677+{
678+ /* The folio was referenced previously ... */
679+ if (folio_test_young(folio) || folio_test_referenced(folio))
680+ return true;
681+ /* ... or the PTE mapping was recently used */
682+ return pte_young(pteval) || mmu_notifier_test_young(vma->vm_mm, addr);
683+}
684+
685+static void count_collapse_event(unsigned int order, enum vm_event_item vm_event,
686+ enum mthp_stat_item mthp_event)
687+{
688+ if (is_pmd_order(order))
689+ count_vm_event(vm_event);
690+ count_mthp_stat(order, mthp_event);
691+}
692+
668693 static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
669694 unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
670695 unsigned int order, struct list_head *compound_pagelist)
@@ -685,9 +710,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · 14 unmodified lines
685710 if (pte_none_or_zero(pteval)) {
686711 if (++none_or_zero > max_ptes_none) {
687712 result = SCAN_EXCEED_NONE_PTE;
688− if (is_pmd_order(order))
689− count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
690− count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_NONE);
713+ count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE,
714+ MTHP_STAT_COLLAPSE_EXCEED_NONE);
691715 goto out;
692716 }
693717 continue;
@@ -729,9 +753,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · 35 unmodified lines
729753 */
730754 if (++shared > max_ptes_shared) {
731755 result = SCAN_EXCEED_SHARED_PTE;
732− if (is_pmd_order(order))
733− count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
734− count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SHARED);
756+ count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE,
757+ MTHP_STAT_COLLAPSE_EXCEED_SHARED);
735758 goto out;
736759 }
737760 }
@@ -759,8 +782,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · 21 unmodified lines
759782
760783 /*
761784 * We can do it before folio_isolate_lru because the
762− * folio can't be freed from under us. NOTE: PG_lock
763− * is needed to serialize against split_huge_page
785+ * folio can't be freed from under us. NOTE: folio lock
786+ * is needed to serialize against split_huge_page()
764787 * when invoked from the VM.
765788 */
766789 if (!folio_trylock(folio)) {
@@ -786,7 +809,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · 19 unmodified lines
786809 }
787810
788811 /*
789− * Isolate the page to avoid collapsing an hugepage
812+ * Isolate the folio to avoid collapsing a hugepage
790813 * currently in use by the VM.
791814 */
792815 if (!folio_isolate_lru(folio)) {
@@ -803,14 +826,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · 10 unmodified lines
803826 if (folio_test_large(folio))
804827 list_add_tail(&folio->lru, compound_pagelist);
805828 next:
806− /*
807− * If collapse was initiated by khugepaged, check that there is
808− * enough young pte to justify collapsing the page
809− */
810829 if (cc->is_khugepaged &&
811− (pte_young(pteval) || folio_test_young(folio) ||
812− folio_test_referenced(folio) ||
813− mmu_notifier_test_young(vma->vm_mm, addr)))
830+ folio_pte_referenced(folio, vma, addr, pteval))
814831 referenced++;
815832 }
816833
@@ -904,7 +921,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, · 87 unmodified lines
904921 * Re-establish the PMD to point to the original page table
905922 * entry. Restoring PMD needs to be done prior to releasing
906923 * pages. Since pages are still isolated and locked here,
907− * acquiring anon_vma_lock_write is unnecessary.
924+ * acquiring anon_vma_lock_write() is unnecessary.
908925 */
909926 pmd_ptl = pmd_lock(vma->vm_mm, pmd);
910927 pmd_populate(vma->vm_mm, pmd, pmd_pgtable(orig_pmd));
@@ -1078,9 +1095,9 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l · 167 unmodified lines
10781095 return SCAN_VMA_CHECK;
10791096 /*
10801097 * Anon VMA expected, the address may be unmapped then
1081− * remapped to file after khugepaged reaquired the mmap_lock.
1098+ * remapped to file after khugepaged reacquired the mmap_lock.
10821099 *
1083− * thp_vma_allowable_orders may return true for qualified file
1100+ * thp_vma_allowable_orders() may return true for qualified file
10841101 * vmas.
10851102 */
10861103 if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap)))
@@ -1136,7 +1153,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm, · 49 unmodified lines
11361153
11371154 /*
11381155 * Bring missing pages in from swap, to complete THP collapse.
1139− * Only done if khugepaged_scan_pmd believes it is worthwhile.
1156+ * Only done if collapse_scan_pmd() believes it is worthwhile.
11401157 *
11411158 * For mTHP orders the function bails on the first swap entry, because
11421159 * faulting pages back in during collapse could re-populate PTEs that
@@ -1204,7 +1221,7 @@ static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm, · 61 unmodified lines
12041221 pte = NULL;
12051222
12061223 /*
1207− * do_swap_page returns VM_FAULT_RETRY with released mmap_lock.
1224+ * do_swap_page() returns VM_FAULT_RETRY with released mmap_lock.
12081225 * Note we treat VM_FAULT_RETRY as VM_FAULT_ERROR here because
12091226 * we do not retry here and swap entry will remain in pagetable
12101227 * resulting in later failure.
@@ -1247,15 +1264,12 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru · 36 unmodified lines
12471264 folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
12481265 if (!folio) {
12491266 *foliop = NULL;
1250− if (is_pmd_order(order))
1251− count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
1252− count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC_FAILED);
1267+ count_collapse_event(order, THP_COLLAPSE_ALLOC_FAILED,
1268+ MTHP_STAT_COLLAPSE_ALLOC_FAILED);
12531269 return SCAN_ALLOC_HUGE_PAGE_FAIL;
12541270 }
12551271
1256− if (is_pmd_order(order))
1257− count_vm_event(THP_COLLAPSE_ALLOC);
1258− count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC);
1272+ count_collapse_event(order, THP_COLLAPSE_ALLOC, MTHP_STAT_COLLAPSE_ALLOC);
12591273
12601274 if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {
12611275 folio_put(folio);
@@ -1271,7 +1285,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru · 9 unmodified lines
12711285 }
12721286
12731287 /*
1274− * collapse_huge_page expects the mmap_lock to be unlocked before entering and
1288+ * collapse_huge_page() expects the mmap_lock to be unlocked before entering and
12751289 * will always return with the lock unlocked, to avoid holding the mmap_lock
12761290 * while allocating a THP, as that could trigger direct reclaim/compaction.
12771291 * Note that the VMA must be rechecked after grabbing the mmap_lock again.
@@ -1318,7 +1332,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s · 40 unmodified lines
13181332
13191333 if (unmapped) {
13201334 /*
1321− * __collapse_huge_page_swapin will return with mmap_lock
1335+ * __collapse_huge_page_swapin() will return with mmap_lock
13221336 * released when it fails. So we jump out_nolock directly in
13231337 * that case. Continuing to collapse causes inconsistency.
13241338 */
@@ -1331,8 +1345,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s · 6 unmodified lines
13311345 mmap_read_unlock(mm);
13321346 /*
13331347 * Prevent all access to pagetables with the exception of
1334− * gup_fast later handled by the ptep_clear_flush and the VM
1335− * handled by the anon_vma lock + PG_lock.
1348+ * gup_fast later handled by the pmdp_collapse_flush() and the VM
1349+ * handled by the anon_vma lock + folio lock.
13361350 *
13371351 * UFFDIO_MOVE is prevented to race as well thanks to the
13381352 * mmap_lock.
@@ -1389,9 +1403,9 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s · 50 unmodified lines
13891403 spin_lock(pmd_ptl);
13901404 VM_WARN_ON_ONCE(!pmd_none(*pmd));
13911405 /*
1392− * We can only use set_pmd_at when establishing
1406+ * We can only use set_pmd_at() when establishing
13931407 * hugepmds and never for establishing regular pmds that
1394− * points to regular pagetables. Use pmd_populate for that
1408+ * points to regular pagetables. Use pmd_populate() for that
13951409 */
13961410 pmd_populate(mm, pmd, pmd_pgtable(_pmd));
13971411 spin_unlock(pmd_ptl);
@@ -1449,10 +1463,10 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s · 51 unmodified lines
14491463
14501464 result = SCAN_SUCCEED;
14511465 out_up_write:
1452− if (anon_vma_locked)
1453− anon_vma_unlock_write(vma->anon_vma);
14541466 if (pte)
14551467 pte_unmap(pte);
1468+ if (anon_vma_locked)
1469+ anon_vma_unlock_write(vma->anon_vma);
14561470 mmap_write_unlock(mm);
14571471 out_nolock:
14581472 if (folio)
@@ -1617,15 +1631,14 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 158 unmodified lines
16171631 goto out;
16181632 }
16191633
1620− bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
1621− memset(cc->node_load, 0, sizeof(cc->node_load));
1622− nodes_clear(cc->alloc_nmask);
1634+ collapse_control_init_scan(cc);
16231635
16241636 enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
16251637
16261638 /*
16271639 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
1628− * scan all pages to populate the bitmap for mTHP collapse.
1640+ * scan all pages to populate the bitmap for mTHP collapse. The bitmap
1641+ * is then checked again in mthp_collapse() for each attempted order.
16291642 */
16301643 if (enabled_orders != BIT(HPAGE_PMD_ORDER))
16311644 max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
@@ -1647,9 +1660,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 15 unmodified lines
16471660 if (pte_none_or_zero(pteval)) {
16481661 if (++none_or_zero > max_ptes_none) {
16491662 result = SCAN_EXCEED_NONE_PTE;
1650− count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
1651− count_mthp_stat(HPAGE_PMD_ORDER,
1652− MTHP_STAT_COLLAPSE_EXCEED_NONE);
1663+ count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE,
1664+ MTHP_STAT_COLLAPSE_EXCEED_NONE);
16531665 goto out_unmap;
16541666 }
16551667 continue;
@@ -1657,9 +1669,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 1 unmodified line
16571669 if (!pte_present(pteval)) {
16581670 if (++unmapped > max_ptes_swap) {
16591671 result = SCAN_EXCEED_SWAP_PTE;
1660− count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
1661− count_mthp_stat(HPAGE_PMD_ORDER,
1662− MTHP_STAT_COLLAPSE_EXCEED_SWAP);
1672+ count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE,
1673+ MTHP_STAT_COLLAPSE_EXCEED_SWAP);
16631674 goto out_unmap;
16641675 }
16651676 /*
@@ -1716,9 +1727,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 50 unmodified lines
17161727 if (folio_maybe_mapped_shared(folio)) {
17171728 if (++shared > max_ptes_shared) {
17181729 result = SCAN_EXCEED_SHARED_PTE;
1719− count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
1720− count_mthp_stat(HPAGE_PMD_ORDER,
1721− MTHP_STAT_COLLAPSE_EXCEED_SHARED);
1730+ count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE,
1731+ MTHP_STAT_COLLAPSE_EXCEED_SHARED);
17221732 goto out_unmap;
17231733 }
17241734 }
@@ -1749,26 +1759,17 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 24 unmodified lines
17491759 /*
17501760 * Check if the page has any GUP (or other external) pins.
17511761 *
1752− * Here the check may be racy:
1753− * it may see folio_mapcount() > folio_ref_count().
1754− * But such case is ephemeral we could always retry collapse
1755− * later. However it may report false positive if the page
1756− * has excessive GUP pins (i.e. 512). Anyway the same check
1757− * will be done again later the risk seems low.
1762+ * Here the check is racy, but such cases are ephemeral and
1763+ * we can always retry collapse later. Anyway the same
1764+ * check will be done again later, so the risk seems to be low.
17581765 */
17591766 if (folio_expected_ref_count(folio) != folio_ref_count(folio)) {
17601767 result = SCAN_PAGE_COUNT;
17611768 goto out_unmap;
17621769 }
17631770
1764− /*
1765− * If collapse was initiated by khugepaged, check that there is
1766− * enough young pte to justify collapsing the page
1767− */
17681771 if (cc->is_khugepaged &&
1769− (pte_young(pteval) || folio_test_young(folio) ||
1770− folio_test_referenced(folio) ||
1771− mmu_notifier_test_young(vma->vm_mm, addr)))
1772+ folio_pte_referenced(folio, vma, addr, pteval))
17721773 referenced++;
17731774 }
17741775 if (cc->is_khugepaged &&
@@ -1781,7 +1782,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · 6 unmodified lines
17811782 out_unmap:
17821783 pte_unmap_unlock(pte, ptl);
17831784 if (result == SCAN_SUCCEED) {
1784− /* collapse_huge_page expects the lock to be dropped before calling */
1785+ /* collapse_huge_page() expects the lock to be dropped before calling */
17851786 mmap_read_unlock(mm);
17861787 result = mthp_collapse(mm, start_addr, referenced,
17871788 unmapped, cc, enabled_orders);
@@ -2691,8 +2692,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, · 903 unmodified lines
26912692
26922693 present = 0;
26932694 swap = 0;
2694− memset(cc->node_load, 0, sizeof(cc->node_load));
2695− nodes_clear(cc->alloc_nmask);
2695+ collapse_control_init_scan(cc);
26962696 rcu_read_lock();
26972697 xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
26982698 if (xas_retry(&xas, folio))
mm/kmemleak.c+92−13
@@ -151,6 +151,8 @@ struct kmemleak_object { · 150 unmodified lines
151151 int min_count;
152152 /* the total number of pointers found pointing to this object */
153153 int count;
154+ /* consecutive scans the object has been seen unreferenced */
155+ unsigned int unref_scans;
154156 /* checksum for detecting modified objects */
155157 u32 checksum;
156158 depot_stack_handle_t trace_handle;
@@ -175,6 +177,8 @@ struct kmemleak_object { · 18 unmodified lines
175177 #define OBJECT_PHYS (1 << 4)
176178 /* flag set for per-CPU pointers */
177179 #define OBJECT_PERCPU (1 << 5)
180+/* flag set on an object left unreferenced by the full scan, pending confirmation */
181+#define OBJECT_SUSPECT (1 << 6)
178182
179183 /* set when __remove_object() called */
180184 #define DELSTATE_REMOVED (1 << 0)
@@ -232,9 +236,15 @@ static unsigned long max_percpu_addr; · 51 unmodified lines
232236 static struct task_struct *scan_thread;
233237 /* used to avoid reporting of recently allocated objects */
234238 static unsigned long jiffies_min_age;
239+/* consecutive scans an object must stay unreferenced before reporting */
240+static unsigned int min_unref_scans =
241+ IS_ENABLED(CONFIG_DEBUG_KMEMLEAK_VERBOSE) ? 2 : 1;
242+module_param(min_unref_scans, uint, 0644);
235243 static unsigned long jiffies_last_scan;
236244 /* delay between automatic memory scannings */
237245 static unsigned long jiffies_scan_wait;
246+/* number of objects flagged OBJECT_SUSPECT during the current scan */
247+static int nr_suspects;
238248 /* enables or disables the task stacks scanning */
239249 static int kmemleak_stack_scan = 1;
240250 /* protects the memory scanning, parameters and debug/kmemleak file access */
@@ -688,6 +698,7 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) · 447 unmodified lines
688698 object->excess_ref = 0;
689699 object->count = 0; /* white color initially */
690700 object->checksum = ~0;
701+ object->unref_scans = 0;
691702 object->del_state = 0;
692703
693704 /* task information */
@@ -1440,6 +1451,11 @@ static void update_refs(struct kmemleak_object *object) · 746 unmodified lines
14401451 */
14411452 object->count++;
14421453 if (color_gray(object)) {
1454+ /* referenced after all, no longer a suspect */
1455+ if (object->flags & OBJECT_SUSPECT) {
1456+ object->flags &= ~OBJECT_SUSPECT;
1457+ nr_suspects--;
1458+ }
14431459 /* put_object() called when removing from gray_list */
14441460 WARN_ON(!get_object(object));
14451461 list_add_tail(&object->gray_list, &gray_list);
@@ -1571,7 +1587,7 @@ static int scan_large_block(void *start, void *end) · 125 unmodified lines
15711587 if (scan_block(start, next, NULL))
15721588 return 1;
15731589 start = next;
1574− cond_resched();
1590+ cond_resched_tasks_rcu_qs();
15751591 }
15761592
15771593 return 0;
@@ -1608,7 +1624,7 @@ static void scan_object(struct kmemleak_object *object) · 30 unmodified lines
16081624 scan_block(start, end, object);
16091625
16101626 raw_spin_unlock_irqrestore(&object->lock, flags);
1611− cond_resched();
1627+ cond_resched_tasks_rcu_qs();
16121628 raw_spin_lock_irqsave(&object->lock, flags);
16131629 if (!(object->flags & OBJECT_ALLOCATED))
16141630 break;
@@ -1630,7 +1646,7 @@ static void scan_object(struct kmemleak_object *object) · 15 unmodified lines
16301646 break;
16311647
16321648 raw_spin_unlock_irqrestore(&object->lock, flags);
1633− cond_resched();
1649+ cond_resched_tasks_rcu_qs();
16341650 raw_spin_lock_irqsave(&object->lock, flags);
16351651 } while (object->flags & OBJECT_ALLOCATED);
16361652 } else {
@@ -1658,7 +1674,7 @@ static void scan_gray_list(void) · 21 unmodified lines
16581674 */
16591675 object = list_entry(gray_list.next, typeof(*object), gray_list);
16601676 while (&object->gray_list != &gray_list) {
1661− cond_resched();
1677+ cond_resched_tasks_rcu_qs();
16621678
16631679 /* may add new objects to the list */
16641680 if (!scan_should_stop())
@@ -1693,7 +1709,7 @@ static void kmemleak_cond_resched(struct kmemleak_object *object) · 28 unmodified lines
16931709 raw_spin_unlock_irq(&kmemleak_lock);
16941710
16951711 rcu_read_unlock();
1696− cond_resched();
1712+ cond_resched_tasks_rcu_qs();
16971713 rcu_read_lock();
16981714
16991715 raw_spin_lock_irq(&kmemleak_lock);
@@ -1738,7 +1754,7 @@ static void kmemleak_scan_task_stacks(void) · 38 unmodified lines
17381754 }
17391755 put_task_struct(p);
17401756 }
1741− cond_resched();
1757+ cond_resched_tasks_rcu_qs();
17421758 } while (pid && !stop);
17431759 }
17441760
@@ -1844,16 +1860,16 @@ static void dedup_flush(struct xarray *dedup) · 99 unmodified lines
18441860 * kernel's standard allocators. This function must be called with the
18451861 * scan_mutex held.
18461862 */
1847−static void kmemleak_scan(void)
1863+static int __kmemleak_scan(bool full)
18481864 {
18491865 struct kmemleak_object *object;
18501866 struct zone *zone;
18511867 int __maybe_unused i;
1852− struct xarray dedup;
1853− int new_leaks = 0;
18541868 int stop = 0;
18551869
18561870 jiffies_last_scan = jiffies;
1871+ if (full)
1872+ nr_suspects = 0;
18571873
18581874 /* prepare the kmemleak_object's */
18591875 rcu_read_lock();
@@ -1881,8 +1897,13 @@ static void kmemleak_scan(void) · 21 unmodified lines
18811897 __paint_it(object, KMEMLEAK_BLACK);
18821898 }
18831899
1900+ /* referenced last scan: restart the unreferenced run */
1901+ if (!color_white(object))
1902+ object->unref_scans = 0;
18841903 /* reset the reference count (whiten the object) */
18851904 object->count = 0;
1905+ if (full)
1906+ object->flags &= ~OBJECT_SUSPECT;
18861907 if (color_gray(object) && get_object(object))
18871908 list_add_tail(&object->gray_list, &gray_list);
18881909
@@ -1915,7 +1936,7 @@ static void kmemleak_scan(void) · 26 unmodified lines
19151936 struct page *page = pfn_to_online_page(pfn);
19161937
19171938 if (!(pfn & 63))
1918− cond_resched();
1939+ cond_resched_tasks_rcu_qs();
19191940
19201941 if (!page)
19211942 continue;
@@ -1950,6 +1971,10 @@ static void kmemleak_scan(void) · 28 unmodified lines
19501971 scan_gray:
19511972 scan_gray_list();
19521973
1974+ /* a confirmation scan does not look for modified objects */
1975+ if (!full)
1976+ return nr_suspects;
1977+
19531978 /*
19541979 * Check for new or unreferenced objects modified since the previous
19551980 * scan and color them gray until the next scan.
@@ -1972,6 +1997,11 @@ scan_gray: · 16 unmodified lines
19721997 /* color it gray temporarily */
19731998 object->count = object->min_count;
19741999 list_add_tail(&object->gray_list, &gray_list);
2000+ } else if (unreferenced_object(object) &&
2001+ !(object->flags & OBJECT_REPORTED)) {
2002+ /* flag the objects left unreferenced by this scan */
2003+ object->flags |= OBJECT_SUSPECT;
2004+ nr_suspects++;
19752005 }
19762006 raw_spin_unlock_irq(&object->lock);
19772007 }
@@ -1982,12 +2012,62 @@ scan_gray: · 4 unmodified lines
19822012 */
19832013 scan_gray_list();
19842014
2015+ return nr_suspects;
2016+}
2017+
2018+/*
2019+ * Promote a suspected object to a reported leak once it has stayed
2020+ * unreferenced for min_unref_scans consecutive scans. Called with
2021+ * object->lock held; returns true when the object is newly reported.
2022+ */
2023+static bool confirm_leak(struct kmemleak_object *object)
2024+{
2025+ if (!unreferenced_object(object) ||
2026+ !(object->flags & OBJECT_SUSPECT) ||
2027+ (object->flags & OBJECT_REPORTED))
2028+ return false;
2029+
2030+ object->unref_scans += 1;
2031+ if (object->unref_scans < min_unref_scans)
2032+ return false;
2033+
2034+ object->flags |= OBJECT_REPORTED;
2035+ return true;
2036+}
2037+
2038+/*
2039+ * Scan the memory and report the unreferenced objects as leaks. Must be
2040+ * called with the scan_mutex held.
2041+ */
2042+static void kmemleak_scan(void)
2043+{
2044+ struct kmemleak_object *object;
2045+ struct xarray dedup;
2046+ int new_leaks = 0;
2047+
2048+ /*
2049+ * Full scan. Objects left unreferenced are flagged OBJECT_SUSPECT and
2050+ * counted in the return value; nothing to confirm or report otherwise.
2051+ */
2052+ if (!__kmemleak_scan(true))
2053+ return;
2054+
19852055 /*
19862056 * If scanning was stopped do not report any new unreferenced objects.
19872057 */
19882058 if (scan_should_stop())
19892059 return;
19902060
2061+ /*
2062+ * A live object whose only reference is moved by, for example, a
2063+ * concurrent RCU update can be missed for one scan and reported as a
2064+ * transient false positive. Scan again and only report the objects
2065+ * left unreferenced (still flagged OBJECT_SUSPECT) by both scans.
2066+ */
2067+ __kmemleak_scan(false);
2068+ if (scan_should_stop())
2069+ return;
2070+
19912071 /*
19922072 * Scanning result reporting. When verbose printing is enabled, dedupe
19932073 * by stackdepot trace_handle so each unique backtrace is logged once
@@ -2014,9 +2094,8 @@ scan_gray: · 20 unmodified lines
20142094 raw_spin_lock_irq(&object->lock);
20152095 trace_handle = 0;
20162096 dedup_print = false;
2017− if (unreferenced_object(object) &&
2018− !(object->flags & OBJECT_REPORTED)) {
2019− object->flags |= OBJECT_REPORTED;
2097+
2098+ if (confirm_leak(object)) {
20202099 if (kmemleak_verbose) {
20212100 trace_handle = object->trace_handle;
20222101 dedup_print = true;
mm/ksm.c+7−9
@@ -959,10 +959,9 @@ enum ksm_get_folio_flags { · 958 unmodified lines
959959 * seconds or even minutes: much too unresponsive. So instead we use a
960960 * "keyhole reference": access to the ksm page from the stable node peeps
961961 * out through its keyhole to see if that page still holds the right key,
962− * pointing back to this stable node. This relies on freeing a PageAnon
963− * page to reset its page->mapping to NULL, and relies on no other use of
964− * a page to put something that might look like our key in page->mapping.
965− * is on its way to being freed; but it is an anomaly to bear in mind.
962+ * pointing back to this stable node. This relies on freeing an anon
963+ * folio to reset its mapping to NULL, and relies on no other use of a
964+ * folio to put something that might look like our key in its mapping.
966965 */
967966 static struct folio *ksm_get_folio(struct ksm_stable_node *stable_node,
968967 enum ksm_get_folio_flags flags)
@@ -1625,7 +1624,7 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item, · 656 unmodified lines
16251624 * stable_tree, break_cow() will clean it up.
16261625 */
16271626 rmap_item->anon_vma = vma->anon_vma;
1628− rmap_item->linear_page_index = linear_page_index(vma, rmap_item->address);
1627+ rmap_item->linear_page_index = linear_anon_page_index(vma, rmap_item->address);
16291628 get_anon_vma(vma->anon_vma);
16301629 out:
16311630 mmap_read_unlock(mm);
@@ -3060,10 +3059,9 @@ int __ksm_enter(struct mm_struct *mm) · 1428 unmodified lines
30603059
30613060 slot = &mm_slot->slot;
30623061
3062+ spin_lock(&ksm_mmlist_lock);
30633063 /* Check ksm_run too? Would need tighter locking */
30643064 needs_wakeup = list_empty(&ksm_mm_head.slot.mm_node);
3065−
3066− spin_lock(&ksm_mmlist_lock);
30673065 mm_slot_insert(mm_slots_hash, mm, slot);
30683066 /*
30693067 * When KSM_RUN_MERGE (or KSM_RUN_STOP),
@@ -3152,7 +3150,7 @@ struct folio *ksm_might_need_to_copy(struct folio *folio, · 82 unmodified lines
31523150 return folio; /* no need to copy it */
31533151 } else if (!anon_vma) {
31543152 return folio; /* no need to copy it */
3155− } else if (folio->index == linear_page_index(vma, addr) &&
3153+ } else if (folio->index == linear_anon_page_index(vma, addr) &&
31563154 anon_vma->root == vma->anon_vma->root) {
31573155 return folio; /* still no need to copy it */
31583156 }
@@ -3222,7 +3220,7 @@ again: · 63 unmodified lines
32223220 /*
32233221 * Currently, KSM folios are always small folios, so it's
32243222 * sufficient to search for a single page. We can simply use
3225− * the linear_page_index of the original de-duplicate
3223+ * the linear_anon_page_index of the original de-duplicate
32263224 * anonymous page that we remembered in the rmap_item while
32273225 * de-duplicating. Note that mremap() always de-duplicates KSM
32283226 * folios: so if there was mremap() in our parent or our child,
mm/madvise.c+9−8
@@ -32,6 +32,7 @@ · 31 unmodified lines
3232 #include <linux/leafops.h>
3333 #include <linux/shmem_fs.h>
3434 #include <linux/mmu_notifier.h>
35+#include <linux/swap_ops.h>
3536
3637 #include <asm/tlb.h>
3738
@@ -188,7 +189,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, · 150 unmodified lines
188189 unsigned long end, struct mm_walk *walk)
189190 {
190191 struct vm_area_struct *vma = walk->private;
191− struct swap_iocb *splug = NULL;
192+ struct swap_io_ctx ctx = {};
192193 pte_t *ptep = NULL;
193194 spinlock_t *ptl;
194195 unsigned long addr;
@@ -212,15 +213,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, · 17 unmodified lines
212213 pte_unmap_unlock(ptep, ptl);
213214 ptep = NULL;
214215
215− folio = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE,
216− vma, addr, &splug);
216+ folio = read_swap_cache_async(&ctx, entry, GFP_HIGHUSER_MOVABLE,
217+ vma, addr);
217218 if (folio)
218219 folio_put(folio);
219220 }
220221
221222 if (ptep)
222223 pte_unmap_unlock(ptep, ptl);
223− swap_read_unplug(splug);
224+ swap_read_submit(&ctx);
224225 cond_resched();
225226
226227 return 0;
@@ -238,7 +239,7 @@ static void shmem_swapin_range(struct vm_area_struct *vma, · 11 unmodified lines
238239 XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start));
239240 pgoff_t end_index = linear_page_index(vma, end) - 1;
240241 struct folio *folio;
241− struct swap_iocb *splug = NULL;
242+ struct swap_io_ctx ctx = {};
242243
243244 rcu_read_lock();
244245 xas_for_each(&xas, folio, end_index) {
@@ -257,15 +258,15 @@ static void shmem_swapin_range(struct vm_area_struct *vma, · 12 unmodified lines
257258 xas_pause(&xas);
258259 rcu_read_unlock();
259260
260− folio = read_swap_cache_async(entry, mapping_gfp_mask(mapping),
261− vma, addr, &splug);
261+ folio = read_swap_cache_async(&ctx, entry,
262+ mapping_gfp_mask(mapping), vma, addr);
262263 if (folio)
263264 folio_put(folio);
264265
265266 rcu_read_lock();
266267 }
267268 rcu_read_unlock();
268− swap_read_unplug(splug);
269+ swap_read_submit(&ctx);
269270 }
270271 #endif /* CONFIG_SWAP */
271272
mm/memcontrol-v1.c+2−2
@@ -2287,8 +2287,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 2286 unmodified lines
22872287 for_each_online_pgdat(pgdat) {
22882288 mz = memcg->nodeinfo[pgdat->node_id];
22892289
2290− anon_cost += mz->lruvec.anon_cost;
2291− file_cost += mz->lruvec.file_cost;
2290+ anon_cost += mz->lruvec.cost[WORKINGSET_ANON].count;
2291+ file_cost += mz->lruvec.cost[WORKINGSET_FILE].count;
22922292 }
22932293 seq_buf_printf(s, "anon_cost %lu\n", anon_cost);
22942294 seq_buf_printf(s, "file_cost %lu\n", file_cost);
mm/memcontrol-v1.h+1−0
@@ -4,6 +4,7 @@ · 3 unmodified lines
44 #define __MM_MEMCONTROL_V1_H
55
66 #include <linux/cgroup-defs.h>
7+#include <linux/memcontrol.h>
78
89 /* Cgroup v1 and v2 common declarations */
910
mm/memcontrol.c+66−3
@@ -63,6 +63,7 @@ · 62 unmodified lines
6363 #include <linux/sched/isolation.h>
6464 #include <linux/kmemleak.h>
6565 #include "internal.h"
66+#include "swap.h"
6667 #include "swap_table.h"
6768 #include <net/sock.h>
6869 #include <net/ip.h>
@@ -398,6 +399,7 @@ static const unsigned int memcg_node_stat_items[] = { · 329 unmodified lines
398399 NR_SHMEM_THPS,
399400 NR_FILE_THPS,
400401 NR_ANON_THPS,
402+ NR_VMSCAN_WRITE,
401403 NR_VMALLOC,
402404 NR_KERNEL_STACK_KB,
403405 NR_PAGETABLE,
@@ -424,6 +426,8 @@ static const unsigned int memcg_node_stat_items[] = { · 20 unmodified lines
424426 PGSCAN_PROACTIVE,
425427 PGSCAN_ANON,
426428 PGSCAN_FILE,
429+ PGROTATE_ANON,
430+ PGROTATE_FILE,
427431 PGREFILL,
428432 #ifdef CONFIG_HUGETLB_PAGE
429433 NR_HUGETLB,
@@ -507,6 +511,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) · 77 unmodified lines
507511 return x;
508512 }
509513
514+/**
515+ * lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling
516+ * @lruvec: the LRU vector to read from
517+ * @idx: the node_stat_item to read
518+ *
519+ * Returns the raw state[idx] value cast to unsigned long, skipping the
520+ * clamp-negative-to-zero step in lruvec_page_state(). Intended for callers
521+ * that snapshot a monotonically-incremented counter and subtract two
522+ * samples: unsigned modular arithmetic then yields the correct delta across
523+ * a signed-long wraparound (a real hazard on 32-bit) that the clamp would
524+ * otherwise turn into a huge spurious delta.
525+ *
526+ * Do NOT use for non-monotonic page-count reads where a transient negative
527+ * reading from per-CPU delta skew must present as zero.
528+ *
529+ * XXX: This helper (and its node/global peers) exists because some
530+ * monotonically-incremented event counters are stored in
531+ * enum node_stat_item.
532+ */
533+unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec,
534+ enum node_stat_item idx)
535+{
536+ struct mem_cgroup_per_node *pn;
537+ int i;
538+
539+ if (mem_cgroup_disabled())
540+ return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
541+
542+ i = memcg_stats_index(idx);
543+ if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
544+ return 0;
545+
546+ pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
547+ return (unsigned long)READ_ONCE(pn->lruvec_stats->state[i]);
548+}
549+
510550 unsigned long lruvec_page_state_local(struct lruvec *lruvec,
511551 enum node_stat_item idx)
512552 {
@@ -2100,7 +2140,12 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) · 1587 unmodified lines
21002140
21012141 stock_pages = READ_ONCE(stock->nr_pages[i]);
21022142 if (stock_pages >= nr_pages) {
2103− WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages);
2143+ stock_pages -= nr_pages;
2144+ WRITE_ONCE(stock->nr_pages[i], stock_pages);
2145+ if (!stock_pages) {
2146+ css_put(&memcg->css);
2147+ WRITE_ONCE(stock->cached[i], NULL);
2148+ }
21042149 ret = true;
21052150 }
21062151 break;
@@ -2653,6 +2698,19 @@ retry: · 546 unmodified lines
26532698 if (!gfpflags_allow_blocking(gfp_mask))
26542699 goto nomem;
26552700
2701+ /*
2702+ * OOM victim still needs to charge memory to exit. OOM reaper should
2703+ * help but it might fail on mmap_lock contention. If the victim is a
2704+ * large thread group then all exiting threads might compete on oom_lock
2705+ * just to learn that there is nothing really killable anymore. Bail
2706+ * out early and fail the charge to expedite their exit. They are
2707+ * considered fully reclaimed by the oom reaper and they shouldn't
2708+ * contribute further charges.
2709+ */
2710+ if (tsk_is_oom_victim(current) &&
2711+ mm_flags_test(MMF_OOM_SKIP, current->signal->oom_mm))
2712+ goto nomem;
2713+
26562714 __memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
26572715 raised_max_event = true;
26582716
@@ -4177,11 +4235,10 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) · 1518 unmodified lines
41774235 #endif
41784236 page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
41794237 if (parent) {
4180− WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
4181−
41824238 page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl);
41834239 page_counter_init(&memcg->swap, &parent->swap, false);
41844240 #ifdef CONFIG_MEMCG_V1
4241+ WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
41854242 memcg->memory.track_failcnt = !memcg_on_dfl;
41864243 memcg->memsw.track_failcnt = !memcg_on_dfl;
41874244 WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable));
@@ -4801,6 +4858,9 @@ static ssize_t memory_high_write(struct kernfs_open_file *of, · 613 unmodified lines
48014858 unsigned long nr_pages = page_counter_read(&memcg->memory);
48024859 unsigned long reclaimed;
48034860
4861+ if (high != READ_ONCE(memcg->memory.high))
4862+ break;
4863+
48044864 if (nr_pages <= high)
48054865 break;
48064866
@@ -4856,6 +4916,9 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, · 49 unmodified lines
48564916 for (;;) {
48574917 unsigned long nr_pages = page_counter_read(&memcg->memory);
48584918
4919+ if (max != READ_ONCE(memcg->memory.max))
4920+ break;
4921+
48594922 if (nr_pages <= max)
48604923 break;
48614924
mm/memory-failure.c+2−2
@@ -620,7 +620,7 @@ static void add_to_kill_fsdax(struct task_struct *tsk, const struct page *p, · 619 unmodified lines
620620 struct vm_area_struct *vma,
621621 struct list_head *to_kill, pgoff_t pgoff)
622622 {
623− unsigned long addr = vma_address(vma, pgoff, 1);
623+ unsigned long addr = vma_filebacked_address(vma, pgoff, 1);
624624 __add_to_kill(tsk, p, vma, to_kill, addr);
625625 }
626626
@@ -2265,7 +2265,7 @@ static void add_to_kill_pgoff(struct task_struct *tsk, · 1638 unmodified lines
22652265 }
22662266
22672267 /* Check for pgoff not backed by struct page */
2268− tk->addr = vma_address(vma, pgoff, 1);
2268+ tk->addr = vma_filebacked_address(vma, pgoff, 1);
22692269 tk->size_shift = PAGE_SHIFT;
22702270
22712271 if (tk->addr == -EFAULT)
mm/memory.c+101−39
@@ -519,9 +519,52 @@ static bool is_bad_page_map_ratelimited(void) · 518 unmodified lines
519519 return false;
520520 }
521521
522+static void ptval_bytes_to_hex_str(char *buf, size_t buf_size, const void *entry, size_t entry_size)
523+{
524+ if (WARN_ON_ONCE(buf_size < entry_size * 2 + 1)) {
525+ snprintf(buf, buf_size, "overflow");
526+ return;
527+ }
528+
529+ switch (entry_size) {
530+ case sizeof(u32):
531+ snprintf(buf, buf_size, "%08x", *(const u32 *)entry);
532+ break;
533+ case sizeof(u64):
534+ snprintf(buf, buf_size, "%016llx", *(const u64 *)entry);
535+ break;
536+#if defined(__SIZEOF_INT128__)
537+ case sizeof(u128):
538+ snprintf(buf, buf_size, "%016llx%016llx",
539+ (unsigned long long)(*(const u128 *)entry >> 64),
540+ (unsigned long long)*(const u128 *)entry);
541+ break;
542+#endif
543+ default:
544+ snprintf(buf, buf_size, "unsupported");
545+ break;
546+ }
547+}
548+
549+#define ptval_to_str(buf, val) \
550+ do { \
551+ auto __val = (val); \
552+ \
553+ ptval_bytes_to_hex_str((buf), sizeof(buf), &__val, sizeof(__val)); \
554+ } while (0)
555+
556+#if defined(__SIZEOF_INT128__)
557+#define PTVAL_STR_MAX (32 + 1) /* Max 128-bit value in hex + NUL */
558+#else
559+#define PTVAL_STR_MAX (16 + 1) /* Max 64-bit value in hex + NUL */
560+#endif
561+
522562 static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long addr)
523563 {
524− unsigned long long pgdv, p4dv, pudv, pmdv;
564+ char pgd_str[PTVAL_STR_MAX];
565+ char p4d_str[PTVAL_STR_MAX];
566+ char pud_str[PTVAL_STR_MAX];
567+ char pmd_str[PTVAL_STR_MAX];
525568 p4d_t p4d, *p4dp;
526569 pud_t pud, *pudp;
527570 pmd_t pmd, *pmdp;
@@ -532,34 +575,34 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add · 4 unmodified lines
532575 * see locking requirements for print_bad_page_map().
533576 */
534577 pgdp = pgd_offset(mm, addr);
535− pgdv = pgd_val(*pgdp);
578+ ptval_to_str(pgd_str, pgd_val(*pgdp));
536579
537580 if (!pgd_present(*pgdp) || pgd_leaf(*pgdp)) {
538− pr_alert("pgd:%08llx\n", pgdv);
581+ pr_alert("pgd:%s\n", pgd_str);
539582 return;
540583 }
541584
542585 p4dp = p4d_offset(pgdp, addr);
543586 p4d = p4dp_get(p4dp);
544− p4dv = p4d_val(p4d);
587+ ptval_to_str(p4d_str, p4d_val(p4d));
545588
546589 if (!p4d_present(p4d) || p4d_leaf(p4d)) {
547− pr_alert("pgd:%08llx p4d:%08llx\n", pgdv, p4dv);
590+ pr_alert("pgd:%s p4d:%s\n", pgd_str, p4d_str);
548591 return;
549592 }
550593
551594 pudp = pud_offset(p4dp, addr);
552595 pud = pudp_get(pudp);
553− pudv = pud_val(pud);
596+ ptval_to_str(pud_str, pud_val(pud));
554597
555598 if (!pud_present(pud) || pud_leaf(pud)) {
556− pr_alert("pgd:%08llx p4d:%08llx pud:%08llx\n", pgdv, p4dv, pudv);
599+ pr_alert("pgd:%s p4d:%s pud:%s\n", pgd_str, p4d_str, pud_str);
557600 return;
558601 }
559602
560603 pmdp = pmd_offset(pudp, addr);
561604 pmd = pmdp_get(pmdp);
562− pmdv = pmd_val(pmd);
605+ ptval_to_str(pmd_str, pmd_val(pmd));
563606
564607 /*
565608 * Dumping the PTE would be nice, but it's tricky with CONFIG_HIGHPTE,
@@ -567,8 +610,7 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add · 1 unmodified line
567610 * doing another map would be bad. print_bad_page_map() should
568611 * already take care of printing the PTE.
569612 */
570− pr_alert("pgd:%08llx p4d:%08llx pud:%08llx pmd:%08llx\n", pgdv,
571− p4dv, pudv, pmdv);
613+ pr_alert("pgd:%s p4d:%s pud:%s pmd:%s\n", pgd_str, p4d_str, pud_str, pmd_str);
572614 }
573615
574616 /*
@@ -584,25 +626,34 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add · 9 unmodified lines
584626 * page table lock.
585627 */
586628 static void print_bad_page_map(struct vm_area_struct *vma,
587− unsigned long addr, unsigned long long entry, struct page *page,
588− enum pgtable_level level)
629+ unsigned long addr, const void *entry, size_t entry_size,
630+ struct page *page, enum pgtable_level level)
589631 {
590632 struct address_space *mapping;
591− pgoff_t index;
633+ char entry_str[PTVAL_STR_MAX];
634+ pgoff_t index, anon_index;
592635
593636 if (is_bad_page_map_ratelimited())
594637 return;
595638
596639 mapping = vma->vm_file ? vma->vm_file->f_mapping : NULL;
597640 index = linear_page_index(vma, addr);
641+ anon_index = __linear_anon_page_index(vma, addr);
598642
599− pr_alert("BUG: Bad page map in process %s %s:%08llx", current->comm,
600− pgtable_level_to_str(level), entry);
643+ ptval_bytes_to_hex_str(entry_str, sizeof(entry_str), entry, entry_size);
644+ pr_alert("BUG: Bad page map in process %s %s:%s", current->comm,
645+ pgtable_level_to_str(level), entry_str);
601646 __print_bad_page_map_pgtable(vma->vm_mm, addr);
602647 if (page)
603648 dump_page(page, "bad page map");
604− pr_alert("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n",
605− (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index);
649+ pr_alert("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px",
650+ (void *)addr, vma->vm_flags, vma->anon_vma, mapping);
651+ if (!vma_is_cow_mapping(vma) || index == anon_index) {
652+ pr_cont(" index:%lx\n", index);
653+ } else {
654+ pr_cont(" index:%lx (file) %lx (anon)\n", index, anon_index);
655+ }
656+
606657 pr_alert("file:%pD fault:%ps mmap:%ps mmap_prepare: %ps read_folio:%ps\n",
607658 vma->vm_file,
608659 vma->vm_ops ? vma->vm_ops->fault : NULL,
@@ -627,8 +678,13 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) · 18 unmodified lines
627678 }
628679 }
629680
630−#define print_bad_pte(vma, addr, pte, page) \
631− print_bad_page_map(vma, addr, pte_val(pte), page, PGTABLE_LEVEL_PTE)
681+static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
682+ pte_t pte, struct page *page)
683+{
684+ auto entry = pte_val(pte);
685+
686+ print_bad_page_map(vma, addr, &entry, sizeof(entry), page, PGTABLE_LEVEL_PTE);
687+}
632688
633689 /**
634690 * __vm_normal_page() - Get the "struct page" associated with a page table entry.
@@ -636,8 +692,9 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) · 1 unmodified line
636692 * @addr: The address where the page table entry is mapped.
637693 * @pfn: The PFN stored in the page table entry.
638694 * @special: Whether the page table entry is marked "special".
639− * @level: The page table level for error reporting purposes only.
640695 * @entry: The page table entry value for error reporting purposes only.
696+ * @entry_size: The size of @entry.
697+ * @level: The page table level for error reporting purposes only.
641698 *
642699 * "Special" mappings do not wish to be associated with a "struct page" (either
643700 * it doesn't exist, or it exists but they don't want to touch it). In this
@@ -697,7 +754,7 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) · 53 unmodified lines
697754 */
698755 static inline struct page *__vm_normal_page(struct vm_area_struct *vma,
699756 unsigned long addr, unsigned long pfn, bool special,
700− unsigned long long entry, enum pgtable_level level)
757+ const void *entry, size_t entry_size, enum pgtable_level level)
701758 {
702759 if (pgtable_level_has_pxx_special(level)) {
703760 if (unlikely(special)) {
@@ -710,7 +767,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, · 6 unmodified lines
710767 if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn))
711768 return NULL;
712769
713− print_bad_page_map(vma, addr, entry, NULL, level);
770+ print_bad_page_map(vma, addr, entry, entry_size, NULL, level);
714771 return NULL;
715772 }
716773 /*
@@ -730,7 +787,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, · 13 unmodified lines
730787 /* Only CoW'ed anon folios are "normal". */
731788 if (pfn == index)
732789 return NULL;
733− if (!is_cow_mapping(vma->vm_flags))
790+ if (!vma_is_cow_mapping(vma))
734791 return NULL;
735792 }
736793 }
@@ -741,7 +798,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, · 4 unmodified lines
741798
742799 if (unlikely(pfn > highest_memmap_pfn)) {
743800 /* Corrupted page table entry. */
744− print_bad_page_map(vma, addr, entry, NULL, level);
801+ print_bad_page_map(vma, addr, entry, entry_size, NULL, level);
745802 return NULL;
746803 }
747804 /*
@@ -767,8 +824,10 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, · 19 unmodified lines
767824 struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
768825 pte_t pte)
769826 {
827+ auto entry = pte_val(pte);
828+
770829 return __vm_normal_page(vma, addr, pte_pfn(pte), pte_special(pte),
771− pte_val(pte), PGTABLE_LEVEL_PTE);
830+ &entry, sizeof(entry), PGTABLE_LEVEL_PTE);
772831 }
773832
774833 /**
@@ -809,8 +868,10 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, · 34 unmodified lines
809868 struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
810869 pmd_t pmd)
811870 {
871+ auto entry = pmd_val(pmd);
872+
812873 return __vm_normal_page(vma, addr, pmd_pfn(pmd), pmd_special(pmd),
813− pmd_val(pmd), PGTABLE_LEVEL_PMD);
874+ &entry, sizeof(entry), PGTABLE_LEVEL_PMD);
814875 }
815876
816877 /**
@@ -850,8 +911,10 @@ struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, · 33 unmodified lines
850911 struct page *vm_normal_page_pud(struct vm_area_struct *vma,
851912 unsigned long addr, pud_t pud)
852913 {
914+ auto entry = pud_val(pud);
915+
853916 return __vm_normal_page(vma, addr, pud_pfn(pud), pud_special(pud),
854− pud_val(pud), PGTABLE_LEVEL_PUD);
917+ &entry, sizeof(entry), PGTABLE_LEVEL_PUD);
855918 }
856919 #endif
857920
@@ -946,7 +1009,6 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 88 unmodified lines
9461009 pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma,
9471010 struct vm_area_struct *src_vma, unsigned long addr, int *rss)
9481011 {
949− vm_flags_t vm_flags = dst_vma->vm_flags;
9501012 pte_t orig_pte = ptep_get(src_pte);
9511013 softleaf_t entry = softleaf_from_pte(orig_pte);
9521014 pte_t pte = orig_pte;
@@ -970,7 +1032,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 17 unmodified lines
9701032 rss[mm_counter(folio)]++;
9711033
9721034 if (!softleaf_is_migration_read(entry) &&
973− is_cow_mapping(vm_flags)) {
1035+ vma_is_cow_mapping(dst_vma)) {
9741036 /*
9751037 * COW mappings require pages in both parent and child
9761038 * to be set to read. A previously exclusive entry is
@@ -1011,7 +1073,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 34 unmodified lines
10111073 * save and restore device driver state).
10121074 */
10131075 if (softleaf_is_device_private_write(entry) &&
1014− is_cow_mapping(vm_flags)) {
1076+ vma_is_cow_mapping(dst_vma)) {
10151077 entry = make_readable_device_private_entry(
10161078 swp_offset(entry));
10171079 pte = swp_entry_to_pte(entry);
@@ -1026,7 +1088,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 8 unmodified lines
10261088 * exclusive entries currently only support private writable
10271089 * (ie. COW) mappings.
10281090 */
1029− VM_BUG_ON(!is_cow_mapping(src_vma->vm_flags));
1091+ VM_BUG_ON(!vma_is_cow_mapping(src_vma));
10301092 if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte))
10311093 return -EBUSY;
10321094 return -ENOENT;
@@ -1125,7 +1187,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · 92 unmodified lines
11251187 }
11261188
11271189 /* If it's a COW mapping, write protect it both processes. */
1128− if (is_cow_mapping(src_vma->vm_flags) && writable) {
1190+ if (vma_is_cow_mapping(src_vma) && writable) {
11291191 wrprotect_ptes(src_mm, addr, src_pte, nr);
11301192 pte = pte_wrprotect(pte);
11311193 }
@@ -1546,9 +1608,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) · 414 unmodified lines
15461608 * We need to invalidate the secondary MMU mappings only when
15471609 * there could be a permission downgrade on the ptes of the
15481610 * parent mm. And a permission downgrade will only happen if
1549− * is_cow_mapping() returns true.
1611+ * vma_is_cow_mapping() returns true.
15501612 */
1551− is_cow = is_cow_mapping(src_vma->vm_flags);
1613+ is_cow = vma_is_cow_mapping(src_vma);
15521614
15531615 if (is_cow) {
15541616 mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
@@ -2381,7 +2443,7 @@ static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma) · 826 unmodified lines
23812443 if (mm_forbids_zeropage(vma->vm_mm))
23822444 return false;
23832445 /* zeropages in COW mappings are common and unproblematic. */
2384− if (is_cow_mapping(vma->vm_flags))
2446+ if (vma_is_cow_mapping(vma))
23852447 return true;
23862448 /* Mappings that do not allow for writable PTEs are unproblematic. */
23872449 if (!(vma->vm_flags & (VM_WRITE | VM_MAYWRITE)))
@@ -2832,7 +2894,7 @@ vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, · 444 unmodified lines
28322894 BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
28332895 BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
28342896 (VM_PFNMAP|VM_MIXEDMAP));
2835− BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
2897+ BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma));
28362898 BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn));
28372899
28382900 if (addr < vma->vm_start || addr >= vma->vm_end)
@@ -3244,7 +3306,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, · 405 unmodified lines
32443306 unsigned long size)
32453307 {
32463308 const unsigned long end = addr + PAGE_ALIGN(size);
3247− const bool is_cow = is_cow_mapping(vma->vm_flags);
3309+ const bool is_cow = vma_is_cow_mapping(vma);
32483310 int err;
32493311
32503312 err = get_remap_pgoff(is_cow, addr, end, vma->vm_start, vma->vm_end,
@@ -6744,7 +6806,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, · 3493 unmodified lines
67446806 * FAULT_FLAG_UNSHARE only applies to COW mappings. Let's
67456807 * just treat it like an ordinary read-fault otherwise.
67466808 */
6747− if (!is_cow_mapping(vma->vm_flags))
6809+ if (!vma_is_cow_mapping(vma))
67486810 *flags &= ~FAULT_FLAG_UNSHARE;
67496811 } else if (*flags & FAULT_FLAG_WRITE) {
67506812 /* Write faults on read-only mappings are impossible ... */
@@ -6752,7 +6814,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, · 1 unmodified line
67526814 return VM_FAULT_SIGSEGV;
67536815 /* ... and FOLL_FORCE only applies to COW mappings. */
67546816 if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE) &&
6755− !is_cow_mapping(vma->vm_flags)))
6817+ !vma_is_cow_mapping(vma)))
67566818 return VM_FAULT_SIGSEGV;
67576819 }
67586820 #ifdef CONFIG_PER_VMA_LOCK
mm/mempolicy.c+7−16
@@ -844,7 +844,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, · 843 unmodified lines
844844 return false;
845845
846846 /* Also skip shared copy-on-write folios */
847− if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio))
847+ if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
848848 return false;
849849
850850 /* Folios are pinned and can't be migrated */
@@ -2060,24 +2060,15 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, · 1209 unmodified lines
20602060 bool vma_policy_mof(struct vm_area_struct *vma)
20612061 {
20622062 struct mempolicy *pol;
2063+ pgoff_t ilx;
2064+ bool mof;
20632065
2064− if (vma->vm_ops && vma->vm_ops->get_policy) {
2065− bool ret = false;
2066− pgoff_t ilx; /* ignored here */
2067−
2068− pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx);
2069− if (pol && (pol->flags & MPOL_F_MOF))
2070− ret = true;
2071− mpol_cond_put(pol);
2072−
2073− return ret;
2074− }
2075−
2076− pol = vma->vm_policy;
2066+ pol = __get_vma_policy(vma, vma->vm_start, &ilx);
20772067 if (!pol)
20782068 pol = get_task_policy(current);
2079−
2080− return pol->flags & MPOL_F_MOF;
2069+ mof = pol->flags & MPOL_F_MOF;
2070+ mpol_cond_put(pol);
2071+ return mof;
20812072 }
20822073
20832074 bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
mm/migrate.c+10−11
@@ -356,23 +356,18 @@ static bool remove_migration_pte(struct folio *folio, · 355 unmodified lines
356356
357357 while (page_vma_mapped_walk(&pvmw)) {
358358 rmap_t rmap_flags = RMAP_NONE;
359− pte_t old_pte;
360− pte_t pte;
359+ unsigned long idx = 0;
361360 softleaf_t entry;
362361 struct page *new;
363− unsigned long idx = 0;
364−
365− /* pgoff is invalid for ksm pages, but they are never large */
366− if (folio_test_large(folio) && !folio_test_hugetlb(folio))
367− idx = linear_page_index(vma, pvmw.address) - pvmw.pgoff;
368− new = folio_page(folio, idx);
362+ pte_t old_pte;
363+ pte_t pte;
369364
370365 #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
371366 /* PMD-mapped THP migration entry */
372367 if (!pvmw.pte) {
373368 VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) ||
374369 !folio_test_pmd_mappable(folio), folio);
375− remove_migration_pmd(&pvmw, new);
370+ remove_migration_pmd(&pvmw, folio);
376371 continue;
377372 }
378373 #endif
@@ -381,14 +376,18 @@ static bool remove_migration_pte(struct folio *folio, · 2 unmodified lines
381376 pvmw.pte);
382377 else
383378 old_pte = ptep_get(pvmw.pte);
379+
380+ entry = softleaf_from_pte(old_pte);
381+ if (folio_test_large(folio) && !folio_test_hugetlb(folio))
382+ idx = softleaf_to_pfn(entry) - pvmw.pfn;
383+
384384 if (rmap_walk_arg->map_unused_to_zeropage &&
385385 try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx))
386386 continue;
387387
388388 folio_get(folio);
389+ new = folio_page(folio, idx);
389390 pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
390−
391− entry = softleaf_from_pte(old_pte);
392391 if (!softleaf_is_migration_young(entry))
393392 pte = pte_mkold(pte);
394393 if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry))
mm/migrate_device.c+14−1
@@ -882,7 +882,7 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, · 881 unmodified lines
882882
883883 if (flush) {
884884 pte_free(vma->vm_mm, pgtable);
885− flush_cache_page(vma, addr, addr + HPAGE_PMD_SIZE);
885+ flush_cache_range(vma, addr, addr + HPAGE_PMD_SIZE);
886886 pmdp_invalidate(vma, addr, pmdp);
887887 } else {
888888 pgtable_trans_huge_deposit(vma->vm_mm, pmdp, pgtable);
@@ -1193,6 +1193,13 @@ static void __migrate_device_pages(unsigned long *src_pfns, · 304 unmodified lines
11931193 MIGRATE_PFN_COMPOUND);
11941194 goto next;
11951195 }
1196+
1197+ /*
1198+ * reset nr so that only first after-split folio
1199+ * is processed below
1200+ */
1201+ VM_WARN_ON_ONCE(folio_test_large(folio));
1202+ nr = 1;
11961203 } else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) &&
11971204 (dst_pfns[i] & MIGRATE_PFN_COMPOUND) &&
11981205 !(src_pfns[i] & MIGRATE_PFN_COMPOUND)) {
@@ -1232,6 +1239,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, · 33 unmodified lines
12321239 folio = page_folio(migrate_pfn_to_page(src_pfns[i+j]));
12331240 newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j]));
12341241
1242+ /*
1243+ * folio_free_swap() removed the folio from the swap
1244+ * cache. Refresh the saved mapping before migration.
1245+ */
1246+ mapping = folio_mapping(folio);
1247+
12351248 r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt);
12361249 if (r)
12371250 src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE;
mm/mm_init.c+0−1
@@ -1339,7 +1339,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) · 1338 unmodified lines
13391339 for (i = 0; i < NR_VMSCAN_THROTTLE; i++)
13401340 init_waitqueue_head(&pgdat->reclaim_wait[i]);
13411341
1342− pgdat_page_ext_init(pgdat);
13431342 lruvec_init(&pgdat->__lruvec);
13441343 }
13451344
mm/mmzone.c+1−0
@@ -79,6 +79,7 @@ void lruvec_init(struct lruvec *lruvec) · 78 unmodified lines
7979
8080 memset(lruvec, 0, sizeof(struct lruvec));
8181 spin_lock_init(&lruvec->lru_lock);
82+ spin_lock_init(&lruvec->cost_lock);
8283 zswap_lruvec_state_init(lruvec);
8384
8485 for_each_lru(lru)
mm/mremap.c+5−3
@@ -264,7 +264,7 @@ static int move_ptes(struct pagetable_move_control *pmc, · 263 unmodified lines
264264
265265 for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE,
266266 new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) {
267− VM_WARN_ON_ONCE(!pte_none(*new_ptep));
267+ VM_WARN_ON_ONCE(!pte_none(ptep_get(new_ptep)));
268268
269269 nr_ptes = 1;
270270 max_nr_ptes = (old_end - old_addr) >> PAGE_SHIFT;
@@ -1265,7 +1265,9 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) · 994 unmodified lines
12651265 static int copy_vma_and_data(struct vma_remap_struct *vrm,
12661266 struct vm_area_struct **new_vma_ptr)
12671267 {
1268− const unsigned long new_pgoff = linear_page_index(vrm->vma, vrm->addr);
1268+ const pgoff_t new_pgoff = linear_page_index(vrm->vma, vrm->addr);
1269+ const pgoff_t new_anon_pgoff =
1270+ __linear_anon_page_index(vrm->vma, vrm->addr);
12691271 struct vm_area_struct *vma = vrm->vma;
12701272 struct vm_area_struct *new_vma;
12711273 unsigned long moved_len;
@@ -1273,7 +1275,7 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, · 1 unmodified line
12731275 PAGETABLE_MOVE(pmc, NULL, NULL, vrm->addr, vrm->new_addr, vrm->old_len);
12741276
12751277 new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff,
1276− &pmc.need_rmap_locks);
1278+ new_anon_pgoff, &pmc.need_rmap_locks);
12771279 if (!new_vma) {
12781280 vrm_uncharge(vrm);
12791281 *new_vma_ptr = NULL;
mm/nommu.c+4−0
@@ -1393,6 +1393,10 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, · 1392 unmodified lines
13931393 setup_vma_to_mm(vma, mm);
13941394 setup_vma_to_mm(new, mm);
13951395 vma_iter_store_new(vmi, new);
1396+
1397+ /* vmi should point lower address */
1398+ if (new_below)
1399+ vma_next(vmi);
13961400 mm->map_count++;
13971401 return 0;
13981402
mm/page-writeback.c+5−0
@@ -2762,6 +2762,11 @@ EXPORT_SYMBOL(folio_redirty_for_writepage); · 2761 unmodified lines
27622762 * in this folio. Truncation will block on the page table lock as it
27632763 * unmaps pages before removing the folio from its mapping.
27642764 *
2765+ * .. DANGER::
2766+ * Do not use this on a folio obtained from a function like
2767+ * get_user_pages_fast() without holding appropriate locks; you might want to
2768+ * use set_page_dirty_lock() or folio_mark_dirty_lock() instead.
2769+ *
27652770 * Return: True if the folio was newly dirtied, false if it was already dirty.
27662771 */
27672772 bool folio_mark_dirty(struct folio *folio)
mm/page_alloc.c+20−8
@@ -6853,8 +6853,8 @@ static int sysctl_min_slab_ratio_sysctl_handler(const struct ctl_table *table, i · 6852 unmodified lines
68536853
68546854 /*
68556855 * lowmem_reserve_ratio_sysctl_handler - just a wrapper around
6856− * proc_dointvec() so that we can call setup_per_zone_lowmem_reserve()
6857− * whenever sysctl_lowmem_reserve_ratio changes.
6856+ * proc_dointvec_minmax() so that we can call
6857+ * setup_per_zone_lowmem_reserve() when the sysctl is written.
68586858 *
68596859 * The reserve ratio obviously has absolutely no relation with the
68606860 * minimum watermarks. The lowmem reserve ratio can only make sense
@@ -6863,16 +6863,27 @@ static int sysctl_min_slab_ratio_sysctl_handler(const struct ctl_table *table, i · 2 unmodified lines
68636863 static int lowmem_reserve_ratio_sysctl_handler(const struct ctl_table *table,
68646864 int write, void *buffer, size_t *length, loff_t *ppos)
68656865 {
6866− int i;
6866+ struct ctl_table tmp = *table;
6867+ int ratio[ARRAY_SIZE(sysctl_lowmem_reserve_ratio)];
6868+ int rc;
68676869
6868− proc_dointvec_minmax(table, write, buffer, length, ppos);
6870+ if (!write)
6871+ return proc_dointvec_minmax(table, write, buffer, length, ppos);
68696872
6870− for (i = 0; i < MAX_NR_ZONES; i++) {
6871− if (sysctl_lowmem_reserve_ratio[i] < 1)
6872− sysctl_lowmem_reserve_ratio[i] = 0;
6873− }
6873+ /*
6874+ * proc_dointvec_max() works incrementally. Use a buffer and only set
6875+ * the values if all of them parse cleanly.
6876+ */
6877+ memcpy(ratio, sysctl_lowmem_reserve_ratio, sizeof(ratio));
6878+ tmp.data = ratio;
6879+
6880+ rc = proc_dointvec_minmax(&tmp, write, buffer, length, ppos);
6881+ if (rc)
6882+ return rc;
68746883
6884+ memcpy(sysctl_lowmem_reserve_ratio, ratio, sizeof(ratio));
68756885 setup_per_zone_lowmem_reserve();
6886+
68766887 return 0;
68776888 }
68786889
@@ -6971,6 +6982,7 @@ static const struct ctl_table page_alloc_sysctl_table[] = { · 92 unmodified lines
69716982 .maxlen = sizeof(sysctl_lowmem_reserve_ratio),
69726983 .mode = 0644,
69736984 .proc_handler = lowmem_reserve_ratio_sysctl_handler,
6985+ .extra1 = SYSCTL_ZERO,
69746986 },
69756987 #ifdef CONFIG_NUMA
69766988 {
mm/page_ext.c+0−9
@@ -164,11 +164,6 @@ void __init page_ext_init_flatmem_late(void) · 163 unmodified lines
164164 invoke_init_callbacks();
165165 }
166166
167−void __meminit pgdat_page_ext_init(struct pglist_data *pgdat)
168−{
169− pgdat->node_page_ext = NULL;
170−}
171−
172167 static struct page_ext *lookup_page_ext(const struct page *page)
173168 {
174169 unsigned long pfn = page_to_pfn(page);
@@ -494,10 +489,6 @@ oom: · 319 unmodified lines
494489 panic("Out of memory");
495490 }
496491
497−void __meminit pgdat_page_ext_init(struct pglist_data *pgdat)
498−{
499−}
500−
501492 #endif
502493
503494 /**
mm/page_io.c+281−300
@@ -25,57 +25,10 @@ · 24 unmodified lines
2525 #include <linux/sched/task.h>
2626 #include <linux/delayacct.h>
2727 #include <linux/zswap.h>
28+#include <linux/swap_ops.h>
2829 #include "swap.h"
2930 #include "swap_table.h"
3031
31−static void __end_swap_bio_write(struct bio *bio)
32−{
33− struct folio *folio = bio_first_folio_all(bio);
34−
35− if (bio->bi_status) {
36− /*
37− * We failed to write the page out to swap-space.
38− * Re-dirty the page in order to avoid it being reclaimed.
39− * Also print a dire warning that things will go BAD (tm)
40− * very quickly.
41− *
42− * Also clear PG_reclaim to avoid folio_rotate_reclaimable()
43− */
44− folio_mark_dirty(folio);
45− pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n",
46− MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
47− (unsigned long long)bio->bi_iter.bi_sector);
48− folio_clear_reclaim(folio);
49− }
50− folio_end_writeback(folio);
51−}
52−
53−static void end_swap_bio_write(struct bio *bio)
54−{
55− __end_swap_bio_write(bio);
56− bio_put(bio);
57−}
58−
59−static void __end_swap_bio_read(struct bio *bio)
60−{
61− struct folio *folio = bio_first_folio_all(bio);
62−
63− if (bio->bi_status) {
64− pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n",
65− MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
66− (unsigned long long)bio->bi_iter.bi_sector);
67− } else {
68− folio_mark_uptodate(folio);
69− }
70− folio_unlock(folio);
71−}
72−
73−static void end_swap_bio_read(struct bio *bio)
74−{
75− __end_swap_bio_read(bio);
76− bio_put(bio);
77−}
78−
7932 int generic_swapfile_activate(struct swap_info_struct *sis,
8033 struct file *swap_file,
8134 sector_t *span)
@@ -248,7 +201,7 @@ static void swap_zeromap_folio_clear(struct folio *folio) · 166 unmodified lines
248201 * We may have stale swap cache pages in memory: notice
249202 * them here and get rid of the unnecessary final write.
250203 */
251−int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
204+int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
252205 {
253206 int ret = 0;
254207
@@ -295,39 +248,44 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug) · 40 unmodified lines
295248 }
296249 rcu_read_unlock();
297250
298− __swap_writepage(folio, swap_plug);
251+ __swap_writepage(ctx, folio);
299252 return 0;
300253 out_unlock:
301254 folio_unlock(folio);
302255 return ret;
303256 }
304257
305−static inline void count_swpout_vm_event(struct folio *folio)
258+#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
259+static struct cgroup_subsys_state *folio_memcg_blkg_css(struct folio *folio)
306260 {
307−#ifdef CONFIG_TRANSPARENT_HUGEPAGE
308− if (unlikely(folio_test_pmd_mappable(folio))) {
309− count_memcg_folio_events(folio, THP_SWPOUT, 1);
310− count_vm_event(THP_SWPOUT);
261+ return cgroup_e_css(folio_memcg(folio)->css.cgroup, &io_cgrp_subsys);
262+}
263+
264+static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio)
265+{
266+ bool can_merge = true;
267+
268+ if (folio_memcg_charged(folio) != folio_memcg_charged(prev_folio))
269+ return false;
270+ if (folio_memcg_charged(folio)) {
271+ rcu_read_lock();
272+ if (folio_memcg_blkg_css(folio) !=
273+ folio_memcg_blkg_css(prev_folio))
274+ can_merge = false;
275+ rcu_read_unlock();
311276 }
312−#endif
313− count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
314− count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
315− count_vm_events(PSWPOUT, folio_nr_pages(folio));
277+ return can_merge;
316278 }
317279
318−#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
319280 static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
320281 {
321282 struct cgroup_subsys_state *css;
322− struct mem_cgroup *memcg;
323283
324284 if (!folio_memcg_charged(folio))
325285 return;
326−
327286 rcu_read_lock();
328− memcg = folio_memcg(folio);
329− css = cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys);
330− if (!css || !css_tryget(css))
287+ css = folio_memcg_blkg_css(folio);
288+ if (css && !css_tryget(css))
331289 css = NULL;
332290 rcu_read_unlock();
333291
@@ -336,15 +294,13 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) · 2 unmodified lines
336294 css_put(css);
337295 }
338296 #else
297+static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio)
298+{
299+ return true;
300+}
339301 #define bio_associate_blkg_from_page(bio, folio) do { } while (0)
340302 #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */
341303
342−struct swap_iocb {
343− struct kiocb iocb;
344− struct bio_vec bvecs[SWAP_CLUSTER_MAX];
345− int nr_bvecs;
346− int len;
347−};
348304 static mempool_t *sio_pool;
349305
350306 int sio_pool_init(void)
@@ -360,168 +316,74 @@ int sio_pool_init(void) · 9 unmodified lines
360316 return 0;
361317 }
362318
363−static void sio_write_complete(struct kiocb *iocb, long ret)
319+static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
320+ int rw)
364321 {
365− struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
366− struct page *page = sio->bvecs[0].bv_page;
367− int p;
368−
369− if (ret != sio->len) {
370− /*
371− * In the case of swap-over-nfs, this can be a
372− * temporary failure if the system has limited
373− * memory for allocating transmit buffers.
374− * Mark the page dirty and avoid
375− * folio_rotate_reclaimable but rate-limit the
376− * messages.
377− */
378− pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
379− ret, swap_dev_pos(page_swap_entry(page)));
380− for (p = 0; p < sio->nr_bvecs; p++) {
381− page = sio->bvecs[p].bv_page;
382− set_page_dirty(page);
383− ClearPageReclaim(page);
384− }
385− }
386−
387− for (p = 0; p < sio->nr_bvecs; p++)
388− end_page_writeback(sio->bvecs[p].bv_page);
322+ struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
323+ struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
324+ struct folio *prev_folio = bvec_folio(last_bv);
325+ size_t prev_folio_size = folio_size(prev_folio);
389326
390− mempool_free(sio, sio_pool);
327+ if (ctx->sis != sis)
328+ return false;
329+ return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
391330 }
392331
393−static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
332+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
394333 {
395− struct swap_iocb *sio = swap_plug ? *swap_plug : NULL;
396334 struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
397− struct file *swap_file = sis->swap_file;
398− loff_t pos = swap_dev_pos(folio->swap);
399−
400− count_swpout_vm_event(folio);
401− folio_start_writeback(folio);
402− folio_unlock(folio);
403− if (sio) {
404− if (sio->iocb.ki_filp != swap_file ||
405− sio->iocb.ki_pos + sio->len != pos) {
406− swap_write_unplug(sio);
407− sio = NULL;
408− }
335+ struct swap_iocb *sio = ctx->sio;
336+
337+ if (sio && !swap_can_merge(ctx, folio, rw)) {
338+ if (rw == WRITE)
339+ swap_write_submit(ctx);
340+ else
341+ swap_read_submit(ctx);
342+ sio = ctx->sio;
409343 }
344+
410345 if (!sio) {
411− sio = mempool_alloc(sio_pool, GFP_NOIO);
412− init_sync_kiocb(&sio->iocb, swap_file);
413− sio->iocb.ki_complete = sio_write_complete;
414− sio->iocb.ki_pos = pos;
346+ ctx->sis = sis;
347+ ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
415348 sio->nr_bvecs = 0;
416349 sio->len = 0;
417350 }
418351 bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
419352 sio->len += folio_size(folio);
420− sio->nr_bvecs += 1;
421− if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !swap_plug) {
422− swap_write_unplug(sio);
423− sio = NULL;
424− }
425− if (swap_plug)
426− *swap_plug = sio;
427−}
428−
429−static void swap_writepage_bdev_sync(struct folio *folio,
430− struct swap_info_struct *sis)
431−{
432− struct bio_vec bv;
433− struct bio bio;
434−
435− bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_WRITE | REQ_SWAP);
436− bio.bi_iter.bi_sector = swap_folio_sector(folio);
437− bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
438−
439− bio_associate_blkg_from_page(&bio, folio);
440− count_swpout_vm_event(folio);
441−
442− folio_start_writeback(folio);
443− folio_unlock(folio);
444353
445− submit_bio_wait(&bio);
446− __end_swap_bio_write(&bio);
447−}
448−
449−static void swap_writepage_bdev_async(struct folio *folio,
450− struct swap_info_struct *sis)
451−{
452− struct bio *bio;
453−
454− bio = bio_alloc(sis->bdev, 1, REQ_OP_WRITE | REQ_SWAP, GFP_NOIO);
455− bio->bi_iter.bi_sector = swap_folio_sector(folio);
456− bio->bi_end_io = end_swap_bio_write;
457− bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
458−
459− bio_associate_blkg_from_page(bio, folio);
460− count_swpout_vm_event(folio);
461− folio_start_writeback(folio);
462− folio_unlock(folio);
463− submit_bio(bio);
464−}
465−
466−void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
467−{
468− struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
469−
470− VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
471− /*
472− * ->flags can be updated non-atomically,
473− * but that will never affect SWP_FS_OPS, so the data_race
474− * is safe.
475− */
476− if (data_race(sis->flags & SWP_FS_OPS))
477− swap_writepage_fs(folio, swap_plug);
478354 /*
479− * ->flags can be updated non-atomically,
480− * but that will never affect SWP_SYNCHRONOUS_IO, so the data_race
481− * is safe.
355+ * Write out the iocb if we filled it, or if the device is synchronous.
356+ *
357+ * The latter is to work around expectations in the classic LRU code
358+ * which make synchronous clearing of the folio writeback flag in the
359+ * reclaim path beneficial.
482360 */
483− else if (data_race(sis->flags & SWP_SYNCHRONOUS_IO))
484− swap_writepage_bdev_sync(folio, sis);
485− else
486− swap_writepage_bdev_async(folio, sis);
487−}
488−
489−void swap_write_unplug(struct swap_iocb *sio)
490−{
491− struct iov_iter from;
492− struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
493− int ret;
494−
495− iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len);
496− ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
497− if (ret != -EIOCBQUEUED)
498− sio_write_complete(&sio->iocb, ret);
361+ if (++sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) ||
362+ (rw == WRITE && (sis->flags & SWP_SYNCHRONOUS_IO))) {
363+ if (rw == WRITE)
364+ swap_write_submit(ctx);
365+ else
366+ swap_read_submit(ctx);
367+ }
499368 }
500369
501−static void sio_read_complete(struct kiocb *iocb, long ret)
370+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
502371 {
503− struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
504− int p;
505−
506− if (ret == sio->len) {
507− for (p = 0; p < sio->nr_bvecs; p++) {
508− struct folio *folio = bvec_folio(&sio->bvecs[p]);
509−
510− count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
511− count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
512− folio_mark_uptodate(folio);
513− folio_unlock(folio);
514− }
515− count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
516− } else {
517− for (p = 0; p < sio->nr_bvecs; p++) {
518− struct folio *folio = bvec_folio(&sio->bvecs[p]);
372+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
519373
520− folio_unlock(folio);
521− }
522− pr_alert_ratelimited("Read-error on swap-device\n");
374+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
375+ if (unlikely(folio_test_pmd_mappable(folio))) {
376+ count_memcg_folio_events(folio, THP_SWPOUT, 1);
377+ count_vm_event(THP_SWPOUT);
523378 }
524− mempool_free(sio, sio_pool);
379+#endif
380+ count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
381+ count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
382+ count_vm_events(PSWPOUT, folio_nr_pages(folio));
383+
384+ folio_start_writeback(folio);
385+ folio_unlock(folio);
386+ swap_add_folio(ctx, folio, WRITE);
525387 }
526388
527389 /*
@@ -587,78 +449,7 @@ static bool swap_read_folio_zeromap(struct folio *folio) · 59 unmodified lines
587449 return true;
588450 }
589451
590−static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
591−{
592− struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
593− struct swap_iocb *sio = NULL;
594− loff_t pos = swap_dev_pos(folio->swap);
595−
596− if (plug)
597− sio = *plug;
598− if (sio) {
599− if (sio->iocb.ki_filp != sis->swap_file ||
600− sio->iocb.ki_pos + sio->len != pos) {
601− swap_read_unplug(sio);
602− sio = NULL;
603− }
604− }
605− if (!sio) {
606− sio = mempool_alloc(sio_pool, GFP_KERNEL);
607− init_sync_kiocb(&sio->iocb, sis->swap_file);
608− sio->iocb.ki_pos = pos;
609− sio->iocb.ki_complete = sio_read_complete;
610− sio->nr_bvecs = 0;
611− sio->len = 0;
612− }
613− bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
614− sio->len += folio_size(folio);
615− sio->nr_bvecs += 1;
616− if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !plug) {
617− swap_read_unplug(sio);
618− sio = NULL;
619− }
620− if (plug)
621− *plug = sio;
622−}
623−
624−static void swap_read_folio_bdev_sync(struct folio *folio,
625− struct swap_info_struct *sis)
626−{
627− struct bio_vec bv;
628− struct bio bio;
629−
630− bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ);
631− bio.bi_iter.bi_sector = swap_folio_sector(folio);
632− bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
633− /*
634− * Keep this task valid during swap readpage because the oom killer may
635− * attempt to access it in the page fault retry time check.
636− */
637− get_task_struct(current);
638− count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
639− count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
640− count_vm_events(PSWPIN, folio_nr_pages(folio));
641− submit_bio_wait(&bio);
642− __end_swap_bio_read(&bio);
643− put_task_struct(current);
644−}
645−
646−static void swap_read_folio_bdev_async(struct folio *folio,
647− struct swap_info_struct *sis)
648−{
649− struct bio *bio;
650−
651− bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL);
652− bio->bi_iter.bi_sector = swap_folio_sector(folio);
653− bio->bi_end_io = end_swap_bio_read;
654− bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
655− count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
656− count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
657− count_vm_events(PSWPIN, folio_nr_pages(folio));
658− submit_bio(bio);
659−}
660−
661−void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
452+void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
662453 {
663454 struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
664455 bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
@@ -691,14 +482,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug) · 26 unmodified lines
691482
692483 /* We have to read from slower devices. Increase zswap protection. */
693484 zswap_folio_swapin(folio);
694−
695− if (data_race(sis->flags & SWP_FS_OPS)) {
696− swap_read_folio_fs(folio, plug);
697− } else if (synchronous) {
698− swap_read_folio_bdev_sync(folio, sis);
699− } else {
700− swap_read_folio_bdev_async(folio, sis);
701− }
485+ swap_add_folio(ctx, folio, READ);
702486
703487 finish:
704488 if (workingset) {
@@ -708,14 +492,211 @@ finish: · 3 unmodified lines
708492 delayacct_swapin_end();
709493 }
710494
711−void __swap_read_unplug(struct swap_iocb *sio)
495+static void swap_write_end(struct swap_iocb *sio, bool failed)
712496 {
713− struct iov_iter from;
714− struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
715− int ret;
497+ int p;
498+
499+ for (p = 0; p < sio->nr_bvecs; p++) {
500+ struct page *page = sio->bvecs[p].bv_page;
501+
502+ if (failed) {
503+ set_page_dirty(page);
504+ ClearPageReclaim(page);
505+ }
506+ end_page_writeback(page);
507+ }
508+ mempool_free(sio, sio_pool);
509+}
510+
511+static void swap_fs_write_complete(struct kiocb *iocb, long ret)
512+{
513+ struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
514+ bool failed = ret != sio->len;
515+
516+ if (failed) {
517+ struct page *page = sio->bvecs[0].bv_page;
518+
519+ /*
520+ * In the case of swap-over-nfs, this can be a temporary failure
521+ * if the system has limited memory for allocating transmit
522+ * buffers. Mark the page dirty and avoid
523+ * folio_rotate_reclaimable but rate-limit the messages.
524+ */
525+ pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
526+ ret, swap_dev_pos(page_swap_entry(page)));
527+ }
528+
529+ swap_write_end(sio, failed);
530+}
716531
717− iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len);
718− ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
719− if (ret != -EIOCBQUEUED)
720− sio_read_complete(&sio->iocb, ret);
532+static void end_swap_bio_write(struct bio *bio)
533+{
534+ struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio);
535+ bool failed = !!bio->bi_status;
536+
537+ if (failed)
538+ pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n",
539+ MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
540+ (unsigned long long)bio->bi_iter.bi_sector);
541+ bio_uninit(bio);
542+ swap_write_end(sio, failed);
543+}
544+
545+static void swap_read_end(struct swap_iocb *sio, bool failed)
546+{
547+ int p;
548+
549+ for (p = 0; p < sio->nr_bvecs; p++) {
550+ struct folio *folio = bvec_folio(&sio->bvecs[p]);
551+
552+ if (!failed) {
553+ count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
554+ count_memcg_folio_events(folio, PSWPIN,
555+ folio_nr_pages(folio));
556+ folio_mark_uptodate(folio);
557+ }
558+ folio_unlock(folio);
559+ }
560+
561+ if (!failed)
562+ count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
563+
564+ mempool_free(sio, sio_pool);
565+}
566+
567+static void swap_fs_read_complete(struct kiocb *iocb, long ret)
568+{
569+ struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
570+ bool failed = ret != sio->len;
571+
572+ if (failed)
573+ pr_alert_ratelimited("Read-error on swap-device\n");
574+ swap_read_end(sio, failed);
575+}
576+
577+static void swap_bio_read_end_io(struct bio *bio)
578+{
579+ struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio);
580+ bool failed = !!bio->bi_status;
581+
582+ if (failed)
583+ pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n",
584+ MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
585+ (unsigned long long)bio->bi_iter.bi_sector);
586+ bio_uninit(bio);
587+ swap_read_end(sio, failed);
588+}
589+
590+static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
591+{
592+ struct swap_iocb *sio = ctx->sio;
593+ struct bio *bio = &sio->bio;
594+
595+ bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
596+ REQ_OP_WRITE | REQ_SWAP);
597+ bio->bi_iter.bi_size = sio->len;
598+ bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
599+ bio_associate_blkg_from_page(bio, bio_first_folio_all(bio));
600+
601+ if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
602+ submit_bio_wait(bio);
603+ end_swap_bio_write(bio);
604+ } else {
605+ bio->bi_end_io = end_swap_bio_write;
606+ submit_bio(bio);
607+ }
608+}
609+
610+static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
611+{
612+ struct swap_iocb *sio = ctx->sio;
613+ struct bio *bio = &sio->bio;
614+
615+ bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
616+ REQ_OP_READ);
617+ bio->bi_iter.bi_size = sio->len;
618+ bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
619+
620+ if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
621+ /*
622+ * Keep this task valid during swap readpage because the oom
623+ * killer may attempt to access it in the page fault retry
624+ * time check.
625+ */
626+ get_task_struct(current);
627+ submit_bio_wait(bio);
628+ swap_bio_read_end_io(bio);
629+ put_task_struct(current);
630+ } else {
631+ bio->bi_end_io = swap_bio_read_end_io;
632+ submit_bio(bio);
633+ }
634+}
635+
636+static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
637+ size_t prev_folio_size, int rw)
638+{
639+ if (swap_folio_sector(folio) !=
640+ swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
641+ return false;
642+ if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
643+ return false;
644+ return true;
645+}
646+
647+const struct swap_ops swap_bdev_ops = {
648+ .submit_write = swap_bdev_submit_write,
649+ .submit_read = swap_bdev_submit_read,
650+ .can_merge = swap_bdev_can_merge,
651+};
652+
653+void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
654+{
655+ struct swap_iocb *sio = ctx->sio;
656+
657+ init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
658+ sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
659+ if (rw == WRITE)
660+ sio->iocb.ki_complete = swap_fs_write_complete;
661+ else
662+ sio->iocb.ki_complete = swap_fs_read_complete;
663+
664+ iov_iter_bvec(iter, rw == WRITE ? ITER_SOURCE : ITER_DEST,
665+ sio->bvecs, sio->nr_bvecs, sio->len);
666+}
667+EXPORT_SYMBOL_GPL(swap_fs_prepare_rw);
668+
669+bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
670+ size_t prev_folio_size, int rw)
671+{
672+ return swap_dev_pos(folio->swap) ==
673+ swap_dev_pos(prev_folio->swap) + prev_folio_size;
674+}
675+EXPORT_SYMBOL_GPL(swap_fs_can_merge);
676+
677+int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops)
678+{
679+ sis->ops = ops;
680+ return add_swap_extent(sis, 0, sis->max, 0);
681+}
682+EXPORT_SYMBOL_GPL(swap_fs_activate);
683+
684+void swap_write_submit(struct swap_io_ctx *ctx)
685+{
686+ if (!ctx->sio)
687+ return;
688+ count_vm_events(NRSWPOUT, 1);
689+ ctx->sis->ops->submit_write(ctx);
690+ ctx->sio = NULL;
691+ ctx->sis = NULL;
692+}
693+
694+void swap_read_submit(struct swap_io_ctx *ctx)
695+{
696+ if (!ctx->sio)
697+ return;
698+ count_vm_events(NRSWPIN, 1);
699+ ctx->sis->ops->submit_read(ctx);
700+ ctx->sio = NULL;
701+ ctx->sis = NULL;
721702 }
mm/page_reporting.c+18−10
@@ -48,7 +48,11 @@ MODULE_PARM_DESC(page_reporting_order, "Set page reporting order"); · 47 unmodified lines
4848 */
4949 EXPORT_SYMBOL_GPL(page_reporting_order);
5050
51−#define PAGE_REPORTING_DELAY (2 * HZ)
51+static unsigned int page_reporting_delay_ms = 2 * MSEC_PER_SEC;
52+module_param(page_reporting_delay_ms, uint, 0644);
53+MODULE_PARM_DESC(page_reporting_delay_ms,
54+ "Set page reporting delay in milliseconds");
55+
5256 static struct page_reporting_dev_info __rcu *pr_dev_info __read_mostly;
5357
5458 enum {
@@ -57,6 +61,13 @@ enum { · 2 unmodified lines
5761 PAGE_REPORTING_ACTIVE
5862 };
5963
64+/* schedule work for page reporting */
65+static void page_reporting_schedule_work(struct page_reporting_dev_info *prdev)
66+{
67+ queue_delayed_work(system_freezable_wq, &prdev->work,
68+ msecs_to_jiffies(page_reporting_delay_ms));
69+}
70+
6071 /* request page reporting */
6172 static void
6273 __page_reporting_request(struct page_reporting_dev_info *prdev)
@@ -77,12 +88,10 @@ __page_reporting_request(struct page_reporting_dev_info *prdev) · 14 unmodified lines
7788 return;
7889
7990 /*
80− * Delay the start of work to allow a sizable queue to build. For
81− * now we are limiting this to running no more than once every
82− * couple of seconds.
91+ * Delay the start of work to allow a sizable queue to build.
92+ * We limit this based on page_reporting_delay_ms.
8393 */
84− queue_delayed_work(system_freezable_wq, &prdev->work,
85− PAGE_REPORTING_DELAY);
94+ page_reporting_schedule_work(prdev);
8695 }
8796
8897 /* notify prdev of free page reporting request */
@@ -337,13 +346,12 @@ static void page_reporting_process(struct work_struct *work) · 248 unmodified lines
337346 err_out:
338347 /*
339348 * If the state has reverted back to requested then there may be
340− * additional pages to be processed. We will defer for 2s to allow
341− * more pages to accumulate.
349+ * additional pages to be processed. We will defer by
350+ * page_reporting_delay_ms to allow more pages to accumulate.
342351 */
343352 state = atomic_cmpxchg(&prdev->state, state, PAGE_REPORTING_IDLE);
344353 if (state == PAGE_REPORTING_REQUESTED)
345− queue_delayed_work(system_freezable_wq, &prdev->work,
346− PAGE_REPORTING_DELAY);
354+ page_reporting_schedule_work(prdev);
347355 }
348356
349357 static DEFINE_MUTEX(page_reporting_mutex);
mm/page_vma_mapped.c+5−1
@@ -356,6 +356,7 @@ unsigned long page_mapped_in_vma(const struct page *page, · 355 unmodified lines
356356 struct vm_area_struct *vma)
357357 {
358358 const struct folio *folio = page_folio(page);
359+ const pgoff_t pgoff = page_pgoff(folio, page);
359360 struct page_vma_mapped_walk pvmw = {
360361 .pfn = page_to_pfn(page),
361362 .nr_pages = 1,
@@ -363,7 +364,10 @@ unsigned long page_mapped_in_vma(const struct page *page, · 1 unmodified line
363364 .flags = PVMW_SYNC,
364365 };
365366
366− pvmw.address = vma_address(vma, page_pgoff(folio, page), 1);
367+ if (folio_test_anon(folio))
368+ pvmw.address = vma_anon_address(vma, pgoff, 1);
369+ else
370+ pvmw.address = vma_filebacked_address(vma, pgoff, 1);
367371 if (pvmw.address == -EFAULT)
368372 goto out;
369373 if (!page_vma_mapped_walk(&pvmw))
mm/pgalloc-track.h+3−0
@@ -2,6 +2,9 @@ · 1 unmodified line
22 #ifndef _LINUX_PGALLOC_TRACK_H
33 #define _LINUX_PGALLOC_TRACK_H
44
5+#include <linux/mm.h>
6+#include <linux/pgtable.h>
7+
58 #if defined(CONFIG_MMU)
69 static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd,
710 unsigned long address,
mm/rmap.c+25−18
@@ -350,7 +350,7 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, · 349 unmodified lines
350350 * Now link the anon_vma's back to the newly inserted AVCs.
351351 * Note that all anon_vma's share the same root.
352352 */
353− anon_vma_lock_write(src->anon_vma);
353+ anon_vma_lock_write(active_anon_vma);
354354 list_for_each_entry_reverse(avc, &dst->anon_vma_chain, same_vma) {
355355 struct anon_vma *anon_vma = avc->anon_vma;
356356
@@ -865,14 +865,15 @@ unsigned long page_address_in_vma(const struct folio *folio, · 508 unmodified lines
865865 if (!vma->anon_vma || !anon_vma ||
866866 vma->anon_vma->root != anon_vma->root)
867867 return -EFAULT;
868+ /* KSM folios don't reach here because of the !anon_vma check */
869+ return vma_anon_address(vma, page_pgoff(folio, page), 1);
868870 } else if (!vma->vm_file) {
869871 return -EFAULT;
870872 } else if (vma->vm_file->f_mapping != folio->mapping) {
871873 return -EFAULT;
872874 }
873875
874− /* KSM folios don't reach here because of the !anon_vma check */
875− return vma_address(vma, page_pgoff(folio, page), 1);
876+ return vma_filebacked_address(vma, page_pgoff(folio, page), 1);
876877 }
877878
878879 /*
@@ -907,7 +908,7 @@ out: · 28 unmodified lines
907908 struct folio_referenced_arg {
908909 int mapcount;
909910 int referenced;
910− vm_flags_t vm_flags;
911+ vma_flags_t vma_flags;
911912 struct mem_cgroup *memcg;
912913 };
913914
@@ -926,7 +927,7 @@ static bool folio_referenced_one(struct folio *folio, · 12 unmodified lines
926927 address = pvmw.address;
927928 nr = 1;
928929
929− if (vma->vm_flags & VM_LOCKED) {
930+ if (vma_test(vma, VMA_LOCKED_BIT)) {
930931 ptes++;
931932 pra->mapcount--;
932933
@@ -947,7 +948,7 @@ static bool folio_referenced_one(struct folio *folio, · 14 unmodified lines
947948 /* Restore the mlock which got missed */
948949 mlock_vma_folio(folio, vma);
949950 page_vma_mapped_walk_done(&pvmw);
950− pra->vm_flags |= VM_LOCKED;
951+ vma_flags_set(&pra->vma_flags, VMA_LOCKED_BIT);
951952 return false; /* To break the loop */
952953 }
953954
@@ -1015,8 +1016,11 @@ static bool folio_referenced_one(struct folio *folio, · 61 unmodified lines
10151016 referenced++;
10161017
10171018 if (referenced) {
1019+ vma_flags_t vma_flags = vma->flags;
1020+
10181021 pra->referenced++;
1019− pra->vm_flags |= vma->vm_flags & ~VM_LOCKED;
1022+ vma_flags_clear(&vma_flags, VMA_LOCKED_BIT);
1023+ vma_flags_set_mask(&pra->vma_flags, vma_flags);
10201024 }
10211025
10221026 if (!pra->mapcount)
@@ -1054,7 +1058,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) · 31 unmodified lines
10541058 * @folio: The folio to test.
10551059 * @is_locked: Caller holds lock on the folio.
10561060 * @memcg: target memory cgroup
1057− * @vm_flags: A combination of all the vma->vm_flags which referenced the folio.
1061+ * @vma_flags: A combination of all the vma->flags which referenced the folio.
10581062 *
10591063 * Quick test_and_clear_referenced for all mappings of a folio,
10601064 *
@@ -1062,7 +1066,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) · 1 unmodified line
10621066 * the function bailed out due to rmap lock contention.
10631067 */
10641068 int folio_referenced(struct folio *folio, int is_locked,
1065− struct mem_cgroup *memcg, vm_flags_t *vm_flags)
1069+ struct mem_cgroup *memcg, vma_flags_t *vma_flags)
10661070 {
10671071 bool we_locked = false;
10681072 struct folio_referenced_arg pra = {
@@ -1078,7 +1082,7 @@ int folio_referenced(struct folio *folio, int is_locked, · 9 unmodified lines
10781082 };
10791083
10801084 VM_WARN_ON_ONCE_FOLIO(folio_is_zone_device(folio), folio);
1081− *vm_flags = 0;
1085+ vma_flags_clear_all(vma_flags);
10821086 if (!pra.mapcount)
10831087 return 0;
10841088
@@ -1092,7 +1096,7 @@ int folio_referenced(struct folio *folio, int is_locked, · 7 unmodified lines
10921096 }
10931097
10941098 rmap_walk(folio, &rwc);
1095− *vm_flags = pra.vm_flags;
1099+ vma_flags_set_mask(vma_flags, pra.vma_flags);
10961100
10971101 if (we_locked)
10981102 folio_unlock(folio);
@@ -1239,6 +1243,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, · 140 unmodified lines
12391243 .vma = vma,
12401244 .address = address,
12411245 .flags = PVMW_SYNC,
1246+ .pgoff_is_anon = false,
12421247 };
12431248
12441249 state->cleaned += page_vma_mkclean_one(&pvmw);
@@ -1316,12 +1321,13 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, · 71 unmodified lines
13161321 .pgoff = pgoff,
13171322 .vma = vma,
13181323 .flags = PVMW_SYNC,
1324+ .pgoff_is_anon = false,
13191325 };
13201326
13211327 if (invalid_mkclean_vma(vma, NULL))
13221328 return 0;
13231329
1324− pvmw.address = vma_address(vma, pgoff, nr_pages);
1330+ pvmw.address = vma_filebacked_address(vma, pgoff, nr_pages);
13251331 VM_BUG_ON_VMA(pvmw.address == -EFAULT, vma);
13261332
13271333 return page_vma_mkclean_one(&pvmw);
@@ -1482,7 +1488,7 @@ static void __folio_set_anon(struct folio *folio, struct vm_area_struct *vma, · 154 unmodified lines
14821488 */
14831489 anon_vma = (void *) anon_vma + FOLIO_MAPPING_ANON;
14841490 WRITE_ONCE(folio->mapping, (struct address_space *) anon_vma);
1485− folio->index = linear_page_index(vma, address);
1491+ folio->index = linear_anon_page_index(vma, address);
14861492 }
14871493
14881494 /**
@@ -1509,8 +1515,8 @@ static void __page_check_anon_rmap(const struct folio *folio, · 20 unmodified lines
15091515 */
15101516 VM_BUG_ON_FOLIO(folio_anon_vma(folio)->root != vma->anon_vma->root,
15111517 folio);
1512− VM_BUG_ON_PAGE(page_pgoff(folio, page) != linear_page_index(vma, address),
1513− page);
1518+ VM_BUG_ON_PAGE(page_pgoff(folio, page) !=
1519+ linear_anon_page_index(vma, address), page);
15141520 }
15151521
15161522 static __always_inline void __folio_add_anon_rmap(struct folio *folio,
@@ -3035,10 +3041,10 @@ static void rmap_walk_anon(struct folio *folio, · 1518 unmodified lines
30353041 pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
30363042 anon_rmap_tree_foreach(avc, anon_vma, pgoff_start, pgoff_end) {
30373043 struct vm_area_struct *vma = avc->vma;
3038− unsigned long address = vma_address(vma, pgoff_start,
3044+ const unsigned long address = vma_anon_address(vma, pgoff_start,
30393045 folio_nr_pages(folio));
30403046
3041− VM_BUG_ON_VMA(address == -EFAULT, vma);
3047+ VM_WARN_ON_ONCE_VMA(address == -EFAULT, vma);
30423048 cond_resched();
30433049
30443050 if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
@@ -3098,7 +3104,8 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping, · 53 unmodified lines
30983104 }
30993105 lookup:
31003106 mapping_rmap_tree_foreach(vma, mapping, pgoff_start, pgoff_end) {
3101− unsigned long address = vma_address(vma, pgoff_start, nr_pages);
3107+ unsigned long address = vma_filebacked_address(vma, pgoff_start,
3108+ nr_pages);
31023109
31033110 VM_BUG_ON_VMA(address == -EFAULT, vma);
31043111 cond_resched();
mm/shmem.c+24−5
@@ -41,6 +41,7 @@ · 40 unmodified lines
4141 #include <linux/swapfile.h>
4242 #include <linux/iversion.h>
4343 #include <linux/unicode.h>
44+#include <linux/swap_ops.h>
4445 #include "swap.h"
4546
4647 static struct vfsmount *shm_mnt __ro_after_init;
@@ -1597,13 +1598,13 @@ start_over: · 1550 unmodified lines
15971598
15981599 /**
15991600 * shmem_writeout - Write the folio to swap
1601+ * @ctx: swap I/O context
16001602 * @folio: The folio to write
1601− * @plug: swap plug
16021603 * @folio_list: list to put back folios on split
16031604 *
16041605 * Move the folio from the page cache to the swap cache.
16051606 */
1606−int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
1607+int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
16071608 struct list_head *folio_list)
16081609 {
16091610 struct address_space *mapping = folio->mapping;
@@ -1715,7 +1716,7 @@ try_split: · 105 unmodified lines
17151716 shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
17161717
17171718 BUG_ON(folio_mapped(folio));
1718− error = swap_writeout(folio, plug);
1719+ error = swap_writeout(ctx, folio);
17191720 if (error != AOP_WRITEPAGE_ACTIVATE) {
17201721 /* folio has been unlocked */
17211722 return error;
@@ -1751,7 +1752,17 @@ redirty: · 29 unmodified lines
17511752 folio_mark_dirty(folio);
17521753 return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */
17531754 }
1754−EXPORT_SYMBOL_GPL(shmem_writeout);
1755+
1756+int shmem_write_folio(struct folio *folio)
1757+{
1758+ struct swap_io_ctx ctx = {};
1759+ int err;
1760+
1761+ err = shmem_writeout(&ctx, folio, NULL);
1762+ swap_write_submit(&ctx);
1763+ return err;
1764+}
1765+EXPORT_SYMBOL_GPL(shmem_write_folio);
17551766
17561767 #if defined(CONFIG_NUMA) && defined(CONFIG_TMPFS)
17571768 static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
@@ -3617,6 +3628,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, · 1859 unmodified lines
36173628 struct shmem_inode_info *info = SHMEM_I(inode);
36183629 struct shmem_falloc shmem_falloc;
36193630 pgoff_t start, index, end, undo_fallocend;
3631+ loff_t aligned_end;
36203632 int error;
36213633
36223634 if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
@@ -3673,8 +3685,15 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, · 50 unmodified lines
36733685 goto out;
36743686 }
36753687
3688+ /* Check for wraparound */
3689+ if (check_add_overflow(offset + len, (loff_t)PAGE_SIZE - 1,
3690+ &aligned_end)) {
3691+ error = -EFBIG;
3692+ goto out;
3693+ }
3694+
36763695 start = offset >> PAGE_SHIFT;
3677− end = (offset + len + PAGE_SIZE - 1) >> PAGE_SHIFT;
3696+ end = aligned_end >> PAGE_SHIFT;
36783697 /* Try to avoid a swapstorm if len is impossible to satisfy */
36793698 if (sbinfo->max_blocks && end - start > sbinfo->max_blocks) {
36803699 error = -ENOSPC;
mm/show_mem.c+6−6
@@ -309,8 +309,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, · 308 unmodified lines
309309 " min:%lukB"
310310 " low:%lukB"
311311 " high:%lukB"
312− " reserved_highatomic:%luKB"
313− " free_highatomic:%luKB"
312+ " reserved_highatomic:%lukB"
313+ " free_highatomic:%lukB"
314314 " active_anon:%lukB"
315315 " inactive_anon:%lukB"
316316 " active_file:%lukB"
@@ -323,7 +323,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, · 6 unmodified lines
323323 " mlocked:%lukB"
324324 " bounce:%lukB"
325325 " free_pcp:%lukB"
326− " local_pcp:%ukB"
326+ " local_pcp:%lukB"
327327 " free_cma:%lukB"
328328 "\n",
329329 zone->name,
@@ -350,7 +350,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, · 20 unmodified lines
350350 K(zone_page_state(zone, NR_MLOCK)),
351351 0UL,
352352 K(free_pcp),
353− K(this_cpu_read(zone->per_cpu_pageset->count)),
353+ K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)),
354354 K(zone_page_state(zone, NR_FREE_CMA_PAGES)));
355355 printk("lowmem_reserve[]:");
356356 for (i = 0; i < MAX_NR_ZONES; i++)
@@ -400,7 +400,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, · 43 unmodified lines
400400 hugetlb_show_meminfo_node(nid);
401401 }
402402
403− printk("%ld total pagecache pages\n", global_node_page_state(NR_FILE_PAGES));
403+ printk("%lu total pagecache pages\n", global_node_page_state(NR_FILE_PAGES));
404404
405405 show_swap_cache_info();
406406 }
@@ -430,7 +430,7 @@ void __show_mem(unsigned int filter, const nodemask_t *nodemask, · 23 unmodified lines
430430 printk("%lu pages cma reserved\n", totalcma_pages);
431431 #endif
432432 #ifdef CONFIG_MEMORY_FAILURE
433− printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages));
433+ printk("%ld pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages));
434434 #endif
435435 #ifdef CONFIG_MEM_ALLOC_PROFILING
436436 static DEFINE_SPINLOCK(mem_alloc_profiling_spinlock);
mm/shuffle.h+2−0
@@ -2,7 +2,9 @@ · 1 unmodified line
22 // Copyright(c) 2018 Intel Corporation. All rights reserved.
33 #ifndef _MM_SHUFFLE_H
44 #define _MM_SHUFFLE_H
5+
56 #include <linux/jump_label.h>
7+#include <linux/mmzone.h>
68
79 #define SHUFFLE_ORDER MAX_PAGE_ORDER
810
mm/sparse.c+4−14
@@ -104,7 +104,7 @@ int __meminit sparse_index_init(unsigned long section_nr, int nid) · 103 unmodified lines
104104 return 0;
105105 }
106106 #else /* !SPARSEMEM_EXTREME */
107−int sparse_index_init(unsigned long section_nr, int nid)
107+int __meminit sparse_index_init(unsigned long section_nr, int nid)
108108 {
109109 return 0;
110110 }
@@ -127,7 +127,7 @@ static inline int sparse_early_nid(struct mem_section *section) · 16 unmodified lines
127127 }
128128
129129 /* Validate the physical addressing limitations of the model */
130−static void __meminit mminit_validate_memmodel_limits(unsigned long *start_pfn,
130+static void __init mminit_validate_memmodel_limits(unsigned long *start_pfn,
131131 unsigned long *end_pfn)
132132 {
133133 unsigned long max_sparsemem_pfn = (DIRECT_MAP_PHYSMEM_END + 1) >> PAGE_SHIFT;
@@ -213,16 +213,6 @@ static void __init memblocks_present(void) · 79 unmodified lines
213213 memory_present(nid, start, end);
214214 }
215215
216−static unsigned long usemap_size(void)
217−{
218− return BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS) * sizeof(unsigned long);
219−}
220−
221−size_t mem_section_usage_size(void)
222−{
223− return sizeof(struct mem_section_usage) + usemap_size();
224−}
225−
226216 #ifdef CONFIG_SPARSEMEM_VMEMMAP
227217 unsigned long __init section_map_size(void)
228218 {
@@ -249,8 +239,8 @@ void __weak __meminit vmemmap_populate_print_last(void) · 20 unmodified lines
249239 {
250240 }
251241
252−static void *sparse_usagebuf __meminitdata;
253−static void *sparse_usagebuf_end __meminitdata;
242+static void *sparse_usagebuf __initdata;
243+static void *sparse_usagebuf_end __initdata;
254244
255245 /*
256246 * Helper function that is used for generic section initialization, and
mm/sparse.h+6−0
@@ -47,6 +47,12 @@ static inline void __section_mark_present(struct mem_section *ms, · 46 unmodified lines
4747
4848 ms->section_mem_map |= SECTION_MARKED_PRESENT;
4949 }
50+
51+static inline size_t mem_section_usage_size(void)
52+{
53+ return struct_size_t(struct mem_section_usage, pageblock_flags,
54+ BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS));
55+}
5056 #else
5157 static inline void sparse_init(void) {}
5258 #endif /* CONFIG_SPARSEMEM */
mm/swap.h+31−34
@@ -4,9 +4,13 @@ · 3 unmodified lines
44
55 #include <linux/atomic.h> /* for atomic_long_t */
66 #include <linux/mm.h> /* for PAGE_SHIFT */
7+#include <linux/memcontrol.h> /* for mem_cgroup_swappiness() */
8+#include <linux/swap.h> /* for MAX_SWAPFILES_SHIFT, struct swap_info_struct */
9+
710 struct mempolicy;
811 struct swap_iocb;
912 struct swap_memcg_table;
13+struct swap_io_ctx;
1014
1115 #if defined(MAX_POSSIBLE_PHYSMEM_BITS)
1216 #define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
@@ -76,6 +80,18 @@ enum swap_cluster_flags { · 63 unmodified lines
7680 CLUSTER_FLAG_MAX,
7781 };
7882
83+extern int vm_swappiness;
84+
85+static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
86+{
87+#ifdef CONFIG_MEMCG_V1
88+ if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) &&
89+ !mem_cgroup_disabled() && !mem_cgroup_is_root(memcg))
90+ return READ_ONCE(memcg->swappiness);
91+#endif
92+ return READ_ONCE(vm_swappiness);
93+}
94+
7995 #ifdef CONFIG_SWAP
8096 #include <linux/swapops.h> /* for swp_offset */
8197 #include <linux/blk_types.h> /* for bio_end_io_t */
@@ -238,17 +254,11 @@ extern void __swap_cluster_free_entries(struct swap_info_struct *si, · 156 unmodified lines
238254
239255 /* linux/mm/page_io.c */
240256 int sio_pool_init(void);
241−struct swap_iocb;
242−void swap_read_folio(struct folio *folio, struct swap_iocb **plug);
243−void __swap_read_unplug(struct swap_iocb *plug);
244−static inline void swap_read_unplug(struct swap_iocb *plug)
245−{
246− if (unlikely(plug))
247− __swap_read_unplug(plug);
248−}
249−void swap_write_unplug(struct swap_iocb *sio);
250−int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug);
251−void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug);
257+void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
258+void swap_read_submit(struct swap_io_ctx *ctx);
259+void swap_write_submit(struct swap_io_ctx *ctx);
260+int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
261+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);
252262
253263 /* linux/mm/swap_state.c */
254264 extern struct address_space swap_space __read_mostly;
@@ -315,9 +325,8 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 60 unmodified lines
315325
316326 void show_swap_cache_info(void);
317327 void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr);
318−struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
319− struct vm_area_struct *vma, unsigned long addr,
320− struct swap_iocb **plug);
328+struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry,
329+ gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr);
321330 struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag,
322331 struct mempolicy *mpol, pgoff_t ilx);
323332 struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
@@ -327,18 +336,7 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders, · 3 unmodified lines
327336 void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
328337 unsigned long addr);
329338
330−static inline unsigned int folio_swap_flags(struct folio *folio)
331−{
332− return __swap_entry_to_info(folio->swap)->flags;
333−}
334−
335339 #else /* CONFIG_SWAP */
336−struct swap_iocb;
337−static inline struct swap_cluster_info *swap_cluster_lock(
338− struct swap_info_struct *si, pgoff_t offset, bool irq)
339−{
340− return NULL;
341−}
342340
343341 static inline struct swap_cluster_info *swap_cluster_get_and_lock(
344342 struct folio *folio)
@@ -379,11 +377,11 @@ static inline void folio_put_swap(struct folio *folio, struct page *page) · 34 unmodified lines
379377 {
380378 }
381379
382−static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
380+static inline void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
383381 {
384382 }
385383
386−static inline void swap_write_unplug(struct swap_iocb *sio)
384+static inline void swap_write_submit(struct swap_io_ctx *ctx)
387385 {
388386 }
389387
@@ -425,8 +423,7 @@ static inline void swap_update_readahead(struct folio *folio, · 35 unmodified lines
425423 {
426424 }
427425
428−static inline int swap_writeout(struct folio *folio,
429− struct swap_iocb **swap_plug)
426+static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
430427 {
431428 return 0;
432429 }
@@ -464,11 +461,11 @@ static inline void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 31 unmodified lines
464461 struct folio *old, struct folio *new)
465462 {
466463 }
464+#endif /* CONFIG_SWAP */
467465
468−static inline unsigned int folio_swap_flags(struct folio *folio)
469−{
470− return 0;
471−}
466+extern const struct swap_ops swap_bdev_ops;
467+
468+int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
469+ struct list_head *folio_list);
472470
473−#endif /* CONFIG_SWAP */
474471 #endif /* _MM_SWAP_H */
mm/swap_state.c+32−22
@@ -23,6 +23,7 @@ · 22 unmodified lines
2323 #include <linux/huge_mm.h>
2424 #include <linux/shmem_fs.h>
2525 #include <linux/sysctl.h>
26+#include <linux/swap_ops.h>
2627 #include "internal.h"
2728 #include "swap_table.h"
2829 #include "swap.h"
@@ -638,9 +639,9 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, · 609 unmodified lines
638639 }
639640 }
640641
641−static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
642− struct mempolicy *mpol, pgoff_t ilx,
643− struct swap_iocb **plug, bool readahead)
642+static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
643+ swp_entry_t entry, gfp_t gfp, struct mempolicy *mpol,
644+ pgoff_t ilx, bool readahead)
644645 {
645646 struct folio *folio;
646647
@@ -654,7 +655,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp, · 7 unmodified lines
654655 if (IS_ERR_OR_NULL(folio))
655656 return NULL;
656657
657− swap_read_folio(folio, plug);
658+ swap_read_folio(ctx, folio);
658659 if (readahead) {
659660 folio_set_readahead(folio);
660661 count_vm_event(SWAP_RA);
@@ -682,6 +683,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp, · 21 unmodified lines
682683 struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
683684 struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
684685 {
686+ struct swap_io_ctx ctx = {};
685687 struct folio *folio;
686688
687689 do {
@@ -694,7 +696,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders, · 6 unmodified lines
694696 if (IS_ERR(folio))
695697 return folio;
696698
697− swap_read_folio(folio, NULL);
699+ swap_read_folio(&ctx, folio);
700+ swap_read_submit(&ctx);
698701 return folio;
699702 }
700703
@@ -704,9 +707,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders, · 3 unmodified lines
704707 * A failure return means that either the page allocation failed or that
705708 * the swap entry is no longer in use.
706709 */
707−struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
708− struct vm_area_struct *vma, unsigned long addr,
709− struct swap_iocb **plug)
710+struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry,
711+ gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr)
710712 {
711713 struct swap_info_struct *si;
712714 struct mempolicy *mpol;
@@ -718,13 +720,24 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 5 unmodified lines
718720 return NULL;
719721
720722 mpol = get_vma_policy(vma, addr, 0, &ilx);
721− folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false);
723+ folio = swap_cache_read_folio(ctx, entry, gfp_mask, mpol, ilx, false);
722724 mpol_cond_put(mpol);
723725
724726 put_swap_device(si);
725727 return folio;
726728 }
727729
730+static struct folio *swap_cache_read_folio_sync(swp_entry_t entry, gfp_t gfp,
731+ struct mempolicy *mpol, pgoff_t ilx)
732+{
733+ struct swap_io_ctx ctx = {};
734+ struct folio *folio;
735+
736+ folio = swap_cache_read_folio(&ctx, entry, gfp, mpol, ilx, false);
737+ swap_read_submit(&ctx);
738+ return folio;
739+}
740+
728741 static unsigned int __swapin_nr_pages(unsigned long prev_offset,
729742 unsigned long offset,
730743 int hits,
@@ -813,8 +826,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 82 unmodified lines
813826 unsigned long start_offset, end_offset;
814827 unsigned long mask;
815828 struct swap_info_struct *si = __swap_entry_to_info(entry);
829+ struct swap_io_ctx ctx = {};
816830 struct blk_plug plug;
817− struct swap_iocb *splug = NULL;
818831 swp_entry_t ra_entry;
819832
820833 mask = swapin_nr_pages(offset) - 1;
@@ -833,17 +846,16 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 12 unmodified lines
833846 for (offset = start_offset; offset <= end_offset ; offset++) {
834847 /* Ok, do the async read-ahead now */
835848 ra_entry = swp_entry(swp_type(entry), offset);
836− folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx,
837− &splug, offset != entry_offset);
849+ folio = swap_cache_read_folio(&ctx, ra_entry, gfp_mask, mpol,
850+ ilx, offset != entry_offset);
838851 if (!folio)
839852 continue;
840853 folio_put(folio);
841854 }
842855 blk_finish_plug(&plug);
843− swap_read_unplug(splug);
856+ swap_read_submit(&ctx);
844857 skip:
845− /* The page was likely read above, so no need for plugging here */
846− return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
858+ return swap_cache_read_folio_sync(entry, gfp_mask, mpol, ilx);
847859 }
848860
849861 static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
@@ -903,8 +915,8 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start, · 53 unmodified lines
903915 static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
904916 struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf)
905917 {
918+ struct swap_io_ctx ctx = {};
906919 struct blk_plug plug;
907− struct swap_iocb *splug = NULL;
908920 struct folio *folio;
909921 pte_t *pte = NULL, pentry;
910922 int win;
@@ -943,8 +955,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 32 unmodified lines
943955 if (!si)
944956 continue;
945957 }
946− folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx,
947− &splug, addr != vmf->address);
958+ folio = swap_cache_read_folio(&ctx, entry, gfp_mask, mpol, ilx,
959+ addr != vmf->address);
948960 if (si)
949961 put_swap_device(si);
950962 if (!folio)
@@ -954,12 +966,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 3 unmodified lines
954966 if (pte)
955967 pte_unmap(pte);
956968 blk_finish_plug(&plug);
957− swap_read_unplug(splug);
969+ swap_read_submit(&ctx);
958970 skip:
959971 /* The folio was likely read above, so no need for plugging here */
960− folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
961− NULL, false);
962− return folio;
972+ return swap_cache_read_folio_sync(targ_entry, gfp_mask, mpol, targ_ilx);
963973 }
964974
965975 /**
mm/swapfile.c+16−8
@@ -1899,11 +1899,11 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry) · 1898 unmodified lines
18991899
19001900 return si;
19011901 bad_nofile:
1902− pr_err("%s: %s%08lx\n", __func__, Bad_file, entry.val);
1902+ pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_file, entry.val);
19031903 out:
19041904 return NULL;
19051905 put_out:
1906− pr_err("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
1906+ pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
19071907 percpu_ref_put(&si->users);
19081908 return NULL;
19091909 }
@@ -2959,6 +2959,12 @@ static int setup_swap_extents(struct swap_info_struct *sis, · 1049 unmodified lines
29592959 struct inode *inode = mapping->host;
29602960 int ret;
29612961
2962+ ret = sio_pool_init();
2963+ if (ret)
2964+ return ret;
2965+
2966+ sis->ops = &swap_bdev_ops;
2967+
29622968 if (S_ISBLK(inode->i_mode)) {
29632969 ret = add_swap_extent(sis, 0, sis->max, 0);
29642970 *span = sis->pages;
@@ -2970,11 +2976,6 @@ static int setup_swap_extents(struct swap_info_struct *sis, · 5 unmodified lines
29702976 if (ret < 0)
29712977 return ret;
29722978 sis->flags |= SWP_ACTIVATED;
2973− if ((sis->flags & SWP_FS_OPS) &&
2974− sio_pool_init() != 0) {
2975− destroy_swap_extents(sis, swap_file);
2976− return -ENOMEM;
2977− }
29782979 return ret;
29792980 }
29802981
@@ -3668,6 +3669,13 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · 687 unmodified lines
36683669 error = -EBUSY;
36693670 goto bad_swap_unlock_inode;
36703671 }
3672+ if (IS_ENCRYPTED(inode)) {
3673+ pr_warn_once(
3674+ "Filesystem-level encrypted swapfile '%s' is unsupported. Create a loop device over it, or use dm-crypt\n",
3675+ name->name);
3676+ error = -EINVAL;
3677+ goto bad_swap_unlock_inode;
3678+ }
36713679
36723680 /*
36733681 * The swap subsystem needs a major overhaul to support this.
@@ -3875,7 +3883,7 @@ int swap_dup_entry_direct(swp_entry_t entry) · 201 unmodified lines
38753883
38763884 si = swap_entry_to_info(entry);
38773885 if (WARN_ON_ONCE(!si)) {
3878− pr_err("%s%08lx\n", Bad_file, entry.val);
3886+ pr_err_ratelimited("%s%08lx\n", Bad_file, entry.val);
38793887 return -EINVAL;
38803888 }
38813889
mm/userfaultfd.c+2−2
@@ -1352,7 +1352,7 @@ static long move_present_ptes(struct mm_struct *mm, · 1351 unmodified lines
13521352 }
13531353
13541354 folio_move_anon_rmap(src_folio, dst_vma);
1355− src_folio->index = linear_page_index(dst_vma, dst_addr);
1355+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
13561356
13571357 orig_dst_pte = folio_mk_pte(src_folio, dst_vma->vm_page_prot);
13581358 /* Set soft dirty bit so userspace can notice the pte was moved */
@@ -1428,7 +1428,7 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, · 69 unmodified lines
14281428 */
14291429 if (src_folio) {
14301430 folio_move_anon_rmap(src_folio, dst_vma);
1431− src_folio->index = linear_page_index(dst_vma, dst_addr);
1431+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
14321432 } else {
14331433 /*
14341434 * Check if the swap entry is cached after acquiring the src_pte
mm/vma.c+79−34
@@ -18,6 +18,7 @@ struct mmap_state { · 17 unmodified lines
1818 unsigned long addr;
1919 unsigned long end;
2020 pgoff_t pgoff;
21+ pgoff_t anon_pgoff;
2122 unsigned long pglen;
2223 union {
2324 vm_flags_t vm_flags;
@@ -46,13 +47,14 @@ struct mmap_state { · 22 unmodified lines
4647 bool file_doesnt_need_get :1;
4748 };
4849
49−#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, vma_flags_, file_) \
50+#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, anon_pgoff_, vma_flags_, file_) \
5051 struct mmap_state name = { \
5152 .mm = mm_, \
5253 .vmi = vmi_, \
5354 .addr = addr_, \
5455 .end = (addr_) + (len_), \
5556 .pgoff = pgoff_, \
57+ .anon_pgoff = anon_pgoff_, \
5658 .pglen = PHYS_PFN(len_), \
5759 .vma_flags = vma_flags_, \
5860 .file = file_, \
@@ -67,6 +69,7 @@ struct mmap_state { · 8 unmodified lines
6769 .end = (map_)->end, \
6870 .vma_flags = (map_)->vma_flags, \
6971 .pgoff = (map_)->pgoff, \
72+ .anon_pgoff = (map_)->anon_pgoff, \
7073 .file = (map_)->file, \
7174 .prev = (map_)->prev, \
7275 .middle = vma_, \
@@ -82,10 +85,11 @@ static void __vma_set_range(struct vm_area_struct *vma, unsigned long start, · 9 unmodified lines
8285 }
8386
8487 static void vma_set_range(struct vm_area_struct *vma, unsigned long start,
85− unsigned long end, pgoff_t pgoff)
88+ unsigned long end, pgoff_t pgoff, pgoff_t anon_pgoff)
8689 {
8790 __vma_set_range(vma, start, end);
8891 vma_set_pgoff(vma, pgoff);
92+ vma_set_anon_pgoff(vma, anon_pgoff);
8993 }
9094
9195 /* Was this VMA ever forked from a parent, i.e. maybe contains CoW mappings? */
@@ -200,6 +204,25 @@ static void init_multi_vma_prep(struct vma_prepare *vp, · 108 unmodified lines
200204 vp->skip_vma_uprobe = true;
201205 }
202206
207+/*
208+ * Does this merge require that adjacent VMAs must have adjacent anonymous page
209+ * offsets in addition to having adjacent vma->vm_pgoff?
210+ *
211+ * This is only required for MAP_PRIVATE-file backed mappings as the page offset
212+ * for pure anonymous VMAs is equal to the anonymous page offset.
213+ *
214+ * Read-only shared mappings (with VMA_SHARED_BIT cleared) are always unfaulted
215+ * so automatically have correct anonymous page offset (as it is always updated
216+ * on remap).
217+ *
218+ * 'Special' mappings in the sense of VDSO, VVAR etc. have !file but would in
219+ * any case not be candidates for merge nor be mergeable.
220+ */
221+static bool needs_adjacent_anon_pgoff(const struct vma_merge_struct *vmg)
222+{
223+ return vmg->file && vma_flags_is_cow_mapping(&vmg->vma_flags);
224+}
225+
203226 /*
204227 * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff)
205228 * in front of (at a lower virtual address and file offset than) the vma.
@@ -221,6 +244,9 @@ static bool can_vma_merge_before(struct vma_merge_struct *vmg) · 15 unmodified lines
221244 return false;
222245 if (vmg_end_pgoff(vmg) != vma_start_pgoff(vmg->next))
223246 return false;
247+ if (needs_adjacent_anon_pgoff(vmg) &&
248+ vmg_end_anon_pgoff(vmg) != vma_start_anon_pgoff(vmg->next))
249+ return false;
224250 return true;
225251 }
226252
@@ -241,6 +267,9 @@ static bool can_vma_merge_after(struct vma_merge_struct *vmg) · 14 unmodified lines
241267 return false;
242268 if (vma_end_pgoff(vmg->prev) != vmg_start_pgoff(vmg))
243269 return false;
270+ if (needs_adjacent_anon_pgoff(vmg) &&
271+ vma_end_anon_pgoff(vmg->prev) != vmg_start_anon_pgoff(vmg))
272+ return false;
244273 return true;
245274 }
246275
@@ -812,7 +841,8 @@ static int commit_merge(struct vma_merge_struct *vmg) · 565 unmodified lines
812841 */
813842 vma_adjust_trans_huge(vma, vmg->start, vmg->end,
814843 vmg->__adjust_middle_start ? vmg->middle : NULL);
815− vma_set_range(vma, vmg->start, vmg->end, vmg_start_pgoff(vmg));
844+ vma_set_range(vma, vmg->start, vmg->end, vmg_start_pgoff(vmg),
845+ vmg_start_anon_pgoff(vmg));
816846 vmg_adjust_set_range(vmg);
817847 vma_iter_store_overwrite(vmg->vmi, vmg->target);
818848
@@ -982,6 +1012,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( · 163 unmodified lines
9821012 vmg->start = prev->vm_start;
9831013 vmg->end = next->vm_end;
9841014 vmg->pgoff = vma_start_pgoff(prev);
1015+ vmg->anon_pgoff = vma_start_anon_pgoff(prev);
9851016
9861017 /*
9871018 * We already ensured anon_vma compatibility above, so now it's
@@ -1000,6 +1031,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( · 12 unmodified lines
10001031 */
10011032 vmg->start = prev->vm_start;
10021033 vmg->pgoff = vma_start_pgoff(prev);
1034+ vmg->anon_pgoff = vma_start_anon_pgoff(prev);
10031035
10041036 if (!vmg->__remove_middle)
10051037 vmg->__adjust_middle_start = true;
@@ -1022,12 +1054,14 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( · 16 unmodified lines
10221054 if (vmg->__remove_middle) {
10231055 vmg->end = next->vm_end;
10241056 vmg->pgoff = vma_start_pgoff(next) - pglen;
1057+ vmg->anon_pgoff = vma_start_anon_pgoff(next) - pglen;
10251058 } else {
10261059 /* We shrink middle and expand next. */
10271060 vmg->__adjust_next_start = true;
10281061 vmg->start = middle->vm_start;
10291062 vmg->end = start;
10301063 vmg->pgoff = vma_start_pgoff(middle);
1064+ vmg->anon_pgoff = vma_start_anon_pgoff(middle);
10311065 }
10321066
10331067 err = dup_anon_vma(next, middle, &anon_dup);
@@ -1137,6 +1171,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg) · 103 unmodified lines
11371171 vmg->start = prev->vm_start;
11381172 vmg->target = prev;
11391173 vmg->pgoff = vma_start_pgoff(prev);
1174+ vmg->anon_pgoff = vma_start_anon_pgoff(prev);
11401175
11411176 /*
11421177 * If this merge would result in removal of the next VMA but we
@@ -1908,23 +1943,27 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) · 765 unmodified lines
19081943 */
19091944 struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
19101945 unsigned long addr, unsigned long len, pgoff_t pgoff,
1911− bool *need_rmap_locks)
1946+ pgoff_t anon_pgoff, bool *need_rmap_locks)
19121947 {
19131948 struct vm_area_struct *vma = *vmap;
1914− unsigned long vma_start = vma->vm_start;
1949+ unsigned long old_vma_start = vma->vm_start;
19151950 struct mm_struct *mm = vma->vm_mm;
19161951 struct vm_area_struct *new_vma;
1917− bool faulted_in_anon_vma = true;
1952+ bool can_self_merge = false;
19181953 VMA_ITERATOR(vmi, mm, addr);
19191954 VMG_VMA_STATE(vmg, &vmi, NULL, vma, addr, addr + len);
19201955
19211956 /*
1922− * If anonymous vma has not yet been faulted, update new pgoff
1923− * to match new location, to increase its chance of merging.
1957+ * If a vma has not yet been faulted, update its anonymous pgoff to
1958+ * match the new location to increase its chance of merging.
19241959 */
1925− if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) {
1926− pgoff = addr >> PAGE_SHIFT;
1927− faulted_in_anon_vma = false;
1960+ if (!vma->anon_vma) {
1961+ anon_pgoff = addr >> PAGE_SHIFT;
1962+
1963+ if (vma_is_anonymous(vma)) {
1964+ pgoff = anon_pgoff;
1965+ can_self_merge = true;
1966+ }
19281967 }
19291968
19301969 /*
@@ -1940,28 +1979,26 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, · 9 unmodified lines
19401979 return NULL; /* should never get here */
19411980
19421981 vmg.pgoff = pgoff;
1982+ vmg.anon_pgoff = anon_pgoff;
19431983 vmg.next = vma_iter_next_rewind(&vmi, NULL);
19441984 new_vma = vma_merge_copied_range(&vmg);
19451985
19461986 if (new_vma) {
1947− /*
1948− * Source vma may have been merged into new_vma
1949− */
1950− if (unlikely(vma_start >= new_vma->vm_start &&
1951− vma_start < new_vma->vm_end)) {
1987+ /* Self-merged and VMA replaced. */
1988+ if (unlikely(new_vma->vm_start < old_vma_start &&
1989+ new_vma->vm_end > old_vma_start)) {
19521990 /*
1953− * The only way we can get a vma_merge with
1954− * self during an mremap is if the vma hasn't
1955− * been faulted in yet and we were allowed to
1956− * reset the dst vma->vm_pgoff to the
1957− * destination address of the mremap to allow
1958− * the merge to happen. mremap must change the
1959− * vm_pgoff linearity between src and dst vmas
1960− * (in turn preventing a vma_merge) to be
1961− * safe. It is only safe to keep the vm_pgoff
1962− * linear if there are no pages mapped yet.
1991+ * The only way a VMA can both self-merge and be
1992+ * replaced is if the remap places the new VMA
1993+ * immediately prior to its old self ('next') and
1994+ * immediately after another VMA ('prev') causing the
1995+ * next to be removed and prev to be expanded to cover
1996+ * the entire range.
1997+ *
1998+ * This should only be possible if the anonymous page
1999+ * offset was updated, i.e. the VMA is unfaulted.
19632000 */
1964− VM_BUG_ON_VMA(faulted_in_anon_vma, new_vma);
2001+ VM_WARN_ON_ONCE_VMA(!can_self_merge, new_vma);
19652002 *vmap = vma = new_vma;
19662003 }
19672004 *need_rmap_locks =
@@ -1970,7 +2007,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, · 2 unmodified lines
19702007 new_vma = vm_area_dup(vma);
19712008 if (!new_vma)
19722009 goto out;
1973− vma_set_range(new_vma, addr, addr + len, pgoff);
2010+ vma_set_range(new_vma, addr, addr + len, pgoff, anon_pgoff);
19742011 if (vma_dup_policy(vma, new_vma))
19752012 goto out_free_vma;
19762013 if (anon_vma_clone(new_vma, vma, VMA_OP_REMAP))
@@ -2036,7 +2073,12 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * · 59 unmodified lines
20362073 if (!vma_flags_empty(&diff))
20372074 return false;
20382075 /* Page offset must align. */
2039− return vma_end_pgoff(a) == vma_start_pgoff(b);
2076+ if (vma_end_pgoff(a) != vma_start_pgoff(b))
2077+ return false;
2078+ /* Only reached from anon path, so either MAP_PRIVATE file or anon. */
2079+ if (vma_end_anon_pgoff(a) != vma_start_anon_pgoff(b))
2080+ return false;
2081+ return true;
20402082 }
20412083
20422084 /*
@@ -2612,7 +2654,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, · 569 unmodified lines
26122654 if (is_anon)
26132655 vma_set_anonymous(vma);
26142656
2615− vma_set_range(vma, map->addr, map->end, map->pgoff);
2657+ vma_set_range(vma, map->addr, map->end, map->pgoff, map->anon_pgoff);
26162658 vma->flags = map->vma_flags;
26172659 vma->vm_page_prot = map->page_prot;
26182660
@@ -2801,7 +2843,8 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, · 182 unmodified lines
28012843 struct vm_area_struct *vma = NULL;
28022844 bool have_mmap_prepare = file && file->f_op->mmap_prepare;
28032845 VMA_ITERATOR(vmi, mm, addr);
2804− MMAP_STATE(map, mm, &vmi, addr, len, pgoff, vma_flags, file);
2846+ const pgoff_t anon_pgoff = addr >> PAGE_SHIFT;
2847+ MMAP_STATE(map, mm, &vmi, addr, len, pgoff, anon_pgoff, vma_flags, file);
28052848 struct vm_area_desc desc = {
28062849 .mm = mm,
28072850 .file = file,
@@ -2946,6 +2989,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, · 138 unmodified lines
29462989 unsigned long addr, unsigned long len, vma_flags_t vma_flags)
29472990 {
29482991 struct mm_struct *mm = current->mm;
2992+ const pgoff_t pgoff = addr >> PAGE_SHIFT;
29492993
29502994 /*
29512995 * Check against address space limits by the changed size
@@ -2970,7 +3014,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, · 18 unmodified lines
29703014 * occur after forking, so the expand will only happen on new VMAs.
29713015 */
29723016 if (vma && vma->vm_end == addr) {
2973− VMG_STATE(vmg, mm, vmi, addr, addr + len, vma_flags, PHYS_PFN(addr));
3017+ VMG_STATE(vmg, mm, vmi, addr, addr + len, vma_flags, pgoff, pgoff);
29743018
29753019 vmg.prev = vma;
29763020 /* vmi is positioned at prev, which this mode expects. */
@@ -2990,7 +3034,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, · 13 unmodified lines
29903034 goto unacct_fail;
29913035
29923036 vma_set_anonymous(vma);
2993− vma_set_range(vma, addr, addr + len, addr >> PAGE_SHIFT);
3037+ vma_set_range(vma, addr, addr + len, pgoff, pgoff);
29943038 vma->flags = vma_flags;
29953039 vma->vm_page_prot = vm_get_page_prot(vma_flags_to_legacy(vma_flags));
29963040 vma_start_write(vma);
@@ -3382,6 +3426,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) · 385 unmodified lines
33823426 WARN_ON_ONCE(vma->anon_vma);
33833427 vma_set_pgoff(vma, vma->vm_start >> PAGE_SHIFT);
33843428 }
3429+ vma_set_anon_pgoff(vma, vma->vm_start >> PAGE_SHIFT);
33853430
33863431 if (vma_link(mm, vma)) {
33873432 if (vma_test(vma, VMA_ACCOUNT_BIT))
@@ -3437,7 +3482,7 @@ struct vm_area_struct *__install_special_mapping( · 49 unmodified lines
34373482
34383483 vma->vm_ops = ops;
34393484 vma->vm_private_data = priv;
3440− vma_set_range(vma, addr, addr + len, 0);
3485+ vma_set_range(vma, addr, addr + len, 0, addr >> PAGE_SHIFT);
34413486
34423487 ret = insert_vm_struct(mm, vma);
34433488 if (ret)
mm/vma.h+59−30
@@ -104,6 +104,7 @@ struct vma_merge_struct { · 103 unmodified lines
104104 unsigned long start;
105105 unsigned long end;
106106 pgoff_t pgoff;
107+ pgoff_t anon_pgoff;
107108
108109 union {
109110 /* Temporary while VMA flags are being converted. */
@@ -237,11 +238,6 @@ static inline bool vmg_nomem(struct vma_merge_struct *vmg) · 127 unmodified lines
237238 return vmg->state == VMA_MERGE_ERROR_NOMEM;
238239 }
239240
240−static inline pgoff_t vmg_start_pgoff(const struct vma_merge_struct *vmg)
241−{
242− return vmg->pgoff;
243−}
244−
245241 static inline pgoff_t vmg_pages(const struct vma_merge_struct *vmg)
246242 {
247243 const unsigned long size = vmg->end - vmg->start;
@@ -249,6 +245,11 @@ static inline pgoff_t vmg_pages(const struct vma_merge_struct *vmg) · 1 unmodified line
249245 return size >> PAGE_SHIFT;
250246 }
251247
248+static inline pgoff_t vmg_start_pgoff(const struct vma_merge_struct *vmg)
249+{
250+ return vmg->pgoff;
251+}
252+
252253 static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg)
253254 {
254255 return vmg_start_pgoff(vmg) + vmg_pages(vmg);
@@ -283,48 +284,76 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) · 28 unmodified lines
283284 vma->vm_pgoff = pgoff;
284285 }
285286
287+static inline pgoff_t vmg_start_anon_pgoff(const struct vma_merge_struct *vmg)
288+{
289+ return vmg->anon_pgoff;
290+}
291+
292+static inline pgoff_t vmg_end_anon_pgoff(const struct vma_merge_struct *vmg)
293+{
294+ return vmg_start_anon_pgoff(vmg) + vmg_pages(vmg);
295+}
296+
297+static inline void __vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff)
298+{
299+#ifdef CONFIG_64BIT
300+ vma->__vm_anon_pgoff_hi = pgoff >> 32;
301+#endif
302+ vma->__vm_anon_pgoff_lo = pgoff & GENMASK(31, 0);
303+}
304+
305+static inline void vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff)
306+{
307+ vma_assert_can_modify(vma);
308+ __vma_set_anon_pgoff(vma, pgoff);
309+}
310+
286311 static inline void vma_add_pgoff(struct vm_area_struct *vma, pgoff_t delta)
287312 {
288313 vma_assert_can_modify(vma);
289314 vma_set_pgoff(vma, vma_start_pgoff(vma) + delta);
315+ vma_set_anon_pgoff(vma, vma_start_anon_pgoff(vma) + delta);
290316 }
291317
292318 static inline void vma_sub_pgoff(struct vm_area_struct *vma, pgoff_t delta)
293319 {
294320 vma_assert_can_modify(vma);
295321 vma_set_pgoff(vma, vma_start_pgoff(vma) - delta);
296−}
322+ vma_set_anon_pgoff(vma, vma_start_anon_pgoff(vma) - delta);
323+}
324+
325+#define VMG_STATE(name, mm_, vmi_, start_, end_, vma_flags_, pgoff_, anon_pgoff_) \
326+ struct vma_merge_struct name = { \
327+ .mm = mm_, \
328+ .vmi = vmi_, \
329+ .start = start_, \
330+ .end = end_, \
331+ .vma_flags = vma_flags_, \
332+ .pgoff = pgoff_, \
333+ .anon_pgoff = anon_pgoff_, \
334+ .state = VMA_MERGE_START, \
335+ }
297336
298−#define VMG_STATE(name, mm_, vmi_, start_, end_, vma_flags_, pgoff_) \
337+#define VMG_VMA_STATE(name, vmi_, prev_, vma_, start_, end_) \
299338 struct vma_merge_struct name = { \
300− .mm = mm_, \
339+ .mm = vma_->vm_mm, \
301340 .vmi = vmi_, \
341+ .prev = prev_, \
342+ .middle = vma_, \
343+ .next = NULL, \
302344 .start = start_, \
303345 .end = end_, \
304− .vma_flags = vma_flags_, \
305− .pgoff = pgoff_, \
346+ .vm_flags = vma_->vm_flags, \
347+ .pgoff = linear_page_index(vma_, start_), \
348+ .anon_pgoff = __linear_anon_page_index(vma_, start_), \
349+ .file = vma_->vm_file, \
350+ .anon_vma = vma_->anon_vma, \
351+ .policy = vma_policy(vma_), \
352+ .uffd_ctx = vma_->vm_userfaultfd_ctx, \
353+ .anon_name = anon_vma_name(vma_), \
306354 .state = VMA_MERGE_START, \
307355 }
308356
309−#define VMG_VMA_STATE(name, vmi_, prev_, vma_, start_, end_) \
310− struct vma_merge_struct name = { \
311− .mm = vma_->vm_mm, \
312− .vmi = vmi_, \
313− .prev = prev_, \
314− .middle = vma_, \
315− .next = NULL, \
316− .start = start_, \
317− .end = end_, \
318− .vm_flags = vma_->vm_flags, \
319− .pgoff = linear_page_index(vma_, start_), \
320− .file = vma_->vm_file, \
321− .anon_vma = vma_->anon_vma, \
322− .policy = vma_policy(vma_), \
323− .uffd_ctx = vma_->vm_userfaultfd_ctx, \
324− .anon_name = anon_vma_name(vma_), \
325− .state = VMA_MERGE_START, \
326− }
327−
328357 #ifdef CONFIG_DEBUG_VM_MAPLE_TREE
329358 void validate_mm(struct mm_struct *mm);
330359 #else
@@ -506,7 +535,7 @@ void unlink_file_vma_batch_add(struct unlink_vma_file_batch *vb, · 175 unmodified lines
506535
507536 struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
508537 unsigned long addr, unsigned long len, pgoff_t pgoff,
509− bool *need_rmap_locks);
538+ pgoff_t anon_pgoff, bool *need_rmap_locks);
510539
511540 struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma);
512541
mm/vma_exec.c+1−1
@@ -41,7 +41,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) · 40 unmodified lines
4141 unsigned long new_end = old_end - shift;
4242 VMA_ITERATOR(vmi, mm, new_start);
4343 VMG_STATE(vmg, mm, &vmi, new_start, old_end, EMPTY_VMA_FLAGS,
44− vma_start_pgoff(vma));
44+ vma_start_pgoff(vma), vma_start_anon_pgoff(vma));
4545 struct vm_area_struct *next;
4646 struct mmu_gather tlb;
4747 PAGETABLE_MOVE(pmc, vma, vma, old_start, new_start, length);
mm/vma_init.c+1−0
@@ -51,6 +51,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, · 50 unmodified lines
5151 dest->vm_end = src->vm_end;
5252 dest->anon_vma = src->anon_vma;
5353 dest->vm_pgoff = vma_start_pgoff(src);
54+ __vma_set_anon_pgoff(dest, vma_start_anon_pgoff(src));
5455 dest->vm_file = src->vm_file;
5556 dest->vm_private_data = src->vm_private_data;
5657 vm_flags_init(dest, src->vm_flags);
mm/vmalloc.c+34−37
@@ -1840,8 +1840,10 @@ va_alloc(struct vmap_area *va, · 1839 unmodified lines
18401840
18411841 /* Update the free vmap_area. */
18421842 ret = va_clip(root, head, va, nva_start_addr, size);
1843− if (WARN_ON_ONCE(ret))
1843+ if (ret) {
1844+ WARN_ON_ONCE(ret != -ENOMEM);
18441845 return ret;
1846+ }
18451847
18461848 return nva_start_addr;
18471849 }
@@ -1914,12 +1916,9 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) · 66 unmodified lines
19141916
19151917 /*
19161918 * Preload this CPU with one extra vmap_area object. It is used
1917− * when fit type of free area is NE_FIT_TYPE. It guarantees that
1918− * a CPU that does an allocation is preloaded.
1919− *
1920− * We do it in non-atomic context, thus it allows us to use more
1921− * permissive allocation masks to be more stable under low memory
1922− * condition and high memory pressure.
1919+ * when fit type of free area is NE_FIT_TYPE. It is best effort
1920+ * pre-loading. If it fails va_clip() may return -ENOMEM from its
1921+ * GFP_NOWAIT fallback.
19231922 */
19241923 if (!this_cpu_read(ne_fit_preload_node))
19251924 va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node);
@@ -3361,7 +3360,7 @@ struct vm_struct *remove_vm_area(const void *addr) · 1435 unmodified lines
33613360 static inline void set_area_direct_map(const struct vm_struct *area,
33623361 int (*set_direct_map)(struct page *page))
33633362 {
3364− int i;
3363+ unsigned long i;
33653364
33663365 /* HUGE_VMALLOC passes small pages to set_direct_map */
33673366 for (i = 0; i < area->nr_pages; i++)
@@ -3377,7 +3376,7 @@ static void vm_reset_perms(struct vm_struct *area) · 9 unmodified lines
33773376 unsigned long start = ULONG_MAX, end = 0;
33783377 unsigned int page_order = vm_area_page_order(area);
33793378 int flush_dmap = 0;
3380− int i;
3379+ unsigned long i;
33813380
33823381 /*
33833382 * Find the start and end range of the direct mappings to make sure that
@@ -3450,10 +3449,10 @@ void vfree_atomic(const void *addr) · 66 unmodified lines
34503449 * Caller is responsible for unmapping (vunmap_range) and KASAN
34513450 * poisoning before calling this.
34523451 */
3453−static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx,
3454− unsigned int end_idx)
3452+static void vm_area_free_pages(struct vm_struct *vm, unsigned long start_idx,
3453+ unsigned long end_idx)
34553454 {
3456− unsigned int i;
3455+ unsigned long i;
34573456
34583457 if (!(vm->flags & VM_MAP_PUT_PAGES)) {
34593458 for (i = start_idx; i < end_idx; i++)
@@ -3665,12 +3664,12 @@ static inline gfp_t vmalloc_gfp_adjust(gfp_t flags, const bool large) · 205 unmodified lines
36653664 return flags;
36663665 }
36673666
3668−static inline unsigned int
3667+static inline unsigned long
36693668 vm_area_alloc_pages(gfp_t gfp, int nid,
3670− unsigned int order, unsigned int nr_pages, struct page **pages)
3669+ unsigned int order, unsigned long nr_pages, struct page **pages)
36713670 {
3672− unsigned int nr_allocated = 0;
3673− unsigned int nr_remaining = nr_pages;
3671+ unsigned long nr_allocated = 0;
3672+ unsigned long nr_remaining = nr_pages;
36743673 unsigned int max_attempt_order = MAX_PAGE_ORDER;
36753674 struct page *page;
36763675 int i;
@@ -3718,7 +3717,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, · 41 unmodified lines
37183717 if (!order) {
37193718 while (nr_allocated < nr_pages) {
37203719 unsigned int nr, nr_pages_request;
3721− int i;
3720+ unsigned long i;
37223721
37233722 /*
37243723 * A maximum allowed request is hard-coded and is 100
@@ -3726,7 +3725,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, · 1 unmodified line
37263725 * long preemption off scenario in the bulk-allocator
37273726 * so the range is [1:100].
37283727 */
3729− nr_pages_request = min(100U, nr_pages - nr_allocated);
3728+ nr_pages_request = min(100UL, nr_pages - nr_allocated);
37303729
37313730 /* memory allocation should consider mempolicy, we can't
37323731 * wrongly use nearest node when nid == NUMA_NO_NODE,
@@ -3872,12 +3871,12 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · 139 unmodified lines
38723871 unsigned long addr = (unsigned long)area->addr;
38733872 unsigned long size = get_vm_area_size(area);
38743873 unsigned long array_size;
3875− unsigned int nr_small_pages = size >> PAGE_SHIFT;
3874+ unsigned long nr_small_pages = size >> PAGE_SHIFT;
38763875 unsigned int page_order;
38773876 unsigned int flags;
38783877 int ret;
38793878
3880− array_size = (unsigned long)nr_small_pages * sizeof(struct page *);
3879+ array_size = nr_small_pages * sizeof(struct page *);
38813880
38823881 /* __GFP_NOFAIL and "noblock" flags are mutually exclusive. */
38833882 if (!gfpflags_allow_blocking(gfp_mask))
@@ -4375,7 +4374,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 491 unmodified lines
43754374 }
43764375
43774376 if (size <= old_size) {
4378− unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
4377+ unsigned long new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
43794378
43804379 /* Zero out "freed" memory, potentially for future realloc. */
43814380 if (want_init_on_free() || want_init_on_alloc(flags))
@@ -4404,7 +4403,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 22 unmodified lines
44044403 !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) &&
44054404 gfp_has_io_fs(flags)) {
44064405 unsigned long addr = (unsigned long)kasan_reset_tag(p);
4407− unsigned int old_nr_pages = vm->nr_pages;
4406+ unsigned long old_nr_pages = vm->nr_pages;
44084407
44094408 /*
44104409 * Use the node lock to synchronize with concurrent
@@ -4417,16 +4416,13 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align · 6 unmodified lines
44174416 spin_unlock(&vn->busy.lock);
44184417
44194418 /* Notify kmemleak of the reduced allocation size before unmapping. */
4420− kmemleak_free_part(
4421− (void *)addr + ((unsigned long)new_nr_pages
4422− << PAGE_SHIFT),
4423− (unsigned long)(old_nr_pages - new_nr_pages)
4424− << PAGE_SHIFT);
4419+ kmemleak_free_part((void *)addr +
4420+ (new_nr_pages << PAGE_SHIFT),
4421+ (old_nr_pages - new_nr_pages)
4422+ << PAGE_SHIFT);
44254423
4426− vunmap_range(addr + ((unsigned long)new_nr_pages
4427− << PAGE_SHIFT),
4428− addr + ((unsigned long)old_nr_pages
4429− << PAGE_SHIFT));
4424+ vunmap_range(addr + (new_nr_pages << PAGE_SHIFT),
4425+ addr + (old_nr_pages << PAGE_SHIFT));
44304426
44314427 vm_area_free_pages(vm, new_nr_pages, old_nr_pages);
44324428 }
@@ -5250,7 +5246,7 @@ bool vmalloc_dump_obj(void *object) · 817 unmodified lines
52505246 struct vmap_area *va;
52515247 struct vmap_node *vn;
52525248 unsigned long addr;
5253− unsigned int nr_pages;
5249+ unsigned long nr_pages;
52545250
52555251 addr = PAGE_ALIGN((unsigned long) object);
52565252 vn = addr_to_node(addr);
@@ -5270,7 +5266,7 @@ bool vmalloc_dump_obj(void *object) · 13 unmodified lines
52705266 nr_pages = vm->nr_pages;
52715267 spin_unlock(&vn->busy.lock);
52725268
5273− pr_cont(" %u-page vmalloc region starting at %#lx allocated at %pS\n",
5269+ pr_cont(" %lu-page vmalloc region starting at %#lx allocated at %pS\n",
52745270 nr_pages, addr, caller);
52755271
52765272 return true;
@@ -5288,16 +5284,17 @@ bool vmalloc_dump_obj(void *object) · 11 unmodified lines
52885284 static void show_numa_info(struct seq_file *m, struct vm_struct *v,
52895285 unsigned int *counters)
52905286 {
5291− unsigned int nr;
52925287 unsigned int step = 1U << vm_area_page_order(v);
5288+ unsigned long i;
5289+ unsigned int nr;
52935290
52945291 if (!counters)
52955292 return;
52965293
52975294 memset(counters, 0, nr_node_ids * sizeof(unsigned int));
52985295
5299− for (nr = 0; nr < v->nr_pages; nr += step)
5300− counters[page_to_nid(v->pages[nr])] += step;
5296+ for (i = 0; i < v->nr_pages; i += step)
5297+ counters[page_to_nid(v->pages[i])] += step;
53015298 for_each_node_state(nr, N_HIGH_MEMORY)
53025299 if (counters[nr])
53035300 seq_printf(m, " N%u=%u", nr, counters[nr]);
@@ -5355,7 +5352,7 @@ static int vmalloc_info_show(struct seq_file *m, void *p) · 51 unmodified lines
53555352 seq_printf(m, " %pS", v->caller);
53565353
53575354 if (v->nr_pages)
5358− seq_printf(m, " pages=%d", v->nr_pages);
5355+ seq_printf(m, " pages=%lu", v->nr_pages);
53595356
53605357 if (v->phys_addr)
53615358 seq_printf(m, " phys=%pa", &v->phys_addr);
mm/vmalloc.h+2−0
@@ -5,6 +5,8 @@ · 4 unmodified lines
55 #ifndef __MM_VMALLOC_H
66 #define __MM_VMALLOC_H
77
8+#include <linux/vmalloc.h>
9+
810 #ifdef CONFIG_MMU
911 void __init vmalloc_init(void);
1012 int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end,
mm/vmscan.c+151−79
@@ -58,6 +58,7 @@ · 57 unmodified lines
5858 #include <linux/random.h>
5959 #include <linux/mmu_notifier.h>
6060 #include <linux/parser.h>
61+#include <linux/swap_ops.h>
6162
6263 #include <asm/tlbflush.h>
6364 #include <asm/div64.h>
@@ -199,6 +200,13 @@ struct scan_control { · 135 unmodified lines
199200 */
200201 int vm_swappiness = 60;
201202
203+static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg)
204+{
205+ if (sc->proactive && sc->proactive_swappiness)
206+ return *sc->proactive_swappiness;
207+ return mem_cgroup_swappiness(memcg);
208+}
209+
202210 #ifdef CONFIG_MEMCG
203211
204212 /* Returns true for reclaim through cgroup limits or cgroup interfaces. */
@@ -239,13 +247,6 @@ static bool writeback_throttling_sane(struct scan_control *sc) · 34 unmodified lines
239247 #endif
240248 return false;
241249 }
242−
243−static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg)
244−{
245− if (sc->proactive && sc->proactive_swappiness)
246− return *sc->proactive_swappiness;
247− return mem_cgroup_swappiness(memcg);
248−}
249250 #else
250251 static bool cgroup_reclaim(struct scan_control *sc)
251252 {
@@ -261,12 +262,13 @@ static bool writeback_throttling_sane(struct scan_control *sc) · 9 unmodified lines
261262 {
262263 return true;
263264 }
265+#endif
264266
265−static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg)
267+static inline bool is_exec_file_folio(const struct folio *folio,
268+ const vma_flags_t *vma_flags)
266269 {
267− return READ_ONCE(vm_swappiness);
270+ return vma_flags_test(vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio);
268271 }
269−#endif
270272
271273 static void set_task_reclaim_state(struct task_struct *task,
272274 struct reclaim_state *rs)
@@ -616,8 +618,8 @@ typedef enum { · 343 unmodified lines
616618 /*
617619 * pageout is called by shrink_folio_list() for each dirty folio.
618620 */
619−static pageout_t pageout(struct folio *folio, struct address_space *mapping,
620− struct swap_iocb **plug, struct list_head *folio_list)
621+static pageout_t pageout(struct swap_io_ctx *ctx, struct address_space *mapping,
622+ struct folio *folio, struct list_head *folio_list)
621623 {
622624 int res;
623625
@@ -653,9 +655,9 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, · 29 unmodified lines
653655 * the split out folios get added back to folio_list.
654656 */
655657 if (shmem_mapping(mapping))
656− res = shmem_writeout(folio, plug, folio_list);
658+ res = shmem_writeout(ctx, folio, folio_list);
657659 else
658− res = swap_writeout(folio, plug);
660+ res = swap_writeout(ctx, folio);
659661
660662 if (res < 0)
661663 handle_write_error(mapping, folio, res);
@@ -669,7 +671,7 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, · 7 unmodified lines
669671 folio_clear_reclaim(folio);
670672
671673 trace_mm_vmscan_write_folio(folio);
672− node_stat_add_folio(folio, NR_VMSCAN_WRITE);
674+ lruvec_stat_mod_folio(folio, NR_VMSCAN_WRITE, folio_nr_pages(folio));
673675 return PAGE_SUCCESS;
674676 }
675677
@@ -835,10 +837,16 @@ enum folio_references { · 159 unmodified lines
835837 * with PG_active set. In contrast, the aging (page table walk) path uses
836838 * folio_update_gen().
837839 */
838−static bool lru_gen_set_refs(struct folio *folio)
840+static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
839841 {
840842 /* see the comment on LRU_REFS_FLAGS */
841843 if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
844+ /* Activate file-backed executable folios after first usage. */
845+ if (is_exec_file_folio(folio, vma_flags)) {
846+ set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
847+ return true;
848+ }
849+
842850 set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
843851 return false;
844852 }
@@ -851,7 +859,7 @@ static bool lru_gen_set_refs(struct folio *folio) · 6 unmodified lines
851859 return true;
852860 }
853861 #else
854−static bool lru_gen_set_refs(struct folio *folio)
862+static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
855863 {
856864 return false;
857865 }
@@ -861,16 +869,16 @@ static enum folio_references folio_check_references(struct folio *folio, · 3 unmodified lines
861869 struct scan_control *sc)
862870 {
863871 int referenced_ptes, referenced_folio;
864− vm_flags_t vm_flags;
872+ vma_flags_t vma_flags;
865873
866874 referenced_ptes = folio_referenced(folio, 1, sc->target_mem_cgroup,
867− &vm_flags);
875+ &vma_flags);
868876
869877 /*
870878 * The supposedly reclaimable folio was found to be in a VM_LOCKED vma.
871879 * Let the folio, now marked Mlocked, be moved to the unevictable list.
872880 */
873− if (vm_flags & VM_LOCKED)
881+ if (vma_flags_test(&vma_flags, VMA_LOCKED_BIT))
874882 return FOLIOREF_ACTIVATE;
875883
876884 /*
@@ -886,7 +894,7 @@ static enum folio_references folio_check_references(struct folio *folio, · 9 unmodified lines
886894 if (!referenced_ptes)
887895 return FOLIOREF_RECLAIM;
888896
889− return lru_gen_set_refs(folio) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
897+ return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
890898 }
891899
892900 referenced_folio = folio_test_clear_referenced(folio);
@@ -914,7 +922,7 @@ static enum folio_references folio_check_references(struct folio *folio, · 21 unmodified lines
914922 /*
915923 * Activate file-backed executable folios after first usage.
916924 */
917− if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio))
925+ if (is_exec_file_folio(folio, &vma_flags))
918926 return FOLIOREF_ACTIVATE;
919927
920928 return FOLIOREF_KEEP;
@@ -1033,16 +1041,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) · 112 unmodified lines
10331041 {
10341042 if (gfp_mask & __GFP_FS)
10351043 return true;
1036− if (!folio_test_swapcache(folio) || !(gfp_mask & __GFP_IO))
1037− return false;
10381044 /*
1039− * We can "enter_fs" for swap-cache with only __GFP_IO
1040− * providing this isn't SWP_FS_OPS.
1041− * ->flags can be updated non-atomically,
1042− * but that will never affect SWP_FS_OPS, so the data_race
1043− * is safe.
1045+ * We can "enter_fs" for swap-cache with only __GFP_IO unless backed by
1046+ * a swapfile that requires GFP_NOFS I/O.
10441047 */
1045− return !data_race(folio_swap_flags(folio) & SWP_FS_OPS);
1048+ if (folio_test_swapcache(folio) && (gfp_mask & __GFP_IO) &&
1049+ !(__swap_entry_to_info(folio->swap)->ops->flags &
1050+ SWAP_OPS_F_REQUIRE_NOFS))
1051+ return true;
1052+ return false;
10461053 }
10471054
10481055 /*
@@ -1059,7 +1066,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, · 10 unmodified lines
10591066 unsigned int nr_reclaimed = 0, nr_demoted = 0;
10601067 unsigned int pgactivate = 0;
10611068 bool do_demote_pass;
1062− struct swap_iocb *plug = NULL;
1069+ struct swap_io_ctx ctx = {};
10631070
10641071 folio_batch_init(&free_folios);
10651072 memset(stat, 0, sizeof(*stat));
@@ -1387,7 +1394,7 @@ retry: · 321 unmodified lines
13871394 * starts and then write it out here.
13881395 */
13891396 try_to_unmap_flush_dirty();
1390− switch (pageout(folio, mapping, &plug, folio_list)) {
1397+ switch (pageout(&ctx, mapping, folio, folio_list)) {
13911398 case PAGE_KEEP:
13921399 goto keep_locked;
13931400 case PAGE_ACTIVATE:
@@ -1406,8 +1413,6 @@ retry: · 12 unmodified lines
14061413 sc->nr_scanned -= (nr_pages - 1);
14071414 nr_pages = 1;
14081415 }
1409− stat->nr_pageout += nr_pages;
1410−
14111416 if (folio_test_writeback(folio))
14121417 goto keep;
14131418 if (folio_test_dirty(folio))
@@ -1577,8 +1582,7 @@ keep: · 163 unmodified lines
15771582 list_splice(&ret_folios, folio_list);
15781583 count_vm_events(PGACTIVATE, pgactivate);
15791584
1580− if (plug)
1581− swap_write_unplug(plug);
1585+ swap_write_submit(&ctx);
15821586 return nr_reclaimed;
15831587 }
15841588
@@ -2031,10 +2035,10 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, · 446 unmodified lines
20312035 item = PGSTEAL_KSWAPD + reclaimer_offset(sc);
20322036 mod_lruvec_state(lruvec, item, nr_reclaimed);
20332037 mod_lruvec_state(lruvec, PGSTEAL_ANON + file, nr_reclaimed);
2038+ if (nr_scanned > nr_reclaimed)
2039+ mod_lruvec_state(lruvec, PGROTATE_ANON + file,
2040+ nr_scanned - nr_reclaimed);
20342041
2035− lruvec_lock_irq(lruvec);
2036− lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout,
2037− nr_scanned - nr_reclaimed);
20382042 handle_reclaim_writeback(nr_taken, pgdat, sc, &stat);
20392043 trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
20402044 nr_scanned, nr_reclaimed, &stat, sc->priority, file);
@@ -2065,7 +2069,7 @@ static void shrink_active_list(unsigned long nr_to_scan, · 24 unmodified lines
20652069 {
20662070 unsigned long nr_taken;
20672071 unsigned long nr_scanned;
2068− vm_flags_t vm_flags;
2072+ vma_flags_t vma_flags;
20692073 LIST_HEAD(l_hold); /* The folios which were snipped off */
20702074 LIST_HEAD(l_active);
20712075 LIST_HEAD(l_inactive);
@@ -2109,7 +2113,7 @@ static void shrink_active_list(unsigned long nr_to_scan, · 37 unmodified lines
21092113
21102114 /* Referenced or rmap lock contention: rotate */
21112115 if (folio_referenced(folio, 0, sc->target_mem_cgroup,
2112− &vm_flags) != 0) {
2116+ &vma_flags) != 0) {
21132117 /*
21142118 * Identify referenced, file-backed active folios and
21152119 * give them one more trip around the active list. So
@@ -2119,7 +2123,7 @@ static void shrink_active_list(unsigned long nr_to_scan, · 3 unmodified lines
21192123 * IO, plus JVM can create lots of anon VM_EXEC folios,
21202124 * so we ignore them here.
21212125 */
2122− if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) {
2126+ if (is_exec_file_folio(folio, &vma_flags)) {
21232127 nr_rotated += folio_nr_pages(folio);
21242128 list_add(&folio->lru, &l_active);
21252129 continue;
@@ -2140,9 +2144,9 @@ static void shrink_active_list(unsigned long nr_to_scan, · 14 unmodified lines
21402144 count_vm_events(PGDEACTIVATE, nr_deactivate);
21412145 count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate);
21422146 mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken);
2147+ if (nr_rotated)
2148+ mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated);
21432149
2144− lruvec_lock_irq(lruvec);
2145− lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated);
21462150 trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
21472151 nr_deactivate, nr_rotated, sc->priority, file);
21482152 }
@@ -2275,8 +2279,10 @@ enum scan_balance { · 126 unmodified lines
22752279
22762280 static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc)
22772281 {
2278− unsigned long file;
2282+ struct lru_cost *anon_cost, *file_cost;
22792283 struct lruvec *target_lruvec;
2284+ unsigned long lrusize;
2285+ unsigned long file;
22802286
22812287 if (lru_gen_enabled() && !lru_gen_switching())
22822288 return;
@@ -2292,11 +2298,69 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) · 9 unmodified lines
22922298
22932299 /*
22942300 * Determine the scan balance between anon and file LRUs.
2301+ *
2302+ * The cost model is based on rotations, refaults and
2303+ * reclaim-driven writes (anon only) on each side.
2304+ *
2305+ * These event counters are monotonic, so each reclaim cycle
2306+ * the delta since the last scan is extracted and incorporated
2307+ * into a decaying average. This ensures currency, as workloads
2308+ * change over time, and avoids overflow in the calculations.
2309+ *
2310+ * Use lruvec_page_state_monotonic() so unsigned subtraction
2311+ * yields the correct delta across a signed-long wraparound of
2312+ * the underlying counter (a real hazard on 32-bit that the
2313+ * clamp in lruvec_page_state() would otherwise turn into a huge
2314+ * spurious delta).
22952315 */
2296− spin_lock_irq(&target_lruvec->lru_lock);
2297− sc->anon_cost = target_lruvec->anon_cost;
2298− sc->file_cost = target_lruvec->file_cost;
2299− spin_unlock_irq(&target_lruvec->lru_lock);
2316+ spin_lock(&target_lruvec->cost_lock);
2317+
2318+ for (int f = 0; f <= 1; f++) {
2319+ struct lru_cost *cost = &target_lruvec->cost[f];
2320+ unsigned long rotated, io, nr_rotated, nr_io;
2321+
2322+ rotated = lruvec_page_state_monotonic(target_lruvec,
2323+ PGROTATE_ANON + f);
2324+ io = lruvec_page_state_monotonic(target_lruvec,
2325+ WORKINGSET_RESTORE_BASE + f);
2326+ if (f == WORKINGSET_ANON)
2327+ io += lruvec_page_state_monotonic(target_lruvec,
2328+ NR_VMSCAN_WRITE);
2329+
2330+ nr_rotated = rotated - cost->last_rotated;
2331+ nr_io = io - cost->last_io;
2332+
2333+ /*
2334+ * Reflect the relative cost of incurring IO and spending
2335+ * CPU time on rotations. This doesn't attempt to make a
2336+ * precise comparison, it just says: if reloads are about
2337+ * comparable between the LRU lists, or rotations are
2338+ * overwhelmingly different between them, adjust scan
2339+ * balance for CPU work.
2340+ */
2341+ cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated;
2342+
2343+ cost->last_rotated = rotated;
2344+ cost->last_io = io;
2345+ }
2346+
2347+ anon_cost = &target_lruvec->cost[WORKINGSET_ANON];
2348+ file_cost = &target_lruvec->cost[WORKINGSET_FILE];
2349+
2350+ lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) +
2351+ lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) +
2352+ lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) +
2353+ lruvec_page_state(target_lruvec, NR_ACTIVE_FILE);
2354+
2355+ while (anon_cost->count + file_cost->count > lrusize / 4) {
2356+ anon_cost->count /= 2;
2357+ file_cost->count /= 2;
2358+ }
2359+
2360+ sc->anon_cost = anon_cost->count;
2361+ sc->file_cost = file_cost->count;
2362+
2363+ spin_unlock(&target_lruvec->cost_lock);
23002364
23012365 /*
23022366 * Target desirable inactive:active list ratios for the anon
@@ -3202,14 +3266,19 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) · 899 unmodified lines
32023266 ******************************************************************************/
32033267
32043268 /* promote pages accessed through page tables */
3205−static int folio_update_gen(struct folio *folio, int gen)
3269+static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags)
32063270 {
32073271 unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
32083272
32093273 VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
32103274
3211− /* see the comment on LRU_REFS_FLAGS */
3212− if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
3275+ /*
3276+ * See the comment on LRU_REFS_FLAGS, and activate file-backed
3277+ * executable folios after first usage to avoid typical IO
3278+ * thrashing from reclaiming.
3279+ */
3280+ if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
3281+ !is_exec_file_folio(folio, vma_flags)) {
32133282 set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
32143283 return -1;
32153284 }
@@ -3442,8 +3511,8 @@ static bool suitable_to_scan(int total, int young) · 226 unmodified lines
34423511 return young * n >= total;
34433512 }
34443513
3445−static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio,
3446− int new_gen, bool dirty)
3514+static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
3515+ struct folio *folio, int new_gen, bool dirty)
34473516 {
34483517 int old_gen;
34493518
@@ -3456,10 +3525,10 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio, · 6 unmodified lines
34563525 folio_mark_dirty(folio);
34573526
34583527 if (walk) {
3459− old_gen = folio_update_gen(folio, new_gen);
3528+ old_gen = folio_update_gen(folio, new_gen, &vma->flags);
34603529 if (old_gen >= 0 && old_gen != new_gen)
34613530 update_batch_size(walk, folio, old_gen, new_gen);
3462− } else if (lru_gen_set_refs(folio)) {
3531+ } else if (lru_gen_set_refs(folio, &vma->flags)) {
34633532 old_gen = folio_lru_gen(folio);
34643533 if (old_gen >= 0 && old_gen != new_gen)
34653534 folio_activate(folio);
@@ -3532,7 +3601,7 @@ restart: · 66 unmodified lines
35323601 continue;
35333602
35343603 if (last != folio) {
3535− walk_update_folio(walk, last, gen, dirty);
3604+ walk_update_folio(walk, args->vma, last, gen, dirty);
35363605
35373606 last = folio;
35383607 dirty = false;
@@ -3545,7 +3614,7 @@ restart: · 6 unmodified lines
35453614 walk->mm_stats[MM_LEAF_YOUNG] += nr;
35463615 }
35473616
3548− walk_update_folio(walk, last, gen, dirty);
3617+ walk_update_folio(walk, args->vma, last, gen, dirty);
35493618 last = NULL;
35503619
35513620 if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
@@ -3623,7 +3692,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area · 71 unmodified lines
36233692 goto next;
36243693
36253694 if (last != folio) {
3626− walk_update_folio(walk, last, gen, dirty);
3695+ walk_update_folio(walk, vma, last, gen, dirty);
36273696
36283697 last = folio;
36293698 dirty = false;
@@ -3637,7 +3706,7 @@ next: · 7 unmodified lines
36373706 i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1;
36383707 } while (i <= MIN_LRU_BATCH);
36393708
3640− walk_update_folio(walk, last, gen, dirty);
3709+ walk_update_folio(walk, vma, last, gen, dirty);
36413710
36423711 lazy_mmu_mode_disable();
36433712 spin_unlock(ptl);
@@ -4192,7 +4261,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) · 548 unmodified lines
41924261 unsigned long end;
41934262 struct lru_gen_mm_walk *walk;
41944263 struct folio *last = NULL;
4195− int young = 1;
4264+ int young = nr;
41964265 pte_t *pte = pvmw->pte;
41974266 unsigned long addr = pvmw->address;
41984267 struct vm_area_struct *vma = pvmw->vma;
@@ -4272,7 +4341,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) · 73 unmodified lines
42724341 continue;
42734342
42744343 if (last != folio) {
4275− walk_update_folio(walk, last, gen, dirty);
4344+ walk_update_folio(walk, vma, last, gen, dirty);
42764345
42774346 last = folio;
42784347 dirty = false;
@@ -4284,7 +4353,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) · 5 unmodified lines
42844353 young += nr;
42854354 }
42864355
4287− walk_update_folio(walk, last, gen, dirty);
4356+ walk_update_folio(walk, vma, last, gen, dirty);
42884357
42894358 lazy_mmu_mode_disable();
42904359
@@ -4566,7 +4635,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, · 275 unmodified lines
45664635 for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
45674636 unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
45684637
4638+ if (!size)
4639+ continue;
4640+
4641+ /* Move the accounting, do not duplicate it. */
45694642 mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
4643+ mem_cgroup_update_lru_size(child_lruvec, lru, zid, -(long)size);
45704644 }
45714645 }
45724646 }
@@ -4580,7 +4654,6 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, · 7 unmodified lines
45804654 static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc,
45814655 int tier_idx)
45824656 {
4583− bool success;
45844657 int gen = folio_lru_gen(folio);
45854658 int type = folio_is_file_lru(folio);
45864659 int zone = folio_zonenum(folio);
@@ -4592,15 +4665,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c · 5 unmodified lines
45924665
45934666 VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
45944667
4595− /* unevictable */
4596− if (!folio_evictable(folio)) {
4597− success = lru_gen_del_folio(lruvec, folio, true);
4598− VM_WARN_ON_ONCE_FOLIO(!success, folio);
4599− folio_set_unevictable(folio);
4600− lruvec_add_folio(lruvec, folio);
4601− __count_vm_events(UNEVICTABLE_PGCULLED, delta);
4602− return true;
4603− }
4668+ /* unevictable: let it through and the generic path will cull it */
4669+ if (!folio_evictable(folio))
4670+ return false;
46044671
46054672 /* promoted */
46064673 if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
@@ -4816,7 +4883,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 209 unmodified lines
48164883 struct reclaim_stat stat;
48174884 struct lru_gen_mm_walk *walk;
48184885 int scanned, reclaimed;
4819− int isolated = 0, type, type_scanned;
4886+ int isolated = 0, nr_isolated = 0, type, type_scanned;
4887+ unsigned long total_reclaimed = 0;
48204888 bool skip_retry = false;
48214889 struct mem_cgroup *memcg = lruvec_memcg(lruvec);
48224890 struct pglist_data *pgdat = lruvec_pgdat(lruvec);
@@ -4828,6 +4896,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 5 unmodified lines
48284896
48294897 scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness,
48304898 &list, &isolated, &type, &type_scanned);
4899+ nr_isolated = isolated;
48314900
48324901 /* Scanning may have emptied the oldest gen, flush it */
48334902 if (scanned)
@@ -4840,6 +4909,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, · 6 unmodified lines
48404909 retry:
48414910 reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false, memcg);
48424911 sc->nr_reclaimed += reclaimed;
4912+ total_reclaimed += reclaimed;
48434913 /* Retry pass is only meant for clean folios without new isolation */
48444914 if (isolated)
48454915 handle_reclaim_writeback(isolated, pgdat, sc, &stat);
@@ -4850,11 +4920,9 @@ retry: · 4 unmodified lines
48504920 list_for_each_entry_safe_reverse(folio, next, &list, lru) {
48514921 DEFINE_MIN_SEQ(lruvec);
48524922
4853− if (!folio_evictable(folio)) {
4854− list_del(&folio->lru);
4855− folio_putback_lru(folio);
4923+ /* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */
4924+ if (!folio_evictable(folio))
48564925 continue;
4857− }
48584926
48594927 /* retry folios that may have missed folio_rotate_reclaimable() */
48604928 if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) &&
@@ -4891,6 +4959,10 @@ retry: · 30 unmodified lines
48914959 goto retry;
48924960 }
48934961
4962+ if (nr_isolated > total_reclaimed)
4963+ mod_lruvec_state(lruvec, PGROTATE_ANON + type,
4964+ nr_isolated - total_reclaimed);
4965+
48944966 return scanned;
48954967 }
48964968
@@ -7209,7 +7281,7 @@ restart: · 2312 unmodified lines
72097281
72107282 /*
72117283 * If the low watermark is met there is no need for processes
7212− * to be throttled on pfmemalloc_wait as they should not be
7284+ * to be throttled on pfmemalloc_wait as they should now be
72137285 * able to safely make forward progress. Wake them
72147286 */
72157287 if (waitqueue_active(&pgdat->pfmemalloc_wait) &&
mm/vmstat.c+18−1
@@ -1025,6 +1025,17 @@ unsigned long node_page_state(struct pglist_data *pgdat, · 1024 unmodified lines
10251025
10261026 return node_page_state_pages(pgdat, item);
10271027 }
1028+
1029+/*
1030+ * Non-clamping variant of node_page_state() intended for callers that
1031+ * snapshot a monotonically-incremented counter and subtract two samples.
1032+ * See global_node_page_state_monotonic() for the rationale.
1033+ */
1034+unsigned long node_page_state_monotonic(struct pglist_data *pgdat,
1035+ enum node_stat_item item)
1036+{
1037+ return (unsigned long)atomic_long_read(&pgdat->vm_stat[item]);
1038+}
10281039 #endif
10291040
10301041 /*
@@ -1290,6 +1301,8 @@ const char * const vmstat_text[] = { · 259 unmodified lines
12901301 [I(PGSCAN_PROACTIVE)] = "pgscan_proactive",
12911302 [I(PGSCAN_ANON)] = "pgscan_anon",
12921303 [I(PGSCAN_FILE)] = "pgscan_file",
1304+ [I(PGROTATE_ANON)] = "pgrotate_anon",
1305+ [I(PGROTATE_FILE)] = "pgrotate_file",
12931306 [I(PGREFILL)] = "pgrefill",
12941307 #ifdef CONFIG_HUGETLB_PAGE
12951308 [I(NR_HUGETLB)] = "nr_hugetlb",
@@ -1489,7 +1502,11 @@ const char * const vmstat_text[] = { · 193 unmodified lines
14891502 #if THREAD_SIZE > 65536
14901503 [I(KSTACK_REST)] = "kstack_rest",
14911504 #endif
1492−#endif
1505+#endif /* CONFIG_DEBUG_STACK_USAGE */
1506+#ifdef CONFIG_SWAP
1507+ [I(NRSWPIN)] = "nrswpin",
1508+ [I(NRSWPOUT)] = "nrswpout",
1509+#endif /* CONFIG_SWAP */
14931510 #undef I
14941511 #endif /* CONFIG_VM_EVENT_COUNTERS */
14951512 };
mm/workingset.c+0−5
@@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow) · 583 unmodified lines
584584 /* Folio was active prior to eviction */
585585 if (workingset) {
586586 folio_set_workingset(folio);
587− /*
588− * XXX: Move to folio_add_lru() when it supports new vs
589− * putback
590− */
591− lru_note_cost_refault(folio);
592587 mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr);
593588 }
594589 out:
mm/zsmalloc.c+7−4
@@ -552,6 +552,11 @@ static int get_size_class_index(int size) · 551 unmodified lines
552552 return min_t(int, ZS_SIZE_CLASSES - 1, idx);
553553 }
554554
555+static struct size_class *lookup_size_class(struct zs_pool *pool, size_t size)
556+{
557+ return pool->size_class[get_size_class_index(size + ZS_HANDLE_SIZE)];
558+}
559+
555560 static inline void class_stat_add(struct size_class *class, int type,
556561 unsigned long cnt)
557562 {
@@ -1117,7 +1122,7 @@ unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size) · 559 unmodified lines
11171122 {
11181123 struct size_class *class;
11191124
1120− class = pool->size_class[get_size_class_index(size)];
1125+ class = lookup_size_class(pool, size);
11211126
11221127 return class->index;
11231128 }
@@ -1407,9 +1412,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp, · 283 unmodified lines
14071412 if (!handle)
14081413 return (unsigned long)ERR_PTR(-ENOMEM);
14091414
1410− /* extra space in chunk to keep the handle */
1411− size += ZS_HANDLE_SIZE;
1412− class = pool->size_class[get_size_class_index(size)];
1415+ class = lookup_size_class(pool, size);
14131416
14141417 /* class->lock effectively protects the zpage migration */
14151418 spin_lock(&class->lock);
mm/zswap.c+29−17
@@ -20,7 +20,7 @@ · 19 unmodified lines
2020 #include <linux/spinlock.h>
2121 #include <linux/types.h>
2222 #include <linux/atomic.h>
23−#include <linux/swap.h>
23+#include <linux/swap_ops.h>
2424 #include <linux/crypto.h>
2525 #include <linux/scatterlist.h>
2626 #include <linux/mempolicy.h>
@@ -665,7 +665,7 @@ static inline int entry_to_nid(struct zswap_entry *entry) · 638 unmodified lines
665665 return page_to_nid(virt_to_page(entry));
666666 }
667667
668−static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry)
668+static void zswap_lru_add(struct zswap_entry *entry)
669669 {
670670 int nid = entry_to_nid(entry);
671671 struct mem_cgroup *memcg;
@@ -684,11 +684,11 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) · 12 unmodified lines
684684 rcu_read_lock();
685685 memcg = mem_cgroup_from_entry(entry);
686686 /* will always succeed */
687− list_lru_add(list_lru, &entry->lru, nid, memcg);
687+ list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
688688 rcu_read_unlock();
689689 }
690690
691−static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *entry)
691+static void zswap_lru_del(struct zswap_entry *entry)
692692 {
693693 int nid = entry_to_nid(entry);
694694 struct mem_cgroup *memcg;
@@ -696,7 +696,7 @@ static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *entry) · 1 unmodified line
696696 rcu_read_lock();
697697 memcg = mem_cgroup_from_entry(entry);
698698 /* will always succeed */
699− list_lru_del(list_lru, &entry->lru, nid, memcg);
699+ list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
700700 rcu_read_unlock();
701701 }
702702
@@ -764,7 +764,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry) · 61 unmodified lines
764764 */
765765 static void zswap_entry_free(struct zswap_entry *entry)
766766 {
767− zswap_lru_del(&zswap_list_lru, entry);
767+ zswap_lru_del(entry);
768768 zs_free(entry->pool->zs_pool, entry->handle);
769769 zswap_pool_put(entry->pool);
770770 if (entry->objcg) {
@@ -992,6 +992,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, · 221 unmodified lines
992992 struct folio *folio;
993993 struct mempolicy *mpol;
994994 struct swap_info_struct *si;
995+ struct swap_io_ctx ctx = {};
995996 int ret = 0;
996997
997998 /* try to allocate swap cache folio */
@@ -1049,7 +1050,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry, · 51 unmodified lines
10491050 folio_set_reclaim(folio);
10501051
10511052 /* start writeback */
1052− __swap_writepage(folio, NULL);
1053+ __swap_writepage(&ctx, folio);
1054+ swap_write_submit(&ctx);
10531055
10541056 out:
10551057 if (ret) {
@@ -1217,7 +1219,7 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker, · 161 unmodified lines
12171219 * Without memcg, use the zswap pool-wide metrics.
12181220 */
12191221 if (!mem_cgroup_disabled()) {
1220− mem_cgroup_flush_stats(memcg);
1222+ mem_cgroup_flush_stats_ratelimited(memcg);
12211223 nr_backing = memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT;
12221224 nr_stored = memcg_page_state(memcg, MEMCG_ZSWAPPED);
12231225 } else {
@@ -1275,6 +1277,14 @@ static struct shrinker *zswap_alloc_shrinker(void) · 51 unmodified lines
12751277 return shrinker;
12761278 }
12771279
1280+/*
1281+ * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of @memcg
1282+ * and write back the reclaimable ones.
1283+ *
1284+ * Return: 0 if at least one entry was written back, -EAGAIN if entries
1285+ * were scanned but none could be written back, or -ENOENT if @memcg has
1286+ * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
1287+ */
12781288 static int shrink_memcg(struct mem_cgroup *memcg)
12791289 {
12801290 int nid, shrunk = 0, scanned = 0;
@@ -1290,13 +1300,14 @@ static int shrink_memcg(struct mem_cgroup *memcg) · 9 unmodified lines
12901300 return -ENOENT;
12911301
12921302 for_each_node_state(nid, N_NORMAL_MEMORY) {
1293− unsigned long nr_to_walk = 1;
1303+ unsigned long nr_to_walk = SWAP_CLUSTER_MAX;
12941304
12951305 shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
12961306 &shrink_memcg_cb, NULL, &nr_to_walk);
1297− scanned += 1 - nr_to_walk;
1307+ scanned += SWAP_CLUSTER_MAX - nr_to_walk;
12981308 }
12991309
1310+ /* Nothing was scanned: every LRU under @memcg was empty. */
13001311 if (!scanned)
13011312 return -ENOENT;
13021313
@@ -1356,11 +1367,12 @@ static void shrink_worker(struct work_struct *w) · 53 unmodified lines
13561367 } while (memcg && !mem_cgroup_tryget_online(memcg));
13571368 spin_unlock(&zswap_shrink_lock);
13581369
1359− if (!memcg) {
1360− /*
1361− * Continue shrinking without incrementing failures if
1362− * we found candidate memcgs in the last tree walk.
1363− */
1370+ /*
1371+ * A NULL memcg ends a full hierarchy pass (except when memcg is
1372+ * disabled, where it is always NULL: fall through to the root LRU).
1373+ * Count a failure only if the last pass found no candidates.
1374+ */
1375+ if (!memcg && !mem_cgroup_disabled()) {
13641376 if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
13651377 break;
13661378
@@ -1379,7 +1391,7 @@ static void shrink_worker(struct work_struct *w) · 12 unmodified lines
13791391 * and failures.
13801392 */
13811393 if (ret == -ENOENT)
1382− continue;
1394+ goto resched;
13831395 ++attempts;
13841396
13851397 if (ret && ++failures == MAX_RECLAIM_RETRIES)
@@ -1461,7 +1473,7 @@ static bool zswap_store_page(struct page *page, · 75 unmodified lines
14611473 entry->referenced = true;
14621474 if (entry->length) {
14631475 INIT_LIST_HEAD(&entry->lru);
1464− zswap_lru_add(&zswap_list_lru, entry);
1476+ zswap_lru_add(entry);
14651477 }
14661478
14671479 return true;
tools/testing/radix-tree/maple.c+2−2
@@ -35234,7 +35234,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt) · 35233 unmodified lines
3523435234 mt_set_non_kernel(1);
3523535235 /* Spanning store */
3523635236 mas_set_range(&mas, 1, 100);
35237− MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL & GFP_NOWAIT) == 0);
35237+ MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_NOWAIT) == 0);
3523835238 allocated = mas_allocated(&mas);
3523935239 height = mas_mt_height(&mas);
3524035240 MT_BUG_ON(mt, allocated != 0);
@@ -35257,7 +35257,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt) · 16 unmodified lines
3525735257 MT_BUG_ON(mt, mas_allocated(&mas) != 0);
3525835258 mas_set_range(&mas, 0, 200);
3525935259 mt_set_non_kernel(1);
35260− MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL & GFP_NOWAIT) == 0);
35260+ MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_NOWAIT) == 0);
3526135261 allocated = mas_allocated(&mas);
3526235262 height = mas_mt_height(&mas);
3526335263 MT_BUG_ON(mt, allocated != 0);
tools/testing/selftests/Makefile+1−0
@@ -1,5 +1,6 @@
11 # SPDX-License-Identifier: GPL-2.0
22 TARGETS += acct
3+TARGETS += alloc_tag
34 TARGETS += alsa
45 TARGETS += amd-pstate
56 TARGETS += arm64
tools/testing/selftests/alloc_tag/Makefile+8−0
@@ -0,0 +1,8 @@
1+# SPDX-License-Identifier: GPL-2.0
2+
3+TEST_GEN_PROGS := allocinfo_ioctl_test
4+
5+CFLAGS += -Wall
6+CFLAGS += $(KHDR_INCLUDES)
7+
8+include ../lib.mk
tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c+548−0
@@ -0,0 +1,548 @@
1+// SPDX-License-Identifier: GPL-2.0-only
2+
3+/* kselftest for allocinfo ioctl
4+ * allocinfo ioctl retrieves allocinfo data through ioctl
5+ * Copyright (C) 2026 Google, Inc.
6+ */
7+
8+#include <errno.h>
9+#include <fcntl.h>
10+#include <stdio.h>
11+#include <stdlib.h>
12+#include <string.h>
13+#include <stdbool.h>
14+#include <unistd.h>
15+#include <sys/ioctl.h>
16+#include <linux/types.h>
17+#include <linux/alloc_tag.h>
18+#include "../kselftest.h"
19+
20+#define MAX_LINE_LEN 512
21+#define ALLOCINFO_PROC "/proc/allocinfo"
22+
23+enum ioctl_ret {
24+ IOCTL_SUCCESS = 0,
25+ IOCTL_FAILURE = 1,
26+ IOCTL_INVALID_DATA = 2,
27+};
28+
29+#define VEC_MAX_ENTRIES 32
30+
31+struct allocinfo_tag_data_vec {
32+ struct allocinfo_tag_data tag[VEC_MAX_ENTRIES];
33+ __u64 count;
34+};
35+
36+static inline int __allocinfo_get_content_id(int dev_fd, struct allocinfo_content_id *params)
37+{
38+ return ioctl(dev_fd, ALLOCINFO_IOC_CONTENT_ID, params);
39+}
40+
41+static inline int __allocinfo_get_at(int dev_fd, struct allocinfo_get_at *params)
42+{
43+ return ioctl(dev_fd, ALLOCINFO_IOC_GET_AT, params);
44+}
45+
46+static inline int __allocinfo_get_next(int dev_fd, struct allocinfo_tag_data *params)
47+{
48+ return ioctl(dev_fd, ALLOCINFO_IOC_GET_NEXT, params);
49+}
50+
51+static bool match_entry(const struct allocinfo_tag_data *procfs_entry,
52+ const struct allocinfo_tag_data *tag_data,
53+ bool match_bytes, bool match_calls, bool match_lineno,
54+ bool match_function, bool match_filename)
55+{
56+ if (match_bytes && tag_data->counter.bytes != procfs_entry->counter.bytes) {
57+ ksft_print_msg("size retrieved through ioctl does not match procfs\n");
58+ return false;
59+ }
60+
61+ if (match_calls && tag_data->counter.calls != procfs_entry->counter.calls) {
62+ ksft_print_msg("call count retrieved through ioctl does not match procfs\n");
63+ return false;
64+ }
65+
66+ if (match_lineno && tag_data->tag.lineno != procfs_entry->tag.lineno) {
67+ ksft_print_msg("lineno retrieved through ioctl does not match procfs\n");
68+ return false;
69+ }
70+
71+ if (match_function &&
72+ strncmp(tag_data->tag.function, procfs_entry->tag.function, ALLOCINFO_STR_SIZE)) {
73+ ksft_print_msg("function retrieved through ioctl does not match procfs\n");
74+ return false;
75+ }
76+
77+ if (match_filename &&
78+ strncmp(tag_data->tag.filename, procfs_entry->tag.filename, ALLOCINFO_STR_SIZE)) {
79+ ksft_print_msg("filename retrieved through ioctl does not match procfs\n");
80+ return false;
81+ }
82+ return true;
83+}
84+
85+static bool match_entries(const struct allocinfo_tag_data_vec *procfs_entries,
86+ const struct allocinfo_tag_data_vec *tags,
87+ bool match_bytes, bool match_calls, bool match_lineno,
88+ bool match_function, bool match_filename)
89+{
90+ __u64 i;
91+
92+ if (procfs_entries->count != tags->count) {
93+ ksft_print_msg("Entry count mismatch. ioctl entries: %llu, proc entries: %llu\n",
94+ tags->count, procfs_entries->count);
95+ return false;
96+ }
97+ for (i = 0; i < procfs_entries->count; i++) {
98+ if (!match_entry(&procfs_entries->tag[i], &tags->tag[i],
99+ match_bytes, match_calls, match_lineno,
100+ match_function, match_filename)) {
101+ ksft_print_msg("%lluth entry does not match.\n", i);
102+ return false;
103+ }
104+ }
105+ return true;
106+}
107+
108+static const char *allocinfo_str(const char *str)
109+{
110+ size_t len = strlen(str);
111+
112+ if (len >= ALLOCINFO_STR_SIZE)
113+ str += (len - ALLOCINFO_STR_SIZE) + 1;
114+ return str;
115+}
116+
117+static void allocinfo_copy_str(char *dest, const char *src)
118+{
119+ strncpy(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE - 1);
120+ dest[ALLOCINFO_STR_SIZE - 1] = '\0';
121+}
122+
123+static int get_filtered_procfs_entries(struct allocinfo_tag_data_vec *procfs_entries,
124+ const struct allocinfo_filter *filter)
125+{
126+ FILE *fp = fopen(ALLOCINFO_PROC, "r");
127+ char line[MAX_LINE_LEN];
128+ int matches;
129+ struct allocinfo_tag_data procfs_entry;
130+
131+ if (!fp) {
132+ ksft_print_msg("Failed to open " ALLOCINFO_PROC " for reading\n");
133+ return 1;
134+ }
135+ memset(procfs_entries, 0, sizeof(*procfs_entries));
136+ while (fgets(line, sizeof(line), fp) && procfs_entries->count < VEC_MAX_ENTRIES) {
137+ char filename[MAX_LINE_LEN];
138+ char function[MAX_LINE_LEN];
139+
140+ memset(&procfs_entry, 0, sizeof(procfs_entry));
141+ matches = sscanf(line, "%llu %llu %[^:]:%llu func:%s",
142+ &procfs_entry.counter.bytes,
143+ &procfs_entry.counter.calls,
144+ filename,
145+ &procfs_entry.tag.lineno,
146+ function);
147+
148+ if (matches != 5)
149+ continue;
150+
151+ allocinfo_copy_str(procfs_entry.tag.filename, filename);
152+ allocinfo_copy_str(procfs_entry.tag.function, function);
153+
154+ if (filter->mask & ALLOCINFO_FILTER_MASK_FILENAME) {
155+ if (strncmp(procfs_entry.tag.filename,
156+ filter->fields.filename, ALLOCINFO_STR_SIZE))
157+ continue;
158+ }
159+ if (filter->mask & ALLOCINFO_FILTER_MASK_FUNCTION) {
160+ if (strncmp(procfs_entry.tag.function,
161+ filter->fields.function, ALLOCINFO_STR_SIZE))
162+ continue;
163+ }
164+ if (filter->mask & ALLOCINFO_FILTER_MASK_LINENO) {
165+ if (procfs_entry.tag.lineno != filter->fields.lineno)
166+ continue;
167+ }
168+ if (filter->mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) {
169+ if (procfs_entry.counter.bytes < filter->min_size)
170+ continue;
171+ }
172+ if (filter->mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) {
173+ if (procfs_entry.counter.bytes > filter->max_size)
174+ continue;
175+ }
176+
177+ memcpy(&procfs_entries->tag[procfs_entries->count++], &procfs_entry,
178+ sizeof(procfs_entry));
179+ }
180+ fclose(fp);
181+ return 0;
182+}
183+
184+static enum ioctl_ret get_filtered_ioctl_entries(struct allocinfo_tag_data_vec *tags,
185+ const struct allocinfo_filter *filter,
186+ __u64 start_pos)
187+{
188+ int fd = open(ALLOCINFO_PROC, O_RDONLY);
189+
190+ if (fd < 0) {
191+ ksft_print_msg("Failed to open " ALLOCINFO_PROC " for IOCTL\n");
192+ return IOCTL_FAILURE;
193+ }
194+
195+ struct allocinfo_content_id start_cont_id, end_cont_id;
196+ struct allocinfo_get_at get_at_params;
197+ const int max_retries = 10;
198+ int retry_count = 0;
199+ int status;
200+
201+ /*
202+ * __allocinfo_get_content_id may return different values if a kernel module was loaded
203+ * between the two calls. If that happens, the data gathered cannot be considered consistent
204+ * and hence needs to be fetched again to avoid flakiness.
205+ */
206+ do {
207+ if (__allocinfo_get_content_id(fd, &start_cont_id)) {
208+ ksft_print_msg("allocinfo_get_content_id failed\n");
209+ status = IOCTL_FAILURE;
210+ break;
211+ }
212+
213+ memset(tags, 0, sizeof(*tags));
214+ memset(&get_at_params, 0, sizeof(get_at_params));
215+ memcpy(&get_at_params.filter, filter, sizeof(*filter));
216+ get_at_params.pos = start_pos;
217+ if (__allocinfo_get_at(fd, &get_at_params)) {
218+ ksft_print_msg("allocinfo_get_at failed\n");
219+ status = IOCTL_FAILURE;
220+ break;
221+ }
222+ memcpy(&tags->tag[tags->count++], &get_at_params.data, sizeof(get_at_params.data));
223+
224+ while (tags->count < VEC_MAX_ENTRIES &&
225+ __allocinfo_get_next(fd, &tags->tag[tags->count]) == 0)
226+ tags->count++;
227+
228+ if (__allocinfo_get_content_id(fd, &end_cont_id)) {
229+ ksft_print_msg("allocinfo_get_content_id failed\n");
230+ status = IOCTL_FAILURE;
231+ break;
232+ }
233+
234+ if (start_cont_id.id == end_cont_id.id) {
235+ status = IOCTL_SUCCESS;
236+ } else {
237+ ksft_print_msg("allocinfo_get_content_id mismatch, retrying...\n");
238+ status = IOCTL_INVALID_DATA;
239+ }
240+ } while (status == IOCTL_INVALID_DATA && retry_count++ < max_retries);
241+
242+ close(fd);
243+ return status;
244+}
245+
246+static int run_filter_test(const struct allocinfo_filter *filter)
247+{
248+ struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags));
249+ struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries));
250+ int ioctl_status;
251+ int ret = KSFT_PASS;
252+
253+ if (!tags || !procfs_entries) {
254+ ksft_print_msg("Memory allocation failed.\n");
255+ ret = KSFT_FAIL;
256+ goto exit;
257+ }
258+
259+ if (get_filtered_procfs_entries(procfs_entries, filter)) {
260+ ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n");
261+ ret = KSFT_SKIP;
262+ goto exit;
263+ }
264+
265+ if (procfs_entries->count == 0) {
266+ ksft_print_msg("No entries found in " ALLOCINFO_PROC ", skipping test\n");
267+ ret = KSFT_SKIP;
268+ goto exit;
269+ }
270+
271+ ioctl_status = get_filtered_ioctl_entries(tags, filter, 0);
272+ if (ioctl_status == IOCTL_INVALID_DATA) {
273+ ksft_print_msg("Trouble retrieving valid IOCTL entries, skipping.\n");
274+ ret = KSFT_SKIP;
275+ goto exit;
276+ }
277+ if (ioctl_status == IOCTL_FAILURE) {
278+ ksft_print_msg("Error retrieving IOCTL entries.\n");
279+ ret = KSFT_FAIL;
280+ goto exit;
281+ }
282+
283+ if (!match_entries(procfs_entries, tags, false, false, true, true, true))
284+ ret = KSFT_FAIL;
285+
286+exit:
287+ free(tags);
288+ free(procfs_entries);
289+ return ret;
290+}
291+
292+static int test_filename_filter(void)
293+{
294+ struct allocinfo_filter filter;
295+ const char *target_filename = "mm/memory.c";
296+
297+ memset(&filter, 0, sizeof(filter));
298+ filter.mask |= ALLOCINFO_FILTER_MASK_FILENAME;
299+ strncpy(filter.fields.filename, target_filename, ALLOCINFO_STR_SIZE);
300+
301+ return run_filter_test(&filter);
302+}
303+
304+static int test_function_filter(void)
305+{
306+ struct allocinfo_filter filter;
307+ const char *target_function = "dup_mm";
308+
309+ memset(&filter, 0, sizeof(filter));
310+ filter.mask |= ALLOCINFO_FILTER_MASK_FUNCTION;
311+ strncpy(filter.fields.function, target_function, ALLOCINFO_STR_SIZE);
312+
313+ return run_filter_test(&filter);
314+}
315+
316+static int test_size_filter(void)
317+{
318+ int fd;
319+ struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags));
320+ struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries));
321+ struct allocinfo_filter filter;
322+ int ret = KSFT_PASS;
323+ __u64 target_size, i, pos;
324+ struct allocinfo_tag_data *found_tag = NULL;
325+ const char *target_function = "do_init_module";
326+ struct allocinfo_content_id start_cont_id, end_cont_id;
327+ int retry = 0;
328+ const int max_retries = 10;
329+
330+ if (!tags || !procfs_entries) {
331+ ksft_print_msg("Memory allocation failed.\n");
332+ ret = KSFT_FAIL;
333+ goto freemem;
334+ }
335+
336+ fd = open(ALLOCINFO_PROC, O_RDONLY);
337+ if (fd < 0) {
338+ ksft_print_msg("Failed to open " ALLOCINFO_PROC ": %s\n", strerror(errno));
339+ ret = KSFT_SKIP;
340+ goto freemem;
341+ }
342+
343+ do {
344+ found_tag = NULL;
345+ pos = 0;
346+
347+ if (__allocinfo_get_content_id(fd, &start_cont_id)) {
348+ ksft_print_msg("allocinfo_get_content_id failed\n");
349+ ret = KSFT_FAIL;
350+ goto exit;
351+ }
352+
353+ memset(&filter, 0, sizeof(filter));
354+ filter.mask |= ALLOCINFO_FILTER_MASK_FUNCTION;
355+ strncpy(filter.fields.function, target_function, ALLOCINFO_STR_SIZE);
356+
357+ if (get_filtered_procfs_entries(procfs_entries, &filter)) {
358+ ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n");
359+ ret = KSFT_SKIP;
360+ goto exit;
361+ }
362+
363+ if (procfs_entries->count == 0) {
364+ ksft_print_msg("Function %s not found in procfs\n", target_function);
365+ ret = KSFT_SKIP;
366+ goto exit;
367+ }
368+
369+ target_size = procfs_entries->tag[0].counter.bytes;
370+
371+ memset(&filter, 0, sizeof(filter));
372+ filter.mask |= ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE;
373+ filter.min_size = target_size;
374+ filter.max_size = target_size;
375+
376+ while (1) {
377+ struct allocinfo_get_at get_at_params;
378+
379+ memset(&get_at_params, 0, sizeof(get_at_params));
380+ memcpy(&get_at_params.filter, &filter, sizeof(filter));
381+ get_at_params.pos = pos;
382+
383+ if (__allocinfo_get_at(fd, &get_at_params))
384+ break;
385+
386+ tags->count = 0;
387+ memcpy(&tags->tag[tags->count++], &get_at_params.data,
388+ sizeof(get_at_params.data));
389+
390+ while (tags->count < VEC_MAX_ENTRIES &&
391+ __allocinfo_get_next(fd, &tags->tag[tags->count]) == 0)
392+ tags->count++;
393+
394+ for (i = 0; i < tags->count; i++) {
395+ if (strcmp(tags->tag[i].tag.function, target_function) == 0) {
396+ found_tag = &tags->tag[i];
397+ break;
398+ }
399+ }
400+
401+ if (found_tag || tags->count < VEC_MAX_ENTRIES)
402+ break;
403+
404+ pos += tags->count;
405+ }
406+
407+ if (__allocinfo_get_content_id(fd, &end_cont_id)) {
408+ ksft_print_msg("allocinfo_get_content_id failed\n");
409+ ret = KSFT_FAIL;
410+ goto exit;
411+ }
412+
413+ if (start_cont_id.id == end_cont_id.id)
414+ break;
415+
416+ ksft_print_msg("Module load detected during size verification, retrying...\n");
417+ } while (retry++ < max_retries);
418+
419+ if (start_cont_id.id == end_cont_id.id && !found_tag) {
420+ ksft_print_msg("Entry with function %s not found in IOCTL results\n",
421+ target_function);
422+ ret = KSFT_FAIL;
423+ } else if (start_cont_id.id != end_cont_id.id) {
424+ ksft_print_msg("Failed to match content_ids for procfs and IOCTL, skipping...\n");
425+ ret = KSFT_SKIP;
426+ } else if (found_tag && found_tag->counter.bytes != target_size) {
427+ ksft_print_msg("IOCTL entry size %llu does not match target size %llu\n",
428+ found_tag->counter.bytes, target_size);
429+ ret = KSFT_FAIL;
430+ }
431+
432+exit:
433+ close(fd);
434+freemem:
435+ free(tags);
436+ free(procfs_entries);
437+ return ret;
438+}
439+
440+static int test_lineno_filter(void)
441+{
442+ struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags));
443+ struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries));
444+ struct allocinfo_filter filter;
445+ enum ioctl_ret ioctl_status;
446+ int ret = KSFT_PASS;
447+ __u64 target_lineno, i;
448+ struct allocinfo_tag_data *target_tag;
449+ bool found = false;
450+
451+ if (!tags || !procfs_entries) {
452+ ksft_print_msg("Memory allocation failed.\n");
453+ ret = KSFT_FAIL;
454+ goto exit;
455+ }
456+
457+ memset(&filter, 0, sizeof(filter));
458+
459+ if (get_filtered_procfs_entries(procfs_entries, &filter)) {
460+ ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n");
461+ ret = KSFT_SKIP;
462+ goto exit;
463+ }
464+ if (procfs_entries->count == 0) {
465+ ksft_print_msg("Could not retrieve procfs entries\n");
466+ ret = KSFT_SKIP;
467+ goto exit;
468+ }
469+ /*
470+ * We depend on the procfs results to determine the line number for the filter before
471+ * making the ioctl query. Hence, we cannot reuse run_filter_test here.
472+ */
473+ target_tag = &procfs_entries->tag[0];
474+ target_lineno = target_tag->tag.lineno;
475+
476+ filter.mask |= ALLOCINFO_FILTER_MASK_LINENO;
477+ filter.fields.lineno = target_lineno;
478+
479+ ioctl_status = get_filtered_ioctl_entries(tags, &filter, 0);
480+ if (ioctl_status == IOCTL_INVALID_DATA) {
481+ ksft_print_msg("Trouble retrieving valid IOCTL entries, skipping.\n");
482+ ret = KSFT_SKIP;
483+ goto exit;
484+ }
485+ if (ioctl_status == IOCTL_FAILURE) {
486+ ksft_print_msg("Error retrieving IOCTL entries.\n");
487+ ret = KSFT_FAIL;
488+ goto exit;
489+ }
490+
491+ for (i = 0; i < tags->count; i++) {
492+ if (tags->tag[i].tag.lineno != target_lineno) {
493+ ksft_print_msg("IOCTL entry %llu has incorrect lineno %llu.\n",
494+ i, tags->tag[i].tag.lineno);
495+ ret = KSFT_FAIL;
496+ goto exit;
497+ }
498+
499+ if (strncmp(tags->tag[i].tag.function, target_tag->tag.function,
500+ ALLOCINFO_STR_SIZE) == 0 &&
501+ strncmp(tags->tag[i].tag.filename, target_tag->tag.filename,
502+ ALLOCINFO_STR_SIZE) == 0)
503+ found = true;
504+ }
505+
506+ if (!found) {
507+ ksft_print_msg("Original procfs entry not found in IOCTL lineno filter results.\n");
508+ ret = KSFT_FAIL;
509+ }
510+
511+exit:
512+ free(tags);
513+ free(procfs_entries);
514+ return ret;
515+}
516+
517+int main(int argc, char *argv[])
518+{
519+ int ret;
520+
521+ ksft_set_plan(4);
522+
523+ ret = test_filename_filter();
524+ if (ret == KSFT_SKIP)
525+ ksft_test_result_skip("Skipping test_filename_filter\n");
526+ else
527+ ksft_test_result(ret == KSFT_PASS, "test_filename_filter\n");
528+
529+ ret = test_function_filter();
530+ if (ret == KSFT_SKIP)
531+ ksft_test_result_skip("Skipping test_function_filter\n");
532+ else
533+ ksft_test_result(ret == KSFT_PASS, "test_function_filter\n");
534+
535+ ret = test_size_filter();
536+ if (ret == KSFT_SKIP)
537+ ksft_test_result_skip("Skipping test_size_filter\n");
538+ else
539+ ksft_test_result(ret == KSFT_PASS, "test_size_filter\n");
540+
541+ ret = test_lineno_filter();
542+ if (ret == KSFT_SKIP)
543+ ksft_test_result_skip("Skipping test_lineno_filter\n");
544+ else
545+ ksft_test_result(ret == KSFT_PASS, "test_lineno_filter\n");
546+
547+ ksft_finished();
548+}
tools/testing/selftests/cgroup/lib/include/cgroup_util.h+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 #endif
88
99 #define MB(x) (x << 20)
10+#define GB(x) ((unsigned long long)(x) << 30)
1011
1112 #define NSEC_PER_USEC 1000L
1213 #define USEC_PER_SEC 1000000L
tools/testing/selftests/cgroup/test_zswap.c+11−2
@@ -20,6 +20,7 @@ static int page_size; · 19 unmodified lines
2020
2121 #define PATH_ZSWAP "/sys/module/zswap"
2222 #define PATH_ZSWAP_ENABLED "/sys/module/zswap/parameters/enabled"
23+#define PATH_ZSWAP_STORED_PAGES "/sys/kernel/debug/zswap/stored_pages"
2324
2425 static int read_int(const char *path, size_t *value)
2526 {
@@ -55,7 +56,7 @@ static int read_min_free_kb(size_t *value) · 29 unmodified lines
5556
5657 static int get_zswap_stored_pages(size_t *value)
5758 {
58− return read_int("/sys/kernel/debug/zswap/stored_pages", value);
59+ return read_int(PATH_ZSWAP_STORED_PAGES, value);
5960 }
6061
6162 static long get_cg_wb_count(const char *cg)
@@ -570,8 +571,16 @@ static int test_no_kmem_bypass(const char *root) · 508 unmodified lines
570571 /* Read sys info and compute test values accordingly */
571572 if (sysinfo(&sys_info) != 0)
572573 return KSFT_FAIL;
573− if (sys_info.totalram > 5000000000)
574+ if (sys_info.totalram > GB(4)) {
575+ ksft_print_msg(
576+ "requires less than 4GB total ram, sys_info.totalram: %.1fGB\n",
577+ (double)sys_info.totalram / GB(1));
574578 return KSFT_SKIP;
579+ }
580+ if (access(PATH_ZSWAP_STORED_PAGES, R_OK)) {
581+ ksft_print_msg("debugfs not mounted at /sys/kernel/debug\n");
582+ return KSFT_SKIP;
583+ }
575584 values = mmap(0, sizeof(struct no_kmem_bypass_child_args), PROT_READ |
576585 PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
577586 if (values == MAP_FAILED)
tools/testing/selftests/mm/.gitignore+9−67
@@ -1,68 +1,10 @@
11 # SPDX-License-Identifier: GPL-2.0-only
2−cow
3−hugepage-mmap
4−hugepage-mremap
5−hugepage-shm
6−hugepage-vmemmap
7−hugetlb-mmap
8−hugetlb-mremap
9−hugetlb-shm
10−hugetlb-vmemmap
11−hugetlb-madvise
12−hugetlb-read-hwpoison
13−hugetlb-soft-offline
14−khugepaged
15−map_hugetlb
16−map_populate
17−thuge-gen
18−compaction_test
19−memory-failure
20−migration
21−mlock2-tests
22−mrelease_test
23−mremap_dontunmap
24−mremap_test
25−on-fault-limit
26−transhuge-stress
27−pagemap_ioctl
28−pfnmap
29−process_madv
30−*.tmp*
31−protection_keys
32−protection_keys_32
33−protection_keys_64
34−madv_populate
35−uffd-stress
36−uffd-unit-tests
37−uffd-wp-mremap
38−mlock-intersect-test
39−mlock-random-test
40−virtual_address_range
41−gup_test
42−va_128TBswitch
43−map_fixed_noreplace
44−write_to_hugetlbfs
45−hmm-tests
46−memfd_secret
47−soft-dirty
48−split_huge_page_test
49−ksm_tests
50−local_config.h
51−local_config.mk
52−ksm_functional_tests
53−mdwe_test
54−gup_longterm
55−mkdirty
56−va_high_addr_switch
57−hugetlb_fault_after_madv
58−hugetlb_madv_vs_map
59−mseal_test
60−droppable
61−hugetlb_dio
62−pkey_sighandler_tests_32
63−pkey_sighandler_tests_64
64−guard-regions
65−merge
66−prctl_thp_disable
67−rmap
68−folio_split_race_test
2+*
3+!/**/
4+!*.c
5+!*.h
6+!*.sh
7+!.gitignore
8+!Makefile
9+!config
10+!settings
tools/testing/selftests/mm/Makefile+4−3
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 # script so kunit knows to run it, and add it to the list below.
66 # If you do not YOUR TESTS WILL NOT RUN IN THE CI.
77
8−LOCAL_HDRS += $(selfdir)/mm/local_config.h $(top_srcdir)/mm/gup_test.h
8+LOCAL_HDRS += $(selfdir)/mm/local_config.h_gen $(top_srcdir)/mm/gup_test.h
99 LOCAL_HDRS += $(selfdir)/mm/mseal_helpers.h
1010
1111 include local_config.mk
@@ -149,6 +149,7 @@ TEST_PROGS += ksft_gup_test.sh · 137 unmodified lines
149149 TEST_PROGS += ksft_hmm.sh
150150 TEST_PROGS += ksft_hugetlb.sh
151151 TEST_PROGS += ksft_hugevm.sh
152+TEST_PROGS += ksft_kmemleak_confirm.sh
152153 TEST_PROGS += ksft_kmemleak_dedup.sh
153154 TEST_PROGS += ksft_ksm.sh
154155 TEST_PROGS += ksft_ksm_numa.sh
@@ -261,11 +262,11 @@ $(OUTPUT)/migration: LDLIBS += -lnuma · 106 unmodified lines
261262
262263 $(OUTPUT)/rmap: LDLIBS += -lnuma
263264
264−local_config.mk local_config.h: check_config.sh
265+local_config.mk local_config.h_gen: check_config.sh
265266 $(call msg,CHK,config,$@)
266267 $(Q)CC="$(CC)" CFLAGS="$(CFLAGS)" ./check_config.sh
267268
268−EXTRA_CLEAN += local_config.mk local_config.h
269+EXTRA_CLEAN += local_config.mk local_config.h_gen
269270
270271 ifeq ($(IOURING_EXTRA_LIBS),)
271272 all: warn_missing_liburing
tools/testing/selftests/mm/check_config.sh+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
44 # Probe for libraries and create header files to record the results. Both C
55 # header files and Makefile include fragments are created.
66
7−OUTPUT_H_FILE=local_config.h
7+OUTPUT_H_FILE=local_config.h_gen
88 OUTPUT_MKFILE=local_config.mk
99
1010 tmpname=$(mktemp)
tools/testing/selftests/mm/compaction_test.c+21−17
@@ -29,30 +29,34 @@ struct map_list { · 28 unmodified lines
2929
3030 int read_memory_info(unsigned long *memfree, unsigned long *hugepagesize)
3131 {
32− char buffer[256] = {0};
33− char *cmd = "cat /proc/meminfo | grep -i memfree | grep -o '[0-9]*'";
34− FILE *cmdfile = popen(cmd, "r");
32+ char buffer[256];
33+ int found = 0;
34+ FILE *file;
35+ int ret = -1;
3536
36− if (!(fgets(buffer, sizeof(buffer), cmdfile))) {
37− ksft_print_msg("Failed to read meminfo: %s\n", strerror(errno));
37+ file = fopen("/proc/meminfo", "r");
38+ if (!file) {
39+ ksft_print_msg("Failed to open /proc/meminfo: %s\n",
40+ strerror(errno));
3841 return -1;
3942 }
4043
41− pclose(cmdfile);
42−
43− *memfree = atoll(buffer);
44− cmd = "cat /proc/meminfo | grep -i hugepagesize | grep -o '[0-9]*'";
45− cmdfile = popen(cmd, "r");
46−
47− if (!(fgets(buffer, sizeof(buffer), cmdfile))) {
48− ksft_print_msg("Failed to read meminfo: %s\n", strerror(errno));
49− return -1;
44+ while (fgets(buffer, sizeof(buffer), file) && found != 2) {
45+ if (sscanf(buffer, "MemFree: %lu kB", memfree) == 1 ||
46+ sscanf(buffer, "Hugepagesize: %lu kB", hugepagesize) == 1)
47+ found++;
5048 }
5149
52− pclose(cmdfile);
53− *hugepagesize = atoll(buffer);
50+ if (ferror(file))
51+ ksft_print_msg("Failed to read /proc/meminfo: %s\n",
52+ strerror(errno));
53+ else if (found != 2)
54+ ksft_print_msg("Failed to parse /proc/meminfo\n");
55+ else
56+ ret = 0;
5457
55− return 0;
58+ fclose(file);
59+ return ret;
5660 }
5761
5862 int prereq(void)
tools/testing/selftests/mm/cow.c+8−3
@@ -21,7 +21,7 @@ · 20 unmodified lines
2121 #include <sys/wait.h>
2222 #include <linux/memfd.h>
2323
24−#include "local_config.h"
24+#include "local_config.h_gen"
2525 #ifdef LOCAL_CONFIG_HAVE_LIBURING
2626 #include <liburing.h>
2727 #endif /* LOCAL_CONFIG_HAVE_LIBURING */
@@ -1718,8 +1718,13 @@ static void run_with_tmpfile(non_anon_test_fn fn, const char *desc) · 1690 unmodified lines
17181718
17191719 /* File consists of a single page filled with zeroes. */
17201720 if (fallocate(fd, 0, 0, pagesize)) {
1721− ksft_perror("fallocate() failed");
1722− log_test_result(KSFT_FAIL);
1721+ if (errno == EOPNOTSUPP) {
1722+ ksft_print_msg("fallocate() not supported by filesystem\n");
1723+ log_test_result(KSFT_SKIP);
1724+ } else {
1725+ ksft_perror("fallocate() failed");
1726+ log_test_result(KSFT_FAIL);
1727+ }
17231728 goto close;
17241729 }
17251730
tools/testing/selftests/mm/folio_split_race_test.c+1−1
@@ -182,7 +182,7 @@ static uint64_t run_iteration(void) · 181 unmodified lines
182182 for (i = 0; i < TOTAL_PAGES; i++)
183183 fill_page(mmap_base, i);
184184
185− if (!check_huge_shmem(mmap_base, NR_PMD_PAGE, pmd_pagesize))
185+ if (!check_huge_shmem(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize))
186186 ksft_exit_fail_msg("No shmem THP is allocated\n");
187187
188188 if (pthread_barrier_init(&ctl.barrier, NULL, NUM_READER_THREADS + 1) != 0)
tools/testing/selftests/mm/guard-regions.c+7−3
@@ -1912,7 +1912,7 @@ TEST_F(guard_regions, hole_punch) · 1911 unmodified lines
19121912 {
19131913 const unsigned long page_size = self->page_size;
19141914 char *ptr;
1915− int i;
1915+ int i, ret;
19161916
19171917 if (variant->backing == ANON_BACKED)
19181918 SKIP(return, "Truncation test specific to file-backed");
@@ -1944,8 +1944,12 @@ TEST_F(guard_regions, hole_punch) · 25 unmodified lines
19441944 }
19451945
19461946 /* Now hole punch the guarded region. */
1947− ASSERT_EQ(madvise(&ptr[3 * page_size], 4 * page_size,
1948− MADV_REMOVE), 0);
1947+ ret = madvise(&ptr[3 * page_size], 4 * page_size, MADV_REMOVE);
1948+ if (ret == -1 && errno == EOPNOTSUPP) {
1949+ ASSERT_EQ(munmap(ptr, 10 * page_size), 0);
1950+ SKIP(return, "MADV_REMOVE not supported by filesystem");
1951+ }
1952+ ASSERT_EQ(ret, 0);
19491953
19501954 /* Ensure guard regions remain. */
19511955 for (i = 0; i < 10; i++) {
tools/testing/selftests/mm/gup_longterm.c+1−1
@@ -21,7 +21,7 @@ · 20 unmodified lines
2121 #include <linux/magic.h>
2222 #include <linux/memfd.h>
2323
24−#include "local_config.h"
24+#include "local_config.h_gen"
2525 #ifdef LOCAL_CONFIG_HAVE_LIBURING
2626 #include <liburing.h>
2727 #endif /* LOCAL_CONFIG_HAVE_LIBURING */
tools/testing/selftests/mm/khugepaged.c+155−49
@@ -26,9 +26,11 @@ · 25 unmodified lines
2626
2727 #define BASE_ADDR ((void *)(1UL << 30))
2828 static unsigned long hpage_pmd_size;
29+static int hpage_pmd_order;
2930 static unsigned long page_size;
3031 static int hpage_pmd_nr;
3132 static int anon_order;
33+static int collapse_order;
3234
3335 #define PID_SMAPS "/proc/self/smaps"
3436 #define TEST_FILE "collapse_test_file"
@@ -51,7 +53,7 @@ struct mem_ops { · 16 unmodified lines
5153 void *(*setup_area)(int nr_hpages);
5254 void (*cleanup_area)(void *p, unsigned long size);
5355 void (*fault)(void *p, unsigned long start, unsigned long end);
54− bool (*check_huge)(void *addr, int nr_hpages);
56+ bool (*check_huge)(void *addr, size_t len, int nr_hpages, unsigned long hpage_size);
5557 const char *name;
5658 };
5759
@@ -69,6 +71,7 @@ struct collapse_context { · 11 unmodified lines
6971 };
7072
7173 static struct collapse_context *khugepaged_context;
74+static struct collapse_context *mthp_khugepaged_context;
7275 static struct collapse_context *madvise_context;
7376
7477 struct file_info {
@@ -121,7 +124,8 @@ static void get_finfo(const char *dir) · 46 unmodified lines
121124 char *str, *end;
122125
123126 finfo.dir = dir;
124− stat(finfo.dir, &path_stat);
127+ if (stat(finfo.dir, &path_stat))
128+ ksft_exit_fail_perror("stat()");
125129 if (!S_ISDIR(path_stat.st_mode))
126130 ksft_exit_fail_msg("%s: Not a directory (%s)\n", __func__, finfo.dir);
127131 if (snprintf(finfo.path, sizeof(finfo.path), "%s/" TEST_FILE,
@@ -138,8 +142,8 @@ static void get_finfo(const char *dir) · 10 unmodified lines
138142 major(path_stat.st_dev), minor(path_stat.st_dev))
139143 >= sizeof(path))
140144 ksft_exit_fail_msg("%s: Pathname is too long\n", __func__);
141− if (read_file(path, buf, sizeof(buf)) < 0)
142− ksft_exit_fail_perror("read_file(read_num)");
145+ if (!read_file(path, buf, sizeof(buf)))
146+ ksft_exit_fail_perror("read_file(uevent)");
143147 if (strstr(buf, "DEVTYPE=disk")) {
144148 /* Found it */
145149 if (snprintf(finfo.dev_queue_read_ahead_path,
@@ -276,7 +280,7 @@ static void *alloc_hpage(struct mem_ops *ops) · 130 unmodified lines
276280 ksft_print_msg("Allocate huge page...");
277281 if (madvise_collapse_retry(p, hpage_pmd_size))
278282 ksft_exit_fail_perror("madvise(MADV_COLLAPSE)");
279− if (!ops->check_huge(p, 1))
283+ if (!ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
280284 ksft_exit_fail_perror("madvise(MADV_COLLAPSE)");
281285 if (madvise(p, hpage_pmd_size, MADV_HUGEPAGE))
282286 ksft_exit_fail_perror("madvise(MADV_HUGEPAGE)");
@@ -310,9 +314,10 @@ static void anon_fault(void *p, unsigned long start, unsigned long end) · 27 unmodified lines
310314 fill_memory(p, start, end);
311315 }
312316
313−static bool anon_check_huge(void *addr, int nr_hpages)
317+static bool anon_check_huge(void *addr, size_t len, int nr_hpages,
318+ unsigned long hpage_size)
314319 {
315− return check_huge_anon(addr, nr_hpages, hpage_pmd_size);
320+ return check_huge_anon(addr, len, nr_hpages, hpage_size);
316321 }
317322
318323 static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup)
@@ -412,13 +417,14 @@ static void file_fault_write(void *p, unsigned long start, unsigned long end) · 93 unmodified lines
412417 ksft_exit_fail_perror("madvise(MADV_POPULATE_WRITE)");
413418 }
414419
415−static bool file_check_huge(void *addr, int nr_hpages)
420+static bool file_check_huge(void *addr, size_t len, int nr_hpages,
421+ unsigned long hpage_size)
416422 {
417423 switch (finfo.type) {
418424 case VMA_FILE:
419− return check_huge_file(addr, nr_hpages, hpage_pmd_size);
425+ return check_huge_file(addr, len, nr_hpages, hpage_size);
420426 case VMA_SHMEM:
421− return check_huge_shmem(addr, nr_hpages, hpage_pmd_size);
427+ return check_huge_shmem(addr, len, nr_hpages, hpage_size);
422428 default:
423429 exit(EXIT_FAILURE);
424430 return false;
@@ -448,9 +454,10 @@ static void shmem_cleanup_area(void *p, unsigned long size) · 23 unmodified lines
448454 close(finfo.fd);
449455 }
450456
451−static bool shmem_check_huge(void *addr, int nr_hpages)
457+static bool shmem_check_huge(void *addr, size_t len, int nr_hpages,
458+ unsigned long hpage_size)
452459 {
453− return check_huge_shmem(addr, nr_hpages, hpage_pmd_size);
460+ return check_huge_shmem(addr, len, nr_hpages, hpage_size);
454461 }
455462
456463 static struct mem_ops __anon_ops = {
@@ -533,7 +540,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages, · 76 unmodified lines
533540 ret = madvise_collapse_retry(p, nr_hpages * hpage_pmd_size);
534541 if (((bool)ret) == expect)
535542 fail("Fail: Bad return value");
536− else if (!ops->check_huge(p, expect ? nr_hpages : 0))
543+ else if (!ops->check_huge(p, nr_hpages * hpage_pmd_size, expect ? nr_hpages : 0, hpage_pmd_size))
537544 fail("Fail: check_huge()");
538545 else
539546 success("OK");
@@ -545,30 +552,31 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages, · 5 unmodified lines
545552 struct mem_ops *ops, bool expect)
546553 {
547554 /* Sanity check */
548− if (!ops->check_huge(p, 0))
555+ if (!ops->check_huge(p, nr_hpages * hpage_pmd_size, 0, hpage_pmd_size))
549556 ksft_exit_fail_msg("Unexpected huge page\n");
550557 __madvise_collapse(msg, p, nr_hpages, ops, expect);
551558 }
552559
553560 #define TICK 500000
554−static bool wait_for_scan(const char *msg, char *p, int nr_hpages,
555− struct mem_ops *ops)
561+static bool wait_for_scan(const char *msg, char *p, size_t len,
562+ int nr_hpages, int collap_order, struct mem_ops *ops)
556563 {
564+ unsigned long hpage_size = page_size << collap_order;
557565 int full_scans;
558566 int timeout = 6; /* 3 seconds */
559567
560568 /* Sanity check */
561− if (!ops->check_huge(p, 0))
569+ if (!ops->check_huge(p, len, 0, hpage_size))
562570 ksft_exit_fail_msg("Unexpected huge page\n");
563571
564− madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE);
572+ madvise(p, len, MADV_HUGEPAGE);
565573
566574 /* Wait until the second full_scan completed */
567575 full_scans = thp_read_num("khugepaged/full_scans") + 2;
568576
569577 ksft_print_msg("%s...", msg);
570578 while (timeout--) {
571− if (ops->check_huge(p, nr_hpages))
579+ if (ops->check_huge(p, len, nr_hpages, hpage_size))
572580 break;
573581 if (thp_read_num("khugepaged/full_scans") >= full_scans)
574582 break;
@@ -582,6 +590,8 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages, · 7 unmodified lines
582590 static void khugepaged_collapse(const char *msg, char *p, int nr_hpages,
583591 struct mem_ops *ops, bool expect)
584592 {
593+ size_t len = nr_hpages * hpage_pmd_size;
594+
585595 /*
586596 * read&write file collapse fails since khugepaged does not flush
587597 * the target dirty folios
@@ -589,7 +599,7 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, · 1 unmodified line
589599 if (!is_tmpfs(ops) && ops == &__read_write_file_write_ops)
590600 expect = false;
591601
592− if (wait_for_scan(msg, p, nr_hpages, ops)) {
602+ if (wait_for_scan(msg, p, len, nr_hpages, hpage_pmd_order, ops)) {
593603 if (expect)
594604 fail("Timeout");
595605 else
@@ -605,10 +615,54 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, · 9 unmodified lines
605615 if (ops != &__anon_ops)
606616 ops->fault(p, 0, nr_hpages * hpage_pmd_size);
607617
608− if (ops->check_huge(p, expect ? nr_hpages : 0))
618+ if (ops->check_huge(p, len, expect ? nr_hpages : 0, hpage_pmd_size))
619+ success("OK");
620+ else
621+ fail("Fail");
622+}
623+
624+static void mthp_khugepaged_collapse(const char *msg, char *p, int nr_hpages,
625+ struct mem_ops *ops, bool expect)
626+{
627+ unsigned long hpage_size = page_size << collapse_order;
628+ struct thp_settings settings = *thp_current_settings();
629+ /* mTHP collpase only allocates PMD sized memory */
630+ size_t len = hpage_pmd_size;
631+
632+ /* Set mTHP setting for mTHP collapse */
633+ if (ops == &__anon_ops) {
634+ settings.thp_enabled = THP_NEVER;
635+ settings.hugepages[collapse_order].enabled = THP_MADVISE;
636+ }
637+
638+ thp_push_settings(&settings);
639+
640+ if (wait_for_scan(msg, p, len, nr_hpages, collapse_order, ops)) {
641+ if (expect)
642+ fail("Timeout");
643+ else
644+ success("OK");
645+
646+ /* Restore THP settings for mTHP collapse. */
647+ thp_pop_settings();
648+ return;
649+ }
650+
651+ /*
652+ * For file and shmem memory, khugepaged only retracts pte entries after
653+ * putting the new hugepage in the page cache. The hugepage must be
654+ * subsequently refaulted to install the pmd mapping for the mm.
655+ */
656+ if (ops != &__anon_ops)
657+ ops->fault(p, 0, nr_hpages * hpage_size);
658+
659+ if (ops->check_huge(p, len, expect ? nr_hpages : 0, hpage_size))
609660 success("OK");
610661 else
611662 fail("Fail");
663+
664+ /* Restore THP settings for mTHP collapse. */
665+ thp_pop_settings();
612666 }
613667
614668 static struct collapse_context __khugepaged_context = {
@@ -617,6 +671,12 @@ static struct collapse_context __khugepaged_context = { · 2 unmodified lines
617671 .name = "khugepaged",
618672 };
619673
674+static struct collapse_context __mthp_khugepaged_context = {
675+ .collapse = &mthp_khugepaged_collapse,
676+ .enforce_pte_scan_limits = true,
677+ .name = "mthp_khugepaged",
678+};
679+
620680 static struct collapse_context __madvise_context = {
621681 .collapse = &madvise_collapse,
622682 .enforce_pte_scan_limits = false,
@@ -634,7 +694,7 @@ static void alloc_at_fault(void) · 11 unmodified lines
634694 p = alloc_mapping(1);
635695 *p = 1;
636696 ksft_print_msg("Allocate huge page on fault...");
637− if (check_huge_anon(p, 1, hpage_pmd_size))
697+ if (check_huge_anon(p, hpage_pmd_size, 1, hpage_pmd_size))
638698 success("OK");
639699 else
640700 fail("Fail");
@@ -643,7 +703,7 @@ static void alloc_at_fault(void) · 2 unmodified lines
643703
644704 madvise(p, page_size, MADV_DONTNEED);
645705 ksft_print_msg("Split huge PMD on MADV_DONTNEED...");
646− if (check_huge_anon(p, 0, hpage_pmd_size))
706+ if (check_huge_anon(p, hpage_pmd_size, 0, hpage_pmd_size))
647707 success("OK");
648708 else
649709 fail("Fail");
@@ -655,10 +715,17 @@ static void alloc_at_fault(void) · 5 unmodified lines
655715 static void collapse_full(struct collapse_context *c, struct mem_ops *ops)
656716 {
657717 void *p;
658− int nr_hpages = 4;
718+ int nr_pmds = 4, nr_hpages = 4;
659719 unsigned long size = nr_hpages * hpage_pmd_size;
660720
661− p = ops->setup_area(nr_hpages);
721+ /* Only try 1 PMD sized range for mTHP collapse. */
722+ if (c == &__mthp_khugepaged_context) {
723+ nr_pmds = 1;
724+ nr_hpages = 1 << (hpage_pmd_order - collapse_order);
725+ size = hpage_pmd_size;
726+ }
727+
728+ p = ops->setup_area(nr_pmds);
662729 ops->fault(p, 0, size);
663730 c->collapse("Collapse multiple fully populated PTE table", p, nr_hpages,
664731 ops, true);
@@ -670,10 +737,31 @@ static void collapse_full(struct collapse_context *c, struct mem_ops *ops) · 5 unmodified lines
670737
671738 static void collapse_empty(struct collapse_context *c, struct mem_ops *ops)
672739 {
740+ int nr_hpages = 1;
741+ void *p;
742+
743+ if (c == &__mthp_khugepaged_context)
744+ nr_hpages = 1 << (hpage_pmd_order - collapse_order);
745+
746+ p = ops->setup_area(1);
747+ c->collapse("Do not collapse empty PTE table", p, nr_hpages, ops, false);
748+ ops->cleanup_area(p, hpage_pmd_size);
749+ ksft_test_result_report(exit_status, "%s\n", __func__);
750+}
751+
752+static void collapse_single_mthp(struct collapse_context *c, struct mem_ops *ops)
753+{
754+ unsigned long hpage_size = page_size << collapse_order;
673755 void *p;
674756
675757 p = ops->setup_area(1);
676− c->collapse("Do not collapse empty PTE table", p, 1, ops, false);
758+ /*
759+ * Only fault collapse_order sized ranges, and only check 1
760+ * collapse_order sized huge page.
761+ */
762+ ops->fault(p, 0, hpage_size);
763+ c->collapse("Collapse PTE table with half PTE entries present",
764+ p, 1, ops, true);
677765 ops->cleanup_area(p, hpage_pmd_size);
678766 ksft_test_result_report(exit_status, "%s\n", __func__);
679767 }
@@ -815,7 +903,7 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, struc · 135 unmodified lines
815903 madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE);
816904 ksft_print_msg("Split huge page leaving single PTE mapping compound page...");
817905 madvise(p + page_size, hpage_pmd_size - page_size, MADV_DONTNEED);
818− if (ops->check_huge(p, 0))
906+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
819907 success("OK");
820908 else
821909 fail("Fail");
@@ -836,7 +924,7 @@ static void collapse_full_of_compound(struct collapse_context *c, struct mem_ops · 14 unmodified lines
836924 ksft_print_msg("Split huge page leaving single PTE page table full of compound pages...");
837925 madvise(p, page_size, MADV_NOHUGEPAGE);
838926 madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE);
839− if (ops->check_huge(p, 0))
927+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
840928 success("OK");
841929 else
842930 fail("Fail");
@@ -858,7 +946,7 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_ops · 15 unmodified lines
858946 for (i = 0; i < hpage_pmd_nr; i++) {
859947 madvise(BASE_ADDR, hpage_pmd_size, MADV_HUGEPAGE);
860948 ops->fault(BASE_ADDR, 0, hpage_pmd_size);
861− if (!ops->check_huge(BASE_ADDR, 1))
949+ if (!ops->check_huge(BASE_ADDR, hpage_pmd_size, 1, hpage_pmd_size))
862950 ksft_exit_fail_msg("Failed to allocate huge page\n");
863951 madvise(BASE_ADDR, hpage_pmd_size, MADV_NOHUGEPAGE);
864952
@@ -881,7 +969,7 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_ops · 16 unmodified lines
881969
882970 ops->cleanup_area(BASE_ADDR, hpage_pmd_size);
883971 ops->fault(p, 0, hpage_pmd_size);
884− if (!ops->check_huge(p, 1))
972+ if (!ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
885973 success("OK");
886974 else
887975 fail("Fail");
@@ -903,7 +991,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) · 15 unmodified lines
903991
904992 ksft_print_msg("Allocate small page...");
905993 ops->fault(p, 0, page_size);
906− if (ops->check_huge(p, 0))
994+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
907995 success("OK");
908996 else
909997 fail("Fail");
@@ -911,7 +999,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) · 1 unmodified line
911999 ksft_print_msg("Share small page over fork()...");
9121000 if (!fork()) {
9131001 /* Do not touch settings on child exit */
914− if (ops->check_huge(p, 0))
1002+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
9151003 success("OK");
9161004 else
9171005 fail("Fail");
@@ -929,7 +1017,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) · 11 unmodified lines
9291017 exit_status = WEXITSTATUS(wstatus);
9301018
9311019 ksft_print_msg("Check if parent still has small page...");
932− if (ops->check_huge(p, 0))
1020+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
9331021 success("OK");
9341022 else
9351023 fail("Fail");
@@ -947,7 +1035,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o · 11 unmodified lines
9471035 ksft_print_msg("Share huge page over fork()...");
9481036 if (!fork()) {
9491037 /* Do not touch settings on child exit */
950− if (ops->check_huge(p, 1))
1038+ if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
9511039 success("OK");
9521040 else
9531041 fail("Fail");
@@ -955,7 +1043,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o · 1 unmodified line
9551043 ksft_print_msg("Split huge page PMD in child process...");
9561044 madvise(p, page_size, MADV_NOHUGEPAGE);
9571045 madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE);
958− if (ops->check_huge(p, 0))
1046+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
9591047 success("OK");
9601048 else
9611049 fail("Fail");
@@ -976,7 +1064,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o · 14 unmodified lines
9761064 exit_status = WEXITSTATUS(wstatus);
9771065
9781066 ksft_print_msg("Check if parent still has huge page...");
979− if (ops->check_huge(p, 1))
1067+ if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
9801068 success("OK");
9811069 else
9821070 fail("Fail");
@@ -995,7 +1083,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops · 12 unmodified lines
9951083 ksft_print_msg("Share huge page over fork()...");
9961084 if (!fork()) {
9971085 /* Do not touch settings on child exit */
998− if (ops->check_huge(p, 1))
1086+ if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
9991087 success("OK");
10001088 else
10011089 fail("Fail");
@@ -1003,7 +1091,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops · 1 unmodified line
10031091 ksft_print_msg("Trigger CoW on page %d of %d...",
10041092 hpage_pmd_nr - max_ptes_shared - 1, hpage_pmd_nr);
10051093 ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared - 1) * page_size);
1006− if (ops->check_huge(p, 0))
1094+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
10071095 success("OK");
10081096 else
10091097 fail("Fail");
@@ -1016,7 +1104,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops · 6 unmodified lines
10161104 hpage_pmd_nr - max_ptes_shared, hpage_pmd_nr);
10171105 ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared) *
10181106 page_size);
1019− if (ops->check_huge(p, 0))
1107+ if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size))
10201108 success("OK");
10211109 else
10221110 fail("Fail");
@@ -1034,7 +1122,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops · 11 unmodified lines
10341122 exit_status = WEXITSTATUS(wstatus);
10351123
10361124 ksft_print_msg("Check if parent still has huge page...");
1037− if (ops->check_huge(p, 1))
1125+ if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size))
10381126 success("OK");
10391127 else
10401128 fail("Fail");
@@ -1075,8 +1163,8 @@ static void madvise_retracted_page_tables(struct collapse_context *c, · 34 unmodified lines
10751163 ops->fault(p, 0, size);
10761164
10771165 /* Let khugepaged collapse and leave pmd cleared */
1078− if (wait_for_scan("Collapse and leave PMD cleared", p, nr_hpages,
1079− ops)) {
1166+ if (wait_for_scan("Collapse and leave PMD cleared", p, size, nr_hpages,
1167+ hpage_pmd_order, ops)) {
10801168 fail("Timeout");
10811169 return;
10821170 }
@@ -1092,17 +1180,19 @@ static void usage(void) · 9 unmodified lines
10921180 {
10931181 fprintf(stderr, "\nUsage: ./khugepaged [OPTIONS] <test type> [dir]\n\n");
10941182 fprintf(stderr, "\t<test type>\t: <context>:<mem_type>\n");
1095− fprintf(stderr, "\t<context>\t: [all|khugepaged|madvise]\n");
1183+ fprintf(stderr, "\t<context>\t: [all|khugepaged|mthp_khugepaged|madvise]\n");
10961184 fprintf(stderr, "\t<mem_type>\t: [all|anon|file|shmem]\n");
10971185 fprintf(stderr, "\n\t\"file,all\" mem_type requires [dir] argument\n");
10981186 fprintf(stderr, "\n\t\"file,all\" mem_type requires a file system\n");
10991187 fprintf(stderr, "\twith PMD-sized large folio support\n");
11001188 fprintf(stderr, "\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n");
11011189 fprintf(stderr, "\tmounted with huge=advise option for khugepaged tests to work\n");
1190+ fprintf(stderr, "\n\tmthp_khugepaged only supports anon mem_type now.\n");
11021191 fprintf(stderr, "\n\tSupported Options:\n");
11031192 fprintf(stderr, "\t\t-h: This help message.\n");
11041193 fprintf(stderr, "\t\t-s: mTHP size, expressed as page order.\n");
11051194 fprintf(stderr, "\t\t Defaults to 0. Use this size for anon or shmem allocations.\n");
1195+ fprintf(stderr, "\t\t-c: collapse order for mTHP collapse, expressed as page order.\n");
11061196 exit(1);
11071197 }
11081198
@@ -1112,11 +1202,14 @@ static void parse_test_type(int argc, char **argv) · 3 unmodified lines
11121202 char *buf;
11131203 const char *token;
11141204
1115− while ((opt = getopt(argc, argv, "s:h")) != -1) {
1205+ while ((opt = getopt(argc, argv, "s:c:h")) != -1) {
11161206 switch (opt) {
11171207 case 's':
11181208 anon_order = atoi(optarg);
11191209 break;
1210+ case 'c':
1211+ collapse_order = atoi(optarg);
1212+ break;
11201213 case 'h':
11211214 default:
11221215 usage();
@@ -1142,6 +1235,10 @@ static void parse_test_type(int argc, char **argv) · 19 unmodified lines
11421235 madvise_context = &__madvise_context;
11431236 } else if (!strcmp(token, "khugepaged")) {
11441237 khugepaged_context = &__khugepaged_context;
1238+ } else if (!strcmp(token, "mthp_khugepaged")) {
1239+ mthp_khugepaged_context = &__mthp_khugepaged_context;
1240+ if (collapse_order <= 0 || collapse_order >= hpage_pmd_order)
1241+ usage();
11451242 } else if (!strcmp(token, "madvise")) {
11461243 madvise_context = &__madvise_context;
11471244 } else {
@@ -1157,14 +1254,20 @@ static void parse_test_type(int argc, char **argv) · 9 unmodified lines
11571254 read_write_file_write_ops = &__read_write_file_write_ops;
11581255 anon_ops = &__anon_ops;
11591256 shmem_ops = &__shmem_ops;
1257+ if (mthp_khugepaged_context)
1258+ usage();
11601259 } else if (!strcmp(buf, "anon")) {
11611260 anon_ops = &__anon_ops;
11621261 } else if (!strcmp(buf, "file")) {
11631262 read_only_file_ops = &__read_only_file_ops;
11641263 read_write_file_read_ops = &__read_write_file_read_ops;
11651264 read_write_file_write_ops = &__read_write_file_write_ops;
1265+ if (mthp_khugepaged_context)
1266+ usage();
11661267 } else if (!strcmp(buf, "shmem")) {
11671268 shmem_ops = &__shmem_ops;
1269+ if (mthp_khugepaged_context)
1270+ usage();
11681271 } else {
11691272 usage();
11701273 }
@@ -1207,7 +1310,6 @@ static int nr_test_cases; · 36 unmodified lines
12071310
12081311 int main(int argc, char **argv)
12091312 {
1210− int hpage_pmd_order;
12111313 struct thp_settings default_settings = {
12121314 .thp_enabled = THP_MADVISE,
12131315 .thp_defrag = THP_DEFRAG_ALWAYS,
@@ -1233,10 +1335,6 @@ int main(int argc, char **argv) · 19 unmodified lines
12331335 if (!thp_is_enabled())
12341336 ksft_exit_skip("Transparent Hugepages not available\n");
12351337
1236− parse_test_type(argc, argv);
1237−
1238− setbuf(stdout, NULL);
1239−
12401338 page_size = getpagesize();
12411339 hpage_pmd_size = read_pmd_pagesize();
12421340 if (!hpage_pmd_size)
@@ -1244,6 +1342,10 @@ int main(int argc, char **argv) · 1 unmodified line
12441342 hpage_pmd_nr = hpage_pmd_size / page_size;
12451343 hpage_pmd_order = __builtin_ctz(hpage_pmd_nr);
12461344
1345+ parse_test_type(argc, argv);
1346+
1347+ setbuf(stdout, NULL);
1348+
12471349 default_settings.khugepaged.max_ptes_none = hpage_pmd_nr - 1;
12481350 default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr / 8;
12491351 default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2;
@@ -1261,6 +1363,7 @@ int main(int argc, char **argv) · 11 unmodified lines
12611363 TEST(collapse_full, khugepaged_context, read_write_file_read_ops);
12621364 TEST(collapse_full, khugepaged_context, read_write_file_write_ops);
12631365 TEST(collapse_full, khugepaged_context, shmem_ops);
1366+ TEST(collapse_full, mthp_khugepaged_context, anon_ops);
12641367 TEST(collapse_full, madvise_context, anon_ops);
12651368 TEST(collapse_full, madvise_context, read_only_file_ops);
12661369 TEST(collapse_full, madvise_context, read_write_file_read_ops);
@@ -1268,8 +1371,11 @@ int main(int argc, char **argv) · 1 unmodified line
12681371 TEST(collapse_full, madvise_context, shmem_ops);
12691372
12701373 TEST(collapse_empty, khugepaged_context, anon_ops);
1374+ TEST(collapse_empty, mthp_khugepaged_context, anon_ops);
12711375 TEST(collapse_empty, madvise_context, anon_ops);
12721376
1377+ TEST(collapse_single_mthp, mthp_khugepaged_context, anon_ops);
1378+
12731379 TEST(collapse_single_pte_entry, khugepaged_context, anon_ops);
12741380 TEST(collapse_single_pte_entry, khugepaged_context, read_only_file_ops);
12751381 TEST(collapse_single_pte_entry, khugepaged_context, read_write_file_read_ops);
tools/testing/selftests/mm/ksft_kmemleak_confirm.sh+130−0
@@ -0,0 +1,130 @@
1+#!/bin/bash
2+# SPDX-License-Identifier: GPL-2.0
3+#
4+# Functional test for kmemleak's N-consecutive-scan leak confirmation
5+# (the min_unref_scans module parameter).
6+#
7+# kmemleak only reports an object once it has stayed unreferenced for
8+# min_unref_scans consecutive scans. A threshold of 1 reports on the first
9+# scan (historical behaviour); higher values filter transient false
10+# positives where a live object's only reference is briefly invisible to a
11+# single scan (e.g. an RCU tree update in flight while the scan runs). The
12+# test loads samples/kmemleak's helper module to create orphan allocations
13+# and, counting only those orphans (matched by their [kmemleak_test]
14+# backtrace so unrelated leaks already present on the system are ignored),
15+# checks that:
16+# - a freshly allocated object is greyed on its first scan (its checksum
17+# settles then), so nothing can be reported before that priming scan;
18+# each case below primes once first,
19+# - at min_unref_scans=1 one scan after priming reports the orphans,
20+# - raising the threshold to 2 needs two scans after priming: one is not
21+# enough, the second reports,
22+# - the parameter reads back what was written.
23+#
24+# The "one post-prime scan is not enough at min_unref_scans=2" check is the
25+# core regression test: raising min_unref_scans must push the report
26+# strictly later. Like ksft_kmemleak_dedup.sh, if the module yields no
27+# detectable orphan at all in the running environment the test skips rather
28+# than failing.
29+#
30+# Author: Breno Leitao <leitao@debian.org>
31+
32+# KTAP output helpers (ktap_skip_all, ktap_exit_fail_msg, ktap_test_pass, ...).
33+DIR="$(dirname "$(readlink -f "$0")")"
34+# shellcheck source=../kselftest/ktap_helpers.sh
35+source "${DIR}"/../kselftest/ktap_helpers.sh
36+
37+KMEMLEAK=/sys/kernel/debug/kmemleak
38+PARAM=/sys/module/kmemleak/parameters/min_unref_scans
39+MODULE=kmemleak-test
40+AGE=6 # seconds; must exceed kmemleak's 5s minimum object age
41+
42+ktap_print_header
43+
44+[ "$(id -u)" -eq 0 ] || { ktap_skip_all "must run as root"; exit "$KSFT_SKIP"; }
45+[ -r "$KMEMLEAK" ] ||
46+ { ktap_skip_all "no kmemleak debugfs (CONFIG_DEBUG_KMEMLEAK)"; exit "$KSFT_SKIP"; }
47+[ -w "$PARAM" ] ||
48+ { ktap_skip_all "min_unref_scans module parameter not present"; exit "$KSFT_SKIP"; }
49+modinfo "$MODULE" >/dev/null 2>&1 ||
50+ { ktap_skip_all "$MODULE not built (CONFIG_SAMPLE_KMEMLEAK)"; exit "$KSFT_SKIP"; }
51+
52+# kmemleak can be present but disabled at runtime (kmemleak=off boot arg,
53+# or it self-disabled after an internal error); a "scan" then returns
54+# EPERM. Probe once and skip if so.
55+echo scan > "$KMEMLEAK" 2>/dev/null ||
56+ { ktap_skip_all "kmemleak is disabled (check dmesg or kmemleak= boot arg)"; exit "$KSFT_SKIP"; }
57+
58+prev=$(cat "$PARAM")
59+# shellcheck disable=SC2317 # invoked indirectly via trap
60+cleanup() {
61+ echo "$prev" > "$PARAM" 2>/dev/null # restore the parameter
62+ echo scan=on > "$KMEMLEAK" 2>/dev/null # re-enable auto scan
63+ rmmod "$MODULE" 2>/dev/null
64+ echo clear > "$KMEMLEAK" 2>/dev/null
65+}
66+trap cleanup EXIT
67+
68+# Stop the automatic scan thread: only our manual scans should advance an
69+# object's consecutive-unreferenced run. An auto scan landing between two
70+# manual scans would change the result and make the test flaky.
71+echo scan=off > "$KMEMLEAK" 2>/dev/null
72+
73+# Create a fresh, aged set of orphan objects from the helper module's init
74+# path (its kmalloc/vmalloc/percpu allocations are dropped right away).
75+# Pre-existing reported leaks are greyed first ("clear") so only our
76+# orphans are counted. The module is left loaded on purpose: once it is
77+# unloaded its symbols are gone, so the orphan backtraces no longer resolve
78+# to [kmemleak_test] and could not be matched below.
79+gen_orphans() {
80+ rmmod "$MODULE" 2>/dev/null
81+ echo clear > "$KMEMLEAK"
82+ modprobe "$MODULE" ||
83+ { ktap_skip_all "failed to load $MODULE"; exit "$KSFT_SKIP"; }
84+ sleep "$AGE"
85+}
86+
87+scan() { echo scan > "$KMEMLEAK"; }
88+
89+# Number of helper-module orphans currently reported by kmemleak. Matching
90+# the module's own backtrace ([kmemleak_test]) keeps the count immune to
91+# unrelated leaks on the running system. kmemleak only lists an object here
92+# once it has been reported, so this reflects the confirmation gating.
93+count_orphans() {
94+ c=$(grep -c '\[kmemleak_test\]' "$KMEMLEAK" 2>/dev/null)
95+ echo "${c:-0}"
96+}
97+
98+# 0) the parameter reads back what was written.
99+echo 3 > "$PARAM"
100+[ "$(cat "$PARAM")" = "3" ] || ktap_exit_fail_msg "min_unref_scans did not read back as 3"
101+
102+# Priming scan: kmemleak greys a freshly allocated object on its first scan
103+# (its checksum settles then), so nothing can be reported until a second
104+# scan. Every case below runs this priming scan before counting.
105+prime() { scan; }
106+
107+# 1) min_unref_scans=1: one scan after priming reports the orphans. This
108+# also establishes that the helper produces detectable orphans here.
109+echo 1 > "$PARAM"
110+gen_orphans
111+prime
112+scan
113+first=$(count_orphans)
114+[ "$first" -gt 0 ] ||
115+ { ktap_skip_all "$MODULE produced no detectable orphans (cannot test min_unref_scans)"; exit "$KSFT_SKIP"; }
116+
117+# 2) min_unref_scans=2: after priming, one scan is not enough (still
118+# gated), the second reports. The gated-scan-zero check is the core
119+# regression.
120+echo 2 > "$PARAM"
121+gen_orphans
122+prime
123+scan; s1=$(count_orphans)
124+scan; s2=$(count_orphans)
125+[ "$s1" -eq 0 ] || ktap_exit_fail_msg "min_unref_scans=2: $s1 orphan(s) after 1 post-prime scan (must be 0)"
126+[ "$s2" -gt 0 ] || ktap_exit_fail_msg "min_unref_scans=2: no report after 2 post-prime scans (false negative)"
127+
128+ktap_set_plan 1
129+ktap_test_pass "min_unref_scans=1 reported $first orphan(s) one scan after priming; =2 held them one scan longer ($s1 after one scan, $s2 after two); param read-back ok"
130+ktap_finished
tools/testing/selftests/mm/memory-failure.c+12−2
@@ -287,8 +287,10 @@ TEST_F(memory_failure, clean_pagecache) · 286 unmodified lines
287287 if (fd < 0)
288288 SKIP(return, "failed to open test file.\n");
289289 fs_type = get_fs_type(fd);
290− if (!fs_type || fs_type == TMPFS_MAGIC)
290+ if (!fs_type || fs_type == TMPFS_MAGIC) {
291+ close(fd);
291292 SKIP(return, "unsupported filesystem :%x\n", fs_type);
293+ }
292294
293295 addr = mmap(0, self->page_size, PROT_READ | PROT_WRITE,
294296 MAP_SHARED, fd, 0);
@@ -327,8 +329,16 @@ TEST_F(memory_failure, dirty_pagecache) · 32 unmodified lines
327329 if (fd < 0)
328330 SKIP(return, "failed to open test file.\n");
329331 fs_type = get_fs_type(fd);
330− if (!fs_type || fs_type == TMPFS_MAGIC)
332+ /*
333+ * MADV_HARD poisoning of dirty page-cache data records an expected
334+ * -EIO in the file mapping. NFS reports this error on close(), so
335+ * skip this variant.
336+ */
337+ if (!fs_type || fs_type == TMPFS_MAGIC ||
338+ (fs_type == NFS_SUPER_MAGIC && variant->type == MADV_HARD)) {
339+ close(fd);
331340 SKIP(return, "unsupported filesystem :%x\n", fs_type);
341+ }
332342
333343 addr = mmap(0, self->page_size, PROT_READ | PROT_WRITE,
334344 MAP_SHARED, fd, 0);
tools/testing/selftests/mm/merge.c+57−0
@@ -1305,6 +1305,63 @@ TEST_F(merge, merge_vmas_with_mseal) · 1304 unmodified lines
13051305 ASSERT_EQ(procmap->query.vma_end, (unsigned long)ptr + 2 * page_size);
13061306 }
13071307
1308+TEST_F(merge, anon_and_page_offset_mismatch_memfd)
1309+{
1310+ struct procmap_fd *procmap = &self->procmap;
1311+ unsigned int page_size = self->page_size;
1312+ char *carveout = self->carveout;
1313+ char *ptr, *ptr2;
1314+ int fd;
1315+
1316+ /* Create a 10 page memfd descriptor. */
1317+ fd = memfd_create("anon_page_offset_test", MFD_CLOEXEC);
1318+ ASSERT_NE(fd, -1);
1319+ ASSERT_EQ(ftruncate(fd, 10 * page_size), 0);
1320+
1321+ /* Map a region using the memfd at page offset 0. */
1322+ ptr = mmap(carveout, 5 * page_size, PROT_READ | PROT_WRITE,
1323+ MAP_FIXED | MAP_PRIVATE, fd, 0);
1324+ ASSERT_NE(ptr, MAP_FAILED);
1325+
1326+ /*
1327+ * Map another separately and trigger a CoW fault at page offset 5:
1328+ *
1329+ * |-----------| |---------|
1330+ * | unfaulted | | faulted |
1331+ * |-----------| |---------|
1332+ */
1333+ ptr2 = mmap(&carveout[10 * page_size], 5 * page_size,
1334+ PROT_READ | PROT_WRITE, MAP_FIXED | MAP_PRIVATE,
1335+ fd, 5 * page_size);
1336+ ASSERT_NE(ptr2, MAP_FAILED);
1337+ ptr2[0] = 'x';
1338+
1339+ /*
1340+ * Now move it in place:
1341+ *
1342+ * |----------|
1343+ * | |
1344+ * v |
1345+ * |-----------| |---------|
1346+ * | unfaulted | | faulted |
1347+ * |-----------| |---------|
1348+ *
1349+ * Because the anonymous page offset of the faulted region is now
1350+ * &carveout[10 * page_size], despite the two regions being mergeable
1351+ * due to file page offset, they are NOT mergeable due to anonymous
1352+ * page offset.
1353+ */
1354+ ptr2 = sys_mremap(ptr2, 5 * page_size, 5 * page_size,
1355+ MREMAP_MAYMOVE | MREMAP_FIXED,
1356+ &carveout[5 * page_size]);
1357+ ASSERT_NE(ptr2, MAP_FAILED);
1358+
1359+ /* Assert that they did not merge. */
1360+ ASSERT_TRUE(find_vma_procmap(procmap, ptr));
1361+ ASSERT_EQ(procmap->query.vma_start, (unsigned long)ptr);
1362+ ASSERT_EQ(procmap->query.vma_end, (unsigned long)ptr + 5 * page_size);
1363+}
1364+
13081365 TEST_F(merge_with_fork, mremap_faulted_to_unfaulted_prev)
13091366 {
13101367 struct procmap_fd *procmap = &self->procmap;
tools/testing/selftests/mm/migration.c+20−17
@@ -7,7 +7,7 @@ · 6 unmodified lines
77 #include "kselftest_harness.h"
88 #include "hugepage_settings.h"
99
10−#include <strings.h>
10+#include <string.h>
1111 #include <pthread.h>
1212 #include <numa.h>
1313 #include <numaif.h>
@@ -20,7 +20,6 @@ · 6 unmodified lines
2020
2121 #define TWOMEG (2<<20)
2222 #define RUNTIME (20)
23−#define MAX_RETRIES 100
2423 #define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1)))
2524
2625 HUGETLB_SETUP_DEFAULT_PAGES(1)
@@ -110,7 +109,7 @@ int migrate(uint64_t *ptr, int n1, int n2) · 83 unmodified lines
110109 int ret, tmp;
111110 int status = 0;
112111 struct timespec ts1, ts2;
113− int failures = 0;
112+ int success = 0;
114113
115114 if (clock_gettime(CLOCK_MONOTONIC, &ts1))
116115 return -1;
@@ -119,29 +118,33 @@ int migrate(uint64_t *ptr, int n1, int n2) · 2 unmodified lines
119118 if (clock_gettime(CLOCK_MONOTONIC, &ts2))
120119 return -1;
121120
122− if (ts2.tv_sec - ts1.tv_sec >= RUNTIME)
123− return 0;
121+ if (ts2.tv_sec - ts1.tv_sec >= RUNTIME) {
122+ /* Reaching both targets verifies a cross-node move. */
123+ if (success >= 2)
124+ return 0;
125+ else
126+ return -2;
127+ }
124128
125129 ret = move_pages(0, 1, (void **) &ptr, &n2, &status,
126130 MPOL_MF_MOVE_ALL);
127− if (ret) {
128− if (ret > 0) {
129− /* Migration is best effort; try again */
130− if (++failures < MAX_RETRIES)
131− continue;
132− printf("Didn't migrate %d pages\n", ret);
133− }
134− else
135− perror("Couldn't migrate pages");
131+ if (ret < 0) {
132+ perror("Couldn't migrate pages");
133+ return ret;
134+ }
135+ /* Migration is best effort. Try again */
136+ if (ret > 0 || status < 0)
137+ continue;
138+ if (status != n2) {
139+ printf("Page is on node %d instead of target node %d\n",
140+ status, n2);
136141 return -2;
137142 }
138− failures = 0;
143+ success++;
139144 tmp = n2;
140145 n2 = n1;
141146 n1 = tmp;
142147 }
143−
144− return 0;
145148 }
146149
147150 void *access_mem(void *ptr)
tools/testing/selftests/mm/mseal_test.c+1−2
@@ -1876,7 +1876,7 @@ int main(void) · 1875 unmodified lines
18761876 if (!pkey_supported())
18771877 ksft_print_msg("PKEY not supported\n");
18781878
1879− ksft_set_plan(88);
1879+ ksft_set_plan(87);
18801880
18811881 test_seal_addseal();
18821882 test_seal_unmapped_start();
@@ -1913,7 +1913,6 @@ int main(void) · 30 unmodified lines
19131913 test_seal_mprotect_partial_mprotect(false);
19141914 test_seal_mprotect_partial_mprotect(true);
19151915
1916− test_seal_mprotect_two_vma_with_gap();
19171916 test_seal_mprotect_two_vma_with_gap();
19181917
19191918 test_seal_mprotect_merge(false);
tools/testing/selftests/mm/pagemap_ioctl.c+1−7
@@ -1085,7 +1085,7 @@ static void unpopulated_written_test(const char *name, char *mem, long size, · 1084 unmodified lines
10851085 memset(mem, 1, size);
10861086 if (use_thp &&
10871087 (madvise(mem, size, MADV_COLLAPSE) ||
1088− !check_huge_anon(mem, size / hpage_size, hpage_size))) {
1088+ !check_huge_anon(mem, size, size / hpage_size, hpage_size))) {
10891089 ksft_test_result_skip("%s could not form a THP\n", name);
10901090 goto out;
10911091 }
@@ -1332,12 +1332,6 @@ int mprotect_tests(void) · 240 unmodified lines
13321332 int ret;
13331333 char *mem, *mem2;
13341334 struct page_region vec;
1335− int pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
1336−
1337− if (pagemap_fd < 0) {
1338− fprintf(stderr, "open() failed\n");
1339− exit(1);
1340− }
13411335
13421336 /* 1. Map two pages */
13431337 mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
tools/testing/selftests/mm/prctl_thp_disable.c+1−1
@@ -67,7 +67,7 @@ static int test_mmap_thp(enum thp_collapse_type madvise_buf, size_t pmdsize) · 66 unmodified lines
6767 /* HACK: make sure we have a separate VMA that we can check reliably. */
6868 mprotect(mem, pmdsize, PROT_READ);
6969
70− ret = check_huge_anon(mem, 1, pmdsize);
70+ ret = check_huge_anon(mem, pmdsize, 1, pmdsize);
7171 munmap(mmap_mem, mmap_size);
7272 return ret;
7373 }
tools/testing/selftests/mm/run_vmtests.sh+2−0
@@ -410,6 +410,8 @@ CATEGORY="thp" run_test ./khugepaged all:shmem · 409 unmodified lines
410410
411411 CATEGORY="thp" run_test ./khugepaged -s 4 all:shmem
412412
413+CATEGORY="thp" run_test ./khugepaged -c 4 mthp_khugepaged:anon
414+
413415 # Try to create XFS if not provided
414416 if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then
415417 if test_selected "thp"; then
tools/testing/selftests/mm/soft-dirty.c+3−2
@@ -103,7 +103,7 @@ static void test_hugepage(int pagemap_fd, int pagesize) · 102 unmodified lines
103103 for (i = 0; i < hpage_len; i++)
104104 map[i] = (char)i;
105105
106− if (check_huge_anon(map, 1, hpage_len)) {
106+ if (check_huge_anon(map, hpage_len, 1, hpage_len)) {
107107 ksft_test_result_pass("Test %s huge page allocation\n", __func__);
108108
109109 clear_softdirty();
@@ -152,7 +152,8 @@ static void test_mprotect(int pagemap_fd, int pagesize, bool anon) · 42 unmodified lines
152152 return;
153153 }
154154 unlink(fname);
155− ftruncate(test_fd, pagesize);
155+ if (ftruncate(test_fd, pagesize) != 0)
156+ ksft_exit_fail_msg("ftruncate failed\n");
156157 map = mmap(NULL, pagesize, PROT_READ|PROT_WRITE,
157158 MAP_SHARED, test_fd, 0);
158159 if (map == MAP_FAILED)
tools/testing/selftests/mm/split_huge_page_test.c+8−131
@@ -104,129 +104,6 @@ fail: · 103 unmodified lines
104104 return false;
105105 }
106106
107−static int vaddr_pageflags_get(char *vaddr, int pagemap_fd, int kpageflags_fd,
108− uint64_t *flags)
109−{
110− unsigned long pfn;
111−
112− pfn = pagemap_get_pfn(pagemap_fd, vaddr);
113−
114− /* non-present PFN */
115− if (pfn == -1UL)
116− return 1;
117−
118− if (pageflags_get(pfn, kpageflags_fd, flags))
119− return -1;
120−
121− return 0;
122−}
123−
124−/*
125− * gather_after_split_folio_orders - scan through [vaddr_start, len) and record
126− * folio orders
127− *
128− * @vaddr_start: start vaddr
129− * @len: range length
130− * @pagemap_fd: file descriptor to /proc/<pid>/pagemap
131− * @kpageflags_fd: file descriptor to /proc/kpageflags
132− * @orders: output folio order array
133− * @nr_orders: folio order array size
134− *
135− * gather_after_split_folio_orders() scan through [vaddr_start, len) and check
136− * all folios within the range and record their orders. All order-0 pages will
137− * be recorded. Non-present vaddr is skipped.
138− *
139− * NOTE: the function is used to check folio orders after a split is performed,
140− * so it assumes [vaddr_start, len) fully maps to after-split folios within that
141− * range.
142− *
143− * Return: 0 - no error, -1 - unhandled cases
144− */
145−static int gather_after_split_folio_orders(char *vaddr_start, size_t len,
146− int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders)
147−{
148− uint64_t page_flags = 0;
149− int cur_order = -1;
150− char *vaddr;
151−
152− if (pagemap_fd == -1 || kpageflags_fd == -1)
153− return -1;
154− if (!orders)
155− return -1;
156− if (nr_orders <= 0)
157− return -1;
158−
159− for (vaddr = vaddr_start; vaddr < vaddr_start + len;) {
160− char *next_folio_vaddr;
161− int status;
162−
163− status = vaddr_pageflags_get(vaddr, pagemap_fd, kpageflags_fd,
164− &page_flags);
165− if (status < 0)
166− return -1;
167−
168− /* skip non present vaddr */
169− if (status == 1) {
170− vaddr += psize();
171− continue;
172− }
173−
174− /* all order-0 pages with possible false postive (non folio) */
175− if (!(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) {
176− orders[0]++;
177− vaddr += psize();
178− continue;
179− }
180−
181− /* skip non thp compound pages */
182− if (!(page_flags & KPF_THP)) {
183− vaddr += psize();
184− continue;
185− }
186−
187− /* vpn points to part of a THP at this point */
188− if (page_flags & KPF_COMPOUND_HEAD)
189− cur_order = 1;
190− else {
191− vaddr += psize();
192− continue;
193− }
194−
195− next_folio_vaddr = vaddr + (1UL << (cur_order + pshift()));
196−
197− if (next_folio_vaddr >= vaddr_start + len)
198− break;
199−
200− while ((status = vaddr_pageflags_get(next_folio_vaddr,
201− pagemap_fd, kpageflags_fd,
202− &page_flags)) >= 0) {
203− /*
204− * non present vaddr, next compound head page, or
205− * order-0 page
206− */
207− if (status == 1 ||
208− (page_flags & KPF_COMPOUND_HEAD) ||
209− !(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) {
210− if (cur_order < nr_orders) {
211− orders[cur_order]++;
212− cur_order = -1;
213− vaddr = next_folio_vaddr;
214− }
215− break;
216− }
217−
218− cur_order++;
219− next_folio_vaddr = vaddr + (1UL << (cur_order + pshift()));
220− }
221−
222− if (status < 0)
223− return status;
224− }
225− if (cur_order > 0 && cur_order < nr_orders)
226− orders[cur_order]++;
227− return 0;
228−}
229−
230107 static int check_after_split_folio_orders(char *vaddr_start, size_t len,
231108 int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders)
232109 {
@@ -240,7 +117,7 @@ static int check_after_split_folio_orders(char *vaddr_start, size_t len, · 7 unmodified lines
240117 ksft_exit_fail_msg("Cannot allocate memory for vaddr_orders");
241118
242119 memset(vaddr_orders, 0, sizeof(int) * nr_orders);
243− status = gather_after_split_folio_orders(vaddr_start, len, pagemap_fd,
120+ status = gather_folio_orders(vaddr_start, len, pagemap_fd,
244121 kpageflags_fd, vaddr_orders, nr_orders);
245122 if (status)
246123 ksft_exit_fail_msg("gather folio info failed\n");
@@ -296,7 +173,7 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp · 49 unmodified lines
296173 unsigned long rss_anon_before, rss_anon_after;
297174 size_t i;
298175
299− if (!check_huge_anon(one_page, nr_hpages, pmd_pagesize))
176+ if (!check_huge_anon(one_page, nr_hpages * pmd_pagesize, nr_hpages, pmd_pagesize))
300177 ksft_exit_fail_msg("No THP is allocated\n");
301178
302179 rss_anon_before = rss_anon();
@@ -311,7 +188,7 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp · 8 unmodified lines
311188 if (one_page[i] != (char)0)
312189 ksft_exit_fail_msg("%ld byte corrupted\n", i);
313190
314− if (!check_huge_anon(one_page, 0, pmd_pagesize))
191+ if (!check_huge_anon(one_page, nr_hpages * pmd_pagesize, 0, pmd_pagesize))
315192 ksft_exit_fail_msg("Still AnonHugePages not split\n");
316193
317194 rss_anon_after = rss_anon();
@@ -347,7 +224,7 @@ static void split_pmd_thp_to_order(int order) · 29 unmodified lines
347224 for (i = 0; i < len; i++)
348225 one_page[i] = (char)i;
349226
350− if (!check_huge_anon(one_page, 4, pmd_pagesize))
227+ if (!check_huge_anon(one_page, 4 * pmd_pagesize, 4, pmd_pagesize))
351228 ksft_exit_fail_msg("No THP is allocated\n");
352229
353230 /* split all THPs */
@@ -366,7 +243,7 @@ static void split_pmd_thp_to_order(int order) · 12 unmodified lines
366243 (pmd_order + 1)))
367244 ksft_exit_fail_msg("Unexpected THP split\n");
368245
369− if (!check_huge_anon(one_page, 0, pmd_pagesize))
246+ if (!check_huge_anon(one_page, 4 * pmd_pagesize, 0, pmd_pagesize))
370247 ksft_exit_fail_msg("Still AnonHugePages not split\n");
371248
372249 ksft_test_result_pass("Split huge pages to order %d successful\n", order);
@@ -393,7 +270,7 @@ static void split_pte_mapped_thp(void) · 20 unmodified lines
393270 for (i = 0; i < thp_area_size; i++)
394271 thp_area[i] = (char)i;
395272
396− if (!check_huge_anon(thp_area, nr_thps, pmd_pagesize)) {
273+ if (!check_huge_anon(thp_area, nr_thps * pmd_pagesize, nr_thps, pmd_pagesize)) {
397274 ksft_test_result_skip("Not all THPs allocated\n");
398275 goto out;
399276 }
@@ -657,7 +534,7 @@ static int create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, · 257 unmodified lines
657534
658535 force_read_pages(*addr, fd_size / pmd_pagesize, pmd_pagesize);
659536
660− if (!check_huge_file(*addr, fd_size / pmd_pagesize, pmd_pagesize)) {
537+ if (!check_huge_file(*addr, fd_size, fd_size / pmd_pagesize, pmd_pagesize)) {
661538 ksft_print_msg("No large pagecache folio generated, please provide a filesystem supporting large folio\n");
662539 munmap(*addr, fd_size);
663540 close(*fd);
@@ -735,7 +612,7 @@ static void split_thp_in_pagecache_to_order_at(size_t fd_size, · 71 unmodified lines
735612 goto out;
736613 }
737614
738− if (!check_huge_file(addr, 0, pmd_pagesize)) {
615+ if (!check_huge_file(addr, fd_size, 0, pmd_pagesize)) {
739616 ksft_print_msg("Still FilePmdMapped not split\n");
740617 err = EXIT_FAILURE;
741618 goto out;
tools/testing/selftests/mm/uffd-common.c+3−1
@@ -194,7 +194,9 @@ static void shmem_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *start, · 193 unmodified lines
194194
195195 static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *p, int expect_nr_hpages)
196196 {
197− if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages,
197+ size_t len = expect_nr_hpages * read_pmd_pagesize();
198+
199+ if (!check_huge_shmem(gopts->area_dst_alias, len, expect_nr_hpages,
198200 read_pmd_pagesize()))
199201 err("Did not find expected %d number of hugepages",
200202 expect_nr_hpages);
tools/testing/selftests/mm/vm_util.c+197−10
@@ -15,6 +15,9 @@ · 14 unmodified lines
1515 #define SMAP_FILE_PATH "/proc/self/smaps"
1616 #define STATUS_FILE_PATH "/proc/self/status"
1717 #define MAX_LINE_LENGTH 500
18+#define PAGEMAP_PATH "/proc/self/pagemap"
19+#define KPAGEFLAGS_PATH "/proc/kpageflags"
20+#define MAX_NR_ORDERS 20
1821
1922 unsigned int __page_size;
2023 unsigned int __page_shift;
@@ -31,7 +34,7 @@ uint64_t pagemap_get_entry(int fd, char *start) · 10 unmodified lines
3134 return entry;
3235 }
3336
34−static uint64_t __pagemap_scan_get_categories(int fd, char *start, struct page_region *r)
37+static int __pagemap_scan_get_categories(int fd, char *start, struct page_region *r)
3538 {
3639 struct pm_scan_arg arg;
3740
@@ -55,7 +58,7 @@ static uint64_t __pagemap_scan_get_categories(int fd, char *start, struct page_r · 17 unmodified lines
5558 static uint64_t pagemap_scan_get_categories(int fd, char *start)
5659 {
5760 struct page_region r;
58− long ret;
61+ int ret;
5962
6063 ret = __pagemap_scan_get_categories(fd, start, &r);
6164 if (ret < 0)
@@ -194,6 +197,125 @@ err_out: · 132 unmodified lines
194197 return rss_anon;
195198 }
196199
200+static int vaddr_pageflags_get(char *vaddr, int pagemap_fd, int kpageflags_fd,
201+ uint64_t *flags)
202+{
203+ unsigned long pfn;
204+
205+ pfn = pagemap_get_pfn(pagemap_fd, vaddr);
206+
207+ /* non-present PFN */
208+ if (pfn == -1UL)
209+ return 1;
210+
211+ if (pageflags_get(pfn, kpageflags_fd, flags))
212+ return -1;
213+
214+ return 0;
215+}
216+
217+/*
218+ * gather_folio_orders - scan through [vaddr_start, len) and record
219+ * folio orders
220+ *
221+ * @vaddr_start: start vaddr
222+ * @len: range length
223+ * @pagemap_fd: file descriptor to /proc/<pid>/pagemap
224+ * @kpageflags_fd: file descriptor to /proc/kpageflags
225+ * @orders: output folio order array
226+ * @nr_orders: folio order array size
227+ *
228+ * gather_folio_orders() scan through [vaddr_start, len) and check
229+ * all folios within the range and record their orders. All order-0 pages will
230+ * be recorded. Non-present vaddr is skipped.
231+ *
232+ * Return: 0 - no error, -1 - unhandled cases
233+ */
234+int gather_folio_orders(char *vaddr_start, size_t len,
235+ int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders)
236+{
237+ uint64_t page_flags = 0;
238+ int cur_order = -1;
239+ char *vaddr;
240+
241+ if (pagemap_fd == -1 || kpageflags_fd == -1)
242+ return -1;
243+ if (!orders)
244+ return -1;
245+ if (nr_orders <= 0)
246+ return -1;
247+
248+ for (vaddr = vaddr_start; vaddr < vaddr_start + len;) {
249+ char *next_folio_vaddr;
250+ int status;
251+
252+ status = vaddr_pageflags_get(vaddr, pagemap_fd, kpageflags_fd,
253+ &page_flags);
254+ if (status < 0)
255+ return -1;
256+
257+ /* skip non present vaddr */
258+ if (status == 1) {
259+ vaddr += psize();
260+ continue;
261+ }
262+
263+ /* all order-0 pages with possible false postive (non folio) */
264+ if (!(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) {
265+ orders[0]++;
266+ vaddr += psize();
267+ continue;
268+ }
269+
270+ /* skip non thp compound pages */
271+ if (!(page_flags & KPF_THP)) {
272+ vaddr += psize();
273+ continue;
274+ }
275+
276+ /* vpn points to part of a THP at this point */
277+ if (page_flags & KPF_COMPOUND_HEAD)
278+ cur_order = 1;
279+ else {
280+ vaddr += psize();
281+ continue;
282+ }
283+
284+ next_folio_vaddr = vaddr + (1UL << (cur_order + pshift()));
285+
286+ if (next_folio_vaddr >= vaddr_start + len)
287+ break;
288+
289+ while ((status = vaddr_pageflags_get(next_folio_vaddr,
290+ pagemap_fd, kpageflags_fd,
291+ &page_flags)) >= 0) {
292+ /*
293+ * non present vaddr, next compound head page, or
294+ * order-0 page
295+ */
296+ if (status == 1 ||
297+ (page_flags & KPF_COMPOUND_HEAD) ||
298+ !(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) {
299+ if (cur_order < nr_orders) {
300+ orders[cur_order]++;
301+ cur_order = -1;
302+ vaddr = next_folio_vaddr;
303+ }
304+ break;
305+ }
306+
307+ cur_order++;
308+ next_folio_vaddr = vaddr + (1UL << (cur_order + pshift()));
309+ }
310+
311+ if (status < 0)
312+ return status;
313+ }
314+ if (cur_order > 0 && cur_order < nr_orders)
315+ orders[cur_order]++;
316+ return 0;
317+}
318+
197319 char *__get_smap_entry(void *addr, const char *pattern, char *buf, size_t len)
198320 {
199321 int ret;
@@ -229,7 +351,7 @@ err_out: · 29 unmodified lines
229351 return entry;
230352 }
231353
232−bool __check_huge(void *addr, char *pattern, int nr_hpages,
354+static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages,
233355 uint64_t hpage_size)
234356 {
235357 char buffer[MAX_LINE_LENGTH];
@@ -247,19 +369,84 @@ err_out: · 11 unmodified lines
247369 return thp == (nr_hpages * (hpage_size >> 10));
248370 }
249371
250−bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size)
372+static bool check_large_folios(void *addr, size_t len, int nr_hpages,
373+ uint64_t hpage_size)
251374 {
252− return __check_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size);
375+ int order = 0, pagesize = getpagesize();
376+ unsigned int nr_pages = hpage_size / pagesize;
377+ int orders[MAX_NR_ORDERS], status;
378+ int pagemap_fd, kpageflags_fd;
379+ bool ret = false;
380+
381+ if (!nr_pages)
382+ ksft_exit_fail_msg("invalid hugepage size\n");
383+
384+ order = 31 - __builtin_clz(nr_pages);
385+ if (!order || order >= MAX_NR_ORDERS)
386+ ksft_exit_fail_msg("invalid order\n");
387+
388+ memset(orders, 0, sizeof(int) * MAX_NR_ORDERS);
389+ pagemap_fd = open(PAGEMAP_PATH, O_RDONLY);
390+ if (pagemap_fd == -1)
391+ ksft_exit_fail_msg("read pagemap fail\n");
392+
393+ kpageflags_fd = open(KPAGEFLAGS_PATH, O_RDONLY);
394+ if (kpageflags_fd == -1) {
395+ close(pagemap_fd);
396+ ksft_exit_fail_msg("read kpageflags fail\n");
397+ }
398+
399+ status = gather_folio_orders(addr, len, pagemap_fd,
400+ kpageflags_fd, orders, MAX_NR_ORDERS);
401+ if (status)
402+ goto out;
403+
404+ if (orders[order] == nr_hpages)
405+ ret = true;
406+
407+out:
408+ close(pagemap_fd);
409+ close(kpageflags_fd);
410+ return ret;
253411 }
254412
255−bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size)
413+bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size)
256414 {
257− return __check_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size);
415+ uint64_t pmd_pagesize = read_pmd_pagesize();
416+
417+ if (!pmd_pagesize)
418+ ksft_exit_fail_msg("reading PMD pagesize failed\n");
419+
420+ if (hpage_size == pmd_pagesize)
421+ return __check_pmd_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size);
422+
423+ return check_large_folios(addr, len, nr_hpages, hpage_size);
258424 }
259425
260−bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size)
426+bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size)
261427 {
262− return __check_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size);
428+ uint64_t pmd_pagesize = read_pmd_pagesize();
429+
430+ if (!pmd_pagesize)
431+ ksft_exit_fail_msg("reading PMD pagesize failed\n");
432+
433+ if (hpage_size == pmd_pagesize)
434+ return __check_pmd_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size);
435+
436+ return check_large_folios(addr, len, nr_hpages, hpage_size);
437+}
438+
439+bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size)
440+{
441+ uint64_t pmd_pagesize = read_pmd_pagesize();
442+
443+ if (!pmd_pagesize)
444+ ksft_exit_fail_msg("reading PMD pagesize failed\n");
445+
446+ if (hpage_size == pmd_pagesize)
447+ return __check_pmd_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size);
448+
449+ return check_large_folios(addr, len, nr_hpages, hpage_size);
263450 }
264451
265452 int64_t allocate_transhuge(void *ptr, int pagemap_fd)
@@ -755,7 +942,7 @@ unsigned long read_num(const char *path) · 489 unmodified lines
755942 {
756943 char buf[21];
757944
758− if (read_file(path, buf, sizeof(buf)) < 0)
945+ if (!read_file(path, buf, sizeof(buf)))
759946 ksft_exit_fail_perror("read_file()");
760947
761948 return strtoul(buf, NULL, 10);
tools/testing/selftests/mm/vm_util.h+5−3
@@ -90,11 +90,13 @@ void clear_softdirty(void); · 89 unmodified lines
9090 bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t len);
9191 uint64_t read_pmd_pagesize(void);
9292 unsigned long rss_anon(void);
93−bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size);
94−bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size);
95−bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size);
93+bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
94+bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
95+bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
9696 int64_t allocate_transhuge(void *ptr, int pagemap_fd);
9797 int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags);
98+int gather_folio_orders(char *vaddr_start, size_t len,
99+ int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders);
98100
99101 int uffd_register(int uffd, void *addr, uint64_t len,
100102 bool miss, bool wp, bool minor);
tools/testing/selftests/proc/proc-maps-race.c+2−1
@@ -490,7 +490,8 @@ static bool query_addr_at(int maps_fd, void *addr, · 489 unmodified lines
490490
491491 static inline bool split_vma(FIXTURE_DATA(proc_maps_race) *self)
492492 {
493− return mmap(self->mod_info->addr, self->page_size, self->mod_info->prot | PROT_EXEC,
493+ /* PROT_NONE differs from both readable neighbors. */
494+ return mmap(self->mod_info->addr, self->page_size, PROT_NONE,
494495 MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) != MAP_FAILED;
495496 }
496497
tools/testing/vma/include/dup.h+62−2
@@ -243,7 +243,7 @@ enum { · 242 unmodified lines
243243 #define VM_NOHUGEPAGE INIT_VM_FLAG(NOHUGEPAGE)
244244 #define VM_MERGEABLE INIT_VM_FLAG(MERGEABLE)
245245 #define VM_STACK INIT_VM_FLAG(STACK)
246−#ifdef CONFIG_STACK_GROWS_UP
246+#ifdef CONFIG_STACK_GROWSUP
247247 #define VM_STACK_EARLY INIT_VM_FLAG(STACK_EARLY)
248248 #define VMA_STACK_EARLY mk_vma_flags(VMA_STACK_EARLY_BIT)
249249 #else
@@ -577,6 +577,7 @@ struct vm_area_struct { · 327 unmodified lines
577577 */
578578 unsigned int vm_lock_seq;
579579 #endif
580+ unsigned int __vm_anon_pgoff_lo;
580581
581582 /*
582583 * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma
@@ -612,6 +613,9 @@ struct vm_area_struct { · 29 unmodified lines
612613 #ifdef CONFIG_PER_VMA_LOCK
613614 /* Unstable RCU readers are allowed to read this. */
614615 refcount_t vm_refcnt;
616+#endif
617+#ifdef CONFIG_64BIT
618+ unsigned int __vm_anon_pgoff_hi;
615619 #endif
616620 /*
617621 * For areas with an address space and backing store,
@@ -1158,6 +1162,17 @@ static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma) · 540 unmodified lines
11581162 return is_shared_maywrite(&vma->flags);
11591163 }
11601164
1165+static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags)
1166+{
1167+ return vma_flags_test(flags, VMA_MAYWRITE_BIT) &&
1168+ !vma_flags_test(flags, VMA_SHARED_BIT);
1169+}
1170+
1171+static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma)
1172+{
1173+ return vma_flags_is_cow_mapping(&vma->flags);
1174+}
1175+
11611176 static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi)
11621177 {
11631178 /*
@@ -1320,6 +1335,28 @@ static inline pgoff_t vma_end_pgoff(const struct vm_area_struct *vma) · 156 unmodified lines
13201335 return vma_start_pgoff(vma) + vma_pages(vma);
13211336 }
13221337
1338+static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma)
1339+{
1340+ pgoff_t pgoff = 0;
1341+
1342+#ifdef CONFIG_64BIT
1343+ pgoff += vma->__vm_anon_pgoff_hi;
1344+ pgoff <<= 32;
1345+#endif
1346+ pgoff += vma->__vm_anon_pgoff_lo;
1347+ return pgoff;
1348+}
1349+
1350+static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma)
1351+{
1352+ return vma_start_anon_pgoff(vma) + vma_pages(vma);
1353+}
1354+
1355+static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma)
1356+{
1357+ return vma_end_anon_pgoff(vma) - 1;
1358+}
1359+
13231360 static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc)
13241361 {
13251362 return file->f_op->mmap_prepare(desc);
@@ -1391,7 +1428,7 @@ static inline void vma_iter_set(struct vma_iterator *vmi, unsigned long addr) · 65 unmodified lines
13911428 mas_set(&vmi->mas, addr);
13921429 }
13931430
1394−static inline bool vma_is_anonymous(struct vm_area_struct *vma)
1431+static inline bool vma_is_anonymous(const struct vm_area_struct *vma)
13951432 {
13961433 return !vma->vm_ops;
13971434 }
@@ -1584,3 +1621,26 @@ static inline pgprot_t vma_get_page_prot(const struct vm_area_struct *vma) · 186 unmodified lines
15841621 {
15851622 return vma_flags_to_page_prot(vma->flags);
15861623 }
1624+
1625+static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
1626+ const unsigned long address)
1627+{
1628+ pgoff_t pgoff;
1629+
1630+ pgoff = linear_page_delta(vma, address);
1631+ pgoff += vma_start_anon_pgoff(vma);
1632+ return pgoff;
1633+}
1634+
1635+static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
1636+ const unsigned long address)
1637+{
1638+ const pgoff_t pgoff = __linear_anon_page_index(vma, address);
1639+
1640+ VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
1641+ /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */
1642+ if (vma_is_anonymous(vma) && !vma->vm_file)
1643+ VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
1644+
1645+ return pgoff;
1646+}
tools/testing/vma/shared.c+2−1
@@ -23,7 +23,8 @@ struct vm_area_struct *alloc_vma(struct mm_struct *mm, · 22 unmodified lines
2323
2424 vma->vm_start = start;
2525 vma->vm_end = end;
26− vma->vm_pgoff = pgoff;
26+ vma_set_pgoff(vma, pgoff);
27+ vma_set_anon_pgoff(vma, start >> PAGE_SHIFT);
2728 vma->flags = vma_flags;
2829 vma_assert_detached(vma);
2930
tools/testing/vma/tests/merge.c+39−10
@@ -45,6 +45,7 @@ void vmg_set_range(struct vma_merge_struct *vmg, unsigned long start, · 44 unmodified lines
4545 vmg->start = start;
4646 vmg->end = end;
4747 vmg->pgoff = pgoff;
48+ vmg->anon_pgoff = start >> PAGE_SHIFT;
4849 vmg->vma_flags = vma_flags;
4950
5051 vmg->just_expand = false;
@@ -108,6 +109,7 @@ static bool test_simple_merge(void) · 57 unmodified lines
108109 .end = 0x2000,
109110 .vma_flags = vma_flags,
110111 .pgoff = 1,
112+ .anon_pgoff = 1,
111113 };
112114
113115 ASSERT_FALSE(attach_vma(&mm, vma_left));
@@ -119,6 +121,7 @@ static bool test_simple_merge(void) · 5 unmodified lines
119121 ASSERT_EQ(vma->vm_start, 0);
120122 ASSERT_EQ(vma->vm_end, 0x3000);
121123 ASSERT_EQ(vma_start_pgoff(vma), 0);
124+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
122125 ASSERT_FLAGS_SAME_MASK(&vma->flags, vma_flags);
123126
124127 detach_free_vma(vma);
@@ -151,6 +154,7 @@ static bool test_simple_modify(void) · 26 unmodified lines
151154 ASSERT_EQ(vma->vm_start, 0x1000);
152155 ASSERT_EQ(vma->vm_end, 0x2000);
153156 ASSERT_EQ(vma_start_pgoff(vma), 1);
157+ ASSERT_EQ(vma_start_anon_pgoff(vma), 1);
154158
155159 /*
156160 * Now walk through the three split VMAs and make sure they are as
@@ -163,6 +167,7 @@ static bool test_simple_modify(void) · 6 unmodified lines
163167 ASSERT_EQ(vma->vm_start, 0);
164168 ASSERT_EQ(vma->vm_end, 0x1000);
165169 ASSERT_EQ(vma_start_pgoff(vma), 0);
170+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
166171
167172 detach_free_vma(vma);
168173 vma_iter_clear(&vmi);
@@ -172,6 +177,7 @@ static bool test_simple_modify(void) · 3 unmodified lines
172177 ASSERT_EQ(vma->vm_start, 0x1000);
173178 ASSERT_EQ(vma->vm_end, 0x2000);
174179 ASSERT_EQ(vma_start_pgoff(vma), 1);
180+ ASSERT_EQ(vma_start_anon_pgoff(vma), 1);
175181
176182 detach_free_vma(vma);
177183 vma_iter_clear(&vmi);
@@ -181,6 +187,7 @@ static bool test_simple_modify(void) · 3 unmodified lines
181187 ASSERT_EQ(vma->vm_start, 0x2000);
182188 ASSERT_EQ(vma->vm_end, 0x3000);
183189 ASSERT_EQ(vma_start_pgoff(vma), 2);
190+ ASSERT_EQ(vma_start_anon_pgoff(vma), 2);
184191
185192 detach_free_vma(vma);
186193 mtree_destroy(&mm.mm_mt);
@@ -210,6 +217,7 @@ static bool test_simple_expand(void) · 23 unmodified lines
210217 ASSERT_EQ(vma->vm_start, 0);
211218 ASSERT_EQ(vma->vm_end, 0x3000);
212219 ASSERT_EQ(vma_start_pgoff(vma), 0);
220+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
213221
214222 detach_free_vma(vma);
215223 mtree_destroy(&mm.mm_mt);
@@ -232,6 +240,7 @@ static bool test_simple_shrink(void) · 16 unmodified lines
232240 ASSERT_EQ(vma->vm_start, 0);
233241 ASSERT_EQ(vma->vm_end, 0x1000);
234242 ASSERT_EQ(vma_start_pgoff(vma), 0);
243+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
235244
236245 detach_free_vma(vma);
237246 mtree_destroy(&mm.mm_mt);
@@ -344,6 +353,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 106 unmodified lines
344353 ASSERT_EQ(vma->vm_start, 0);
345354 ASSERT_EQ(vma->vm_end, 0x5000);
346355 ASSERT_EQ(vma_start_pgoff(vma), 0);
356+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
347357 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
348358 ASSERT_TRUE(vma_write_started(vma));
349359 ASSERT_EQ(mm.map_count, 3);
@@ -365,6 +375,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 15 unmodified lines
365375 ASSERT_EQ(vma->vm_start, 0x6000);
366376 ASSERT_EQ(vma->vm_end, 0x9000);
367377 ASSERT_EQ(vma_start_pgoff(vma), 6);
378+ ASSERT_EQ(vma_start_anon_pgoff(vma), 6);
368379 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
369380 ASSERT_TRUE(vma_write_started(vma));
370381 ASSERT_EQ(mm.map_count, 3);
@@ -385,6 +396,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 14 unmodified lines
385396 ASSERT_EQ(vma->vm_start, 0);
386397 ASSERT_EQ(vma->vm_end, 0x9000);
387398 ASSERT_EQ(vma_start_pgoff(vma), 0);
399+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
388400 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
389401 ASSERT_TRUE(vma_write_started(vma));
390402 ASSERT_EQ(mm.map_count, 2);
@@ -405,6 +417,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 14 unmodified lines
405417 ASSERT_EQ(vma->vm_start, 0xa000);
406418 ASSERT_EQ(vma->vm_end, 0xc000);
407419 ASSERT_EQ(vma_start_pgoff(vma), 0xa);
420+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0xa);
408421 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
409422 ASSERT_TRUE(vma_write_started(vma));
410423 ASSERT_EQ(mm.map_count, 2);
@@ -424,6 +437,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 13 unmodified lines
424437 ASSERT_EQ(vma->vm_start, 0);
425438 ASSERT_EQ(vma->vm_end, 0xc000);
426439 ASSERT_EQ(vma_start_pgoff(vma), 0);
440+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
427441 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
428442 ASSERT_TRUE(vma_write_started(vma));
429443 ASSERT_EQ(mm.map_count, 1);
@@ -444,6 +458,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, · 14 unmodified lines
444458 ASSERT_EQ(vma->vm_start, 0);
445459 ASSERT_EQ(vma->vm_end, 0xc000);
446460 ASSERT_EQ(vma_start_pgoff(vma), 0);
461+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
447462 ASSERT_EQ(vma->anon_vma, &dummy_anon_vma);
448463
449464 detach_free_vma(vma);
@@ -640,7 +655,8 @@ static bool test_vma_merge_with_close(void) · 190 unmodified lines
640655 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
641656 ASSERT_EQ(vma_prev->vm_start, 0);
642657 ASSERT_EQ(vma_prev->vm_end, 0x5000);
643− ASSERT_EQ(vma_prev->vm_pgoff, 0);
658+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
659+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
644660
645661 ASSERT_EQ(cleanup_mm(&mm, &vmi), 2);
646662
@@ -751,7 +767,8 @@ static bool test_vma_merge_with_close(void) · 104 unmodified lines
751767 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
752768 ASSERT_EQ(vma_prev->vm_start, 0);
753769 ASSERT_EQ(vma_prev->vm_end, 0x5000);
754− ASSERT_EQ(vma_prev->vm_pgoff, 0);
770+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
771+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
755772
756773 ASSERT_EQ(cleanup_mm(&mm, &vmi), 2);
757774
@@ -806,6 +823,7 @@ static bool test_vma_merge_new_with_close(void) · 48 unmodified lines
806823 ASSERT_EQ(vma->vm_start, 0);
807824 ASSERT_EQ(vma->vm_end, 0x5000);
808825 ASSERT_EQ(vma_start_pgoff(vma), 0);
826+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
809827 ASSERT_EQ(vma->vm_ops, &vm_ops);
810828 ASSERT_TRUE(vma_write_started(vma));
811829 ASSERT_EQ(mm.map_count, 2);
@@ -861,11 +879,13 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo · 49 unmodified lines
861879 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
862880 ASSERT_EQ(vma_next->vm_start, 0x3000);
863881 ASSERT_EQ(vma_next->vm_end, 0x9000);
864− ASSERT_EQ(vma_next->vm_pgoff, 3);
882+ ASSERT_EQ(vma_start_pgoff(vma_next), 3);
883+ ASSERT_EQ(vma_start_anon_pgoff(vma_next), 3);
865884 ASSERT_EQ(vma_next->anon_vma, &dummy_anon_vma);
866885 ASSERT_EQ(vma->vm_start, 0x2000);
867886 ASSERT_EQ(vma->vm_end, 0x3000);
868887 ASSERT_EQ(vma_start_pgoff(vma), 2);
888+ ASSERT_EQ(vma_start_anon_pgoff(vma), 2);
869889 ASSERT_TRUE(vma_write_started(vma));
870890 ASSERT_TRUE(vma_write_started(vma_next));
871891 ASSERT_EQ(mm.map_count, 2);
@@ -895,7 +915,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo · 23 unmodified lines
895915 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
896916 ASSERT_EQ(vma_next->vm_start, 0x2000);
897917 ASSERT_EQ(vma_next->vm_end, 0x9000);
898− ASSERT_EQ(vma_next->vm_pgoff, 2);
918+ ASSERT_EQ(vma_start_pgoff(vma_next), 2);
919+ ASSERT_EQ(vma_start_anon_pgoff(vma_next), 2);
899920 ASSERT_EQ(vma_next->anon_vma, &dummy_anon_vma);
900921 ASSERT_TRUE(vma_write_started(vma_next));
901922 ASSERT_EQ(mm.map_count, 1);
@@ -927,11 +948,13 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo · 25 unmodified lines
927948 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
928949 ASSERT_EQ(vma_prev->vm_start, 0);
929950 ASSERT_EQ(vma_prev->vm_end, 0x6000);
930− ASSERT_EQ(vma_prev->vm_pgoff, 0);
951+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
952+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
931953 ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma);
932954 ASSERT_EQ(vma->vm_start, 0x6000);
933955 ASSERT_EQ(vma->vm_end, 0x7000);
934956 ASSERT_EQ(vma_start_pgoff(vma), 6);
957+ ASSERT_EQ(vma_start_anon_pgoff(vma), 6);
935958 ASSERT_TRUE(vma_write_started(vma_prev));
936959 ASSERT_TRUE(vma_write_started(vma));
937960 ASSERT_EQ(mm.map_count, 2);
@@ -962,7 +985,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo · 24 unmodified lines
962985 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
963986 ASSERT_EQ(vma_prev->vm_start, 0);
964987 ASSERT_EQ(vma_prev->vm_end, 0x7000);
965− ASSERT_EQ(vma_prev->vm_pgoff, 0);
988+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
989+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
966990 ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma);
967991 ASSERT_TRUE(vma_write_started(vma_prev));
968992 ASSERT_EQ(mm.map_count, 1);
@@ -994,7 +1018,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo · 25 unmodified lines
9941018 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
9951019 ASSERT_EQ(vma_prev->vm_start, 0);
9961020 ASSERT_EQ(vma_prev->vm_end, 0x9000);
997− ASSERT_EQ(vma_prev->vm_pgoff, 0);
1021+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
1022+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
9981023 ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma);
9991024 ASSERT_TRUE(vma_write_started(vma_prev));
10001025 ASSERT_EQ(mm.map_count, 1);
@@ -1124,7 +1149,8 @@ static bool test_anon_vma_non_mergeable(void) · 123 unmodified lines
11241149 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
11251150 ASSERT_EQ(vma_prev->vm_start, 0);
11261151 ASSERT_EQ(vma_prev->vm_end, 0x7000);
1127− ASSERT_EQ(vma_prev->vm_pgoff, 0);
1152+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
1153+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
11281154 ASSERT_TRUE(vma_write_started(vma_prev));
11291155 ASSERT_FALSE(vma_write_started(vma_next));
11301156
@@ -1155,7 +1181,8 @@ static bool test_anon_vma_non_mergeable(void) · 24 unmodified lines
11551181 ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS);
11561182 ASSERT_EQ(vma_prev->vm_start, 0);
11571183 ASSERT_EQ(vma_prev->vm_end, 0x7000);
1158− ASSERT_EQ(vma_prev->vm_pgoff, 0);
1184+ ASSERT_EQ(vma_start_pgoff(vma_prev), 0);
1185+ ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0);
11591186 ASSERT_TRUE(vma_write_started(vma_prev));
11601187 ASSERT_FALSE(vma_write_started(vma_next));
11611188
@@ -1417,6 +1444,7 @@ static bool test_merge_extend(void) · 255 unmodified lines
14171444 ASSERT_EQ(vma->vm_start, 0);
14181445 ASSERT_EQ(vma->vm_end, 0x4000);
14191446 ASSERT_EQ(vma_start_pgoff(vma), 0);
1447+ ASSERT_EQ(vma_start_anon_pgoff(vma), 0);
14201448 ASSERT_TRUE(vma_write_started(vma));
14211449 ASSERT_EQ(mm.map_count, 1);
14221450
@@ -1431,7 +1459,7 @@ static bool test_expand_only_mode(void) · 8 unmodified lines
14311459 struct mm_struct mm = {};
14321460 VMA_ITERATOR(vmi, &mm, 0);
14331461 struct vm_area_struct *vma_prev, *vma;
1434− VMG_STATE(vmg, &mm, &vmi, 0x5000, 0x9000, vma_flags, 5);
1462+ VMG_STATE(vmg, &mm, &vmi, 0x5000, 0x9000, vma_flags, 5, 5);
14351463
14361464 /*
14371465 * Place a VMA prior to the one we're expanding so we assert that we do
@@ -1457,6 +1485,7 @@ static bool test_expand_only_mode(void) · 19 unmodified lines
14571485 ASSERT_EQ(vma->vm_start, 0x3000);
14581486 ASSERT_EQ(vma->vm_end, 0x9000);
14591487 ASSERT_EQ(vma_start_pgoff(vma), 3);
1488+ ASSERT_EQ(vma_start_anon_pgoff(vma), 3);
14601489 ASSERT_TRUE(vma_write_started(vma));
14611490 ASSERT_EQ(vma_iter_addr(&vmi), 0x3000);
14621491 vma_assert_attached(vma);
tools/testing/vma/tests/vma.c+47−3
@@ -33,12 +33,56 @@ static bool test_copy_vma(void) · 32 unmodified lines
3333 struct mm_struct mm = {};
3434 bool need_locks = false;
3535 VMA_ITERATOR(vmi, &mm, 0);
36− struct vm_area_struct *vma, *vma_new, *vma_next;
36+ struct vm_area_struct *vma, *vma_prev, *vma_new, *vma_next, *vma_orig;
37+
38+ /* Move forwards, adjacent to old self - self-merge. */
39+
40+ vma = alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags);
41+ vma_set_anonymous(vma);
42+ vma_orig = vma;
43+ vma_new = copy_vma(&vma, 0x2000, 0x1000, 1, 1, &need_locks);
44+ ASSERT_EQ(vma_new, vma_orig);
45+ ASSERT_EQ(vma, vma_orig);
46+ ASSERT_EQ(vma_new->vm_start, 0x1000);
47+ ASSERT_EQ(vma_new->vm_end, 0x3000);
48+
49+ cleanup_mm(&mm, &vmi);
50+
51+ /* Move backwards, adjacent to old self - self-merge. */
52+
53+ vma = alloc_and_link_vma(&mm, 0x2000, 0x3000, 2, vma_flags);
54+ vma_set_anonymous(vma);
55+ vma_orig = vma;
56+ vma_new = copy_vma(&vma, 0x1000, 0x1000, 2, 2, &need_locks);
57+ ASSERT_EQ(vma_new, vma_orig);
58+ ASSERT_EQ(vma, vma_orig);
59+ ASSERT_EQ(vma_new->vm_start, 0x1000);
60+ ASSERT_EQ(vma_new->vm_end, 0x3000);
61+
62+ cleanup_mm(&mm, &vmi);
63+
64+ /*
65+ * Move backwards between prior VMA and old self - self-merge and vma
66+ * updated to a new VMA.
67+ */
68+
69+ vma_prev = alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags);
70+ vma_set_anonymous(vma_prev);
71+ vma = alloc_and_link_vma(&mm, 0x3000, 0x4000, 3, vma_flags);
72+ vma_set_anonymous(vma);
73+ vma_orig = vma;
74+ vma_new = copy_vma(&vma, 0x2000, 0x1000, 3, 3, &need_locks);
75+ ASSERT_NE(vma_new, vma_orig);
76+ ASSERT_EQ(vma_new, vma);
77+ ASSERT_EQ(vma_new->vm_start, 0x1000);
78+ ASSERT_EQ(vma_new->vm_end, 0x4000);
79+
80+ cleanup_mm(&mm, &vmi);
3781
3882 /* Move backwards and do not merge. */
3983
4084 vma = alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, vma_flags);
41− vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks);
85+ vma_new = copy_vma(&vma, 0, 0x2000, 0, 3, &need_locks);
4286 ASSERT_NE(vma_new, vma);
4387 ASSERT_EQ(vma_new->vm_start, 0);
4488 ASSERT_EQ(vma_new->vm_end, 0x2000);
@@ -51,7 +95,7 @@ static bool test_copy_vma(void) · 6 unmodified lines
5195
5296 vma = alloc_and_link_vma(&mm, 0, 0x2000, 0, vma_flags);
5397 vma_next = alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, vma_flags);
54− vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks);
98+ vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, 4, &need_locks);
5599 vma_assert_attached(vma_new);
56100
57101 ASSERT_EQ(vma_new, vma_next);
tools/testing/vma/vma_internal.h+1−0
@@ -53,6 +53,7 @@ typedef __bitwise unsigned int vm_fault_t; · 52 unmodified lines
5353
5454 #define VM_WARN_ON(_expr) (WARN_ON(_expr))
5555 #define VM_WARN_ON_ONCE(_expr) (WARN_ON_ONCE(_expr))
56+#define VM_WARN_ON_ONCE_VMA(_expr, _vma) (WARN_ON_ONCE(_expr))
5657 #define VM_WARN_ON_VMG(_expr, _vmg) (WARN_ON(_expr))
5758 #define VM_BUG_ON(_expr) (BUG_ON(_expr))
5859 #define VM_BUG_ON_VMA(_expr, _vma) (BUG_ON(_expr))