History

Merge tag 'mm-stable-2023-11-01-14-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton: "Many singleton patches against the MM code. The patch series which are included in this merge do the following: - Kemeng Shi has contributed some compation maintenance work in the series 'Fixes and cleanups to compaction' - Joel Fernandes has a patchset ('Optimize mremap during mutual alignment within PMD') which fixes an obscure issue with mremap()'s pagetable handling during a subsequent exec(), based upon an implementation which Linus suggested - More DAMON/DAMOS maintenance and feature work from SeongJae Park i the following patch series: mm/damon: misc fixups for documents, comments and its tracepoint mm/damon: add a tracepoint for damos apply target regions mm/damon: provide pseudo-moving sum based access rate mm/damon: implement DAMOS apply intervals mm/damon/core-test: Fix memory leaks in core-test mm/damon/sysfs-schemes: Do DAMOS tried regions update for only one apply interval - In the series 'Do not try to access unaccepted memory' Adrian Hunter provides some fixups for the recently-added 'unaccepted memory' feature. To increase the feature's checking coverage. 'Plug a few gaps where RAM is exposed without checking if it is unaccepted memory' - In the series 'cleanups for lockless slab shrink' Qi Zheng has done some maintenance work which is preparation for the lockless slab shrinking code - Qi Zheng has redone the earlier (and reverted) attempt to make slab shrinking lockless in the series 'use refcount+RCU method to implement lockless slab shrink' - David Hildenbrand contributes some maintenance work for the rmap code in the series 'Anon rmap cleanups' - Kefeng Wang does more folio conversions and some maintenance work in the migration code. Series 'mm: migrate: more folio conversion and unification' - Matthew Wilcox has fixed an issue in the buffer_head code which was causing long stalls under some heavy memory/IO loads. Some cleanups were added on the way. Series 'Add and use bdev_getblk()' - In the series 'Use nth_page() in place of direct struct page manipulation' Zi Yan has fixed a potential issue with the direct manipulation of hugetlb page frames - In the series 'mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO' has improved our handling of gigantic pages in the hugetlb vmmemmep optimizaton code. This provides significant boot time improvements when significant amounts of gigantic pages are in use - Matthew Wilcox has sent the series 'Small hugetlb cleanups' - code rationalization and folio conversions in the hugetlb code - Yin Fengwei has improved mlock()'s handling of large folios in the series 'support large folio for mlock' - In the series 'Expose swapcache stat for memcg v1' Liu Shixin has added statistics for memcg v1 users which are available (and useful) under memcg v2 - Florent Revest has enhanced the MDWE (Memory-Deny-Write-Executable) prctl so that userspace may direct the kernel to not automatically propagate the denial to child processes. The series is named 'MDWE without inheritance' - Kefeng Wang has provided the series 'mm: convert numa balancing functions to use a folio' which does what it says - In the series 'mm/ksm: add fork-exec support for prctl' Stefan Roesch makes is possible for a process to propagate KSM treatment across exec() - Huang Ying has enhanced memory tiering's calculation of memory distances. This is used to permit the dax/kmem driver to use 'high bandwidth memory' in addition to Optane Data Center Persistent Memory Modules (DCPMM). The series is named 'memory tiering: calculate abstract distance based on ACPI HMAT' - In the series 'Smart scanning mode for KSM' Stefan Roesch has optimized KSM by teaching it to retain and use some historical information from previous scans - Yosry Ahmed has fixed some inconsistencies in memcg statistics in the series 'mm: memcg: fix tracking of pending stats updates values' - In the series 'Implement IOCTL to get and optionally clear info about PTEs' Peter Xu has added an ioctl to /proc/<pid>/pagemap which permits us to atomically read-then-clear page softdirty state. This is mainly used by CRIU - Hugh Dickins contributed the series 'shmem,tmpfs: general maintenance', a bunch of relatively minor maintenance tweaks to this code - Matthew Wilcox has increased the use of the VMA lock over file-backed page faults in the series 'Handle more faults under the VMA lock'. Some rationalizations of the fault path became possible as a result - In the series 'mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()' David Hildenbrand has implemented some cleanups and folio conversions - In the series 'various improvements to the GUP interface' Lorenzo Stoakes has simplified and improved the GUP interface with an eye to providing groundwork for future improvements - Andrey Konovalov has sent along the series 'kasan: assorted fixes and improvements' which does those things - Some page allocator maintenance work from Kemeng Shi in the series 'Two minor cleanups to break_down_buddy_pages' - In thes series 'New selftest for mm' Breno Leitao has developed another MM self test which tickles a race we had between madvise() and page faults - In the series 'Add folio_end_read' Matthew Wilcox provides cleanups and an optimization to the core pagecache code - Nhat Pham has added memcg accounting for hugetlb memory in the series 'hugetlb memcg accounting' - Cleanups and rationalizations to the pagemap code from Lorenzo Stoakes, in the series 'Abstract vma_merge() and split_vma()' - Audra Mitchell has fixed issues in the procfs page_owner code's new timestamping feature which was causing some misbehaviours. In the series 'Fix page_owner's use of free timestamps' - Lorenzo Stoakes has fixed the handling of new mappings of sealed files in the series 'permit write-sealed memfd read-only shared mappings' - Mike Kravetz has optimized the hugetlb vmemmap optimization in the series 'Batch hugetlb vmemmap modification operations' - Some buffer_head folio conversions and cleanups from Matthew Wilcox in the series 'Finish the create_empty_buffers() transition' - As a page allocator performance optimization Huang Ying has added automatic tuning to the allocator's per-cpu-pages feature, in the series 'mm: PCP high auto-tuning' - Roman Gushchin has contributed the patchset 'mm: improve performance of accounted kernel memory allocations' which improves their performance by ~30% as measured by a micro-benchmark - folio conversions from Kefeng Wang in the series 'mm: convert page cpupid functions to folios' - Some kmemleak fixups in Liu Shixin's series 'Some bugfix about kmemleak' - Qi Zheng has improved our handling of memoryless nodes by keeping them off the allocation fallback list. This is done in the series 'handle memoryless nodes more appropriately' - khugepaged conversions from Vishal Moola in the series 'Some khugepaged folio conversions'" [ bcachefs conflicts with the dynamically allocated shrinkers have been resolved as per Stephen Rothwell in https://lore.kernel.org/all/20230913093553.4290421e@canb.auug.org.au/ with help from Qi Zheng. The clone3 test filtering conflict was half-arsed by yours truly ] * tag 'mm-stable-2023-11-01-14-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (406 commits) mm/damon/sysfs: update monitoring target regions for online input commit mm/damon/sysfs: remove requested targets when online-commit inputs selftests: add a sanity check for zswap Documentation: maple_tree: fix word spelling error mm/vmalloc: fix the unchecked dereference warning in vread_iter() zswap: export compression failure stats Documentation: ubsan: drop "the" from article title mempolicy: migration attempt to match interleave nodes mempolicy: mmap_lock is not needed while migrating folios mempolicy: alloc_pages_mpol() for NUMA policy without vma mm: add page_rmappable_folio() wrapper mempolicy: remove confusing MPOL_MF_LAZY dead code mempolicy: mpol_shared_policy_init() without pseudo-vma mempolicy trivia: use pgoff_t in shared mempolicy tree mempolicy trivia: slightly more consistent naming mempolicy trivia: delete those ancient pr_debug()s mempolicy: fix migrate_pages(2) syscall return nr_failed kernfs: drop shared NUMA mempolicy hooks hugetlbfs: drop shared NUMA mempolicy pretence mm/damon/sysfs-test: add a unit test for damon_sysfs_set_targets() ...

Linus Torvalds committed 3 years ago

Commit ecae0bd2 parents bc3012f 9732336281 files · +11683 −5277Browse files

Unified · Split

281 files
  1. Documentation/ABI/testing/sysfs-kernel-mm-damon +7 −0
  2. Documentation/admin-guide/cgroup-v1/memory.rst +1 −0
  3. Documentation/admin-guide/cgroup-v2.rst +38 −0
  4. Documentation/admin-guide/mm/damon/usage.rst +81 −43
  5. Documentation/admin-guide/mm/ksm.rst +11 −0
  6. Documentation/admin-guide/mm/pagemap.rst +89 −0
  7. Documentation/admin-guide/mm/userfaultfd.rst +35 −0
  8. Documentation/core-api/maple_tree.rst +1 −1
  9. Documentation/dev-tools/kasan.rst +5 −2
  10. Documentation/dev-tools/kcsan.rst +2 −2
  11. Documentation/dev-tools/kmsan.rst +3 −3
  12. Documentation/dev-tools/ubsan.rst +4 −2
  13. Documentation/mm/damon/design.rst +22 −4
  14. Documentation/trace/postprocess/trace-vmscan-postprocess.pl +18 −24
  15. MAINTAINERS +2 −0
  16. arch/alpha/include/asm/bitops.h +20 −0
  17. arch/arm64/include/asm/tlb.h +4 −1
  18. arch/arm64/kernel/mte.c +2 −2
  19. arch/arm64/mm/kasan_init.c +5 −1
  20. arch/loongarch/include/asm/pgalloc.h +1 −0
  21. arch/m68k/include/asm/bitops.h +21 −0
  22. arch/mips/include/asm/bitops.h +24 −1
  23. arch/mips/include/asm/pgalloc.h +1 −0
  24. arch/mips/lib/bitops.c +14 −0
  25. arch/mips/mm/cache.c +1 −1
  26. arch/powerpc/include/asm/bitops.h +5 −16
  27. arch/riscv/include/asm/bitops.h +12 −0
  28. arch/s390/include/asm/bitops.h +10 −0
  29. arch/x86/include/asm/bitops.h +5 −6
  30. arch/x86/kvm/mmu/mmu.c +10 −8
  31. arch/x86/mm/pgtable.c +3 −0
  32. drivers/acpi/acpi_pad.c +1 −1
  33. drivers/acpi/numa/hmat.c +93 −53
  34. drivers/android/binder_alloc.c +18 −12
  35. drivers/base/cacheinfo.c +50 −1
  36. drivers/dax/kmem.c +49 −13
  37. drivers/firmware/efi/unaccepted_memory.c +20 −0
  38. drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +16 −13
  39. drivers/gpu/drm/i915/i915_drv.h +1 −1
  40. drivers/gpu/drm/msm/msm_drv.c +3 −1
  41. drivers/gpu/drm/msm/msm_drv.h +2 −2
  42. drivers/gpu/drm/msm/msm_gem_shrinker.c +20 −13
  43. drivers/gpu/drm/panfrost/panfrost_device.h +1 −1
  44. drivers/gpu/drm/panfrost/panfrost_drv.c +5 −1
  45. drivers/gpu/drm/panfrost/panfrost_gem.h +1 −1
  46. drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +17 −12
  47. drivers/gpu/drm/ttm/ttm_pool.c +31 −9
  48. drivers/md/bcache/bcache.h +1 −1
  49. drivers/md/bcache/btree.c +16 −11
  50. drivers/md/bcache/sysfs.c +2 −1
  51. drivers/md/dm-bufio.c +17 −11
  52. drivers/md/dm-cache-metadata.c +1 −1
  53. drivers/md/dm-zoned-metadata.c +16 −12
  54. drivers/md/raid5.c +15 −11
  55. drivers/md/raid5.h +1 −1
  56. drivers/misc/vmw_balloon.c +12 −26
  57. drivers/virtio/virtio_balloon.c +14 −10
  58. drivers/xen/xenbus/xenbus_probe_backend.c +10 −7
  59. fs/bcachefs/btree_cache.c +12 −10
  60. fs/bcachefs/btree_key_cache.c +12 −9
  61. fs/bcachefs/btree_types.h +2 −2
  62. fs/bcachefs/fs.c +1 −1
  63. fs/bcachefs/sysfs.c +1 −1
  64. fs/btrfs/super.c +1 −1
  65. fs/buffer.c +60 −65
  66. fs/dax.c +12 −12
  67. fs/erofs/utils.c +12 −7
  68. fs/exec.c +1 −3
  69. fs/ext4/ext4.h +1 −1
  70. fs/ext4/extents_status.c +13 −10
  71. fs/ext4/inode.c +5 −9
  72. fs/ext4/move_extent.c +5 −6
  73. fs/ext4/readpage.c +3 −11
  74. fs/ext4/super.c +10 −3
  75. fs/f2fs/super.c +23 −8
  76. fs/gfs2/aops.c +1 −1
  77. fs/gfs2/bmap.c +23 −25
  78. fs/gfs2/glock.c +10 −9
  79. fs/gfs2/main.c +3 −3
  80. fs/gfs2/meta_io.c +25 −36
  81. fs/gfs2/quota.c +38 −24
  82. fs/gfs2/quota.h +2 −1
  83. fs/hugetlbfs/inode.c +23 −61
  84. fs/iomap/buffered-io.c +35 −22
  85. fs/jbd2/journal.c +18 −11
  86. fs/kernfs/file.c +0 −49
  87. fs/kernfs/mount.c +1 −1
  88. fs/mbcache.c +12 −10
  89. fs/mpage.c +1 −2
  90. fs/nfs/nfs42xattr.c +44 −43
  91. fs/nfs/super.c +13 −8
  92. fs/nfsd/filecache.c +13 −10
  93. fs/nfsd/netns.h +2 −2
  94. fs/nfsd/nfs4state.c +11 −8
  95. fs/nfsd/nfscache.c +16 −15
  96. fs/nilfs2/mdt.c +34 −32
  97. fs/nilfs2/page.c +39 −37
  98. fs/nilfs2/page.h +0 −11
  99. fs/nilfs2/segment.c +3 −4
  100. fs/ntfs/aops.c +119 −136
  101. fs/ntfs/file.c +41 −48
  102. fs/ntfs3/file.c +16 −15
  103. fs/ocfs2/aops.c +10 −9
  104. fs/proc/kcore.c +2 −1
  105. fs/proc/root.c +1 −1
  106. fs/proc/task_mmu.c +731 −2
  107. fs/quota/dquot.c +9 −8
  108. fs/reiserfs/inode.c +40 −40
  109. fs/super.c +19 −16
  110. fs/ubifs/super.c +11 −10
  111. fs/ufs/balloc.c +9 −11
  112. fs/ufs/inode.c +12 −13
  113. fs/ufs/util.c +16 −18
  114. fs/ufs/util.h +4 −6
  115. fs/userfaultfd.c +40 −56
  116. fs/xfs/xfs_buf.c +13 −11
  117. fs/xfs/xfs_buf.h +1 −1
  118. fs/xfs/xfs_icache.c +15 −11
  119. fs/xfs/xfs_mount.c +2 −2
  120. fs/xfs/xfs_mount.h +1 −1
  121. fs/xfs/xfs_qm.c +14 −13
  122. fs/xfs/xfs_qm.h +1 −1
  123. include/asm-generic/bitops/instrumented-lock.h +13 −15
  124. include/asm-generic/bitops/lock.h +4 −16
  125. include/asm-generic/pgalloc.h +6 −1
  126. include/linux/bootmem_info.h +1 −1
  127. include/linux/buffer_head.h +54 −27
  128. include/linux/cacheinfo.h +1 −0
  129. include/linux/cgroup-defs.h +5 −0
  130. include/linux/damon.h +60 −14
  131. include/linux/dax.h +5 −5
  132. include/linux/fs.h +3 −3
  133. include/linux/gfp.h +11 −1
  134. include/linux/hugetlb.h +14 −11
  135. include/linux/hugetlb_cgroup.h +0 −11
  136. include/linux/jbd2.h +1 −1
  137. include/linux/memblock.h +9 −0
  138. include/linux/memcontrol.h +65 −28
  139. include/linux/memory-tiers.h +40 −1
  140. include/linux/mempolicy.h +31 −27
  141. include/linux/migrate.h +2 −2
  142. include/linux/mm.h +130 −46
  143. include/linux/mm_inline.h +1 −19
  144. include/linux/mm_types.h +45 −4
  145. include/linux/mmu_notifier.h +8 −1
  146. include/linux/mmzone.h +24 −5
  147. include/linux/page-flags.h +19 −0
  148. include/linux/pagemap.h +3 −30
  149. include/linux/percpu_counter.h +30 −0
  150. include/linux/rmap.h +2 −2
  151. include/linux/sched.h +4 −0
  152. include/linux/sched/coredump.h +16 −3
  153. include/linux/sched/mm.h +4 −0
  154. include/linux/sched/numa_balancing.h +3 −3
  155. include/linux/shmem_fs.h +10 −6
  156. include/linux/shrinker.h +55 −32
  157. include/linux/userfaultfd_k.h +27 −1
  158. include/linux/wait.h +3 −6
  159. include/trace/events/damon.h +42 −3
  160. include/trace/events/migrate.h +14 −10
  161. include/trace/events/vmscan.h +2 −6
  162. include/uapi/linux/fs.h +59 −0
  163. include/uapi/linux/mempolicy.h +1 −1
  164. include/uapi/linux/prctl.h +2 −1
  165. include/uapi/linux/userfaultfd.h +8 −1
  166. ipc/shm.c +8 −13
  167. kernel/cgroup/cgroup.c +14 −1
  168. kernel/events/uprobes.c +2 −2
  169. kernel/exit.c +0 −4
  170. kernel/fork.c +2 −6
  171. kernel/kcsan/kcsan_test.c +3 −6
  172. kernel/kcsan/selftest.c +3 −6
  173. kernel/kthread.c +0 −1
  174. kernel/ptrace.c +1 −1
  175. kernel/rcu/tree.c +12 −9
  176. kernel/rcu/tree_nocb.h +10 −9
  177. kernel/sched/fair.c +6 −6
  178. kernel/sched/wait.c +10 −50
  179. kernel/sys.c +26 −6
  180. lib/percpu_counter.c +79 −0
  181. mm/Kconfig +12 −0
  182. mm/Makefile +2 −2
  183. mm/bootmem_info.c +1 −1
  184. mm/cma.c +1 −1
  185. mm/compaction.c +52 −39
  186. mm/damon/Kconfig +12 −0
  187. mm/damon/core-test.h +28 −1
  188. mm/damon/core.c +226 −69
  189. mm/damon/dbgfs.c +2 −1
  190. mm/damon/lru_sort.c +3 −3
  191. mm/damon/ops-common.c +2 −3
  192. mm/damon/paddr.c +5 −6
  193. mm/damon/reclaim.c +2 −0
  194. mm/damon/sysfs-common.h +2 −0
  195. mm/damon/sysfs-schemes.c +128 −5
  196. mm/damon/sysfs-test.h +86 −0
  197. mm/damon/sysfs.c +83 −40
  198. mm/damon/vaddr.c +13 −9
  199. mm/debug_vm_pgtable.c +2 −2
  200. mm/filemap.c +82 −133
  201. mm/gup.c +13 −3
  202. mm/huge_memory.c +79 −55
  203. mm/hugetlb.c +458 −167
  204. mm/hugetlb_cgroup.c +2 −18
  205. mm/hugetlb_vmemmap.c +252 −49
  206. mm/hugetlb_vmemmap.h +24 −7
  207. mm/internal.h +136 −11
  208. mm/kasan/kasan_test.c +9 −10
  209. mm/kasan/kasan_test_module.c +1 −1
  210. mm/kasan/quarantine.c +3 −1
  211. mm/kasan/report.c +2 −2
  212. mm/kasan/report_generic.c +3 −3
  213. mm/kasan/shadow.c +1 −1
  214. mm/khugepaged.c +68 −69
  215. mm/kmemleak.c +98 −46
  216. mm/kmsan/core.c +31 −96
  217. mm/kmsan/kmsan_test.c +71 −38
  218. mm/kmsan/shadow.c +7 −2
  219. mm/ksm.c +116 −0
  220. mm/madvise.c +6 −27
  221. mm/memblock.c +36 −13
  222. mm/memcontrol.c +341 −60
  223. mm/memfd.c +0 −6
  224. mm/memory-failure.c +16 −13
  225. mm/memory-tiers.c +168 −7
  226. mm/memory.c +179 −123
  227. mm/memory_hotplug.c +6 −2
  228. mm/mempolicy.c +455 −571
  229. mm/migrate.c +117 −108
  230. mm/mlock.c +72 −23
  231. mm/mm_init.c +1 −3
  232. mm/mmap.c +116 −33
  233. mm/mmzone.c +3 −3
  234. mm/mprotect.c +24 −34
  235. mm/mremap.c +86 −21
  236. mm/nommu.c +4 −4
  237. mm/oom_kill.c +8 −8
  238. mm/page-writeback.c +17 −20
  239. mm/page_alloc.c +267 −76
  240. mm/page_io.c +4 −4
  241. mm/page_owner.c +2 −2
  242. mm/percpu.c +3 −5
  243. mm/rmap.c +101 −63
  244. mm/shmem.c +294 −294
  245. mm/show_mem.c +4 −7
  246. mm/shrinker.c +809 −0
  247. mm/shrinker_debug.c +10 −25
  248. mm/slab.h +8 −7
  249. mm/swap.h +5 −4
  250. mm/swap_state.c +57 −35
  251. mm/util.c +1 −0
  252. mm/vmalloc.c +1 −1
  253. mm/vmscan.c +22 −712
  254. mm/vmstat.c +17 −13
  255. mm/workingset.c +16 −13
  256. mm/zsmalloc.c +16 −13
  257. mm/zswap.c +12 −3
  258. net/sunrpc/auth.c +12 −8
  259. tools/include/uapi/linux/fs.h +59 −0
  260. tools/include/uapi/linux/prctl.h +2 −1
  261. tools/mm/page_owner_sort.c +103 −114
  262. tools/testing/radix-tree/linux.c +2 −2
  263. tools/testing/selftests/cgroup/.gitignore +1 −0
  264. tools/testing/selftests/cgroup/Makefile +2 −0
  265. tools/testing/selftests/cgroup/test_hugetlb_memcg.c +234 −0
  266. tools/testing/selftests/cgroup/test_zswap.c +48 −0
  267. tools/testing/selftests/clone3/clone3.c +13 −0
  268. tools/testing/selftests/damon/sysfs.sh +1 −0
  269. tools/testing/selftests/mm/.gitignore +2 −0
  270. tools/testing/selftests/mm/Makefile +3 −1
  271. tools/testing/selftests/mm/config +1 −0
  272. tools/testing/selftests/mm/gup_longterm.c +2 −1
  273. tools/testing/selftests/mm/hugetlb-madvise.c +0 −19
  274. tools/testing/selftests/mm/hugetlb_fault_after_madv.c +73 −0
  275. tools/testing/selftests/mm/ksm_functional_tests.c +65 −1
  276. tools/testing/selftests/mm/mdwe_test.c +122 −15
  277. tools/testing/selftests/mm/mremap_test.c +257 −44
  278. tools/testing/selftests/mm/pagemap_ioctl.c +1660 −0
  279. tools/testing/selftests/mm/run_vmtests.sh +8 −0
  280. tools/testing/selftests/mm/vm_util.c +19 −0
  281. tools/testing/selftests/mm/vm_util.h +1 −0
Documentation/ABI/testing/sysfs-kernel-mm-damon+7−0
@@ -151,6 +151,13 @@ Contact: SeongJae Park <sj@kernel.org> · 150 unmodified lines
151151 Description: Writing to and reading from this file sets and gets the action
152152 of the scheme.
153153
154+What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us
155+Date: Sep 2023
156+Contact: SeongJae Park <sj@kernel.org>
157+Description: Writing a value to this file sets the action apply interval of
158+ the scheme in microseconds. Reading this file returns the
159+ value.
160+
154161 What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/min
155162 Date: Mar 2022
156163 Contact: SeongJae Park <sj@kernel.org>
Documentation/admin-guide/cgroup-v1/memory.rst+1−0
@@ -551,6 +551,7 @@ memory.stat file includes following statistics: · 550 unmodified lines
551551 event happens each time a page is unaccounted from the
552552 cgroup.
553553 swap # of bytes of swap usage
554+ swapcached # of bytes of swap cached in memory
554555 dirty # of bytes that are waiting to get written back to the disk.
555556 writeback # of bytes of file/anon cache that are queued for syncing to
556557 disk.
Documentation/admin-guide/cgroup-v2.rst+38−0
@@ -210,6 +210,35 @@ cgroup v2 currently supports the following mount options. · 209 unmodified lines
210210 relying on the original semantics (e.g. specifying bogusly
211211 high 'bypass' protection values at higher tree levels).
212212
213+ memory_hugetlb_accounting
214+ Count HugeTLB memory usage towards the cgroup's overall
215+ memory usage for the memory controller (for the purpose of
216+ statistics reporting and memory protetion). This is a new
217+ behavior that could regress existing setups, so it must be
218+ explicitly opted in with this mount option.
219+
220+ A few caveats to keep in mind:
221+
222+ * There is no HugeTLB pool management involved in the memory
223+ controller. The pre-allocated pool does not belong to anyone.
224+ Specifically, when a new HugeTLB folio is allocated to
225+ the pool, it is not accounted for from the perspective of the
226+ memory controller. It is only charged to a cgroup when it is
227+ actually used (for e.g at page fault time). Host memory
228+ overcommit management has to consider this when configuring
229+ hard limits. In general, HugeTLB pool management should be
230+ done via other mechanisms (such as the HugeTLB controller).
231+ * Failure to charge a HugeTLB folio to the memory controller
232+ results in SIGBUS. This could happen even if the HugeTLB pool
233+ still has pages available (but the cgroup limit is hit and
234+ reclaim attempt fails).
235+ * Charging HugeTLB memory towards the memory controller affects
236+ memory protection and reclaim dynamics. Any userspace tuning
237+ (of low, min limits for e.g) needs to take this into account.
238+ * HugeTLB pages utilized while this option is not selected
239+ will not be tracked by the memory controller (even if cgroup
240+ v2 is remounted later on).
241+
213242
214243 Organizing Processes and Threads
215244 --------------------------------
@@ -1539,6 +1568,15 @@ PAGE_SIZE multiple when read back. · 1323 unmodified lines
15391568 collapsing an existing range of pages. This counter is not
15401569 present when CONFIG_TRANSPARENT_HUGEPAGE is not set.
15411570
1571+ thp_swpout (npn)
1572+ Number of transparent hugepages which are swapout in one piece
1573+ without splitting.
1574+
1575+ thp_swpout_fallback (npn)
1576+ Number of transparent hugepages which were split before swapout.
1577+ Usually because failed to allocate some continuous swap space
1578+ for the huge page.
1579+
15421580 memory.numa_stat
15431581 A read-only nested-keyed file which exists on non-root cgroups.
15441582
Documentation/admin-guide/mm/damon/usage.rst+81−43
@@ -20,18 +20,18 @@ DAMON provides below interfaces for different users. · 19 unmodified lines
2020 you can write and use your personalized DAMON sysfs wrapper programs that
2121 reads/writes the sysfs files instead of you. The `DAMON user space tool
2222 <https://github.com/awslabs/damo>`_ is one example of such programs.
23−- *debugfs interface. (DEPRECATED!)*
24− :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
25− <sysfs_interface>`. This is deprecated, so users should move to the
26− :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
27− move, please report your usecase to damon@lists.linux.dev and
28− linux-mm@kvack.org.
2923 - *Kernel Space Programming Interface.*
3024 :doc:`This </mm/damon/api>` is for kernel space programmers. Using this,
3125 users can utilize every feature of DAMON most flexibly and efficiently by
3226 writing kernel space DAMON application programs for you. You can even extend
3327 DAMON for various address spaces. For detail, please refer to the interface
3428 :doc:`document </mm/damon/api>`.
29+- *debugfs interface. (DEPRECATED!)*
30+ :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
31+ <sysfs_interface>`. This is deprecated, so users should move to the
32+ :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
33+ move, please report your usecase to damon@lists.linux.dev and
34+ linux-mm@kvack.org.
3535
3636 .. _sysfs_interface:
3737
@@ -76,7 +76,7 @@ comma (","). :: · 38 unmodified lines
7676 │ │ │ │ │ │ │ │ ...
7777 │ │ │ │ │ │ ...
7878 │ │ │ │ │ schemes/nr_schemes
79− │ │ │ │ │ │ 0/action
79+ │ │ │ │ │ │ 0/action,apply_interval_us
8080 │ │ │ │ │ │ │ access_pattern/
8181 │ │ │ │ │ │ │ │ sz/min,max
8282 │ │ │ │ │ │ │ │ nr_accesses/min,max
@@ -105,14 +105,12 @@ having the root permission could use this directory. · 22 unmodified lines
105105 kdamonds/
106106 ---------
107107
108−The monitoring-related information including request specifications and results
109−are called DAMON context. DAMON executes each context with a kernel thread
110−called kdamond, and multiple kdamonds could run in parallel.
111−
112108 Under the ``admin`` directory, one directory, ``kdamonds``, which has files for
113−controlling the kdamonds exist. In the beginning, this directory has only one
114−file, ``nr_kdamonds``. Writing a number (``N``) to the file creates the number
115−of child directories named ``0`` to ``N-1``. Each directory represents each
109+controlling the kdamonds (refer to
110+:ref:`design <damon_design_execution_model_and_data_structures>` for more
111+details) exists. In the beginning, this directory has only one file,
112+``nr_kdamonds``. Writing a number (``N``) to the file creates the number of
113+child directories named ``0`` to ``N-1``. Each directory represents each
116114 kdamond.
117115
118116 kdamonds/<N>/
@@ -150,9 +148,10 @@ kdamonds/<N>/contexts/ · 31 unmodified lines
150148
151149 In the beginning, this directory has only one file, ``nr_contexts``. Writing a
152150 number (``N``) to the file creates the number of child directories named as
153−``0`` to ``N-1``. Each directory represents each monitoring context. At the
154−moment, only one context per kdamond is supported, so only ``0`` or ``1`` can
155−be written to the file.
151+``0`` to ``N-1``. Each directory represents each monitoring context (refer to
152+:ref:`design <damon_design_execution_model_and_data_structures>` for more
153+details). At the moment, only one context per kdamond is supported, so only
154+``0`` or ``1`` can be written to the file.
156155
157156 .. _sysfs_contexts:
158157
@@ -270,8 +269,8 @@ schemes/<N>/ · 111 unmodified lines
270269 ------------
271270
272271 In each scheme directory, five directories (``access_pattern``, ``quotas``,
273−``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and one file
274−(``action``) exist.
272+``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files
273+(``action`` and ``apply_interval``) exist.
275274
276275 The ``action`` file is for setting and getting the scheme's :ref:`action
277276 <damon_design_damos_action>`. The keywords that can be written to and read
@@ -297,6 +296,9 @@ Note that support of each action depends on the running DAMON operations set · 19 unmodified lines
297296 - ``stat``: Do nothing but count the statistics.
298297 Supported by all operations sets.
299298
299+The ``apply_interval_us`` file is for setting and getting the scheme's
300+:ref:`apply_interval <damon_design_damos>` in microseconds.
301+
300302 schemes/<N>/access_pattern/
301303 ---------------------------
302304
@@ -392,7 +394,7 @@ pages of all memory cgroups except ``/having_care_already``.:: · 89 unmodified lines
392394 echo N > 1/matching
393395
394396 Note that ``anon`` and ``memcg`` filters are currently supported only when
395−``paddr`` `implementation <sysfs_contexts>` is being used.
397+``paddr`` :ref:`implementation <sysfs_contexts>` is being used.
396398
397399 Also, memory regions that are filtered out by ``addr`` or ``target`` filters
398400 are not counted as the scheme has tried to those, while regions that filtered
@@ -430,9 +432,9 @@ that reading it returns the total size of the scheme tried regions, and creates · 31 unmodified lines
430432 directories named integer starting from ``0`` under this directory. Each
431433 directory contains files exposing detailed information about each of the memory
432434 region that the corresponding scheme's ``action`` has tried to be applied under
433−this directory, during next :ref:`aggregation interval
434−<sysfs_monitoring_attrs>`. The information includes address range,
435−``nr_accesses``, and ``age`` of the region.
435+this directory, during next :ref:`apply interval <damon_design_damos>` of the
436+corresponding scheme. The information includes address range, ``nr_accesses``,
437+and ``age`` of the region.
436438
437439 Writing ``update_schemes_tried_bytes`` to the relevant ``kdamonds/<N>/state``
438440 file will only update the ``total_bytes`` file, and will not create the
@@ -495,6 +497,62 @@ Please note that it's highly recommended to use user space tools like `damo · 56 unmodified lines
495497 <https://github.com/awslabs/damo>`_ rather than manually reading and writing
496498 the files as above. Above is only for an example.
497499
500+.. _tracepoint:
501+
502+Tracepoints for Monitoring Results
503+==================================
504+
505+Users can get the monitoring results via the :ref:`tried_regions
506+<sysfs_schemes_tried_regions>`. The interface is useful for getting a
507+snapshot, but it could be inefficient for fully recording all the monitoring
508+results. For the purpose, two trace points, namely ``damon:damon_aggregated``
509+and ``damon:damos_before_apply``, are provided. ``damon:damon_aggregated``
510+provides the whole monitoring results, while ``damon:damos_before_apply``
511+provides the monitoring results for regions that each DAMON-based Operation
512+Scheme (:ref:`DAMOS <damon_design_damos>`) is gonna be applied. Hence,
513+``damon:damos_before_apply`` is more useful for recording internal behavior of
514+DAMOS, or DAMOS target access
515+:ref:`pattern <damon_design_damos_access_pattern>` based query-like efficient
516+monitoring results recording.
517+
518+While the monitoring is turned on, you could record the tracepoint events and
519+show results using tracepoint supporting tools like ``perf``. For example::
520+
521+ # echo on > monitor_on
522+ # perf record -e damon:damon_aggregated &
523+ # sleep 5
524+ # kill 9 $(pidof perf)
525+ # echo off > monitor_on
526+ # perf script
527+ kdamond.0 46568 [027] 79357.842179: damon:damon_aggregated: target_id=0 nr_regions=11 122509119488-135708762112: 0 864
528+ [...]
529+
530+Each line of the perf script output represents each monitoring region. The
531+first five fields are as usual other tracepoint outputs. The sixth field
532+(``target_id=X``) shows the ide of the monitoring target of the region. The
533+seventh field (``nr_regions=X``) shows the total number of monitoring regions
534+for the target. The eighth field (``X-Y:``) shows the start (``X``) and end
535+(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the
536+``nr_accesses`` of the region (refer to
537+:ref:`design <damon_design_region_based_sampling>` for more details of the
538+counter). Finally the tenth field (``X``) shows the ``age`` of the region
539+(refer to :ref:`design <damon_design_age_tracking>` for more details of the
540+counter).
541+
542+If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would
543+be somewhat like below::
544+
545+ kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136
546+ [...]
547+
548+Each line of the output represents each monitoring region that each DAMON-based
549+Operation Scheme was about to be applied at the traced time. The first five
550+fields are as usual. It shows the index of the DAMON context (``ctx_idx=X``)
551+of the scheme in the list of the contexts of the context's kdamond, the index
552+of the scheme (``scheme_idx=X``) in the list of the schemes of the context, in
553+addition to the output of ``damon_aggregated`` tracepoint.
554+
555+
498556 .. _debugfs_interface:
499557
500558 debugfs Interface (DEPRECATED!)
@@ -790,23 +848,3 @@ directory by putting the name of the context to the ``rm_contexts`` file. :: · 289 unmodified lines
790848
791849 Note that ``mk_contexts``, ``rm_contexts``, and ``monitor_on`` files are in the
792850 root directory only.
793−
794−
795−.. _tracepoint:
796−
797−Tracepoint for Monitoring Results
798−=================================
799−
800−Users can get the monitoring results via the :ref:`tried_regions
801−<sysfs_schemes_tried_regions>` or a tracepoint, ``damon:damon_aggregated``.
802−While the tried regions directory is useful for getting a snapshot, the
803−tracepoint is useful for getting a full record of the results. While the
804−monitoring is turned on, you could record the tracepoint events and show
805−results using tracepoint supporting tools like ``perf``. For example::
806−
807− # echo on > monitor_on
808− # perf record -e damon:damon_aggregated &
809− # sleep 5
810− # kill 9 $(pidof perf)
811− # echo off > monitor_on
812− # perf script
Documentation/admin-guide/mm/ksm.rst+11−0
@@ -155,6 +155,15 @@ stable_node_chains_prune_millisecs · 154 unmodified lines
155155 scan. It's a noop if not a single KSM page hit the
156156 ``max_page_sharing`` yet.
157157
158+smart_scan
159+ Historically KSM checked every candidate page for each scan. It did
160+ not take into account historic information. When smart scan is
161+ enabled, pages that have previously not been de-duplicated get
162+ skipped. How often these pages are skipped depends on how often
163+ de-duplication has already been tried and failed. By default this
164+ optimization is enabled. The ``pages_skipped`` metric shows how
165+ effective the setting is.
166+
158167 The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``:
159168
160169 general_profit
@@ -169,6 +178,8 @@ pages_unshared · 8 unmodified lines
169178 how many pages unique but repeatedly checked for merging
170179 pages_volatile
171180 how many pages changing too fast to be placed in a tree
181+pages_skipped
182+ how many pages did the "smart" page scanning algorithm skip
172183 full_scans
173184 how many times all mergeable areas have been scanned
174185 stable_node_chains
Documentation/admin-guide/mm/pagemap.rst+89−0
@@ -227,3 +227,92 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is · 226 unmodified lines
227227 always 12 at most architectures). Since Linux 3.11 their meaning changes
228228 after first clear of soft-dirty bits. Since Linux 4.2 they are used for
229229 flags unconditionally.
230+
231+Pagemap Scan IOCTL
232+==================
233+
234+The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally
235+clear the info about page table entries. The following operations are supported
236+in this IOCTL:
237+
238+- Scan the address range and get the memory ranges matching the provided criteria.
239+ This is performed when the output buffer is specified.
240+- Write-protect the pages. The ``PM_SCAN_WP_MATCHING`` is used to write-protect
241+ the pages of interest. The ``PM_SCAN_CHECK_WPASYNC`` aborts the operation if
242+ non-Async Write Protected pages are found. The ``PM_SCAN_WP_MATCHING`` can be
243+ used with or without ``PM_SCAN_CHECK_WPASYNC``.
244+- Both of those operations can be combined into one atomic operation where we can
245+ get and write protect the pages as well.
246+
247+Following flags about pages are currently supported:
248+
249+- ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled
250+- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected
251+- ``PAGE_IS_FILE`` - Page is file backed
252+- ``PAGE_IS_PRESENT`` - Page is present in the memory
253+- ``PAGE_IS_SWAPPED`` - Page is in swapped
254+- ``PAGE_IS_PFNZERO`` - Page has zero PFN
255+- ``PAGE_IS_HUGE`` - Page is THP or Hugetlb backed
256+
257+The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
258+
259+ 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size``
260+ field. This field will be helpful in recognizing the structure if extensions
261+ are done later.
262+ 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING``
263+ and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get
264+ operation is optionally performed depending upon if the output buffer is
265+ provided or not.
266+ 3. The range is specified through ``start`` and ``end``.
267+ 4. The walk can abort before visiting the complete range such as the user buffer
268+ can get full etc. The walk ending address is specified in``end_walk``.
269+ 5. The output buffer of ``struct page_region`` array and size is specified in
270+ ``vec`` and ``vec_len``.
271+ 6. The optional maximum requested pages are specified in the ``max_pages``.
272+ 7. The masks are specified in ``category_mask``, ``category_anyof_mask``,
273+ ``category_inverted`` and ``return_mask``.
274+
275+Find pages which have been written and WP them as well::
276+
277+ struct pm_scan_arg arg = {
278+ .size = sizeof(arg),
279+ .flags = PM_SCAN_CHECK_WPASYNC | PM_SCAN_CHECK_WPASYNC,
280+ ..
281+ .category_mask = PAGE_IS_WRITTEN,
282+ .return_mask = PAGE_IS_WRITTEN,
283+ };
284+
285+Find pages which have been written, are file backed, not swapped and either
286+present or huge::
287+
288+ struct pm_scan_arg arg = {
289+ .size = sizeof(arg),
290+ .flags = 0,
291+ ..
292+ .category_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED,
293+ .category_inverted = PAGE_IS_SWAPPED,
294+ .category_anyof_mask = PAGE_IS_PRESENT | PAGE_IS_HUGE,
295+ .return_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED |
296+ PAGE_IS_PRESENT | PAGE_IS_HUGE,
297+ };
298+
299+The ``PAGE_IS_WRITTEN`` flag can be considered as a better-performing alternative
300+of soft-dirty flag. It doesn't get affected by VMA merging of the kernel and hence
301+the user can find the true soft-dirty pages in case of normal pages. (There may
302+still be extra dirty pages reported for THP or Hugetlb pages.)
303+
304+"PAGE_IS_WRITTEN" category is used with uffd write protect-enabled ranges to
305+implement memory dirty tracking in userspace:
306+
307+ 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
308+ 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
309+ are set by ``UFFDIO_API`` IOCTL.
310+ 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
311+ through ``UFFDIO_REGISTER`` IOCTL.
312+ 4. Then any part of the registered memory or the whole memory region must
313+ be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_WP_MATCHING``
314+ or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
315+ same operation. The former is better in terms of performance.
316+ 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
317+ have been written to since they were last marked and/or optionally write protect
318+ the pages as well.
Documentation/admin-guide/mm/userfaultfd.rst+35−0
@@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls · 243 unmodified lines
244244 support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``
245245 respectively) to configure the mapping this way.
246246
247+If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set,
248+any vma registered with write-protection will work in async mode rather
249+than the default sync mode.
250+
251+In async mode, there will be no message generated when a write operation
252+happens, meanwhile the write-protection will be resolved automatically by
253+the kernel. It can be seen as a more accurate version of soft-dirty
254+tracking and it can be different in a few ways:
255+
256+ - The dirty result will not be affected by vma changes (e.g. vma
257+ merging) because the dirty is only tracked by the pte.
258+
259+ - It supports range operations by default, so one can enable tracking on
260+ any range of memory as long as page aligned.
261+
262+ - Dirty information will not get lost if the pte was zapped due to
263+ various reasons (e.g. during split of a shmem transparent huge page).
264+
265+ - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
266+ set; dirty when uffd-wp bit cleared), it has different semantics on
267+ some of the memory operations. For example: ``MADV_DONTNEED`` on
268+ anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
269+ dirtying of memory by dropping uffd-wp bit during the procedure.
270+
271+The user app can collect the "written/dirty" status by looking up the
272+uffd-wp bit for the pages being interested in /proc/pagemap.
273+
274+The page will not be under track of uffd-wp async mode until the page is
275+explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
276+flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault
277+that was tracked by async mode userfaultfd-wp is invalid.
278+
279+When userfaultfd-wp async mode is used alone, it can be applied to all
280+kinds of memory.
281+
247282 Memory Poisioning Emulation
248283 ---------------------------
249284
Documentation/core-api/maple_tree.rst+1−1
@@ -175,7 +175,7 @@ will return the previous entry which occurs before the entry at index. · 174 unmodified lines
175175 mas_find() will find the first entry which exists at or above index on
176176 the first call, and the next entry from every subsequent calls.
177177
178−mas_find_rev() will find the fist entry which exists at or below the last on
178+mas_find_rev() will find the first entry which exists at or below the last on
179179 the first call, and the previous entry from every subsequent calls.
180180
181181 If the user needs to yield the lock during an operation, then the maple state
Documentation/dev-tools/kasan.rst+5−2
@@ -1,5 +1,8 @@
1−The Kernel Address Sanitizer (KASAN)
2−====================================
1+.. SPDX-License-Identifier: GPL-2.0
2+.. Copyright (C) 2023, Google LLC.
3+
4+Kernel Address Sanitizer (KASAN)
5+================================
36
47 Overview
58 --------
Documentation/dev-tools/kcsan.rst+2−2
@@ -1,8 +1,8 @@
11 .. SPDX-License-Identifier: GPL-2.0
22 .. Copyright (C) 2019, Google LLC.
33
4−The Kernel Concurrency Sanitizer (KCSAN)
5−========================================
4+Kernel Concurrency Sanitizer (KCSAN)
5+====================================
66
77 The Kernel Concurrency Sanitizer (KCSAN) is a dynamic race detector, which
88 relies on compile-time instrumentation, and uses a watchpoint-based sampling
Documentation/dev-tools/kmsan.rst+3−3
@@ -1,9 +1,9 @@
11 .. SPDX-License-Identifier: GPL-2.0
22 .. Copyright (C) 2022, Google LLC.
33
4−===================================
5−The Kernel Memory Sanitizer (KMSAN)
6−===================================
4+===============================
5+Kernel Memory Sanitizer (KMSAN)
6+===============================
77
88 KMSAN is a dynamic error detector aimed at finding uses of uninitialized
99 values. It is based on compiler instrumentation, and is quite similar to the
Documentation/dev-tools/ubsan.rst+4−2
@@ -1,5 +1,7 @@
1−The Undefined Behavior Sanitizer - UBSAN
2−========================================
1+.. SPDX-License-Identifier: GPL-2.0
2+
3+Undefined Behavior Sanitizer - UBSAN
4+====================================
35
46 UBSAN is a runtime undefined behaviour checker.
57
Documentation/mm/damon/design.rst+22−4
@@ -154,6 +154,8 @@ The monitoring overhead of this mechanism will arbitrarily increase as the · 153 unmodified lines
154154 size of the target workload grows.
155155
156156
157+.. _damon_design_region_based_sampling:
158+
157159 Region Based Sampling
158160 ~~~~~~~~~~~~~~~~~~~~~
159161
@@ -163,9 +165,10 @@ assumption (pages in a region have the same access frequencies) is kept, only · 3 unmodified lines
163165 one page in the region is required to be checked. Thus, for each ``sampling
164166 interval``, DAMON randomly picks one page in each region, waits for one
165167 ``sampling interval``, checks whether the page is accessed meanwhile, and
166−increases the access frequency of the region if so. Therefore, the monitoring
167−overhead is controllable by setting the number of regions. DAMON allows users
168−to set the minimum and the maximum number of regions for the trade-off.
168+increases the access frequency counter of the region if so. The counter is
169+called ``nr_regions`` of the region. Therefore, the monitoring overhead is
170+controllable by setting the number of regions. DAMON allows users to set the
171+minimum and the maximum number of regions for the trade-off.
169172
170173 This scheme, however, cannot preserve the quality of the output if the
171174 assumption is not guaranteed.
@@ -190,6 +193,8 @@ In this way, DAMON provides its best-effort quality and minimal overhead while · 18 unmodified lines
190193 keeping the bounds users set for their trade-off.
191194
192195
196+.. _damon_design_age_tracking:
197+
193198 Age Tracking
194199 ~~~~~~~~~~~~
195200
@@ -254,7 +259,8 @@ works, DAMON provides a feature called Data Access Monitoring-based Operation · 58 unmodified lines
254259 Schemes (DAMOS). It lets users specify their desired schemes at a high
255260 level. For such specifications, DAMON starts monitoring, finds regions having
256261 the access pattern of interest, and applies the user-desired operation actions
257−to the regions as soon as found.
262+to the regions, for every user-specified time interval called
263+``apply_interval``.
258264
259265
260266 .. _damon_design_damos_action:
@@ -471,3 +477,15 @@ modules for proactive reclamation and LRU lists manipulation are provided. For · 210 unmodified lines
471477 more detail, please read the usage documents for those
472478 (:doc:`/admin-guide/mm/damon/reclaim` and
473479 :doc:`/admin-guide/mm/damon/lru_sort`).
480+
481+
482+.. _damon_design_execution_model_and_data_structures:
483+
484+Execution Model and Data Structures
485+===================================
486+
487+The monitoring-related information including the monitoring request
488+specification and DAMON-based operation schemes are stored in a data structure
489+called DAMON ``context``. DAMON executes each context with a kernel thread
490+called ``kdamond``. Multiple kdamonds could run in parallel, for different
491+types of monitoring.
Documentation/trace/postprocess/trace-vmscan-postprocess.pl+18−24
@@ -107,14 +107,14 @@ GetOptions( · 106 unmodified lines
107107 );
108108
109109 # Defaults for dynamically discovered regex's
110−my $regex_direct_begin_default = 'order=([0-9]*) may_writepage=([0-9]*) gfp_flags=([A-Z_|]*)';
110+my $regex_direct_begin_default = 'order=([0-9]*) gfp_flags=([A-Z_|]*)';
111111 my $regex_direct_end_default = 'nr_reclaimed=([0-9]*)';
112112 my $regex_kswapd_wake_default = 'nid=([0-9]*) order=([0-9]*)';
113113 my $regex_kswapd_sleep_default = 'nid=([0-9]*)';
114−my $regex_wakeup_kswapd_default = 'nid=([0-9]*) zid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)';
115−my $regex_lru_isolate_default = 'isolate_mode=([0-9]*) classzone_idx=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)';
114+my $regex_wakeup_kswapd_default = 'nid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)';
115+my $regex_lru_isolate_default = 'classzone=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)';
116116 my $regex_lru_shrink_inactive_default = 'nid=([0-9]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) nr_dirty=([0-9]*) nr_writeback=([0-9]*) nr_congested=([0-9]*) nr_immediate=([0-9]*) nr_activate_anon=([0-9]*) nr_activate_file=([0-9]*) nr_ref_keep=([0-9]*) nr_unmap_fail=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)';
117−my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_scanned=([0-9]*) nr_rotated=([0-9]*) priority=([0-9]*)';
117+my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_taken=([0-9]*) nr_active=([0-9]*) nr_deactivated=([0-9]*) nr_referenced=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)' ;
118118 my $regex_writepage_default = 'page=([0-9a-f]*) pfn=([0-9]*) flags=([A-Z_|]*)';
119119
120120 # Dyanically discovered regex
@@ -184,8 +184,7 @@ sub generate_traceevent_regex { · 63 unmodified lines
184184 $regex_direct_begin = generate_traceevent_regex(
185185 "vmscan/mm_vmscan_direct_reclaim_begin",
186186 $regex_direct_begin_default,
187− "order", "may_writepage",
188− "gfp_flags");
187+ "order", "gfp_flags");
189188 $regex_direct_end = generate_traceevent_regex(
190189 "vmscan/mm_vmscan_direct_reclaim_end",
191190 $regex_direct_end_default,
@@ -201,11 +200,11 @@ $regex_kswapd_sleep = generate_traceevent_regex( · 9 unmodified lines
201200 $regex_wakeup_kswapd = generate_traceevent_regex(
202201 "vmscan/mm_vmscan_wakeup_kswapd",
203202 $regex_wakeup_kswapd_default,
204− "nid", "zid", "order", "gfp_flags");
203+ "nid", "order", "gfp_flags");
205204 $regex_lru_isolate = generate_traceevent_regex(
206205 "vmscan/mm_vmscan_lru_isolate",
207206 $regex_lru_isolate_default,
208− "isolate_mode", "classzone_idx", "order",
207+ "classzone", "order",
209208 "nr_requested", "nr_scanned", "nr_skipped", "nr_taken",
210209 "lru");
211210 $regex_lru_shrink_inactive = generate_traceevent_regex(
@@ -218,11 +217,10 @@ $regex_lru_shrink_inactive = generate_traceevent_regex( · 6 unmodified lines
218217 $regex_lru_shrink_active = generate_traceevent_regex(
219218 "vmscan/mm_vmscan_lru_shrink_active",
220219 $regex_lru_shrink_active_default,
221− "nid", "zid",
222− "lru",
223− "nr_scanned", "nr_rotated", "priority");
220+ "nid", "nr_taken", "nr_active", "nr_deactivated", "nr_referenced",
221+ "priority", "flags");
224222 $regex_writepage = generate_traceevent_regex(
225− "vmscan/mm_vmscan_writepage",
223+ "vmscan/mm_vmscan_write_folio",
226224 $regex_writepage_default,
227225 "page", "pfn", "flags");
228226
@@ -371,7 +369,7 @@ EVENT_PROCESS: · 142 unmodified lines
371369 print " $regex_wakeup_kswapd\n";
372370 next;
373371 }
374− my $order = $3;
372+ my $order = $2;
375373 $perprocesspid{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD_PERORDER}[$order]++;
376374 } elsif ($tracepoint eq "mm_vmscan_lru_isolate") {
377375 $details = $6;
@@ -381,18 +379,14 @@ EVENT_PROCESS: · 3 unmodified lines
381379 print " $regex_lru_isolate/o\n";
382380 next;
383381 }
384− my $isolate_mode = $1;
385− my $nr_scanned = $5;
386− my $file = $8;
387−
388− # To closer match vmstat scanning statistics, only count isolate_both
389− # and isolate_inactive as scanning. isolate_active is rotation
390− # isolate_inactive == 1
391− # isolate_active == 2
392− # isolate_both == 3
393− if ($isolate_mode != 2) {
382+ my $nr_scanned = $4;
383+ my $lru = $7;
384+
385+ # To closer match vmstat scanning statistics, only count
386+ # inactive lru as scanning
387+ if ($lru =~ /inactive_/) {
394388 $perprocesspid{$process_pid}->{HIGH_NR_SCANNED} += $nr_scanned;
395− if ($file =~ /_file/) {
389+ if ($lru =~ /_file/) {
396390 $perprocesspid{$process_pid}->{HIGH_NR_FILE_SCANNED} += $nr_scanned;
397391 } else {
398392 $perprocesspid{$process_pid}->{HIGH_NR_ANON_SCANNED} += $nr_scanned;
MAINTAINERS+2−0
@@ -5332,6 +5332,7 @@ S: Maintained · 5331 unmodified lines
53325332 F: mm/memcontrol.c
53335333 F: mm/swap_cgroup.c
53345334 F: tools/testing/selftests/cgroup/memcg_protection.m
5335+F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
53355336 F: tools/testing/selftests/cgroup/test_kmem.c
53365337 F: tools/testing/selftests/cgroup/test_memcontrol.c
53375338
@@ -9754,6 +9755,7 @@ F: include/linux/hugetlb.h · 4416 unmodified lines
97549755 F: mm/hugetlb.c
97559756 F: mm/hugetlb_vmemmap.c
97569757 F: mm/hugetlb_vmemmap.h
9758+F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
97579759
97589760 HVA ST MEDIA DRIVER
97599761 M: Jean-Christophe Trotin <jean-christophe.trotin@foss.st.com>
arch/alpha/include/asm/bitops.h+20−0
@@ -286,6 +286,26 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 285 unmodified lines
286286 #define arch_test_bit generic_test_bit
287287 #define arch_test_bit_acquire generic_test_bit_acquire
288288
289+static inline bool xor_unlock_is_negative_byte(unsigned long mask,
290+ volatile unsigned long *p)
291+{
292+ unsigned long temp, old;
293+
294+ __asm__ __volatile__(
295+ "1: ldl_l %0,%4\n"
296+ " mov %0,%2\n"
297+ " xor %0,%3,%0\n"
298+ " stl_c %0,%1\n"
299+ " beq %0,2f\n"
300+ ".subsection 2\n"
301+ "2: br 1b\n"
302+ ".previous"
303+ :"=&r" (temp), "=m" (*p), "=&r" (old)
304+ :"Ir" (mask), "m" (*p));
305+
306+ return (old & BIT(7)) != 0;
307+}
308+
289309 /*
290310 * ffz = Find First Zero in word. Undefined if no zero exists,
291311 * so code should check against ~0UL first..
arch/arm64/include/asm/tlb.h+4−1
@@ -96,7 +96,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, · 95 unmodified lines
9696 static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
9797 unsigned long addr)
9898 {
99− tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp));
99+ struct ptdesc *ptdesc = virt_to_ptdesc(pudp);
100+
101+ pagetable_pud_dtor(ptdesc);
102+ tlb_remove_ptdesc(tlb, ptdesc);
100103 }
101104 #endif
102105
arch/arm64/kernel/mte.c+2−2
@@ -411,8 +411,8 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, · 410 unmodified lines
411411 struct page *page = get_user_page_vma_remote(mm, addr,
412412 gup_flags, &vma);
413413
414− if (IS_ERR_OR_NULL(page)) {
415− err = page == NULL ? -EIO : PTR_ERR(page);
414+ if (IS_ERR(page)) {
415+ err = PTR_ERR(page);
416416 break;
417417 }
418418
arch/arm64/mm/kasan_init.c+5−1
@@ -300,7 +300,11 @@ void __init kasan_init(void) · 299 unmodified lines
300300 kasan_init_shadow();
301301 kasan_init_depth();
302302 #if defined(CONFIG_KASAN_GENERIC)
303− /* CONFIG_KASAN_SW_TAGS also requires kasan_init_sw_tags(). */
303+ /*
304+ * Generic KASAN is now fully initialized.
305+ * Software and Hardware Tag-Based modes still require
306+ * kasan_init_sw_tags() and kasan_init_hw_tags() correspondingly.
307+ */
304308 pr_info("KernelAddressSanitizer initialized (generic)\n");
305309 #endif
306310 }
arch/loongarch/include/asm/pgalloc.h+1−0
@@ -84,6 +84,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 83 unmodified lines
8484
8585 if (!ptdesc)
8686 return NULL;
87+ pagetable_pud_ctor(ptdesc);
8788 pud = ptdesc_address(ptdesc);
8889
8990 pud_init(pud);
arch/m68k/include/asm/bitops.h+21−0
@@ -319,6 +319,27 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 318 unmodified lines
319319 return test_and_change_bit(nr, addr);
320320 }
321321
322+static inline bool xor_unlock_is_negative_byte(unsigned long mask,
323+ volatile unsigned long *p)
324+{
325+#ifdef CONFIG_COLDFIRE
326+ __asm__ __volatile__ ("eorl %1, %0"
327+ : "+m" (*p)
328+ : "d" (mask)
329+ : "memory");
330+ return *p & (1 << 7);
331+#else
332+ char result;
333+ char *cp = (char *)p + 3; /* m68k is big-endian */
334+
335+ __asm__ __volatile__ ("eor.b %1, %2; smi %0"
336+ : "=d" (result)
337+ : "di" (mask), "o" (*cp)
338+ : "memory");
339+ return result;
340+#endif
341+}
342+
322343 /*
323344 * The true 68020 and more advanced processors support the "bfffo"
324345 * instruction for finding bits. ColdFire and simple 68000 parts
arch/mips/include/asm/bitops.h+24−1
@@ -73,7 +73,8 @@ int __mips_test_and_clear_bit(unsigned long nr, · 72 unmodified lines
7373 volatile unsigned long *addr);
7474 int __mips_test_and_change_bit(unsigned long nr,
7575 volatile unsigned long *addr);
76−
76+bool __mips_xor_is_negative_byte(unsigned long mask,
77+ volatile unsigned long *addr);
7778
7879 /*
7980 * set_bit - Atomically set a bit in memory
@@ -279,6 +280,28 @@ static inline int test_and_change_bit(unsigned long nr, · 199 unmodified lines
279280 return res;
280281 }
281282
283+static inline bool xor_unlock_is_negative_byte(unsigned long mask,
284+ volatile unsigned long *p)
285+{
286+ unsigned long orig;
287+ bool res;
288+
289+ smp_mb__before_atomic();
290+
291+ if (!kernel_uses_llsc) {
292+ res = __mips_xor_is_negative_byte(mask, p);
293+ } else {
294+ orig = __test_bit_op(*p, "%0",
295+ "xor\t%1, %0, %3",
296+ "ir"(mask));
297+ res = (orig & BIT(7)) != 0;
298+ }
299+
300+ smp_llsc_mb();
301+
302+ return res;
303+}
304+
282305 #undef __bit_op
283306 #undef __test_bit_op
284307
arch/mips/include/asm/pgalloc.h+1−0
@@ -95,6 +95,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 94 unmodified lines
9595
9696 if (!ptdesc)
9797 return NULL;
98+ pagetable_pud_ctor(ptdesc);
9899 pud = ptdesc_address(ptdesc);
99100
100101 pud_init(pud);
arch/mips/lib/bitops.c+14−0
@@ -146,3 +146,17 @@ int __mips_test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 145 unmodified lines
146146 return res;
147147 }
148148 EXPORT_SYMBOL(__mips_test_and_change_bit);
149+
150+bool __mips_xor_is_negative_byte(unsigned long mask,
151+ volatile unsigned long *addr)
152+{
153+ unsigned long flags;
154+ unsigned long data;
155+
156+ raw_local_irq_save(flags);
157+ data = *addr;
158+ *addr = data ^ mask;
159+ raw_local_irq_restore(flags);
160+
161+ return (data & BIT(7)) != 0;
162+}
arch/mips/mm/cache.c+1−1
@@ -117,7 +117,7 @@ void __flush_dcache_pages(struct page *page, unsigned int nr) · 116 unmodified lines
117117 * get faulted into the tlb (and thus flushed) anyways.
118118 */
119119 for (i = 0; i < nr; i++) {
120− addr = (unsigned long)kmap_local_page(page + i);
120+ addr = (unsigned long)kmap_local_page(nth_page(page, i));
121121 flush_data_cache_page(addr);
122122 kunmap_local((void *)addr);
123123 }
arch/powerpc/include/asm/bitops.h+5−16
@@ -233,35 +233,24 @@ static inline int arch_test_and_change_bit(unsigned long nr, · 232 unmodified lines
233233 return test_and_change_bits(BIT_MASK(nr), addr + BIT_WORD(nr)) != 0;
234234 }
235235
236−#ifdef CONFIG_PPC64
237−static inline unsigned long
238−clear_bit_unlock_return_word(int nr, volatile unsigned long *addr)
236+static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
237+ volatile unsigned long *p)
239238 {
240239 unsigned long old, t;
241− unsigned long *p = (unsigned long *)addr + BIT_WORD(nr);
242− unsigned long mask = BIT_MASK(nr);
243240
244241 __asm__ __volatile__ (
245242 PPC_RELEASE_BARRIER
246243 "1:" PPC_LLARX "%0,0,%3,0\n"
247− "andc %1,%0,%2\n"
244+ "xor %1,%0,%2\n"
248245 PPC_STLCX "%1,0,%3\n"
249246 "bne- 1b\n"
250247 : "=&r" (old), "=&r" (t)
251248 : "r" (mask), "r" (p)
252249 : "cc", "memory");
253250
254− return old;
251+ return (old & BIT_MASK(7)) != 0;
255252 }
256−
257−/*
258− * This is a special function for mm/filemap.c
259− * Bit 7 corresponds to PG_waiters.
260− */
261−#define arch_clear_bit_unlock_is_negative_byte(nr, addr) \
262− (clear_bit_unlock_return_word(nr, addr) & BIT_MASK(7))
263−
264−#endif /* CONFIG_PPC64 */
253+#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
265254
266255 #include <asm-generic/bitops/non-atomic.h>
267256
arch/riscv/include/asm/bitops.h+12−0
@@ -191,6 +191,18 @@ static inline void __clear_bit_unlock( · 190 unmodified lines
191191 clear_bit_unlock(nr, addr);
192192 }
193193
194+static inline bool xor_unlock_is_negative_byte(unsigned long mask,
195+ volatile unsigned long *addr)
196+{
197+ unsigned long res;
198+ __asm__ __volatile__ (
199+ __AMO(xor) ".rl %0, %2, %1"
200+ : "=r" (res), "+A" (*addr)
201+ : "r" (__NOP(mask))
202+ : "memory");
203+ return (res & BIT(7)) != 0;
204+}
205+
194206 #undef __test_and_op_bit
195207 #undef __op_bit
196208 #undef __NOP
arch/s390/include/asm/bitops.h+10−0
@@ -201,6 +201,16 @@ static inline void arch___clear_bit_unlock(unsigned long nr, · 200 unmodified lines
201201 arch___clear_bit(nr, ptr);
202202 }
203203
204+static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
205+ volatile unsigned long *ptr)
206+{
207+ unsigned long old;
208+
209+ old = __atomic64_xor_barrier(mask, (long *)ptr);
210+ return old & BIT(7);
211+}
212+#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
213+
204214 #include <asm-generic/bitops/instrumented-atomic.h>
205215 #include <asm-generic/bitops/instrumented-non-atomic.h>
206216 #include <asm-generic/bitops/instrumented-lock.h>
arch/x86/include/asm/bitops.h+5−6
@@ -94,18 +94,17 @@ arch___clear_bit(unsigned long nr, volatile unsigned long *addr) · 93 unmodified lines
9494 asm volatile(__ASM_SIZE(btr) " %1,%0" : : ADDR, "Ir" (nr) : "memory");
9595 }
9696
97−static __always_inline bool
98−arch_clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr)
97+static __always_inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
98+ volatile unsigned long *addr)
9999 {
100100 bool negative;
101− asm volatile(LOCK_PREFIX "andb %2,%1"
101+ asm volatile(LOCK_PREFIX "xorb %2,%1"
102102 CC_SET(s)
103103 : CC_OUT(s) (negative), WBYTE_ADDR(addr)
104− : "ir" ((char) ~(1 << nr)) : "memory");
104+ : "iq" ((char)mask) : "memory");
105105 return negative;
106106 }
107−#define arch_clear_bit_unlock_is_negative_byte \
108− arch_clear_bit_unlock_is_negative_byte
107+#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
109108
110109 static __always_inline void
111110 arch___clear_bit_unlock(long nr, volatile unsigned long *addr)
arch/x86/kvm/mmu/mmu.c+10−8
@@ -6800,11 +6800,7 @@ static unsigned long mmu_shrink_count(struct shrinker *shrink, · 6799 unmodified lines
68006800 return percpu_counter_read_positive(&kvm_total_used_mmu_pages);
68016801 }
68026802
6803−static struct shrinker mmu_shrinker = {
6804− .count_objects = mmu_shrink_count,
6805− .scan_objects = mmu_shrink_scan,
6806− .seeks = DEFAULT_SEEKS * 10,
6807−};
6803+static struct shrinker *mmu_shrinker;
68086804
68096805 static void mmu_destroy_caches(void)
68106806 {
@@ -6937,10 +6933,16 @@ int kvm_mmu_vendor_module_init(void) · 126 unmodified lines
69376933 if (percpu_counter_init(&kvm_total_used_mmu_pages, 0, GFP_KERNEL))
69386934 goto out;
69396935
6940− ret = register_shrinker(&mmu_shrinker, "x86-mmu");
6941− if (ret)
6936+ mmu_shrinker = shrinker_alloc(0, "x86-mmu");
6937+ if (!mmu_shrinker)
69426938 goto out_shrinker;
69436939
6940+ mmu_shrinker->count_objects = mmu_shrink_count;
6941+ mmu_shrinker->scan_objects = mmu_shrink_scan;
6942+ mmu_shrinker->seeks = DEFAULT_SEEKS * 10;
6943+
6944+ shrinker_register(mmu_shrinker);
6945+
69446946 return 0;
69456947
69466948 out_shrinker:
@@ -6962,7 +6964,7 @@ void kvm_mmu_vendor_module_exit(void) · 15 unmodified lines
69626964 {
69636965 mmu_destroy_caches();
69646966 percpu_counter_destroy(&kvm_total_used_mmu_pages);
6965− unregister_shrinker(&mmu_shrinker);
6967+ shrinker_free(mmu_shrinker);
69666968 }
69676969
69686970 /*
arch/x86/mm/pgtable.c+3−0
@@ -76,6 +76,9 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) · 75 unmodified lines
7676 #if CONFIG_PGTABLE_LEVELS > 3
7777 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)
7878 {
79+ struct ptdesc *ptdesc = virt_to_ptdesc(pud);
80+
81+ pagetable_pud_dtor(ptdesc);
7982 paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
8083 paravirt_tlb_remove_table(tlb, virt_to_page(pud));
8184 }
drivers/acpi/acpi_pad.c+1−1
@@ -101,7 +101,7 @@ static void round_robin_cpu(unsigned int tsk_index) · 100 unmodified lines
101101 for_each_cpu(cpu, pad_busy_cpus)
102102 cpumask_or(tmp, tmp, topology_sibling_cpumask(cpu));
103103 cpumask_andnot(tmp, cpu_online_mask, tmp);
104− /* avoid HT sibilings if possible */
104+ /* avoid HT siblings if possible */
105105 if (cpumask_empty(tmp))
106106 cpumask_andnot(tmp, cpu_online_mask, pad_busy_cpus);
107107 if (cpumask_empty(tmp)) {
drivers/acpi/numa/hmat.c+93−53
@@ -24,6 +24,7 @@ · 23 unmodified lines
2424 #include <linux/node.h>
2525 #include <linux/sysfs.h>
2626 #include <linux/dax.h>
27+#include <linux/memory-tiers.h>
2728
2829 static u8 hmat_revision;
2930 static int hmat_disable __initdata;
@@ -582,28 +583,25 @@ static int initiators_to_nodemask(unsigned long *p_nodes) · 552 unmodified lines
582583 return 0;
583584 }
584585
585−static void hmat_register_target_initiators(struct memory_target *target)
586+static void hmat_update_target_attrs(struct memory_target *target,
587+ unsigned long *p_nodes, int access)
586588 {
587− static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
588589 struct memory_initiator *initiator;
589− unsigned int mem_nid, cpu_nid;
590+ unsigned int cpu_nid;
590591 struct memory_locality *loc = NULL;
591592 u32 best = 0;
592− bool access0done = false;
593593 int i;
594594
595− mem_nid = pxm_to_node(target->memory_pxm);
595+ bitmap_zero(p_nodes, MAX_NUMNODES);
596596 /*
597− * If the Address Range Structure provides a local processor pxm, link
597+ * If the Address Range Structure provides a local processor pxm, set
598598 * only that one. Otherwise, find the best performance attributes and
599− * register all initiators that match.
599+ * collect all initiators that match.
600600 */
601601 if (target->processor_pxm != PXM_INVAL) {
602602 cpu_nid = pxm_to_node(target->processor_pxm);
603− register_memory_node_under_compute_node(mem_nid, cpu_nid, 0);
604− access0done = true;
605− if (node_state(cpu_nid, N_CPU)) {
606− register_memory_node_under_compute_node(mem_nid, cpu_nid, 1);
603+ if (access == 0 || node_state(cpu_nid, N_CPU)) {
604+ set_bit(target->processor_pxm, p_nodes);
607605 return;
608606 }
609607 }
@@ -617,47 +615,10 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines
617615 * We'll also use the sorting to prime the candidate nodes with known
618616 * initiators.
619617 */
620− bitmap_zero(p_nodes, MAX_NUMNODES);
621618 list_sort(NULL, &initiators, initiator_cmp);
622619 if (initiators_to_nodemask(p_nodes) < 0)
623620 return;
624621
625− if (!access0done) {
626− for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) {
627− loc = localities_types[i];
628− if (!loc)
629− continue;
630−
631− best = 0;
632− list_for_each_entry(initiator, &initiators, node) {
633− u32 value;
634−
635− if (!test_bit(initiator->processor_pxm, p_nodes))
636− continue;
637−
638− value = hmat_initiator_perf(target, initiator,
639− loc->hmat_loc);
640− if (hmat_update_best(loc->hmat_loc->data_type, value, &best))
641− bitmap_clear(p_nodes, 0, initiator->processor_pxm);
642− if (value != best)
643− clear_bit(initiator->processor_pxm, p_nodes);
644− }
645− if (best)
646− hmat_update_target_access(target, loc->hmat_loc->data_type,
647− best, 0);
648− }
649−
650− for_each_set_bit(i, p_nodes, MAX_NUMNODES) {
651− cpu_nid = pxm_to_node(i);
652− register_memory_node_under_compute_node(mem_nid, cpu_nid, 0);
653− }
654− }
655−
656− /* Access 1 ignores Generic Initiators */
657− bitmap_zero(p_nodes, MAX_NUMNODES);
658− if (initiators_to_nodemask(p_nodes) < 0)
659− return;
660−
661622 for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) {
662623 loc = localities_types[i];
663624 if (!loc)
@@ -667,7 +628,7 @@ static void hmat_register_target_initiators(struct memory_target *target) · 3 unmodified lines
667628 list_for_each_entry(initiator, &initiators, node) {
668629 u32 value;
669630
670− if (!initiator->has_cpu) {
631+ if (access == 1 && !initiator->has_cpu) {
671632 clear_bit(initiator->processor_pxm, p_nodes);
672633 continue;
673634 }
@@ -681,14 +642,33 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines
681642 clear_bit(initiator->processor_pxm, p_nodes);
682643 }
683644 if (best)
684− hmat_update_target_access(target, loc->hmat_loc->data_type, best, 1);
645+ hmat_update_target_access(target, loc->hmat_loc->data_type, best, access);
685646 }
647+}
648+
649+static void __hmat_register_target_initiators(struct memory_target *target,
650+ unsigned long *p_nodes,
651+ int access)
652+{
653+ unsigned int mem_nid, cpu_nid;
654+ int i;
655+
656+ mem_nid = pxm_to_node(target->memory_pxm);
657+ hmat_update_target_attrs(target, p_nodes, access);
686658 for_each_set_bit(i, p_nodes, MAX_NUMNODES) {
687659 cpu_nid = pxm_to_node(i);
688− register_memory_node_under_compute_node(mem_nid, cpu_nid, 1);
660+ register_memory_node_under_compute_node(mem_nid, cpu_nid, access);
689661 }
690662 }
691663
664+static void hmat_register_target_initiators(struct memory_target *target)
665+{
666+ static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
667+
668+ __hmat_register_target_initiators(target, p_nodes, 0);
669+ __hmat_register_target_initiators(target, p_nodes, 1);
670+}
671+
692672 static void hmat_register_target_cache(struct memory_target *target)
693673 {
694674 unsigned mem_nid = pxm_to_node(target->memory_pxm);
@@ -780,6 +760,61 @@ static int hmat_callback(struct notifier_block *self, · 85 unmodified lines
780760 return NOTIFY_OK;
781761 }
782762
763+static int hmat_set_default_dram_perf(void)
764+{
765+ int rc;
766+ int nid, pxm;
767+ struct memory_target *target;
768+ struct node_hmem_attrs *attrs;
769+
770+ if (!default_dram_type)
771+ return -EIO;
772+
773+ for_each_node_mask(nid, default_dram_type->nodes) {
774+ pxm = node_to_pxm(nid);
775+ target = find_mem_target(pxm);
776+ if (!target)
777+ continue;
778+ attrs = &target->hmem_attrs[1];
779+ rc = mt_set_default_dram_perf(nid, attrs, "ACPI HMAT");
780+ if (rc)
781+ return rc;
782+ }
783+
784+ return 0;
785+}
786+
787+static int hmat_calculate_adistance(struct notifier_block *self,
788+ unsigned long nid, void *data)
789+{
790+ static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
791+ struct memory_target *target;
792+ struct node_hmem_attrs *perf;
793+ int *adist = data;
794+ int pxm;
795+
796+ pxm = node_to_pxm(nid);
797+ target = find_mem_target(pxm);
798+ if (!target)
799+ return NOTIFY_OK;
800+
801+ mutex_lock(&target_lock);
802+ hmat_update_target_attrs(target, p_nodes, 1);
803+ mutex_unlock(&target_lock);
804+
805+ perf = &target->hmem_attrs[1];
806+
807+ if (mt_perf_to_adistance(perf, adist))
808+ return NOTIFY_OK;
809+
810+ return NOTIFY_STOP;
811+}
812+
813+static struct notifier_block hmat_adist_nb __meminitdata = {
814+ .notifier_call = hmat_calculate_adistance,
815+ .priority = 100,
816+};
817+
783818 static __init void hmat_free_structures(void)
784819 {
785820 struct memory_target *target, *tnext;
@@ -862,8 +897,13 @@ static __init int hmat_init(void) · 76 unmodified lines
862897 hmat_register_targets();
863898
864899 /* Keep the table and structures if the notifier may use them */
865− if (!hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI))
866− return 0;
900+ if (hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI))
901+ goto out_put;
902+
903+ if (!hmat_set_default_dram_perf())
904+ register_mt_adistance_algorithm(&hmat_adist_nb);
905+
906+ return 0;
867907 out_put:
868908 hmat_free_structures();
869909 acpi_put_table(tbl);
drivers/android/binder_alloc.c+18−12
@@ -1053,11 +1053,7 @@ binder_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1052 unmodified lines
10531053 NULL, sc->nr_to_scan);
10541054 }
10551055
1056−static struct shrinker binder_shrinker = {
1057− .count_objects = binder_shrink_count,
1058− .scan_objects = binder_shrink_scan,
1059− .seeks = DEFAULT_SEEKS,
1060−};
1056+static struct shrinker *binder_shrinker;
10611057
10621058 /**
10631059 * binder_alloc_init() - called by binder_open() for per-proc initialization
@@ -1077,19 +1073,29 @@ void binder_alloc_init(struct binder_alloc *alloc) · 13 unmodified lines
10771073
10781074 int binder_alloc_shrinker_init(void)
10791075 {
1080− int ret = list_lru_init(&binder_alloc_lru);
1076+ int ret;
10811077
1082− if (ret == 0) {
1083− ret = register_shrinker(&binder_shrinker, "android-binder");
1084− if (ret)
1085− list_lru_destroy(&binder_alloc_lru);
1078+ ret = list_lru_init(&binder_alloc_lru);
1079+ if (ret)
1080+ return ret;
1081+
1082+ binder_shrinker = shrinker_alloc(0, "android-binder");
1083+ if (!binder_shrinker) {
1084+ list_lru_destroy(&binder_alloc_lru);
1085+ return -ENOMEM;
10861086 }
1087− return ret;
1087+
1088+ binder_shrinker->count_objects = binder_shrink_count;
1089+ binder_shrinker->scan_objects = binder_shrink_scan;
1090+
1091+ shrinker_register(binder_shrinker);
1092+
1093+ return 0;
10881094 }
10891095
10901096 void binder_alloc_shrinker_exit(void)
10911097 {
1092− unregister_shrinker(&binder_shrinker);
1098+ shrinker_free(binder_shrinker);
10931099 list_lru_destroy(&binder_alloc_lru);
10941100 }
10951101
drivers/base/cacheinfo.c+50−1
@@ -898,6 +898,48 @@ err: · 897 unmodified lines
898898 return rc;
899899 }
900900
901+/*
902+ * Calculate the size of the per-CPU data cache slice. This can be
903+ * used to estimate the size of the data cache slice that can be used
904+ * by one CPU under ideal circumstances. UNIFIED caches are counted
905+ * in addition to DATA caches. So, please consider code cache usage
906+ * when use the result.
907+ *
908+ * Because the cache inclusive/non-inclusive information isn't
909+ * available, we just use the size of the per-CPU slice of LLC to make
910+ * the result more predictable across architectures.
911+ */
912+static void update_per_cpu_data_slice_size_cpu(unsigned int cpu)
913+{
914+ struct cpu_cacheinfo *ci;
915+ struct cacheinfo *llc;
916+ unsigned int nr_shared;
917+
918+ if (!last_level_cache_is_valid(cpu))
919+ return;
920+
921+ ci = ci_cacheinfo(cpu);
922+ llc = per_cpu_cacheinfo_idx(cpu, cache_leaves(cpu) - 1);
923+
924+ if (llc->type != CACHE_TYPE_DATA && llc->type != CACHE_TYPE_UNIFIED)
925+ return;
926+
927+ nr_shared = cpumask_weight(&llc->shared_cpu_map);
928+ if (nr_shared)
929+ ci->per_cpu_data_slice_size = llc->size / nr_shared;
930+}
931+
932+static void update_per_cpu_data_slice_size(bool cpu_online, unsigned int cpu)
933+{
934+ unsigned int icpu;
935+
936+ for_each_online_cpu(icpu) {
937+ if (!cpu_online && icpu == cpu)
938+ continue;
939+ update_per_cpu_data_slice_size_cpu(icpu);
940+ }
941+}
942+
901943 static int cacheinfo_cpu_online(unsigned int cpu)
902944 {
903945 int rc = detect_cache_attributes(cpu);
@@ -906,7 +948,12 @@ static int cacheinfo_cpu_online(unsigned int cpu) · 2 unmodified lines
906948 return rc;
907949 rc = cache_add_dev(cpu);
908950 if (rc)
909− free_cache_attributes(cpu);
951+ goto err;
952+ update_per_cpu_data_slice_size(true, cpu);
953+ setup_pcp_cacheinfo();
954+ return 0;
955+err:
956+ free_cache_attributes(cpu);
910957 return rc;
911958 }
912959
@@ -916,6 +963,8 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu) · 3 unmodified lines
916963 cpu_cache_sysfs_exit(cpu);
917964
918965 free_cache_attributes(cpu);
966+ update_per_cpu_data_slice_size(false, cpu);
967+ setup_pcp_cacheinfo();
919968 return 0;
920969 }
921970
drivers/dax/kmem.c+49−13
@@ -49,14 +49,52 @@ struct dax_kmem_data { · 48 unmodified lines
4949 struct resource *res[];
5050 };
5151
52−static struct memory_dev_type *dax_slowmem_type;
52+static DEFINE_MUTEX(kmem_memory_type_lock);
53+static LIST_HEAD(kmem_memory_types);
54+
55+static struct memory_dev_type *kmem_find_alloc_memory_type(int adist)
56+{
57+ bool found = false;
58+ struct memory_dev_type *mtype;
59+
60+ mutex_lock(&kmem_memory_type_lock);
61+ list_for_each_entry(mtype, &kmem_memory_types, list) {
62+ if (mtype->adistance == adist) {
63+ found = true;
64+ break;
65+ }
66+ }
67+ if (!found) {
68+ mtype = alloc_memory_type(adist);
69+ if (!IS_ERR(mtype))
70+ list_add(&mtype->list, &kmem_memory_types);
71+ }
72+ mutex_unlock(&kmem_memory_type_lock);
73+
74+ return mtype;
75+}
76+
77+static void kmem_put_memory_types(void)
78+{
79+ struct memory_dev_type *mtype, *mtn;
80+
81+ mutex_lock(&kmem_memory_type_lock);
82+ list_for_each_entry_safe(mtype, mtn, &kmem_memory_types, list) {
83+ list_del(&mtype->list);
84+ put_memory_type(mtype);
85+ }
86+ mutex_unlock(&kmem_memory_type_lock);
87+}
88+
5389 static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
5490 {
5591 struct device *dev = &dev_dax->dev;
5692 unsigned long total_len = 0;
5793 struct dax_kmem_data *data;
94+ struct memory_dev_type *mtype;
5895 int i, rc, mapped = 0;
5996 int numa_node;
97+ int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
6098
6199 /*
62100 * Ensure good NUMA information for the persistent memory.
@@ -71,6 +109,11 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 8 unmodified lines
71109 return -EINVAL;
72110 }
73111
112+ mt_calc_adistance(numa_node, &adist);
113+ mtype = kmem_find_alloc_memory_type(adist);
114+ if (IS_ERR(mtype))
115+ return PTR_ERR(mtype);
116+
74117 for (i = 0; i < dev_dax->nr_range; i++) {
75118 struct range range;
76119
@@ -88,7 +131,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 11 unmodified lines
88131 return -EINVAL;
89132 }
90133
91− init_node_memory_type(numa_node, dax_slowmem_type);
134+ init_node_memory_type(numa_node, mtype);
92135
93136 rc = -ENOMEM;
94137 data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL);
@@ -167,7 +210,7 @@ err_reg_mgid: · 72 unmodified lines
167210 err_res_name:
168211 kfree(data);
169212 err_dax_kmem_data:
170− clear_node_memory_type(numa_node, dax_slowmem_type);
213+ clear_node_memory_type(numa_node, mtype);
171214 return rc;
172215 }
173216
@@ -219,7 +262,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 45 unmodified lines
219262 * for that. This implies this reference will be around
220263 * till next reboot.
221264 */
222− clear_node_memory_type(node, dax_slowmem_type);
265+ clear_node_memory_type(node, NULL);
223266 }
224267 }
225268 #else
@@ -251,12 +294,6 @@ static int __init dax_kmem_init(void) · 25 unmodified lines
251294 if (!kmem_name)
252295 return -ENOMEM;
253296
254− dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE);
255− if (IS_ERR(dax_slowmem_type)) {
256− rc = PTR_ERR(dax_slowmem_type);
257− goto err_dax_slowmem_type;
258− }
259−
260297 rc = dax_driver_register(&device_dax_kmem_driver);
261298 if (rc)
262299 goto error_dax_driver;
@@ -264,8 +301,7 @@ static int __init dax_kmem_init(void) · 1 unmodified line
264301 return rc;
265302
266303 error_dax_driver:
267− put_memory_type(dax_slowmem_type);
268−err_dax_slowmem_type:
304+ kmem_put_memory_types();
269305 kfree_const(kmem_name);
270306 return rc;
271307 }
@@ -275,7 +311,7 @@ static void __exit dax_kmem_exit(void) · 3 unmodified lines
275311 dax_driver_unregister(&device_dax_kmem_driver);
276312 if (!any_hotremove_failed)
277313 kfree_const(kmem_name);
278− put_memory_type(dax_slowmem_type);
314+ kmem_put_memory_types();
279315 }
280316
281317 MODULE_AUTHOR("Intel Corporation");
drivers/firmware/efi/unaccepted_memory.c+20−0
@@ -3,6 +3,7 @@ · 2 unmodified lines
33 #include <linux/efi.h>
44 #include <linux/memblock.h>
55 #include <linux/spinlock.h>
6+#include <linux/crash_dump.h>
67 #include <asm/unaccepted_memory.h>
78
89 /* Protects unaccepted memory bitmap and accepting_list */
@@ -201,3 +202,22 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · 192 unmodified lines
201202
202203 return ret;
203204 }
205+
206+#ifdef CONFIG_PROC_VMCORE
207+static bool unaccepted_memory_vmcore_pfn_is_ram(struct vmcore_cb *cb,
208+ unsigned long pfn)
209+{
210+ return !pfn_is_unaccepted_memory(pfn);
211+}
212+
213+static struct vmcore_cb vmcore_cb = {
214+ .pfn_is_ram = unaccepted_memory_vmcore_pfn_is_ram,
215+};
216+
217+static int __init unaccepted_memory_init_kdump(void)
218+{
219+ register_vmcore_cb(&vmcore_cb);
220+ return 0;
221+}
222+core_initcall(unaccepted_memory_init_kdump);
223+#endif /* CONFIG_PROC_VMCORE */
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c+16−13
@@ -288,8 +288,7 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915) · 287 unmodified lines
288288 static unsigned long
289289 i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
290290 {
291− struct drm_i915_private *i915 =
292− container_of(shrinker, struct drm_i915_private, mm.shrinker);
291+ struct drm_i915_private *i915 = shrinker->private_data;
293292 unsigned long num_objects;
294293 unsigned long count;
295294
@@ -306,8 +305,8 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 10 unmodified lines
306305 if (num_objects) {
307306 unsigned long avg = 2 * count / num_objects;
308307
309− i915->mm.shrinker.batch =
310− max((i915->mm.shrinker.batch + avg) >> 1,
308+ i915->mm.shrinker->batch =
309+ max((i915->mm.shrinker->batch + avg) >> 1,
311310 128ul /* default SHRINK_BATCH */);
312311 }
313312
@@ -317,8 +316,7 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 3 unmodified lines
317316 static unsigned long
318317 i915_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
319318 {
320− struct drm_i915_private *i915 =
321− container_of(shrinker, struct drm_i915_private, mm.shrinker);
319+ struct drm_i915_private *i915 = shrinker->private_data;
322320 unsigned long freed;
323321
324322 sc->nr_scanned = 0;
@@ -430,12 +428,17 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr · 105 unmodified lines
430428
431429 void i915_gem_driver_register__shrinker(struct drm_i915_private *i915)
432430 {
433− i915->mm.shrinker.scan_objects = i915_gem_shrinker_scan;
434− i915->mm.shrinker.count_objects = i915_gem_shrinker_count;
435− i915->mm.shrinker.seeks = DEFAULT_SEEKS;
436− i915->mm.shrinker.batch = 4096;
437− drm_WARN_ON(&i915->drm, register_shrinker(&i915->mm.shrinker,
438− "drm-i915_gem"));
431+ i915->mm.shrinker = shrinker_alloc(0, "drm-i915_gem");
432+ if (!i915->mm.shrinker) {
433+ drm_WARN_ON(&i915->drm, 1);
434+ } else {
435+ i915->mm.shrinker->scan_objects = i915_gem_shrinker_scan;
436+ i915->mm.shrinker->count_objects = i915_gem_shrinker_count;
437+ i915->mm.shrinker->batch = 4096;
438+ i915->mm.shrinker->private_data = i915;
439+
440+ shrinker_register(i915->mm.shrinker);
441+ }
439442
440443 i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom;
441444 drm_WARN_ON(&i915->drm, register_oom_notifier(&i915->mm.oom_notifier));
@@ -451,7 +454,7 @@ void i915_gem_driver_unregister__shrinker(struct drm_i915_private *i915) · 9 unmodified lines
451454 unregister_vmap_purge_notifier(&i915->mm.vmap_notifier));
452455 drm_WARN_ON(&i915->drm,
453456 unregister_oom_notifier(&i915->mm.oom_notifier));
454− unregister_shrinker(&i915->mm.shrinker);
457+ shrinker_free(i915->mm.shrinker);
455458 }
456459
457460 void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915,
drivers/gpu/drm/i915/i915_drv.h+1−1
@@ -163,7 +163,7 @@ struct i915_gem_mm { · 162 unmodified lines
163163
164164 struct notifier_block oom_notifier;
165165 struct notifier_block vmap_notifier;
166− struct shrinker shrinker;
166+ struct shrinker *shrinker;
167167
168168 #ifdef CONFIG_MMU_NOTIFIER
169169 /**
drivers/gpu/drm/msm/msm_drv.c+3−1
@@ -265,7 +265,9 @@ static int msm_drm_init(struct device *dev, const struct drm_driver *drv) · 264 unmodified lines
265265 if (ret)
266266 goto err_deinit_vram;
267267
268− msm_gem_shrinker_init(ddev);
268+ ret = msm_gem_shrinker_init(ddev);
269+ if (ret)
270+ goto err_msm_uninit;
269271
270272 if (priv->kms_init) {
271273 ret = msm_drm_kms_init(dev, drv);
drivers/gpu/drm/msm/msm_drv.h+2−2
@@ -218,7 +218,7 @@ struct msm_drm_private { · 217 unmodified lines
218218 } vram;
219219
220220 struct notifier_block vmap_notifier;
221− struct shrinker shrinker;
221+ struct shrinker *shrinker;
222222
223223 struct drm_atomic_state *pm_state;
224224
@@ -280,7 +280,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data, · 55 unmodified lines
280280 unsigned long msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan);
281281 #endif
282282
283−void msm_gem_shrinker_init(struct drm_device *dev);
283+int msm_gem_shrinker_init(struct drm_device *dev);
284284 void msm_gem_shrinker_cleanup(struct drm_device *dev);
285285
286286 struct sg_table *msm_gem_prime_get_sg_table(struct drm_gem_object *obj);
drivers/gpu/drm/msm/msm_gem_shrinker.c+20−13
@@ -34,8 +34,7 @@ static bool can_block(struct shrink_control *sc) · 33 unmodified lines
3434 static unsigned long
3535 msm_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
3636 {
37− struct msm_drm_private *priv =
38− container_of(shrinker, struct msm_drm_private, shrinker);
37+ struct msm_drm_private *priv = shrinker->private_data;
3938 unsigned count = priv->lru.dontneed.count;
4039
4140 if (can_swap())
@@ -100,8 +99,7 @@ active_evict(struct drm_gem_object *obj) · 58 unmodified lines
10099 static unsigned long
101100 msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
102101 {
103− struct msm_drm_private *priv =
104− container_of(shrinker, struct msm_drm_private, shrinker);
102+ struct msm_drm_private *priv = shrinker->private_data;
105103 struct {
106104 struct drm_gem_lru *lru;
107105 bool (*shrink)(struct drm_gem_object *obj);
@@ -148,10 +146,11 @@ msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan) · 40 unmodified lines
148146 struct shrink_control sc = {
149147 .nr_to_scan = nr_to_scan,
150148 };
151− int ret;
149+ unsigned long ret = SHRINK_STOP;
152150
153151 fs_reclaim_acquire(GFP_KERNEL);
154− ret = msm_gem_shrinker_scan(&priv->shrinker, &sc);
152+ if (priv->shrinker)
153+ ret = msm_gem_shrinker_scan(priv->shrinker, &sc);
155154 fs_reclaim_release(GFP_KERNEL);
156155
157156 return ret;
@@ -210,16 +209,24 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr) · 52 unmodified lines
210209 *
211210 * This function registers and sets up the msm shrinker.
212211 */
213−void msm_gem_shrinker_init(struct drm_device *dev)
212+int msm_gem_shrinker_init(struct drm_device *dev)
214213 {
215214 struct msm_drm_private *priv = dev->dev_private;
216− priv->shrinker.count_objects = msm_gem_shrinker_count;
217− priv->shrinker.scan_objects = msm_gem_shrinker_scan;
218− priv->shrinker.seeks = DEFAULT_SEEKS;
219− WARN_ON(register_shrinker(&priv->shrinker, "drm-msm_gem"));
215+
216+ priv->shrinker = shrinker_alloc(0, "drm-msm_gem");
217+ if (!priv->shrinker)
218+ return -ENOMEM;
219+
220+ priv->shrinker->count_objects = msm_gem_shrinker_count;
221+ priv->shrinker->scan_objects = msm_gem_shrinker_scan;
222+ priv->shrinker->private_data = priv;
223+
224+ shrinker_register(priv->shrinker);
220225
221226 priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap;
222227 WARN_ON(register_vmap_purge_notifier(&priv->vmap_notifier));
228+
229+ return 0;
223230 }
224231
225232 /**
@@ -232,8 +239,8 @@ void msm_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines
232239 {
233240 struct msm_drm_private *priv = dev->dev_private;
234241
235− if (priv->shrinker.nr_deferred) {
242+ if (priv->shrinker) {
236243 WARN_ON(unregister_vmap_purge_notifier(&priv->vmap_notifier));
237− unregister_shrinker(&priv->shrinker);
244+ shrinker_free(priv->shrinker);
238245 }
239246 }
drivers/gpu/drm/panfrost/panfrost_device.h+1−1
@@ -119,7 +119,7 @@ struct panfrost_device { · 118 unmodified lines
119119
120120 struct mutex shrinker_lock;
121121 struct list_head shrinker_list;
122− struct shrinker shrinker;
122+ struct shrinker *shrinker;
123123
124124 struct panfrost_devfreq pfdevfreq;
125125
drivers/gpu/drm/panfrost/panfrost_drv.c+5−1
@@ -659,10 +659,14 @@ static int panfrost_probe(struct platform_device *pdev) · 658 unmodified lines
659659 if (err < 0)
660660 goto err_out1;
661661
662− panfrost_gem_shrinker_init(ddev);
662+ err = panfrost_gem_shrinker_init(ddev);
663+ if (err)
664+ goto err_out2;
663665
664666 return 0;
665667
668+err_out2:
669+ drm_dev_unregister(ddev);
666670 err_out1:
667671 pm_runtime_disable(pfdev->dev);
668672 panfrost_device_fini(pfdev);
drivers/gpu/drm/panfrost/panfrost_gem.h+1−1
@@ -86,7 +86,7 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo, · 85 unmodified lines
8686 void panfrost_gem_mapping_put(struct panfrost_gem_mapping *mapping);
8787 void panfrost_gem_teardown_mappings_locked(struct panfrost_gem_object *bo);
8888
89−void panfrost_gem_shrinker_init(struct drm_device *dev);
89+int panfrost_gem_shrinker_init(struct drm_device *dev);
9090 void panfrost_gem_shrinker_cleanup(struct drm_device *dev);
9191
9292 #endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c+17−12
@@ -18,8 +18,7 @@ · 17 unmodified lines
1818 static unsigned long
1919 panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
2020 {
21− struct panfrost_device *pfdev =
22− container_of(shrinker, struct panfrost_device, shrinker);
21+ struct panfrost_device *pfdev = shrinker->private_data;
2322 struct drm_gem_shmem_object *shmem;
2423 unsigned long count = 0;
2524
@@ -65,8 +64,7 @@ unlock_mappings: · 39 unmodified lines
6564 static unsigned long
6665 panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
6766 {
68− struct panfrost_device *pfdev =
69− container_of(shrinker, struct panfrost_device, shrinker);
67+ struct panfrost_device *pfdev = shrinker->private_data;
7068 struct drm_gem_shmem_object *shmem, *tmp;
7169 unsigned long freed = 0;
7270
@@ -97,13 +95,21 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) · 24 unmodified lines
9795 *
9896 * This function registers and sets up the panfrost shrinker.
9997 */
100−void panfrost_gem_shrinker_init(struct drm_device *dev)
98+int panfrost_gem_shrinker_init(struct drm_device *dev)
10199 {
102100 struct panfrost_device *pfdev = dev->dev_private;
103− pfdev->shrinker.count_objects = panfrost_gem_shrinker_count;
104− pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan;
105− pfdev->shrinker.seeks = DEFAULT_SEEKS;
106− WARN_ON(register_shrinker(&pfdev->shrinker, "drm-panfrost"));
101+
102+ pfdev->shrinker = shrinker_alloc(0, "drm-panfrost");
103+ if (!pfdev->shrinker)
104+ return -ENOMEM;
105+
106+ pfdev->shrinker->count_objects = panfrost_gem_shrinker_count;
107+ pfdev->shrinker->scan_objects = panfrost_gem_shrinker_scan;
108+ pfdev->shrinker->private_data = pfdev;
109+
110+ shrinker_register(pfdev->shrinker);
111+
112+ return 0;
107113 }
108114
109115 /**
@@ -116,7 +122,6 @@ void panfrost_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines
116122 {
117123 struct panfrost_device *pfdev = dev->dev_private;
118124
119− if (pfdev->shrinker.nr_deferred) {
120− unregister_shrinker(&pfdev->shrinker);
121− }
125+ if (pfdev->shrinker)
126+ shrinker_free(pfdev->shrinker);
122127 }
drivers/gpu/drm/ttm/ttm_pool.c+31−9
@@ -73,7 +73,8 @@ static struct ttm_pool_type global_dma32_uncached[MAX_ORDER + 1]; · 72 unmodified lines
7373
7474 static spinlock_t shrinker_lock;
7575 static struct list_head shrinker_list;
76−static struct shrinker mm_shrinker;
76+static struct shrinker *mm_shrinker;
77+static DECLARE_RWSEM(pool_shrink_rwsem);
7778
7879 /* Allocate pages of size 1 << order with the given gfp_flags */
7980 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
@@ -317,6 +318,7 @@ static unsigned int ttm_pool_shrink(void) · 237 unmodified lines
317318 unsigned int num_pages;
318319 struct page *p;
319320
321+ down_read(&pool_shrink_rwsem);
320322 spin_lock(&shrinker_lock);
321323 pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list);
322324 list_move_tail(&pt->shrinker_list, &shrinker_list);
@@ -329,6 +331,7 @@ static unsigned int ttm_pool_shrink(void) · 6 unmodified lines
329331 } else {
330332 num_pages = 0;
331333 }
334+ up_read(&pool_shrink_rwsem);
332335
333336 return num_pages;
334337 }
@@ -572,6 +575,18 @@ void ttm_pool_init(struct ttm_pool *pool, struct device *dev, · 237 unmodified lines
572575 }
573576 EXPORT_SYMBOL(ttm_pool_init);
574577
578+/**
579+ * ttm_pool_synchronize_shrinkers - Wait for all running shrinkers to complete.
580+ *
581+ * This is useful to guarantee that all shrinker invocations have seen an
582+ * update, before freeing memory, similar to rcu.
583+ */
584+static void ttm_pool_synchronize_shrinkers(void)
585+{
586+ down_write(&pool_shrink_rwsem);
587+ up_write(&pool_shrink_rwsem);
588+}
589+
575590 /**
576591 * ttm_pool_fini - Cleanup a pool
577592 *
@@ -593,7 +608,7 @@ void ttm_pool_fini(struct ttm_pool *pool) · 15 unmodified lines
593608 /* We removed the pool types from the LRU, but we need to also make sure
594609 * that no shrinker is concurrently freeing pages from the pool.
595610 */
596− synchronize_shrinkers();
611+ ttm_pool_synchronize_shrinkers();
597612 }
598613 EXPORT_SYMBOL(ttm_pool_fini);
599614
@@ -734,8 +749,8 @@ static int ttm_pool_debugfs_shrink_show(struct seq_file *m, void *data) · 134 unmodified lines
734749 struct shrink_control sc = { .gfp_mask = GFP_NOFS };
735750
736751 fs_reclaim_acquire(GFP_KERNEL);
737− seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(&mm_shrinker, &sc),
738− ttm_pool_shrinker_scan(&mm_shrinker, &sc));
752+ seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(mm_shrinker, &sc),
753+ ttm_pool_shrinker_scan(mm_shrinker, &sc));
739754 fs_reclaim_release(GFP_KERNEL);
740755
741756 return 0;
@@ -779,10 +794,17 @@ int ttm_pool_mgr_init(unsigned long num_pages) · 37 unmodified lines
779794 &ttm_pool_debugfs_shrink_fops);
780795 #endif
781796
782− mm_shrinker.count_objects = ttm_pool_shrinker_count;
783− mm_shrinker.scan_objects = ttm_pool_shrinker_scan;
784− mm_shrinker.seeks = 1;
785− return register_shrinker(&mm_shrinker, "drm-ttm_pool");
797+ mm_shrinker = shrinker_alloc(0, "drm-ttm_pool");
798+ if (!mm_shrinker)
799+ return -ENOMEM;
800+
801+ mm_shrinker->count_objects = ttm_pool_shrinker_count;
802+ mm_shrinker->scan_objects = ttm_pool_shrinker_scan;
803+ mm_shrinker->seeks = 1;
804+
805+ shrinker_register(mm_shrinker);
806+
807+ return 0;
786808 }
787809
788810 /**
@@ -802,6 +824,6 @@ void ttm_pool_mgr_fini(void) · 13 unmodified lines
802824 ttm_pool_type_fini(&global_dma32_uncached[i]);
803825 }
804826
805− unregister_shrinker(&mm_shrinker);
827+ shrinker_free(mm_shrinker);
806828 WARN_ON(!list_empty(&shrinker_list));
807829 }
drivers/md/bcache/bcache.h+1−1
@@ -543,7 +543,7 @@ struct cache_set { · 542 unmodified lines
543543 struct bio_set bio_split;
544544
545545 /* For the btree cache */
546− struct shrinker shrink;
546+ struct shrinker *shrink;
547547
548548 /* For the btree cache and anything allocation related */
549549 struct mutex bucket_lock;
drivers/md/bcache/btree.c+16−11
@@ -667,7 +667,7 @@ out_unlock: · 666 unmodified lines
667667 static unsigned long bch_mca_scan(struct shrinker *shrink,
668668 struct shrink_control *sc)
669669 {
670− struct cache_set *c = container_of(shrink, struct cache_set, shrink);
670+ struct cache_set *c = shrink->private_data;
671671 struct btree *b, *t;
672672 unsigned long i, nr = sc->nr_to_scan;
673673 unsigned long freed = 0;
@@ -734,7 +734,7 @@ out: · 60 unmodified lines
734734 static unsigned long bch_mca_count(struct shrinker *shrink,
735735 struct shrink_control *sc)
736736 {
737− struct cache_set *c = container_of(shrink, struct cache_set, shrink);
737+ struct cache_set *c = shrink->private_data;
738738
739739 if (c->shrinker_disabled)
740740 return 0;
@@ -752,8 +752,8 @@ void bch_btree_cache_free(struct cache_set *c) · 11 unmodified lines
752752
753753 closure_init_stack(&cl);
754754
755− if (c->shrink.list.next)
756− unregister_shrinker(&c->shrink);
755+ if (c->shrink)
756+ shrinker_free(c->shrink);
757757
758758 mutex_lock(&c->bucket_lock);
759759
@@ -828,14 +828,19 @@ int bch_btree_cache_alloc(struct cache_set *c) · 68 unmodified lines
828828 c->verify_data = NULL;
829829 #endif
830830
831− c->shrink.count_objects = bch_mca_count;
832− c->shrink.scan_objects = bch_mca_scan;
833− c->shrink.seeks = 4;
834− c->shrink.batch = c->btree_pages * 2;
831+ c->shrink = shrinker_alloc(0, "md-bcache:%pU", c->set_uuid);
832+ if (!c->shrink) {
833+ pr_warn("bcache: %s: could not allocate shrinker\n", __func__);
834+ return 0;
835+ }
836+
837+ c->shrink->count_objects = bch_mca_count;
838+ c->shrink->scan_objects = bch_mca_scan;
839+ c->shrink->seeks = 4;
840+ c->shrink->batch = c->btree_pages * 2;
841+ c->shrink->private_data = c;
835842
836− if (register_shrinker(&c->shrink, "md-bcache:%pU", c->set_uuid))
837− pr_warn("bcache: %s: could not register shrinker\n",
838− __func__);
843+ shrinker_register(c->shrink);
839844
840845 return 0;
841846 }
drivers/md/bcache/sysfs.c+2−1
@@ -866,7 +866,8 @@ STORE(__bch_cache_set) · 865 unmodified lines
866866
867867 sc.gfp_mask = GFP_KERNEL;
868868 sc.nr_to_scan = strtoul_or_return(buf);
869− c->shrink.scan_objects(&c->shrink, &sc);
869+ if (c->shrink)
870+ c->shrink->scan_objects(c->shrink, &sc);
870871 }
871872
872873 sysfs_strtoul_clamp(congested_read_threshold_us,
drivers/md/dm-bufio.c+17−11
@@ -963,7 +963,7 @@ struct dm_bufio_client { · 962 unmodified lines
963963
964964 sector_t start;
965965
966− struct shrinker shrinker;
966+ struct shrinker *shrinker;
967967 struct work_struct shrink_work;
968968 atomic_long_t need_shrink;
969969
@@ -2368,7 +2368,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 1398 unmodified lines
23682368 {
23692369 struct dm_bufio_client *c;
23702370
2371− c = container_of(shrink, struct dm_bufio_client, shrinker);
2371+ c = shrink->private_data;
23722372 atomic_long_add(sc->nr_to_scan, &c->need_shrink);
23732373 queue_work(dm_bufio_wq, &c->shrink_work);
23742374
@@ -2377,7 +2377,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 2 unmodified lines
23772377
23782378 static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
23792379 {
2380− struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker);
2380+ struct dm_bufio_client *c = shrink->private_data;
23812381 unsigned long count = cache_total(&c->cache);
23822382 unsigned long retain_target = get_retain_buffers(c);
23832383 unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink);
@@ -2490,14 +2490,20 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign · 106 unmodified lines
24902490 INIT_WORK(&c->shrink_work, shrink_work);
24912491 atomic_long_set(&c->need_shrink, 0);
24922492
2493− c->shrinker.count_objects = dm_bufio_shrink_count;
2494− c->shrinker.scan_objects = dm_bufio_shrink_scan;
2495− c->shrinker.seeks = 1;
2496− c->shrinker.batch = 0;
2497− r = register_shrinker(&c->shrinker, "dm-bufio:(%u:%u)",
2498− MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));
2499− if (r)
2493+ c->shrinker = shrinker_alloc(0, "dm-bufio:(%u:%u)",
2494+ MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));
2495+ if (!c->shrinker) {
2496+ r = -ENOMEM;
25002497 goto bad;
2498+ }
2499+
2500+ c->shrinker->count_objects = dm_bufio_shrink_count;
2501+ c->shrinker->scan_objects = dm_bufio_shrink_scan;
2502+ c->shrinker->seeks = 1;
2503+ c->shrinker->batch = 0;
2504+ c->shrinker->private_data = c;
2505+
2506+ shrinker_register(c->shrinker);
25012507
25022508 mutex_lock(&dm_bufio_clients_lock);
25032509 dm_bufio_client_count++;
@@ -2537,7 +2543,7 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c) · 33 unmodified lines
25372543
25382544 drop_buffers(c);
25392545
2540− unregister_shrinker(&c->shrinker);
2546+ shrinker_free(c->shrinker);
25412547 flush_work(&c->shrink_work);
25422548
25432549 mutex_lock(&dm_bufio_clients_lock);
drivers/md/dm-cache-metadata.c+1−1
@@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) · 1827 unmodified lines
18281828 * Replacement block manager (new_bm) is created and old_bm destroyed outside of
18291829 * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of
18301830 * shrinker associated with the block manager's bufio client vs cmd root_lock).
1831− * - must take shrinker_rwsem without holding cmd->root_lock
1831+ * - must take shrinker_mutex without holding cmd->root_lock
18321832 */
18331833 new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT,
18341834 CACHE_MAX_CONCURRENT_LOCKS);
drivers/md/dm-zoned-metadata.c+16−12
@@ -187,7 +187,7 @@ struct dmz_metadata { · 186 unmodified lines
187187 struct rb_root mblk_rbtree;
188188 struct list_head mblk_lru_list;
189189 struct list_head mblk_dirty_list;
190− struct shrinker mblk_shrinker;
190+ struct shrinker *mblk_shrinker;
191191
192192 /* Zone allocation management */
193193 struct mutex map_lock;
@@ -615,7 +615,7 @@ static unsigned long dmz_shrink_mblock_cache(struct dmz_metadata *zmd, · 421 unmodified lines
615615 static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink,
616616 struct shrink_control *sc)
617617 {
618− struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker);
618+ struct dmz_metadata *zmd = shrink->private_data;
619619
620620 return atomic_read(&zmd->nr_mblks);
621621 }
@@ -626,7 +626,7 @@ static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, · 4 unmodified lines
626626 static unsigned long dmz_mblock_shrinker_scan(struct shrinker *shrink,
627627 struct shrink_control *sc)
628628 {
629− struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker);
629+ struct dmz_metadata *zmd = shrink->private_data;
630630 unsigned long count;
631631
632632 spin_lock(&zmd->mblk_lock);
@@ -2936,19 +2936,23 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, · 2303 unmodified lines
29362936 */
29372937 zmd->min_nr_mblks = 2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks * 16;
29382938 zmd->max_nr_mblks = zmd->min_nr_mblks + 512;
2939− zmd->mblk_shrinker.count_objects = dmz_mblock_shrinker_count;
2940− zmd->mblk_shrinker.scan_objects = dmz_mblock_shrinker_scan;
2941− zmd->mblk_shrinker.seeks = DEFAULT_SEEKS;
29422939
29432940 /* Metadata cache shrinker */
2944− ret = register_shrinker(&zmd->mblk_shrinker, "dm-zoned-meta:(%u:%u)",
2945− MAJOR(dev->bdev->bd_dev),
2946− MINOR(dev->bdev->bd_dev));
2947− if (ret) {
2948− dmz_zmd_err(zmd, "Register metadata cache shrinker failed");
2941+ zmd->mblk_shrinker = shrinker_alloc(0, "dm-zoned-meta:(%u:%u)",
2942+ MAJOR(dev->bdev->bd_dev),
2943+ MINOR(dev->bdev->bd_dev));
2944+ if (!zmd->mblk_shrinker) {
2945+ ret = -ENOMEM;
2946+ dmz_zmd_err(zmd, "Allocate metadata cache shrinker failed");
29492947 goto err;
29502948 }
29512949
2950+ zmd->mblk_shrinker->count_objects = dmz_mblock_shrinker_count;
2951+ zmd->mblk_shrinker->scan_objects = dmz_mblock_shrinker_scan;
2952+ zmd->mblk_shrinker->private_data = zmd;
2953+
2954+ shrinker_register(zmd->mblk_shrinker);
2955+
29522956 dmz_zmd_info(zmd, "DM-Zoned metadata version %d", zmd->sb_version);
29532957 for (i = 0; i < zmd->nr_devs; i++)
29542958 dmz_print_dev(zmd, i);
@@ -2995,7 +2999,7 @@ err: · 40 unmodified lines
29952999 */
29963000 void dmz_dtr_metadata(struct dmz_metadata *zmd)
29973001 {
2998− unregister_shrinker(&zmd->mblk_shrinker);
3002+ shrinker_free(zmd->mblk_shrinker);
29993003 dmz_cleanup_metadata(zmd);
30003004 kfree(zmd);
30013005 }
drivers/md/raid5.c+15−11
@@ -7378,7 +7378,7 @@ static void free_conf(struct r5conf *conf) · 7377 unmodified lines
73787378
73797379 log_exit(conf);
73807380
7381− unregister_shrinker(&conf->shrinker);
7381+ shrinker_free(conf->shrinker);
73827382 free_thread_groups(conf);
73837383 shrink_stripes(conf);
73847384 raid5_free_percpu(conf);
@@ -7426,7 +7426,7 @@ static int raid5_alloc_percpu(struct r5conf *conf) · 41 unmodified lines
74267426 static unsigned long raid5_cache_scan(struct shrinker *shrink,
74277427 struct shrink_control *sc)
74287428 {
7429− struct r5conf *conf = container_of(shrink, struct r5conf, shrinker);
7429+ struct r5conf *conf = shrink->private_data;
74307430 unsigned long ret = SHRINK_STOP;
74317431
74327432 if (mutex_trylock(&conf->cache_size_mutex)) {
@@ -7447,7 +7447,7 @@ static unsigned long raid5_cache_scan(struct shrinker *shrink, · 14 unmodified lines
74477447 static unsigned long raid5_cache_count(struct shrinker *shrink,
74487448 struct shrink_control *sc)
74497449 {
7450− struct r5conf *conf = container_of(shrink, struct r5conf, shrinker);
7450+ struct r5conf *conf = shrink->private_data;
74517451
74527452 if (conf->max_nr_stripes < conf->min_nr_stripes)
74537453 /* unlikely, but not impossible */
@@ -7682,18 +7682,22 @@ static struct r5conf *setup_conf(struct mddev *mddev) · 228 unmodified lines
76827682 * it reduces the queue depth and so can hurt throughput.
76837683 * So set it rather large, scaled by number of devices.
76847684 */
7685− conf->shrinker.seeks = DEFAULT_SEEKS * conf->raid_disks * 4;
7686− conf->shrinker.scan_objects = raid5_cache_scan;
7687− conf->shrinker.count_objects = raid5_cache_count;
7688− conf->shrinker.batch = 128;
7689− conf->shrinker.flags = 0;
7690− ret = register_shrinker(&conf->shrinker, "md-raid5:%s", mdname(mddev));
7691− if (ret) {
7692− pr_warn("md/raid:%s: couldn't register shrinker.\n",
7685+ conf->shrinker = shrinker_alloc(0, "md-raid5:%s", mdname(mddev));
7686+ if (!conf->shrinker) {
7687+ ret = -ENOMEM;
7688+ pr_warn("md/raid:%s: couldn't allocate shrinker.\n",
76937689 mdname(mddev));
76947690 goto abort;
76957691 }
76967692
7693+ conf->shrinker->seeks = DEFAULT_SEEKS * conf->raid_disks * 4;
7694+ conf->shrinker->scan_objects = raid5_cache_scan;
7695+ conf->shrinker->count_objects = raid5_cache_count;
7696+ conf->shrinker->batch = 128;
7697+ conf->shrinker->private_data = conf;
7698+
7699+ shrinker_register(conf->shrinker);
7700+
76977701 sprintf(pers_name, "raid%d", mddev->new_level);
76987702 rcu_assign_pointer(conf->thread,
76997703 md_register_thread(raid5d, mddev, pers_name));
drivers/md/raid5.h+1−1
@@ -670,7 +670,7 @@ struct r5conf { · 669 unmodified lines
670670 wait_queue_head_t wait_for_stripe;
671671 wait_queue_head_t wait_for_overlap;
672672 unsigned long cache_state;
673− struct shrinker shrinker;
673+ struct shrinker *shrinker;
674674 int pool_size; /* number of disks in stripeheads in pool */
675675 spinlock_t device_lock;
676676 struct disk_info *disks;
drivers/misc/vmw_balloon.c+12−26
@@ -380,16 +380,7 @@ struct vmballoon { · 379 unmodified lines
380380 /**
381381 * @shrinker: shrinker interface that is used to avoid over-inflation.
382382 */
383− struct shrinker shrinker;
384−
385− /**
386− * @shrinker_registered: whether the shrinker was registered.
387− *
388− * The shrinker interface does not handle gracefully the removal of
389− * shrinker that was not registered before. This indication allows to
390− * simplify the unregistration process.
391− */
392− bool shrinker_registered;
383+ struct shrinker *shrinker;
393384 };
394385
395386 static struct vmballoon balloon;
@@ -1568,29 +1559,27 @@ static unsigned long vmballoon_shrinker_count(struct shrinker *shrinker, · 1172 unmodified lines
15681559
15691560 static void vmballoon_unregister_shrinker(struct vmballoon *b)
15701561 {
1571− if (b->shrinker_registered)
1572− unregister_shrinker(&b->shrinker);
1573− b->shrinker_registered = false;
1562+ shrinker_free(b->shrinker);
1563+ b->shrinker = NULL;
15741564 }
15751565
15761566 static int vmballoon_register_shrinker(struct vmballoon *b)
15771567 {
1578− int r;
1579−
15801568 /* Do nothing if the shrinker is not enabled */
15811569 if (!vmwballoon_shrinker_enable)
15821570 return 0;
15831571
1584− b->shrinker.scan_objects = vmballoon_shrinker_scan;
1585− b->shrinker.count_objects = vmballoon_shrinker_count;
1586− b->shrinker.seeks = DEFAULT_SEEKS;
1572+ b->shrinker = shrinker_alloc(0, "vmw-balloon");
1573+ if (!b->shrinker)
1574+ return -ENOMEM;
15871575
1588− r = register_shrinker(&b->shrinker, "vmw-balloon");
1576+ b->shrinker->scan_objects = vmballoon_shrinker_scan;
1577+ b->shrinker->count_objects = vmballoon_shrinker_count;
1578+ b->shrinker->private_data = b;
15891579
1590− if (r == 0)
1591− b->shrinker_registered = true;
1580+ shrinker_register(b->shrinker);
15921581
1593− return r;
1582+ return 0;
15941583 }
15951584
15961585 /*
@@ -1883,7 +1872,7 @@ static int __init vmballoon_init(void) · 286 unmodified lines
18831872
18841873 error = vmballoon_register_shrinker(&balloon);
18851874 if (error)
1886− goto fail;
1875+ return error;
18871876
18881877 /*
18891878 * Initialization of compaction must be done after the call to
@@ -1905,9 +1894,6 @@ static int __init vmballoon_init(void) · 15 unmodified lines
19051894 vmballoon_debugfs_init(&balloon);
19061895
19071896 return 0;
1908−fail:
1909− vmballoon_unregister_shrinker(&balloon);
1910− return error;
19111897 }
19121898
19131899 /*
drivers/virtio/virtio_balloon.c+14−10
@@ -111,7 +111,7 @@ struct virtio_balloon { · 110 unmodified lines
111111 struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
112112
113113 /* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */
114− struct shrinker shrinker;
114+ struct shrinker *shrinker;
115115
116116 /* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */
117117 struct notifier_block oom_nb;
@@ -820,8 +820,7 @@ static unsigned long shrink_free_pages(struct virtio_balloon *vb, · 702 unmodified lines
820820 static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker,
821821 struct shrink_control *sc)
822822 {
823− struct virtio_balloon *vb = container_of(shrinker,
824− struct virtio_balloon, shrinker);
823+ struct virtio_balloon *vb = shrinker->private_data;
825824
826825 return shrink_free_pages(vb, sc->nr_to_scan);
827826 }
@@ -829,8 +828,7 @@ static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, · 1 unmodified line
829828 static unsigned long virtio_balloon_shrinker_count(struct shrinker *shrinker,
830829 struct shrink_control *sc)
831830 {
832− struct virtio_balloon *vb = container_of(shrinker,
833− struct virtio_balloon, shrinker);
831+ struct virtio_balloon *vb = shrinker->private_data;
834832
835833 return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES;
836834 }
@@ -851,16 +849,22 @@ static int virtio_balloon_oom_notify(struct notifier_block *nb, · 14 unmodified lines
851849
852850 static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb)
853851 {
854− unregister_shrinker(&vb->shrinker);
852+ shrinker_free(vb->shrinker);
855853 }
856854
857855 static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)
858856 {
859− vb->shrinker.scan_objects = virtio_balloon_shrinker_scan;
860− vb->shrinker.count_objects = virtio_balloon_shrinker_count;
861− vb->shrinker.seeks = DEFAULT_SEEKS;
857+ vb->shrinker = shrinker_alloc(0, "virtio-balloon");
858+ if (!vb->shrinker)
859+ return -ENOMEM;
862860
863− return register_shrinker(&vb->shrinker, "virtio-balloon");
861+ vb->shrinker->scan_objects = virtio_balloon_shrinker_scan;
862+ vb->shrinker->count_objects = virtio_balloon_shrinker_count;
863+ vb->shrinker->private_data = vb;
864+
865+ shrinker_register(vb->shrinker);
866+
867+ return 0;
864868 }
865869
866870 static int virtballoon_probe(struct virtio_device *vdev)
drivers/xen/xenbus/xenbus_probe_backend.c+10−7
@@ -284,13 +284,9 @@ static unsigned long backend_shrink_memory_count(struct shrinker *shrinker, · 283 unmodified lines
284284 return 0;
285285 }
286286
287−static struct shrinker backend_memory_shrinker = {
288− .count_objects = backend_shrink_memory_count,
289− .seeks = DEFAULT_SEEKS,
290−};
291−
292287 static int __init xenbus_probe_backend_init(void)
293288 {
289+ struct shrinker *backend_memory_shrinker;
294290 static struct notifier_block xenstore_notifier = {
295291 .notifier_call = backend_probe_and_watch
296292 };
@@ -305,8 +301,15 @@ static int __init xenbus_probe_backend_init(void) · 8 unmodified lines
305301
306302 register_xenstore_notifier(&xenstore_notifier);
307303
308− if (register_shrinker(&backend_memory_shrinker, "xen-backend"))
309− pr_warn("shrinker registration failed\n");
304+ backend_memory_shrinker = shrinker_alloc(0, "xen-backend");
305+ if (!backend_memory_shrinker) {
306+ pr_warn("shrinker allocation failed\n");
307+ return 0;
308+ }
309+
310+ backend_memory_shrinker->count_objects = backend_shrink_memory_count;
311+
312+ shrinker_register(backend_memory_shrinker);
310313
311314 return 0;
312315 }
fs/bcachefs/btree_cache.c+12−10
@@ -285,8 +285,7 @@ static int btree_node_write_and_reclaim(struct bch_fs *c, struct btree *b) · 284 unmodified lines
285285 static unsigned long bch2_btree_cache_scan(struct shrinker *shrink,
286286 struct shrink_control *sc)
287287 {
288− struct bch_fs *c = container_of(shrink, struct bch_fs,
289− btree_cache.shrink);
288+ struct bch_fs *c = shrink->private_data;
290289 struct btree_cache *bc = &c->btree_cache;
291290 struct btree *b, *t;
292291 unsigned long nr = sc->nr_to_scan;
@@ -384,8 +383,7 @@ out_nounlock: · 91 unmodified lines
384383 static unsigned long bch2_btree_cache_count(struct shrinker *shrink,
385384 struct shrink_control *sc)
386385 {
387− struct bch_fs *c = container_of(shrink, struct bch_fs,
388− btree_cache.shrink);
386+ struct bch_fs *c = shrink->private_data;
389387 struct btree_cache *bc = &c->btree_cache;
390388
391389 if (bch2_btree_shrinker_disabled)
@@ -400,7 +398,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 8 unmodified lines
400398 struct btree *b;
401399 unsigned i, flags;
402400
403− unregister_shrinker(&bc->shrink);
401+ shrinker_free(bc->shrink);
404402
405403 /* vfree() can allocate memory: */
406404 flags = memalloc_nofs_save();
@@ -454,6 +452,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 47 unmodified lines
454452 int bch2_fs_btree_cache_init(struct bch_fs *c)
455453 {
456454 struct btree_cache *bc = &c->btree_cache;
455+ struct shrinker *shrink;
457456 unsigned i;
458457 int ret = 0;
459458
@@ -473,12 +472,15 @@ int bch2_fs_btree_cache_init(struct bch_fs *c) · 13 unmodified lines
473472
474473 mutex_init(&c->verify_lock);
475474
476− bc->shrink.count_objects = bch2_btree_cache_count;
477− bc->shrink.scan_objects = bch2_btree_cache_scan;
478− bc->shrink.seeks = 4;
479− ret = register_shrinker(&bc->shrink, "%s/btree_cache", c->name);
480− if (ret)
475+ shrink = shrinker_alloc(0, "%s/btree_cache", c->name);
476+ if (!shrink)
481477 goto err;
478+ bc->shrink = shrink;
479+ shrink->count_objects = bch2_btree_cache_count;
480+ shrink->scan_objects = bch2_btree_cache_scan;
481+ shrink->seeks = 4;
482+ shrink->private_data = c;
483+ shrinker_register(shrink);
482484
483485 return 0;
484486 err:
fs/bcachefs/btree_key_cache.c+12−9
@@ -834,8 +834,7 @@ void bch2_btree_key_cache_drop(struct btree_trans *trans, · 833 unmodified lines
834834 static unsigned long bch2_btree_key_cache_scan(struct shrinker *shrink,
835835 struct shrink_control *sc)
836836 {
837− struct bch_fs *c = container_of(shrink, struct bch_fs,
838− btree_key_cache.shrink);
837+ struct bch_fs *c = shrink->private_data;
839838 struct btree_key_cache *bc = &c->btree_key_cache;
840839 struct bucket_table *tbl;
841840 struct bkey_cached *ck, *t;
@@ -932,8 +931,7 @@ out: · 90 unmodified lines
932931 static unsigned long bch2_btree_key_cache_count(struct shrinker *shrink,
933932 struct shrink_control *sc)
934933 {
935− struct bch_fs *c = container_of(shrink, struct bch_fs,
936− btree_key_cache.shrink);
934+ struct bch_fs *c = shrink->private_data;
937935 struct btree_key_cache *bc = &c->btree_key_cache;
938936 long nr = atomic_long_read(&bc->nr_keys) -
939937 atomic_long_read(&bc->nr_dirty);
@@ -953,7 +951,7 @@ void bch2_fs_btree_key_cache_exit(struct btree_key_cache *bc) · 13 unmodified lines
953951 int cpu;
954952 #endif
955953
956− unregister_shrinker(&bc->shrink);
954+ shrinker_free(bc->shrink);
957955
958956 mutex_lock(&bc->lock);
959957
@@ -1027,6 +1025,7 @@ void bch2_fs_btree_key_cache_init_early(struct btree_key_cache *c) · 67 unmodified lines
10271025 int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc)
10281026 {
10291027 struct bch_fs *c = container_of(bc, struct bch_fs, btree_key_cache);
1028+ struct shrinker *shrink;
10301029
10311030 #ifdef __KERNEL__
10321031 bc->pcpu_freed = alloc_percpu(struct btree_key_cache_freelist);
@@ -1039,11 +1038,15 @@ int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc) · 6 unmodified lines
10391038
10401039 bc->table_init_done = true;
10411040
1042− bc->shrink.seeks = 0;
1043− bc->shrink.count_objects = bch2_btree_key_cache_count;
1044− bc->shrink.scan_objects = bch2_btree_key_cache_scan;
1045− if (register_shrinker(&bc->shrink, "%s/btree_key_cache", c->name))
1041+ shrink = shrinker_alloc(0, "%s/btree_key_cache", c->name);
1042+ if (!shrink)
10461043 return -BCH_ERR_ENOMEM_fs_btree_cache_init;
1044+ bc->shrink = shrink;
1045+ shrink->seeks = 0;
1046+ shrink->count_objects = bch2_btree_key_cache_count;
1047+ shrink->scan_objects = bch2_btree_key_cache_scan;
1048+ shrink->private_data = c;
1049+ shrinker_register(shrink);
10471050 return 0;
10481051 }
10491052
fs/bcachefs/btree_types.h+2−2
@@ -163,7 +163,7 @@ struct btree_cache { · 162 unmodified lines
163163 unsigned used;
164164 unsigned reserve;
165165 atomic_t dirty;
166− struct shrinker shrink;
166+ struct shrinker *shrink;
167167
168168 /*
169169 * If we need to allocate memory for a new btree node and that
@@ -321,7 +321,7 @@ struct btree_key_cache { · 151 unmodified lines
321321 bool table_init_done;
322322 struct list_head freed_pcpu;
323323 struct list_head freed_nonpcpu;
324− struct shrinker shrink;
324+ struct shrinker *shrink;
325325 unsigned shrink_iter;
326326 struct btree_key_cache_freelist __percpu *pcpu_freed;
327327
fs/bcachefs/fs.c+1−1
@@ -1904,7 +1904,7 @@ got_sb: · 1903 unmodified lines
19041904 sb->s_flags |= SB_POSIXACL;
19051905 #endif
19061906
1907− sb->s_shrink.seeks = 0;
1907+ sb->s_shrink->seeks = 0;
19081908
19091909 vinode = bch2_vfs_inode_get(c, BCACHEFS_ROOT_SUBVOL_INUM);
19101910 ret = PTR_ERR_OR_ZERO(vinode);
fs/bcachefs/sysfs.c+1−1
@@ -494,7 +494,7 @@ STORE(bch2_fs) · 493 unmodified lines
494494
495495 sc.gfp_mask = GFP_KERNEL;
496496 sc.nr_to_scan = strtoul_or_return(buf);
497− c->btree_cache.shrink.scan_objects(&c->btree_cache.shrink, &sc);
497+ c->btree_cache.shrink->scan_objects(c->btree_cache.shrink, &sc);
498498 }
499499
500500 if (attr == &sysfs_btree_wakeup)
fs/btrfs/super.c+1−1
@@ -1472,7 +1472,7 @@ static struct dentry *btrfs_mount_root(struct file_system_type *fs_type, · 1471 unmodified lines
14721472 error = -EBUSY;
14731473 } else {
14741474 snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev);
1475− shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", fs_type->name,
1475+ shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fs_type->name,
14761476 s->s_id);
14771477 btrfs_sb(s)->bdev_holder = fs_type;
14781478 error = btrfs_fill_super(s, fs_devices, data);
fs/buffer.c+60−65
@@ -282,13 +282,7 @@ static void end_buffer_async_read(struct buffer_head *bh, int uptodate) · 281 unmodified lines
282282 } while (tmp != bh);
283283 spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
284284
285− /*
286− * If all of the buffers are uptodate then we can set the page
287− * uptodate.
288− */
289− if (folio_uptodate)
290− folio_mark_uptodate(folio);
291− folio_unlock(folio);
285+ folio_end_read(folio, folio_uptodate);
292286 return;
293287
294288 still_busy:
@@ -915,16 +909,12 @@ int remove_inode_buffers(struct inode *inode) · 620 unmodified lines
915909 * which may not fail from ordinary buffer allocations.
916910 */
917911 struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
918− bool retry)
912+ gfp_t gfp)
919913 {
920914 struct buffer_head *bh, *head;
921− gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
922915 long offset;
923916 struct mem_cgroup *memcg, *old_memcg;
924917
925− if (retry)
926− gfp |= __GFP_NOFAIL;
927−
928918 /* The folio lock pins the memcg */
929919 memcg = folio_memcg(folio);
930920 old_memcg = set_active_memcg(memcg);
@@ -967,7 +957,11 @@ EXPORT_SYMBOL_GPL(folio_alloc_buffers); · 36 unmodified lines
967957 struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
968958 bool retry)
969959 {
970− return folio_alloc_buffers(page_folio(page), size, retry);
960+ gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
961+ if (retry)
962+ gfp |= __GFP_NOFAIL;
963+
964+ return folio_alloc_buffers(page_folio(page), size, gfp);
971965 }
972966 EXPORT_SYMBOL_GPL(alloc_page_buffers);
973967
@@ -1043,20 +1037,11 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 69 unmodified lines
10431037 struct buffer_head *bh;
10441038 sector_t end_block;
10451039 int ret = 0;
1046− gfp_t gfp_mask;
1047−
1048− gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp;
1049−
1050− /*
1051− * XXX: __getblk_slow() can not really deal with failure and
1052− * will endlessly loop on improvised global reclaim. Prefer
1053− * looping in the allocator rather than here, at least that
1054− * code knows what it's doing.
1055− */
1056− gfp_mask |= __GFP_NOFAIL;
10571040
10581041 folio = __filemap_get_folio(inode->i_mapping, index,
1059− FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp_mask);
1042+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
1043+ if (IS_ERR(folio))
1044+ return PTR_ERR(folio);
10601045
10611046 bh = folio_buffers(folio);
10621047 if (bh) {
@@ -1069,7 +1054,10 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 6 unmodified lines
10691054 goto failed;
10701055 }
10711056
1072− bh = folio_alloc_buffers(folio, size, true);
1057+ ret = -ENOMEM;
1058+ bh = folio_alloc_buffers(folio, size, gfp | __GFP_ACCOUNT);
1059+ if (!bh)
1060+ goto failed;
10731061
10741062 /*
10751063 * Link the folio to the buffers and initialise them. Take the
@@ -1420,33 +1408,36 @@ __find_get_block(struct block_device *bdev, sector_t block, unsigned size) · 344 unmodified lines
14201408 }
14211409 EXPORT_SYMBOL(__find_get_block);
14221410
1423−/*
1424− * __getblk_gfp() will locate (and, if necessary, create) the buffer_head
1425− * which corresponds to the passed block_device, block and size. The
1426− * returned buffer has its reference count incremented.
1411+/**
1412+ * bdev_getblk - Get a buffer_head in a block device's buffer cache.
1413+ * @bdev: The block device.
1414+ * @block: The block number.
1415+ * @size: The size of buffer_heads for this @bdev.
1416+ * @gfp: The memory allocation flags to use.
14271417 *
1428− * __getblk_gfp() will lock up the machine if grow_dev_page's
1429− * try_to_free_buffers() attempt is failing. FIXME, perhaps?
1418+ * Return: The buffer head, or NULL if memory could not be allocated.
14301419 */
1431−struct buffer_head *
1432−__getblk_gfp(struct block_device *bdev, sector_t block,
1433− unsigned size, gfp_t gfp)
1420+struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
1421+ unsigned size, gfp_t gfp)
14341422 {
14351423 struct buffer_head *bh = __find_get_block(bdev, block, size);
14361424
1437− might_sleep();
1438− if (bh == NULL)
1439− bh = __getblk_slow(bdev, block, size, gfp);
1440− return bh;
1425+ might_alloc(gfp);
1426+ if (bh)
1427+ return bh;
1428+
1429+ return __getblk_slow(bdev, block, size, gfp);
14411430 }
1442−EXPORT_SYMBOL(__getblk_gfp);
1431+EXPORT_SYMBOL(bdev_getblk);
14431432
14441433 /*
14451434 * Do async read-ahead on a buffer..
14461435 */
14471436 void __breadahead(struct block_device *bdev, sector_t block, unsigned size)
14481437 {
1449− struct buffer_head *bh = __getblk(bdev, block, size);
1438+ struct buffer_head *bh = bdev_getblk(bdev, block, size,
1439+ GFP_NOWAIT | __GFP_MOVABLE);
1440+
14501441 if (likely(bh)) {
14511442 bh_readahead(bh, REQ_RAHEAD);
14521443 brelse(bh);
@@ -1470,7 +1461,17 @@ struct buffer_head * · 17 unmodified lines
14701461 __bread_gfp(struct block_device *bdev, sector_t block,
14711462 unsigned size, gfp_t gfp)
14721463 {
1473− struct buffer_head *bh = __getblk_gfp(bdev, block, size, gfp);
1464+ struct buffer_head *bh;
1465+
1466+ gfp |= mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
1467+
1468+ /*
1469+ * Prefer looping in the allocator rather than here, at least that
1470+ * code knows what it's doing.
1471+ */
1472+ gfp |= __GFP_NOFAIL;
1473+
1474+ bh = bdev_getblk(bdev, block, size, gfp);
14741475
14751476 if (likely(bh) && !buffer_uptodate(bh))
14761477 bh = __bread_slow(bh);
@@ -1640,12 +1641,13 @@ EXPORT_SYMBOL(block_invalidate_folio); · 163 unmodified lines
16401641 * block_dirty_folio() via private_lock. try_to_free_buffers
16411642 * is already excluded via the folio lock.
16421643 */
1643−void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize,
1644− unsigned long b_state)
1644+struct buffer_head *create_empty_buffers(struct folio *folio,
1645+ unsigned long blocksize, unsigned long b_state)
16451646 {
16461647 struct buffer_head *bh, *head, *tail;
1648+ gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL;
16471649
1648− head = folio_alloc_buffers(folio, blocksize, true);
1650+ head = folio_alloc_buffers(folio, blocksize, gfp);
16491651 bh = head;
16501652 do {
16511653 bh->b_state |= b_state;
@@ -1667,13 +1669,8 @@ void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize, · 15 unmodified lines
16671669 }
16681670 folio_attach_private(folio, head);
16691671 spin_unlock(&folio->mapping->private_lock);
1670−}
1671−EXPORT_SYMBOL(folio_create_empty_buffers);
16721672
1673−void create_empty_buffers(struct page *page,
1674− unsigned long blocksize, unsigned long b_state)
1675−{
1676− folio_create_empty_buffers(page_folio(page), blocksize, b_state);
1673+ return head;
16771674 }
16781675 EXPORT_SYMBOL(create_empty_buffers);
16791676
@@ -1768,13 +1765,15 @@ static struct buffer_head *folio_create_buffers(struct folio *folio, · 88 unmodified lines
17681765 struct inode *inode,
17691766 unsigned int b_state)
17701767 {
1768+ struct buffer_head *bh;
1769+
17711770 BUG_ON(!folio_test_locked(folio));
17721771
1773− if (!folio_buffers(folio))
1774− folio_create_empty_buffers(folio,
1775− 1 << READ_ONCE(inode->i_blkbits),
1776− b_state);
1777− return folio_buffers(folio);
1772+ bh = folio_buffers(folio);
1773+ if (!bh)
1774+ bh = create_empty_buffers(folio,
1775+ 1 << READ_ONCE(inode->i_blkbits), b_state);
1776+ return bh;
17781777 }
17791778
17801779 /*
@@ -2425,12 +2424,10 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) · 644 unmodified lines
24252424
24262425 if (!nr) {
24272426 /*
2428− * All buffers are uptodate - we can set the folio uptodate
2429− * as well. But not if get_block() returned an error.
2427+ * All buffers are uptodate or get_block() returned an
2428+ * error when trying to map them - we can finish the read.
24302429 */
2431− if (!page_error)
2432− folio_mark_uptodate(folio);
2433− folio_unlock(folio);
2430+ folio_end_read(folio, !page_error);
24342431 return 0;
24352432 }
24362433
@@ -2676,10 +2673,8 @@ int block_truncate_page(struct address_space *mapping, · 239 unmodified lines
26762673 return PTR_ERR(folio);
26772674
26782675 bh = folio_buffers(folio);
2679− if (!bh) {
2680− folio_create_empty_buffers(folio, blocksize, 0);
2681− bh = folio_buffers(folio);
2682− }
2676+ if (!bh)
2677+ bh = create_empty_buffers(folio, blocksize, 0);
26832678
26842679 /* Find the buffer that contains "offset" */
26852680 offset = offset_in_folio(folio, from);
fs/dax.c+12−12
@@ -412,23 +412,23 @@ static struct page *dax_busy_page(void *entry) · 411 unmodified lines
412412 return NULL;
413413 }
414414
415−/*
416− * dax_lock_page - Lock the DAX entry corresponding to a page
417− * @page: The page whose entry we want to lock
415+/**
416+ * dax_lock_folio - Lock the DAX entry corresponding to a folio
417+ * @folio: The folio whose entry we want to lock
418418 *
419419 * Context: Process context.
420− * Return: A cookie to pass to dax_unlock_page() or 0 if the entry could
420+ * Return: A cookie to pass to dax_unlock_folio() or 0 if the entry could
421421 * not be locked.
422422 */
423−dax_entry_t dax_lock_page(struct page *page)
423+dax_entry_t dax_lock_folio(struct folio *folio)
424424 {
425425 XA_STATE(xas, NULL, 0);
426426 void *entry;
427427
428− /* Ensure page->mapping isn't freed while we look at it */
428+ /* Ensure folio->mapping isn't freed while we look at it */
429429 rcu_read_lock();
430430 for (;;) {
431− struct address_space *mapping = READ_ONCE(page->mapping);
431+ struct address_space *mapping = READ_ONCE(folio->mapping);
432432
433433 entry = NULL;
434434 if (!mapping || !dax_mapping(mapping))
@@ -447,11 +447,11 @@ dax_entry_t dax_lock_page(struct page *page) · 12 unmodified lines
447447
448448 xas.xa = &mapping->i_pages;
449449 xas_lock_irq(&xas);
450− if (mapping != page->mapping) {
450+ if (mapping != folio->mapping) {
451451 xas_unlock_irq(&xas);
452452 continue;
453453 }
454− xas_set(&xas, page->index);
454+ xas_set(&xas, folio->index);
455455 entry = xas_load(&xas);
456456 if (dax_is_locked(entry)) {
457457 rcu_read_unlock();
@@ -467,10 +467,10 @@ dax_entry_t dax_lock_page(struct page *page) · 9 unmodified lines
467467 return (dax_entry_t)entry;
468468 }
469469
470−void dax_unlock_page(struct page *page, dax_entry_t cookie)
470+void dax_unlock_folio(struct folio *folio, dax_entry_t cookie)
471471 {
472− struct address_space *mapping = page->mapping;
473− XA_STATE(xas, &mapping->i_pages, page->index);
472+ struct address_space *mapping = folio->mapping;
473+ XA_STATE(xas, &mapping->i_pages, folio->index);
474474
475475 if (S_ISCHR(mapping->host->i_mode))
476476 return;
fs/erofs/utils.c+12−7
@@ -264,19 +264,24 @@ static unsigned long erofs_shrink_scan(struct shrinker *shrink, · 263 unmodified lines
264264 return freed;
265265 }
266266
267−static struct shrinker erofs_shrinker_info = {
268− .scan_objects = erofs_shrink_scan,
269− .count_objects = erofs_shrink_count,
270− .seeks = DEFAULT_SEEKS,
271−};
267+static struct shrinker *erofs_shrinker_info;
272268
273269 int __init erofs_init_shrinker(void)
274270 {
275− return register_shrinker(&erofs_shrinker_info, "erofs-shrinker");
271+ erofs_shrinker_info = shrinker_alloc(0, "erofs-shrinker");
272+ if (!erofs_shrinker_info)
273+ return -ENOMEM;
274+
275+ erofs_shrinker_info->count_objects = erofs_shrink_count;
276+ erofs_shrinker_info->scan_objects = erofs_shrink_scan;
277+
278+ shrinker_register(erofs_shrinker_info);
279+
280+ return 0;
276281 }
277282
278283 void erofs_exit_shrinker(void)
279284 {
280− unregister_shrinker(&erofs_shrinker_info);
285+ shrinker_free(erofs_shrinker_info);
281286 }
282287 #endif /* !CONFIG_EROFS_FS_ZIP */
fs/exec.c+1−3
@@ -713,7 +713,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 712 unmodified lines
713713 * process cleanup to remove whatever mess we made.
714714 */
715715 if (length != move_page_tables(vma, old_start,
716− vma, new_start, length, false))
716+ vma, new_start, length, false, true))
717717 return -ENOMEM;
718718
719719 lru_add_drain();
@@ -986,8 +986,6 @@ static int exec_mmap(struct mm_struct *mm) · 266 unmodified lines
986986 tsk = current;
987987 old_mm = current->mm;
988988 exec_mm_release(tsk, old_mm);
989− if (old_mm)
990− sync_mm_rss(old_mm);
991989
992990 ret = down_write_killable(&tsk->signal->exec_update_lock);
993991 if (ret)
fs/ext4/ext4.h+1−1
@@ -1664,7 +1664,7 @@ struct ext4_sb_info { · 1663 unmodified lines
16641664 __u32 s_csum_seed;
16651665
16661666 /* Reclaim extents from extent status tree */
1667− struct shrinker s_es_shrinker;
1667+ struct shrinker *s_es_shrinker;
16681668 struct list_head s_es_list; /* List of inodes with reclaimable extents */
16691669 long s_es_nr_inode;
16701670 struct ext4_es_stats s_es_stats;
fs/ext4/extents_status.c+13−10
@@ -1632,7 +1632,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 1631 unmodified lines
16321632 unsigned long nr;
16331633 struct ext4_sb_info *sbi;
16341634
1635− sbi = container_of(shrink, struct ext4_sb_info, s_es_shrinker);
1635+ sbi = shrink->private_data;
16361636 nr = percpu_counter_read_positive(&sbi->s_es_stats.es_stats_shk_cnt);
16371637 trace_ext4_es_shrink_count(sbi->s_sb, sc->nr_to_scan, nr);
16381638 return nr;
@@ -1641,8 +1641,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 2 unmodified lines
16411641 static unsigned long ext4_es_scan(struct shrinker *shrink,
16421642 struct shrink_control *sc)
16431643 {
1644− struct ext4_sb_info *sbi = container_of(shrink,
1645− struct ext4_sb_info, s_es_shrinker);
1644+ struct ext4_sb_info *sbi = shrink->private_data;
16461645 int nr_to_scan = sc->nr_to_scan;
16471646 int ret, nr_shrunk;
16481647
@@ -1726,13 +1725,17 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi) · 77 unmodified lines
17261725 if (err)
17271726 goto err3;
17281727
1729− sbi->s_es_shrinker.scan_objects = ext4_es_scan;
1730− sbi->s_es_shrinker.count_objects = ext4_es_count;
1731− sbi->s_es_shrinker.seeks = DEFAULT_SEEKS;
1732− err = register_shrinker(&sbi->s_es_shrinker, "ext4-es:%s",
1733− sbi->s_sb->s_id);
1734− if (err)
1728+ sbi->s_es_shrinker = shrinker_alloc(0, "ext4-es:%s", sbi->s_sb->s_id);
1729+ if (!sbi->s_es_shrinker) {
1730+ err = -ENOMEM;
17351731 goto err4;
1732+ }
1733+
1734+ sbi->s_es_shrinker->scan_objects = ext4_es_scan;
1735+ sbi->s_es_shrinker->count_objects = ext4_es_count;
1736+ sbi->s_es_shrinker->private_data = sbi;
1737+
1738+ shrinker_register(sbi->s_es_shrinker);
17361739
17371740 return 0;
17381741 err4:
@@ -1752,7 +1755,7 @@ void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi) · 13 unmodified lines
17521755 percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses);
17531756 percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt);
17541757 percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt);
1755− unregister_shrinker(&sbi->s_es_shrinker);
1758+ shrinker_free(sbi->s_es_shrinker);
17561759 }
17571760
17581761 /*
fs/ext4/inode.c+5−9
@@ -1032,10 +1032,8 @@ static int ext4_block_write_begin(struct folio *folio, loff_t pos, unsigned len, · 1031 unmodified lines
10321032 BUG_ON(from > to);
10331033
10341034 head = folio_buffers(folio);
1035− if (!head) {
1036− create_empty_buffers(&folio->page, blocksize, 0);
1037− head = folio_buffers(folio);
1038− }
1035+ if (!head)
1036+ head = create_empty_buffers(folio, blocksize, 0);
10391037 bbits = ilog2(blocksize);
10401038 block = (sector_t)folio->index << (PAGE_SHIFT - bbits);
10411039
@@ -1165,7 +1163,7 @@ retry_grab: · 123 unmodified lines
11651163 * starting the handle.
11661164 */
11671165 if (!folio_buffers(folio))
1168− create_empty_buffers(&folio->page, inode->i_sb->s_blocksize, 0);
1166+ create_empty_buffers(folio, inode->i_sb->s_blocksize, 0);
11691167
11701168 folio_unlock(folio);
11711169
@@ -3655,10 +3653,8 @@ static int __ext4_block_zero_page_range(handle_t *handle, · 2483 unmodified lines
36553653 iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);
36563654
36573655 bh = folio_buffers(folio);
3658− if (!bh) {
3659− create_empty_buffers(&folio->page, blocksize, 0);
3660− bh = folio_buffers(folio);
3661− }
3656+ if (!bh)
3657+ bh = create_empty_buffers(folio, blocksize, 0);
36623658
36633659 /* Find the buffer that contains "offset" */
36643660 pos = blocksize;
fs/ext4/move_extent.c+5−6
@@ -183,10 +183,8 @@ mext_page_mkuptodate(struct folio *folio, unsigned from, unsigned to) · 182 unmodified lines
183183
184184 blocksize = i_blocksize(inode);
185185 head = folio_buffers(folio);
186− if (!head) {
187− create_empty_buffers(&folio->page, blocksize, 0);
188− head = folio_buffers(folio);
189− }
186+ if (!head)
187+ head = create_empty_buffers(folio, blocksize, 0);
190188
191189 block = (sector_t)folio->index << (PAGE_SHIFT - inode->i_blkbits);
192190 for (bh = head, block_start = 0; bh != head || !block_start;
@@ -380,9 +378,10 @@ data_copy: · 187 unmodified lines
380378 }
381379 /* Perform all necessary steps similar write_begin()/write_end()
382380 * but keeping in mind that i_size will not change */
383− if (!folio_buffers(folio[0]))
384− create_empty_buffers(&folio[0]->page, 1 << orig_inode->i_blkbits, 0);
385381 bh = folio_buffers(folio[0]);
382+ if (!bh)
383+ bh = create_empty_buffers(folio[0],
384+ 1 << orig_inode->i_blkbits, 0);
386385 for (i = 0; i < data_offset_in_page; i++)
387386 bh = bh->b_this_page;
388387 for (i = 0; i < block_len_in_page; i++) {
fs/ext4/readpage.c+3−11
@@ -70,15 +70,8 @@ static void __read_end_io(struct bio *bio) · 69 unmodified lines
7070 {
7171 struct folio_iter fi;
7272
73− bio_for_each_folio_all(fi, bio) {
74− struct folio *folio = fi.folio;
75−
76− if (bio->bi_status)
77− folio_clear_uptodate(folio);
78− else
79− folio_mark_uptodate(folio);
80− folio_unlock(folio);
81− }
73+ bio_for_each_folio_all(fi, bio)
74+ folio_end_read(fi.folio, bio->bi_status == 0);
8275 if (bio->bi_private)
8376 mempool_free(bio->bi_private, bio_post_read_ctx_pool);
8477 bio_put(bio);
@@ -336,8 +329,7 @@ int ext4_mpage_readpages(struct inode *inode, · 251 unmodified lines
336329 if (ext4_need_verity(inode, folio->index) &&
337330 !fsverity_verify_folio(folio))
338331 goto set_error_page;
339− folio_mark_uptodate(folio);
340− folio_unlock(folio);
332+ folio_end_read(folio, true);
341333 continue;
342334 }
343335 } else if (fully_mapped) {
fs/ext4/super.c+10−3
@@ -244,18 +244,25 @@ static struct buffer_head *__ext4_sb_bread_gfp(struct super_block *sb, · 243 unmodified lines
244244 struct buffer_head *ext4_sb_bread(struct super_block *sb, sector_t block,
245245 blk_opf_t op_flags)
246246 {
247− return __ext4_sb_bread_gfp(sb, block, op_flags, __GFP_MOVABLE);
247+ gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping,
248+ ~__GFP_FS) | __GFP_MOVABLE;
249+
250+ return __ext4_sb_bread_gfp(sb, block, op_flags, gfp);
248251 }
249252
250253 struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb,
251254 sector_t block)
252255 {
253− return __ext4_sb_bread_gfp(sb, block, 0, 0);
256+ gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping,
257+ ~__GFP_FS);
258+
259+ return __ext4_sb_bread_gfp(sb, block, 0, gfp);
254260 }
255261
256262 void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block)
257263 {
258− struct buffer_head *bh = sb_getblk_gfp(sb, block, 0);
264+ struct buffer_head *bh = bdev_getblk(sb->s_bdev, block,
265+ sb->s_blocksize, GFP_NOWAIT | __GFP_NOWARN);
259266
260267 if (likely(bh)) {
261268 if (trylock_buffer(bh))
fs/f2fs/super.c+23−8
@@ -83,11 +83,26 @@ void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate, · 82 unmodified lines
8383 #endif
8484
8585 /* f2fs-wide shrinker description */
86−static struct shrinker f2fs_shrinker_info = {
87− .scan_objects = f2fs_shrink_scan,
88− .count_objects = f2fs_shrink_count,
89− .seeks = DEFAULT_SEEKS,
90−};
86+static struct shrinker *f2fs_shrinker_info;
87+
88+static int __init f2fs_init_shrinker(void)
89+{
90+ f2fs_shrinker_info = shrinker_alloc(0, "f2fs-shrinker");
91+ if (!f2fs_shrinker_info)
92+ return -ENOMEM;
93+
94+ f2fs_shrinker_info->count_objects = f2fs_shrink_count;
95+ f2fs_shrinker_info->scan_objects = f2fs_shrink_scan;
96+
97+ shrinker_register(f2fs_shrinker_info);
98+
99+ return 0;
100+}
101+
102+static void f2fs_exit_shrinker(void)
103+{
104+ shrinker_free(f2fs_shrinker_info);
105+}
91106
92107 enum {
93108 Opt_gc_background,
@@ -4940,7 +4955,7 @@ static int __init init_f2fs_fs(void) · 4846 unmodified lines
49404955 err = f2fs_init_sysfs();
49414956 if (err)
49424957 goto free_garbage_collection_cache;
4943− err = register_shrinker(&f2fs_shrinker_info, "f2fs-shrinker");
4958+ err = f2fs_init_shrinker();
49444959 if (err)
49454960 goto free_sysfs;
49464961 err = register_filesystem(&f2fs_fs_type);
@@ -4985,7 +5000,7 @@ free_root_stats: · 38 unmodified lines
49855000 f2fs_destroy_root_stats();
49865001 unregister_filesystem(&f2fs_fs_type);
49875002 free_shrinker:
4988− unregister_shrinker(&f2fs_shrinker_info);
5003+ f2fs_exit_shrinker();
49895004 free_sysfs:
49905005 f2fs_exit_sysfs();
49915006 free_garbage_collection_cache:
@@ -5017,7 +5032,7 @@ static void __exit exit_f2fs_fs(void) · 25 unmodified lines
50175032 f2fs_destroy_post_read_processing();
50185033 f2fs_destroy_root_stats();
50195034 unregister_filesystem(&f2fs_fs_type);
5020− unregister_shrinker(&f2fs_shrinker_info);
5035+ f2fs_exit_shrinker();
50215036 f2fs_exit_sysfs();
50225037 f2fs_destroy_garbage_collection_cache();
50235038 f2fs_destroy_extent_cache();
fs/gfs2/aops.c+1−1
@@ -130,7 +130,7 @@ static int __gfs2_jdata_write_folio(struct folio *folio, · 129 unmodified lines
130130 if (folio_test_checked(folio)) {
131131 folio_clear_checked(folio);
132132 if (!folio_buffers(folio)) {
133− folio_create_empty_buffers(folio,
133+ create_empty_buffers(folio,
134134 inode->i_sb->s_blocksize,
135135 BIT(BH_Dirty)|BIT(BH_Uptodate));
136136 }
fs/gfs2/bmap.c+23−25
@@ -43,53 +43,51 @@ struct metapath { · 42 unmodified lines
4343 static int punch_hole(struct gfs2_inode *ip, u64 offset, u64 length);
4444
4545 /**
46− * gfs2_unstuffer_page - unstuff a stuffed inode into a block cached by a page
46+ * gfs2_unstuffer_folio - unstuff a stuffed inode into a block cached by a folio
4747 * @ip: the inode
4848 * @dibh: the dinode buffer
4949 * @block: the block number that was allocated
50− * @page: The (optional) page. This is looked up if @page is NULL
50+ * @folio: The folio.
5151 *
5252 * Returns: errno
5353 */
54−
55−static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh,
56− u64 block, struct page *page)
54+static int gfs2_unstuffer_folio(struct gfs2_inode *ip, struct buffer_head *dibh,
55+ u64 block, struct folio *folio)
5756 {
5857 struct inode *inode = &ip->i_inode;
5958
60− if (!PageUptodate(page)) {
61− void *kaddr = kmap(page);
59+ if (!folio_test_uptodate(folio)) {
60+ void *kaddr = kmap_local_folio(folio, 0);
6261 u64 dsize = i_size_read(inode);
6362
6463 memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);
65− memset(kaddr + dsize, 0, PAGE_SIZE - dsize);
66− kunmap(page);
64+ memset(kaddr + dsize, 0, folio_size(folio) - dsize);
65+ kunmap_local(kaddr);
6766
68− SetPageUptodate(page);
67+ folio_mark_uptodate(folio);
6968 }
7069
7170 if (gfs2_is_jdata(ip)) {
72− struct buffer_head *bh;
71+ struct buffer_head *bh = folio_buffers(folio);
7372
74− if (!page_has_buffers(page))
75− create_empty_buffers(page, BIT(inode->i_blkbits),
76− BIT(BH_Uptodate));
73+ if (!bh)
74+ bh = create_empty_buffers(folio,
75+ BIT(inode->i_blkbits), BIT(BH_Uptodate));
7776
78− bh = page_buffers(page);
7977 if (!buffer_mapped(bh))
8078 map_bh(bh, inode->i_sb, block);
8179
8280 set_buffer_uptodate(bh);
8381 gfs2_trans_add_data(ip->i_gl, bh);
8482 } else {
85− set_page_dirty(page);
83+ folio_mark_dirty(folio);
8684 gfs2_ordered_add_inode(ip);
8785 }
8886
8987 return 0;
9088 }
9189
92−static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page)
90+static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct folio *folio)
9391 {
9492 struct buffer_head *bh, *dibh;
9593 struct gfs2_dinode *di;
@@ -118,7 +116,7 @@ static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page) · 22 unmodified lines
118116 dibh, sizeof(struct gfs2_dinode));
119117 brelse(bh);
120118 } else {
121− error = gfs2_unstuffer_page(ip, dibh, block, page);
119+ error = gfs2_unstuffer_folio(ip, dibh, block, folio);
122120 if (error)
123121 goto out_brelse;
124122 }
@@ -157,17 +155,17 @@ out_brelse: · 32 unmodified lines
157155 int gfs2_unstuff_dinode(struct gfs2_inode *ip)
158156 {
159157 struct inode *inode = &ip->i_inode;
160− struct page *page;
158+ struct folio *folio;
161159 int error;
162160
163161 down_write(&ip->i_rw_mutex);
164− page = grab_cache_page(inode->i_mapping, 0);
165− error = -ENOMEM;
166− if (!page)
162+ folio = filemap_grab_folio(inode->i_mapping, 0);
163+ error = PTR_ERR(folio);
164+ if (IS_ERR(folio))
167165 goto out;
168− error = __gfs2_unstuff_inode(ip, page);
169− unlock_page(page);
170− put_page(page);
166+ error = __gfs2_unstuff_inode(ip, folio);
167+ folio_unlock(folio);
168+ folio_put(folio);
171169 out:
172170 up_write(&ip->i_rw_mutex);
173171 return error;
fs/gfs2/glock.c+10−9
@@ -2041,11 +2041,7 @@ static unsigned long gfs2_glock_shrink_count(struct shrinker *shrink, · 2040 unmodified lines
20412041 return vfs_pressure_ratio(atomic_read(&lru_count));
20422042 }
20432043
2044−static struct shrinker glock_shrinker = {
2045− .seeks = DEFAULT_SEEKS,
2046− .count_objects = gfs2_glock_shrink_count,
2047− .scan_objects = gfs2_glock_shrink_scan,
2048−};
2044+static struct shrinker *glock_shrinker;
20492045
20502046 /**
20512047 * glock_hash_walk - Call a function for glock in a hash bucket
@@ -2465,13 +2461,18 @@ int __init gfs2_glock_init(void) · 413 unmodified lines
24652461 return -ENOMEM;
24662462 }
24672463
2468− ret = register_shrinker(&glock_shrinker, "gfs2-glock");
2469− if (ret) {
2464+ glock_shrinker = shrinker_alloc(0, "gfs2-glock");
2465+ if (!glock_shrinker) {
24702466 destroy_workqueue(glock_workqueue);
24712467 rhashtable_destroy(&gl_hash_table);
2472− return ret;
2468+ return -ENOMEM;
24732469 }
24742470
2471+ glock_shrinker->count_objects = gfs2_glock_shrink_count;
2472+ glock_shrinker->scan_objects = gfs2_glock_shrink_scan;
2473+
2474+ shrinker_register(glock_shrinker);
2475+
24752476 for (i = 0; i < GLOCK_WAIT_TABLE_SIZE; i++)
24762477 init_waitqueue_head(glock_wait_table + i);
24772478
@@ -2480,7 +2481,7 @@ int __init gfs2_glock_init(void) · 2 unmodified lines
24802481
24812482 void gfs2_glock_exit(void)
24822483 {
2483− unregister_shrinker(&glock_shrinker);
2484+ shrinker_free(glock_shrinker);
24842485 rhashtable_destroy(&gl_hash_table);
24852486 destroy_workqueue(glock_workqueue);
24862487 }
fs/gfs2/main.c+3−3
@@ -147,7 +147,7 @@ static int __init init_gfs2_fs(void) · 146 unmodified lines
147147 if (!gfs2_trans_cachep)
148148 goto fail_cachep8;
149149
150− error = register_shrinker(&gfs2_qd_shrinker, "gfs2-qd");
150+ error = gfs2_qd_shrinker_init();
151151 if (error)
152152 goto fail_shrinker;
153153
@@ -196,7 +196,7 @@ fail_wq3: · 42 unmodified lines
196196 fail_wq2:
197197 destroy_workqueue(gfs2_recovery_wq);
198198 fail_wq1:
199− unregister_shrinker(&gfs2_qd_shrinker);
199+ gfs2_qd_shrinker_exit();
200200 fail_shrinker:
201201 kmem_cache_destroy(gfs2_trans_cachep);
202202 fail_cachep8:
@@ -229,7 +229,7 @@ fail_lru: · 26 unmodified lines
229229
230230 static void __exit exit_gfs2_fs(void)
231231 {
232− unregister_shrinker(&gfs2_qd_shrinker);
232+ gfs2_qd_shrinker_exit();
233233 gfs2_glock_exit();
234234 gfs2_unregister_debugfs();
235235 unregister_filesystem(&gfs2_fs_type);
fs/gfs2/meta_io.c+25−36
@@ -115,7 +115,7 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 114 unmodified lines
115115 {
116116 struct address_space *mapping = gfs2_glock2aspace(gl);
117117 struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
118− struct page *page;
118+ struct folio *folio;
119119 struct buffer_head *bh;
120120 unsigned int shift;
121121 unsigned long index;
@@ -129,36 +129,31 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 7 unmodified lines
129129 bufnum = blkno - (index << shift); /* block buf index within page */
130130
131131 if (create) {
132− for (;;) {
133− page = grab_cache_page(mapping, index);
134− if (page)
135− break;
136− yield();
137− }
138− if (!page_has_buffers(page))
139− create_empty_buffers(page, sdp->sd_sb.sb_bsize, 0);
132+ folio = __filemap_get_folio(mapping, index,
133+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
134+ mapping_gfp_mask(mapping) | __GFP_NOFAIL);
135+ bh = folio_buffers(folio);
136+ if (!bh)
137+ bh = create_empty_buffers(folio,
138+ sdp->sd_sb.sb_bsize, 0);
140139 } else {
141− page = find_get_page_flags(mapping, index,
142− FGP_LOCK|FGP_ACCESSED);
143− if (!page)
140+ folio = __filemap_get_folio(mapping, index,
141+ FGP_LOCK | FGP_ACCESSED, 0);
142+ if (IS_ERR(folio))
144143 return NULL;
145− if (!page_has_buffers(page)) {
146− bh = NULL;
147− goto out_unlock;
148− }
144+ bh = folio_buffers(folio);
149145 }
150146
151− /* Locate header for our buffer within our page */
152− for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page)
153− /* Do nothing */;
154− get_bh(bh);
147+ if (!bh)
148+ goto out_unlock;
155149
150+ bh = get_nth_bh(bh, bufnum);
156151 if (!buffer_mapped(bh))
157152 map_bh(bh, sdp->sd_vfs, blkno);
158153
159154 out_unlock:
160− unlock_page(page);
161− put_page(page);
155+ folio_unlock(folio);
156+ folio_put(folio);
162157
163158 return bh;
164159 }
@@ -405,26 +400,20 @@ static struct buffer_head *gfs2_getjdatabuf(struct gfs2_inode *ip, u64 blkno) · 240 unmodified lines
405400 {
406401 struct address_space *mapping = ip->i_inode.i_mapping;
407402 struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
408− struct page *page;
403+ struct folio *folio;
409404 struct buffer_head *bh;
410405 unsigned int shift = PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift;
411406 unsigned long index = blkno >> shift; /* convert block to page */
412407 unsigned int bufnum = blkno - (index << shift);
413408
414− page = find_get_page_flags(mapping, index, FGP_LOCK|FGP_ACCESSED);
415− if (!page)
409+ folio = __filemap_get_folio(mapping, index, FGP_LOCK | FGP_ACCESSED, 0);
410+ if (IS_ERR(folio))
416411 return NULL;
417− if (!page_has_buffers(page)) {
418− unlock_page(page);
419− put_page(page);
420− return NULL;
421− }
422− /* Locate header for our buffer within our page */
423− for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page)
424− /* Do nothing */;
425− get_bh(bh);
426− unlock_page(page);
427− put_page(page);
412+ bh = folio_buffers(folio);
413+ if (bh)
414+ bh = get_nth_bh(bh, bufnum);
415+ folio_unlock(folio);
416+ folio_put(folio);
428417 return bh;
429418 }
430419
fs/gfs2/quota.c+38−24
@@ -196,13 +196,26 @@ static unsigned long gfs2_qd_shrink_count(struct shrinker *shrink, · 195 unmodified lines
196196 return vfs_pressure_ratio(list_lru_shrink_count(&gfs2_qd_lru, sc));
197197 }
198198
199−struct shrinker gfs2_qd_shrinker = {
200− .count_objects = gfs2_qd_shrink_count,
201− .scan_objects = gfs2_qd_shrink_scan,
202− .seeks = DEFAULT_SEEKS,
203− .flags = SHRINKER_NUMA_AWARE,
204−};
199+static struct shrinker *gfs2_qd_shrinker;
200+
201+int __init gfs2_qd_shrinker_init(void)
202+{
203+ gfs2_qd_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "gfs2-qd");
204+ if (!gfs2_qd_shrinker)
205+ return -ENOMEM;
206+
207+ gfs2_qd_shrinker->count_objects = gfs2_qd_shrink_count;
208+ gfs2_qd_shrinker->scan_objects = gfs2_qd_shrink_scan;
209+
210+ shrinker_register(gfs2_qd_shrinker);
205211
212+ return 0;
213+}
214+
215+void gfs2_qd_shrinker_exit(void)
216+{
217+ shrinker_free(gfs2_qd_shrinker);
218+}
206219
207220 static u64 qd2index(struct gfs2_quota_data *qd)
208221 {
@@ -736,7 +749,7 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 527 unmodified lines
736749 struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode);
737750 struct inode *inode = &ip->i_inode;
738751 struct address_space *mapping = inode->i_mapping;
739− struct page *page;
752+ struct folio *folio;
740753 struct buffer_head *bh;
741754 u64 blk;
742755 unsigned bsize = sdp->sd_sb.sb_bsize, bnum = 0, boff = 0;
@@ -745,15 +758,15 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 2 unmodified lines
745758 blk = index << (PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift);
746759 boff = off % bsize;
747760
748− page = grab_cache_page(mapping, index);
749− if (!page)
750− return -ENOMEM;
751− if (!page_has_buffers(page))
752− create_empty_buffers(page, bsize, 0);
761+ folio = filemap_grab_folio(mapping, index);
762+ if (IS_ERR(folio))
763+ return PTR_ERR(folio);
764+ bh = folio_buffers(folio);
765+ if (!bh)
766+ bh = create_empty_buffers(folio, bsize, 0);
753767
754− bh = page_buffers(page);
755− for(;;) {
756− /* Find the beginning block within the page */
768+ for (;;) {
769+ /* Find the beginning block within the folio */
757770 if (pg_off >= ((bnum * bsize) + bsize)) {
758771 bh = bh->b_this_page;
759772 bnum++;
@@ -766,9 +779,10 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 6 unmodified lines
766779 goto unlock_out;
767780 /* If it's a newly allocated disk block, zero it */
768781 if (buffer_new(bh))
769− zero_user(page, bnum * bsize, bh->b_size);
782+ folio_zero_range(folio, bnum * bsize,
783+ bh->b_size);
770784 }
771− if (PageUptodate(page))
785+ if (folio_test_uptodate(folio))
772786 set_buffer_uptodate(bh);
773787 if (bh_read(bh, REQ_META | REQ_PRIO) < 0)
774788 goto unlock_out;
@@ -784,17 +798,17 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 9 unmodified lines
784798 break;
785799 }
786800
787− /* Write to the page, now that we have setup the buffer(s) */
788− memcpy_to_page(page, off, buf, bytes);
789− flush_dcache_page(page);
790− unlock_page(page);
791− put_page(page);
801+ /* Write to the folio, now that we have setup the buffer(s) */
802+ memcpy_to_folio(folio, off, buf, bytes);
803+ flush_dcache_folio(folio);
804+ folio_unlock(folio);
805+ folio_put(folio);
792806
793807 return 0;
794808
795809 unlock_out:
796− unlock_page(page);
797− put_page(page);
810+ folio_unlock(folio);
811+ folio_put(folio);
798812 return -EIO;
799813 }
800814
fs/gfs2/quota.h+2−1
@@ -60,7 +60,8 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip, · 59 unmodified lines
6060 }
6161
6262 extern const struct quotactl_ops gfs2_quotactl_ops;
63−extern struct shrinker gfs2_qd_shrinker;
63+int __init gfs2_qd_shrinker_init(void);
64+void gfs2_qd_shrinker_exit(void);
6465 extern struct list_lru gfs2_qd_lru;
6566 extern void __init gfs2_quota_hash_init(void);
6667
fs/hugetlbfs/inode.c+23−61
@@ -83,29 +83,6 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = { · 82 unmodified lines
8383 {}
8484 };
8585
86−#ifdef CONFIG_NUMA
87−static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma,
88− struct inode *inode, pgoff_t index)
89−{
90− vma->vm_policy = mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy,
91− index);
92−}
93−
94−static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma)
95−{
96− mpol_cond_put(vma->vm_policy);
97−}
98−#else
99−static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma,
100− struct inode *inode, pgoff_t index)
101−{
102−}
103−
104−static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma)
105−{
106−}
107−#endif
108−
10986 /*
11087 * Mask used when checking the page offset value passed in via system
11188 * calls. This value will be converted to a loff_t which is signed.
@@ -135,7 +112,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) · 23 unmodified lines
135112 vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND);
136113 vma->vm_ops = &hugetlb_vm_ops;
137114
138− ret = seal_check_future_write(info->seals, vma);
115+ ret = seal_check_write(info->seals, vma);
139116 if (ret)
140117 return ret;
141118
@@ -295,7 +272,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 153 unmodified lines
295272 size_t res = 0;
296273
297274 /* First subpage to start the loop. */
298− page += offset / PAGE_SIZE;
275+ page = nth_page(page, offset / PAGE_SIZE);
299276 offset %= PAGE_SIZE;
300277 while (1) {
301278 if (is_raw_hwpoison_page_in_hugepage(page))
@@ -309,7 +286,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 7 unmodified lines
309286 break;
310287 offset += n;
311288 if (offset == PAGE_SIZE) {
312− page++;
289+ page = nth_page(page, 1);
313290 offset = 0;
314291 }
315292 }
@@ -334,7 +311,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 18 unmodified lines
334311 ssize_t retval = 0;
335312
336313 while (iov_iter_count(to)) {
337− struct page *page;
314+ struct folio *folio;
338315 size_t nr, copied, want;
339316
340317 /* nr is the maximum number of bytes to copy from this page */
@@ -352,18 +329,18 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 11 unmodified lines
352329 }
353330 nr = nr - offset;
354331
355− /* Find the page */
356− page = find_lock_page(mapping, index);
357− if (unlikely(page == NULL)) {
332+ /* Find the folio */
333+ folio = filemap_lock_hugetlb_folio(h, mapping, index);
334+ if (IS_ERR(folio)) {
358335 /*
359336 * We have a HOLE, zero out the user-buffer for the
360337 * length of the hole or request.
361338 */
362339 copied = iov_iter_zero(nr, to);
363340 } else {
364− unlock_page(page);
341+ folio_unlock(folio);
365342
366− if (!PageHWPoison(page))
343+ if (!folio_test_has_hwpoisoned(folio))
367344 want = nr;
368345 else {
369346 /*
@@ -371,19 +348,19 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 1 unmodified line
371348 * touching the 1st raw HWPOISON subpage after
372349 * offset.
373350 */
374− want = adjust_range_hwpoison(page, offset, nr);
351+ want = adjust_range_hwpoison(&folio->page, offset, nr);
375352 if (want == 0) {
376− put_page(page);
353+ folio_put(folio);
377354 retval = -EIO;
378355 break;
379356 }
380357 }
381358
382359 /*
383− * We have the page, copy it to user space buffer.
360+ * We have the folio, copy it to user space buffer.
384361 */
385− copied = copy_page_to_iter(page, offset, want, to);
386− put_page(page);
362+ copied = copy_folio_to_iter(folio, offset, want, to);
363+ folio_put(folio);
387364 }
388365 offset += copied;
389366 retval += copied;
@@ -661,21 +638,20 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 271 unmodified lines
661638 {
662639 struct hstate *h = hstate_inode(inode);
663640 struct address_space *mapping = &inode->i_data;
664− const pgoff_t start = lstart >> huge_page_shift(h);
665− const pgoff_t end = lend >> huge_page_shift(h);
641+ const pgoff_t end = lend >> PAGE_SHIFT;
666642 struct folio_batch fbatch;
667643 pgoff_t next, index;
668644 int i, freed = 0;
669645 bool truncate_op = (lend == LLONG_MAX);
670646
671647 folio_batch_init(&fbatch);
672− next = start;
648+ next = lstart >> PAGE_SHIFT;
673649 while (filemap_get_folios(mapping, &next, end - 1, &fbatch)) {
674650 for (i = 0; i < folio_batch_count(&fbatch); ++i) {
675651 struct folio *folio = fbatch.folios[i];
676652 u32 hash = 0;
677653
678− index = folio->index;
654+ index = folio->index >> huge_page_order(h);
679655 hash = hugetlb_fault_mutex_hash(mapping, index);
680656 mutex_lock(&hugetlb_fault_mutex_table[hash]);
681657
@@ -693,7 +669,9 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 11 unmodified lines
693669 }
694670
695671 if (truncate_op)
696− (void)hugetlb_unreserve_pages(inode, start, LONG_MAX, freed);
672+ (void)hugetlb_unreserve_pages(inode,
673+ lstart >> huge_page_shift(h),
674+ LONG_MAX, freed);
697675 }
698676
699677 static void hugetlbfs_evict_inode(struct inode *inode)
@@ -741,7 +719,7 @@ static void hugetlbfs_zero_partial_page(struct hstate *h, · 41 unmodified lines
741719 pgoff_t idx = start >> huge_page_shift(h);
742720 struct folio *folio;
743721
744− folio = filemap_lock_folio(mapping, idx);
722+ folio = filemap_lock_hugetlb_folio(h, mapping, idx);
745723 if (IS_ERR(folio))
746724 return;
747725
@@ -852,8 +830,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 104 unmodified lines
852830
853831 /*
854832 * Initialize a pseudo vma as this is required by the huge page
855− * allocation routines. If NUMA is configured, use page index
856− * as input to create an allocation policy.
833+ * allocation routines.
857834 */
858835 vma_init(&pseudo_vma, mm);
859836 vm_flags_init(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
@@ -886,7 +863,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 26 unmodified lines
886863 mutex_lock(&hugetlb_fault_mutex_table[hash]);
887864
888865 /* See if already present in mapping to avoid alloc/free */
889− folio = filemap_get_folio(mapping, index);
866+ folio = filemap_get_folio(mapping, index << huge_page_order(h));
890867 if (!IS_ERR(folio)) {
891868 folio_put(folio);
892869 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
@@ -901,9 +878,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 8 unmodified lines
901878 * folios in these areas, we need to consume the reserves
902879 * to keep reservation accounting consistent.
903880 */
904− hugetlb_set_vma_policy(&pseudo_vma, inode, index);
905881 folio = alloc_hugetlb_folio(&pseudo_vma, addr, 0);
906− hugetlb_drop_vma_policy(&pseudo_vma);
907882 if (IS_ERR(folio)) {
908883 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
909884 error = PTR_ERR(folio);
@@ -1282,18 +1257,6 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) · 372 unmodified lines
12821257 hugetlbfs_inc_free_inodes(sbinfo);
12831258 return NULL;
12841259 }
1285−
1286− /*
1287− * Any time after allocation, hugetlbfs_destroy_inode can be called
1288− * for the inode. mpol_free_shared_policy is unconditionally called
1289− * as part of hugetlbfs_destroy_inode. So, initialize policy here
1290− * in case of a quick call to destroy.
1291− *
1292− * Note that the policy is initialized even if we are creating a
1293− * private inode. This simplifies hugetlbfs_destroy_inode.
1294− */
1295− mpol_shared_policy_init(&p->policy, NULL);
1296−
12971260 return &p->vfs_inode;
12981261 }
12991262
@@ -1305,7 +1268,6 @@ static void hugetlbfs_free_inode(struct inode *inode) · 5 unmodified lines
13051268 static void hugetlbfs_destroy_inode(struct inode *inode)
13061269 {
13071270 hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb));
1308− mpol_free_shared_policy(&HUGETLBFS_I(inode)->policy);
13091271 }
13101272
13111273 static const struct address_space_operations hugetlbfs_aops = {
fs/iomap/buffered-io.c+35−22
@@ -29,9 +29,9 @@ typedef int (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length); · 28 unmodified lines
2929 * and I/O completions.
3030 */
3131 struct iomap_folio_state {
32− atomic_t read_bytes_pending;
33− atomic_t write_bytes_pending;
3432 spinlock_t state_lock;
33+ unsigned int read_bytes_pending;
34+ atomic_t write_bytes_pending;
3535
3636 /*
3737 * Each block has two bits in this bitmap:
@@ -57,30 +57,32 @@ static inline bool ifs_block_is_uptodate(struct iomap_folio_state *ifs, · 19 unmodified lines
5757 return test_bit(block, ifs->state);
5858 }
5959
60−static void ifs_set_range_uptodate(struct folio *folio,
60+static bool ifs_set_range_uptodate(struct folio *folio,
6161 struct iomap_folio_state *ifs, size_t off, size_t len)
6262 {
6363 struct inode *inode = folio->mapping->host;
6464 unsigned int first_blk = off >> inode->i_blkbits;
6565 unsigned int last_blk = (off + len - 1) >> inode->i_blkbits;
6666 unsigned int nr_blks = last_blk - first_blk + 1;
67− unsigned long flags;
6867
69− spin_lock_irqsave(&ifs->state_lock, flags);
7068 bitmap_set(ifs->state, first_blk, nr_blks);
71− if (ifs_is_fully_uptodate(folio, ifs))
72− folio_mark_uptodate(folio);
73− spin_unlock_irqrestore(&ifs->state_lock, flags);
69+ return ifs_is_fully_uptodate(folio, ifs);
7470 }
7571
7672 static void iomap_set_range_uptodate(struct folio *folio, size_t off,
7773 size_t len)
7874 {
7975 struct iomap_folio_state *ifs = folio->private;
76+ unsigned long flags;
77+ bool uptodate = true;
8078
81− if (ifs)
82− ifs_set_range_uptodate(folio, ifs, off, len);
83− else
79+ if (ifs) {
80+ spin_lock_irqsave(&ifs->state_lock, flags);
81+ uptodate = ifs_set_range_uptodate(folio, ifs, off, len);
82+ spin_unlock_irqrestore(&ifs->state_lock, flags);
83+ }
84+
85+ if (uptodate)
8486 folio_mark_uptodate(folio);
8587 }
8688
@@ -181,7 +183,7 @@ static void ifs_free(struct folio *folio) · 94 unmodified lines
181183
182184 if (!ifs)
183185 return;
184− WARN_ON_ONCE(atomic_read(&ifs->read_bytes_pending));
186+ WARN_ON_ONCE(ifs->read_bytes_pending != 0);
185187 WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending));
186188 WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) !=
187189 folio_test_uptodate(folio));
@@ -248,20 +250,28 @@ static void iomap_adjust_read_range(struct inode *inode, struct folio *folio, · 60 unmodified lines
248250 *lenp = plen;
249251 }
250252
251−static void iomap_finish_folio_read(struct folio *folio, size_t offset,
253+static void iomap_finish_folio_read(struct folio *folio, size_t off,
252254 size_t len, int error)
253255 {
254256 struct iomap_folio_state *ifs = folio->private;
257+ bool uptodate = !error;
258+ bool finished = true;
255259
256− if (unlikely(error)) {
257− folio_clear_uptodate(folio);
258− folio_set_error(folio);
259− } else {
260− iomap_set_range_uptodate(folio, offset, len);
260+ if (ifs) {
261+ unsigned long flags;
262+
263+ spin_lock_irqsave(&ifs->state_lock, flags);
264+ if (!error)
265+ uptodate = ifs_set_range_uptodate(folio, ifs, off, len);
266+ ifs->read_bytes_pending -= len;
267+ finished = !ifs->read_bytes_pending;
268+ spin_unlock_irqrestore(&ifs->state_lock, flags);
261269 }
262270
263− if (!ifs || atomic_sub_and_test(len, &ifs->read_bytes_pending))
264− folio_unlock(folio);
271+ if (error)
272+ folio_set_error(folio);
273+ if (finished)
274+ folio_end_read(folio, uptodate);
265275 }
266276
267277 static void iomap_read_end_io(struct bio *bio)
@@ -358,8 +368,11 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, · 90 unmodified lines
358368 }
359369
360370 ctx->cur_folio_in_bio = true;
361− if (ifs)
362− atomic_add(plen, &ifs->read_bytes_pending);
371+ if (ifs) {
372+ spin_lock_irq(&ifs->state_lock);
373+ ifs->read_bytes_pending += plen;
374+ spin_unlock_irq(&ifs->state_lock);
375+ }
363376
364377 sector = iomap_sector(iomap, pos);
365378 if (!ctx->bio ||
fs/jbd2/journal.c+18−11
@@ -1290,7 +1290,7 @@ static int jbd2_min_tag_size(void) · 1289 unmodified lines
12901290 static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink,
12911291 struct shrink_control *sc)
12921292 {
1293− journal_t *journal = container_of(shrink, journal_t, j_shrinker);
1293+ journal_t *journal = shrink->private_data;
12941294 unsigned long nr_to_scan = sc->nr_to_scan;
12951295 unsigned long nr_shrunk;
12961296 unsigned long count;
@@ -1316,7 +1316,7 @@ static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, · 19 unmodified lines
13161316 static unsigned long jbd2_journal_shrink_count(struct shrinker *shrink,
13171317 struct shrink_control *sc)
13181318 {
1319− journal_t *journal = container_of(shrink, journal_t, j_shrinker);
1319+ journal_t *journal = shrink->private_data;
13201320 unsigned long count;
13211321
13221322 count = percpu_counter_read_positive(&journal->j_checkpoint_jh_count);
@@ -1588,14 +1588,21 @@ static journal_t *journal_init_common(struct block_device *bdev, · 265 unmodified lines
15881588 goto err_cleanup;
15891589
15901590 journal->j_shrink_transaction = NULL;
1591− journal->j_shrinker.scan_objects = jbd2_journal_shrink_scan;
1592− journal->j_shrinker.count_objects = jbd2_journal_shrink_count;
1593− journal->j_shrinker.seeks = DEFAULT_SEEKS;
1594− journal->j_shrinker.batch = journal->j_max_transaction_buffers;
1595− err = register_shrinker(&journal->j_shrinker, "jbd2-journal:(%u:%u)",
1596− MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));
1597− if (err)
1591+
1592+ journal->j_shrinker = shrinker_alloc(0, "jbd2-journal:(%u:%u)",
1593+ MAJOR(bdev->bd_dev),
1594+ MINOR(bdev->bd_dev));
1595+ if (!journal->j_shrinker) {
1596+ err = -ENOMEM;
15981597 goto err_cleanup;
1598+ }
1599+
1600+ journal->j_shrinker->scan_objects = jbd2_journal_shrink_scan;
1601+ journal->j_shrinker->count_objects = jbd2_journal_shrink_count;
1602+ journal->j_shrinker->batch = journal->j_max_transaction_buffers;
1603+ journal->j_shrinker->private_data = journal;
1604+
1605+ shrinker_register(journal->j_shrinker);
15991606
16001607 return journal;
16011608
@@ -2172,9 +2179,9 @@ int jbd2_journal_destroy(journal_t *journal) · 570 unmodified lines
21722179 brelse(journal->j_sb_buffer);
21732180 }
21742181
2175− if (journal->j_shrinker.flags & SHRINKER_REGISTERED) {
2182+ if (journal->j_shrinker) {
21762183 percpu_counter_destroy(&journal->j_checkpoint_jh_count);
2177− unregister_shrinker(&journal->j_shrinker);
2184+ shrinker_free(journal->j_shrinker);
21782185 }
21792186 if (journal->j_proc_entry)
21802187 jbd2_stats_proc_exit(journal);
fs/kernfs/file.c+0−49
@@ -429,60 +429,11 @@ static int kernfs_vma_access(struct vm_area_struct *vma, unsigned long addr, · 428 unmodified lines
429429 return ret;
430430 }
431431
432−#ifdef CONFIG_NUMA
433−static int kernfs_vma_set_policy(struct vm_area_struct *vma,
434− struct mempolicy *new)
435−{
436− struct file *file = vma->vm_file;
437− struct kernfs_open_file *of = kernfs_of(file);
438− int ret;
439−
440− if (!of->vm_ops)
441− return 0;
442−
443− if (!kernfs_get_active(of->kn))
444− return -EINVAL;
445−
446− ret = 0;
447− if (of->vm_ops->set_policy)
448− ret = of->vm_ops->set_policy(vma, new);
449−
450− kernfs_put_active(of->kn);
451− return ret;
452−}
453−
454−static struct mempolicy *kernfs_vma_get_policy(struct vm_area_struct *vma,
455− unsigned long addr)
456−{
457− struct file *file = vma->vm_file;
458− struct kernfs_open_file *of = kernfs_of(file);
459− struct mempolicy *pol;
460−
461− if (!of->vm_ops)
462− return vma->vm_policy;
463−
464− if (!kernfs_get_active(of->kn))
465− return vma->vm_policy;
466−
467− pol = vma->vm_policy;
468− if (of->vm_ops->get_policy)
469− pol = of->vm_ops->get_policy(vma, addr);
470−
471− kernfs_put_active(of->kn);
472− return pol;
473−}
474−
475−#endif
476−
477432 static const struct vm_operations_struct kernfs_vm_ops = {
478433 .open = kernfs_vma_open,
479434 .fault = kernfs_vma_fault,
480435 .page_mkwrite = kernfs_vma_page_mkwrite,
481436 .access = kernfs_vma_access,
482−#ifdef CONFIG_NUMA
483− .set_policy = kernfs_vma_set_policy,
484− .get_policy = kernfs_vma_get_policy,
485−#endif
486437 };
487438
488439 static int kernfs_fop_mmap(struct file *file, struct vm_area_struct *vma)
fs/kernfs/mount.c+1−1
@@ -265,7 +265,7 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k · 264 unmodified lines
265265 sb->s_time_gran = 1;
266266
267267 /* sysfs dentries and inodes don't require IO to create */
268− sb->s_shrink.seeks = 0;
268+ sb->s_shrink->seeks = 0;
269269
270270 /* get root inode, initialize and unlock it */
271271 down_read(&kf_root->kernfs_rwsem);
fs/mbcache.c+12−10
@@ -37,7 +37,7 @@ struct mb_cache { · 36 unmodified lines
3737 struct list_head c_list;
3838 /* Number of entries in cache */
3939 unsigned long c_entry_count;
40− struct shrinker c_shrink;
40+ struct shrinker *c_shrink;
4141 /* Work for shrinking when the cache has too many entries */
4242 struct work_struct c_shrink_work;
4343 };
@@ -293,8 +293,7 @@ EXPORT_SYMBOL(mb_cache_entry_touch); · 249 unmodified lines
293293 static unsigned long mb_cache_count(struct shrinker *shrink,
294294 struct shrink_control *sc)
295295 {
296− struct mb_cache *cache = container_of(shrink, struct mb_cache,
297− c_shrink);
296+ struct mb_cache *cache = shrink->private_data;
298297
299298 return cache->c_entry_count;
300299 }
@@ -333,8 +332,7 @@ static unsigned long mb_cache_shrink(struct mb_cache *cache, · 32 unmodified lines
333332 static unsigned long mb_cache_scan(struct shrinker *shrink,
334333 struct shrink_control *sc)
335334 {
336− struct mb_cache *cache = container_of(shrink, struct mb_cache,
337− c_shrink);
335+ struct mb_cache *cache = shrink->private_data;
338336 return mb_cache_shrink(cache, sc->nr_to_scan);
339337 }
340338
@@ -377,15 +375,19 @@ struct mb_cache *mb_cache_create(int bucket_bits) · 36 unmodified lines
377375 for (i = 0; i < bucket_count; i++)
378376 INIT_HLIST_BL_HEAD(&cache->c_hash[i]);
379377
380− cache->c_shrink.count_objects = mb_cache_count;
381− cache->c_shrink.scan_objects = mb_cache_scan;
382− cache->c_shrink.seeks = DEFAULT_SEEKS;
383− if (register_shrinker(&cache->c_shrink, "mbcache-shrinker")) {
378+ cache->c_shrink = shrinker_alloc(0, "mbcache-shrinker");
379+ if (!cache->c_shrink) {
384380 kfree(cache->c_hash);
385381 kfree(cache);
386382 goto err_out;
387383 }
388384
385+ cache->c_shrink->count_objects = mb_cache_count;
386+ cache->c_shrink->scan_objects = mb_cache_scan;
387+ cache->c_shrink->private_data = cache;
388+
389+ shrinker_register(cache->c_shrink);
390+
389391 INIT_WORK(&cache->c_shrink_work, mb_cache_shrink_worker);
390392
391393 return cache;
@@ -406,7 +408,7 @@ void mb_cache_destroy(struct mb_cache *cache) · 14 unmodified lines
406408 {
407409 struct mb_cache_entry *entry, *next;
408410
409− unregister_shrinker(&cache->c_shrink);
411+ shrinker_free(cache->c_shrink);
410412
411413 /*
412414 * We don't bother with any locking. Cache must not be used at this
fs/mpage.c+1−2
@@ -119,8 +119,7 @@ static void map_buffer_to_folio(struct folio *folio, struct buffer_head *bh, · 118 unmodified lines
119119 folio_mark_uptodate(folio);
120120 return;
121121 }
122− create_empty_buffers(&folio->page, i_blocksize(inode), 0);
123− head = folio_buffers(folio);
122+ head = create_empty_buffers(folio, i_blocksize(inode), 0);
124123 }
125124
126125 page_bh = head;
fs/nfs/nfs42xattr.c+44−43
@@ -796,28 +796,9 @@ static unsigned long nfs4_xattr_cache_scan(struct shrinker *shrink, · 795 unmodified lines
796796 static unsigned long nfs4_xattr_entry_scan(struct shrinker *shrink,
797797 struct shrink_control *sc);
798798
799−static struct shrinker nfs4_xattr_cache_shrinker = {
800− .count_objects = nfs4_xattr_cache_count,
801− .scan_objects = nfs4_xattr_cache_scan,
802− .seeks = DEFAULT_SEEKS,
803− .flags = SHRINKER_MEMCG_AWARE,
804−};
805−
806−static struct shrinker nfs4_xattr_entry_shrinker = {
807− .count_objects = nfs4_xattr_entry_count,
808− .scan_objects = nfs4_xattr_entry_scan,
809− .seeks = DEFAULT_SEEKS,
810− .batch = 512,
811− .flags = SHRINKER_MEMCG_AWARE,
812−};
813−
814−static struct shrinker nfs4_xattr_large_entry_shrinker = {
815− .count_objects = nfs4_xattr_entry_count,
816− .scan_objects = nfs4_xattr_entry_scan,
817− .seeks = 1,
818− .batch = 512,
819− .flags = SHRINKER_MEMCG_AWARE,
820−};
799+static struct shrinker *nfs4_xattr_cache_shrinker;
800+static struct shrinker *nfs4_xattr_entry_shrinker;
801+static struct shrinker *nfs4_xattr_large_entry_shrinker;
821802
822803 static enum lru_status
823804 cache_lru_isolate(struct list_head *item,
@@ -943,7 +924,7 @@ nfs4_xattr_entry_scan(struct shrinker *shrink, struct shrink_control *sc) · 119 unmodified lines
943924 struct nfs4_xattr_entry *entry;
944925 struct list_lru *lru;
945926
946− lru = (shrink == &nfs4_xattr_large_entry_shrinker) ?
927+ lru = (shrink == nfs4_xattr_large_entry_shrinker) ?
947928 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;
948929
949930 freed = list_lru_shrink_walk(lru, sc, entry_lru_isolate, &dispose);
@@ -971,7 +952,7 @@ nfs4_xattr_entry_count(struct shrinker *shrink, struct shrink_control *sc) · 21 unmodified lines
971952 unsigned long count;
972953 struct list_lru *lru;
973954
974− lru = (shrink == &nfs4_xattr_large_entry_shrinker) ?
955+ lru = (shrink == nfs4_xattr_large_entry_shrinker) ?
975956 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;
976957
977958 count = list_lru_shrink_count(lru, sc);
@@ -991,18 +972,34 @@ static void nfs4_xattr_cache_init_once(void *p) · 13 unmodified lines
991972 INIT_LIST_HEAD(&cache->dispose);
992973 }
993974
994−static int nfs4_xattr_shrinker_init(struct shrinker *shrinker,
995− struct list_lru *lru, const char *name)
975+typedef unsigned long (*count_objects_cb)(struct shrinker *s,
976+ struct shrink_control *sc);
977+typedef unsigned long (*scan_objects_cb)(struct shrinker *s,
978+ struct shrink_control *sc);
979+
980+static int __init nfs4_xattr_shrinker_init(struct shrinker **shrinker,
981+ struct list_lru *lru, const char *name,
982+ count_objects_cb count,
983+ scan_objects_cb scan, long batch, int seeks)
996984 {
997− int ret = 0;
985+ int ret;
998986
999− ret = register_shrinker(shrinker, name);
1000− if (ret)
987+ *shrinker = shrinker_alloc(SHRINKER_MEMCG_AWARE, name);
988+ if (!*shrinker)
989+ return -ENOMEM;
990+
991+ ret = list_lru_init_memcg(lru, *shrinker);
992+ if (ret) {
993+ shrinker_free(*shrinker);
1001994 return ret;
995+ }
1002996
1003− ret = list_lru_init_memcg(lru, shrinker);
1004− if (ret)
1005− unregister_shrinker(shrinker);
997+ (*shrinker)->count_objects = count;
998+ (*shrinker)->scan_objects = scan;
999+ (*shrinker)->batch = batch;
1000+ (*shrinker)->seeks = seeks;
1001+
1002+ shrinker_register(*shrinker);
10061003
10071004 return ret;
10081005 }
@@ -1010,7 +1007,7 @@ static int nfs4_xattr_shrinker_init(struct shrinker *shrinker, · 1 unmodified line
10101007 static void nfs4_xattr_shrinker_destroy(struct shrinker *shrinker,
10111008 struct list_lru *lru)
10121009 {
1013− unregister_shrinker(shrinker);
1010+ shrinker_free(shrinker);
10141011 list_lru_destroy(lru);
10151012 }
10161013
@@ -1026,27 +1023,31 @@ int __init nfs4_xattr_cache_init(void) · 9 unmodified lines
10261023 return -ENOMEM;
10271024
10281025 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_cache_shrinker,
1029− &nfs4_xattr_cache_lru,
1030− "nfs-xattr_cache");
1026+ &nfs4_xattr_cache_lru, "nfs-xattr_cache",
1027+ nfs4_xattr_cache_count,
1028+ nfs4_xattr_cache_scan, 0, DEFAULT_SEEKS);
10311029 if (ret)
10321030 goto out1;
10331031
10341032 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_entry_shrinker,
1035− &nfs4_xattr_entry_lru,
1036− "nfs-xattr_entry");
1033+ &nfs4_xattr_entry_lru, "nfs-xattr_entry",
1034+ nfs4_xattr_entry_count,
1035+ nfs4_xattr_entry_scan, 512, DEFAULT_SEEKS);
10371036 if (ret)
10381037 goto out2;
10391038
10401039 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_large_entry_shrinker,
10411040 &nfs4_xattr_large_entry_lru,
1042− "nfs-xattr_large_entry");
1041+ "nfs-xattr_large_entry",
1042+ nfs4_xattr_entry_count,
1043+ nfs4_xattr_entry_scan, 512, 1);
10431044 if (!ret)
10441045 return 0;
10451046
1046− nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker,
1047+ nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker,
10471048 &nfs4_xattr_entry_lru);
10481049 out2:
1049− nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker,
1050+ nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker,
10501051 &nfs4_xattr_cache_lru);
10511052 out1:
10521053 kmem_cache_destroy(nfs4_xattr_cache_cachep);
@@ -1056,11 +1057,11 @@ out1: · 3 unmodified lines
10561057
10571058 void nfs4_xattr_cache_exit(void)
10581059 {
1059− nfs4_xattr_shrinker_destroy(&nfs4_xattr_large_entry_shrinker,
1060+ nfs4_xattr_shrinker_destroy(nfs4_xattr_large_entry_shrinker,
10601061 &nfs4_xattr_large_entry_lru);
1061− nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker,
1062+ nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker,
10621063 &nfs4_xattr_entry_lru);
1063− nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker,
1064+ nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker,
10641065 &nfs4_xattr_cache_lru);
10651066 kmem_cache_destroy(nfs4_xattr_cache_cachep);
10661067 }
fs/nfs/super.c+13−8
@@ -129,11 +129,7 @@ static void nfs_ssc_unregister_ops(void) · 128 unmodified lines
129129 }
130130 #endif /* CONFIG_NFS_V4_2 */
131131
132−static struct shrinker acl_shrinker = {
133− .count_objects = nfs_access_cache_count,
134− .scan_objects = nfs_access_cache_scan,
135− .seeks = DEFAULT_SEEKS,
136−};
132+static struct shrinker *acl_shrinker;
137133
138134 /*
139135 * Register the NFS filesystems
@@ -153,9 +149,18 @@ int __init register_nfs_fs(void) · 13 unmodified lines
153149 ret = nfs_register_sysctl();
154150 if (ret < 0)
155151 goto error_2;
156− ret = register_shrinker(&acl_shrinker, "nfs-acl");
157− if (ret < 0)
152+
153+ acl_shrinker = shrinker_alloc(0, "nfs-acl");
154+ if (!acl_shrinker) {
155+ ret = -ENOMEM;
158156 goto error_3;
157+ }
158+
159+ acl_shrinker->count_objects = nfs_access_cache_count;
160+ acl_shrinker->scan_objects = nfs_access_cache_scan;
161+
162+ shrinker_register(acl_shrinker);
163+
159164 #ifdef CONFIG_NFS_V4_2
160165 nfs_ssc_register_ops();
161166 #endif
@@ -175,7 +180,7 @@ error_0: · 13 unmodified lines
175180 */
176181 void __exit unregister_nfs_fs(void)
177182 {
178− unregister_shrinker(&acl_shrinker);
183+ shrinker_free(acl_shrinker);
179184 nfs_unregister_sysctl();
180185 unregister_nfs4_fs();
181186 #ifdef CONFIG_NFS_V4_2
fs/nfsd/filecache.c+13−10
@@ -521,11 +521,7 @@ nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) · 520 unmodified lines
521521 return ret;
522522 }
523523
524−static struct shrinker nfsd_file_shrinker = {
525− .scan_objects = nfsd_file_lru_scan,
526− .count_objects = nfsd_file_lru_count,
527− .seeks = 1,
528−};
524+static struct shrinker *nfsd_file_shrinker;
529525
530526 /**
531527 * nfsd_file_cond_queue - conditionally unhash and queue a nfsd_file
@@ -746,12 +742,19 @@ nfsd_file_cache_init(void) · 214 unmodified lines
746742 goto out_err;
747743 }
748744
749− ret = register_shrinker(&nfsd_file_shrinker, "nfsd-filecache");
750− if (ret) {
751− pr_err("nfsd: failed to register nfsd_file_shrinker: %d\n", ret);
745+ nfsd_file_shrinker = shrinker_alloc(0, "nfsd-filecache");
746+ if (!nfsd_file_shrinker) {
747+ ret = -ENOMEM;
748+ pr_err("nfsd: failed to allocate nfsd_file_shrinker\n");
752749 goto out_lru;
753750 }
754751
752+ nfsd_file_shrinker->count_objects = nfsd_file_lru_count;
753+ nfsd_file_shrinker->scan_objects = nfsd_file_lru_scan;
754+ nfsd_file_shrinker->seeks = 1;
755+
756+ shrinker_register(nfsd_file_shrinker);
757+
755758 ret = lease_register_notifier(&nfsd_file_lease_notifier);
756759 if (ret) {
757760 pr_err("nfsd: unable to register lease notifier: %d\n", ret);
@@ -774,7 +777,7 @@ out: · 16 unmodified lines
774777 out_notifier:
775778 lease_unregister_notifier(&nfsd_file_lease_notifier);
776779 out_shrinker:
777− unregister_shrinker(&nfsd_file_shrinker);
780+ shrinker_free(nfsd_file_shrinker);
778781 out_lru:
779782 list_lru_destroy(&nfsd_file_lru);
780783 out_err:
@@ -891,7 +894,7 @@ nfsd_file_cache_shutdown(void) · 110 unmodified lines
891894 return;
892895
893896 lease_unregister_notifier(&nfsd_file_lease_notifier);
894− unregister_shrinker(&nfsd_file_shrinker);
897+ shrinker_free(nfsd_file_shrinker);
895898 /*
896899 * make sure all callers of nfsd_file_lru_cb are done before
897900 * calling nfsd_file_cache_purge
fs/nfsd/netns.h+2−2
@@ -177,7 +177,7 @@ struct nfsd_net { · 176 unmodified lines
177177 /* size of cache when we saw the longest hash chain */
178178 unsigned int longest_chain_cachesize;
179179
180− struct shrinker nfsd_reply_cache_shrinker;
180+ struct shrinker *nfsd_reply_cache_shrinker;
181181
182182 /* tracking server-to-server copy mounts */
183183 spinlock_t nfsd_ssc_lock;
@@ -195,7 +195,7 @@ struct nfsd_net { · 11 unmodified lines
195195 int nfs4_max_clients;
196196
197197 atomic_t nfsd_courtesy_clients;
198− struct shrinker nfsd_client_shrinker;
198+ struct shrinker *nfsd_client_shrinker;
199199 struct work_struct nfsd_shrinker_work;
200200 };
201201
fs/nfsd/nfs4state.c+11−8
@@ -4452,8 +4452,7 @@ static unsigned long · 4451 unmodified lines
44524452 nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
44534453 {
44544454 int count;
4455− struct nfsd_net *nn = container_of(shrink,
4456− struct nfsd_net, nfsd_client_shrinker);
4455+ struct nfsd_net *nn = shrink->private_data;
44574456
44584457 count = atomic_read(&nn->nfsd_courtesy_clients);
44594458 if (!count)
@@ -8235,12 +8234,16 @@ static int nfs4_state_create_net(struct net *net) · 3775 unmodified lines
82358234 INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker);
82368235 get_net(net);
82378236
8238− nn->nfsd_client_shrinker.scan_objects = nfsd4_state_shrinker_scan;
8239− nn->nfsd_client_shrinker.count_objects = nfsd4_state_shrinker_count;
8240− nn->nfsd_client_shrinker.seeks = DEFAULT_SEEKS;
8241−
8242− if (register_shrinker(&nn->nfsd_client_shrinker, "nfsd-client"))
8237+ nn->nfsd_client_shrinker = shrinker_alloc(0, "nfsd-client");
8238+ if (!nn->nfsd_client_shrinker)
82438239 goto err_shrinker;
8240+
8241+ nn->nfsd_client_shrinker->scan_objects = nfsd4_state_shrinker_scan;
8242+ nn->nfsd_client_shrinker->count_objects = nfsd4_state_shrinker_count;
8243+ nn->nfsd_client_shrinker->private_data = nn;
8244+
8245+ shrinker_register(nn->nfsd_client_shrinker);
8246+
82448247 return 0;
82458248
82468249 err_shrinker:
@@ -8338,7 +8341,7 @@ nfs4_state_shutdown_net(struct net *net) · 91 unmodified lines
83388341 struct list_head *pos, *next, reaplist;
83398342 struct nfsd_net *nn = net_generic(net, nfsd_net_id);
83408343
8341− unregister_shrinker(&nn->nfsd_client_shrinker);
8344+ shrinker_free(nn->nfsd_client_shrinker);
83428345 cancel_work(&nn->nfsd_shrinker_work);
83438346 cancel_delayed_work_sync(&nn->laundromat_work);
83448347 locks_end_grace(&nn->nfsd4_manager);
fs/nfsd/nfscache.c+16−15
@@ -201,26 +201,29 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 200 unmodified lines
201201 {
202202 unsigned int hashsize;
203203 unsigned int i;
204− int status = 0;
205204
206205 nn->max_drc_entries = nfsd_cache_size_limit();
207206 atomic_set(&nn->num_drc_entries, 0);
208207 hashsize = nfsd_hashsize(nn->max_drc_entries);
209208 nn->maskbits = ilog2(hashsize);
210209
211− nn->nfsd_reply_cache_shrinker.scan_objects = nfsd_reply_cache_scan;
212− nn->nfsd_reply_cache_shrinker.count_objects = nfsd_reply_cache_count;
213− nn->nfsd_reply_cache_shrinker.seeks = 1;
214− status = register_shrinker(&nn->nfsd_reply_cache_shrinker,
215− "nfsd-reply:%s", nn->nfsd_name);
216− if (status)
217− return status;
218−
219210 nn->drc_hashtbl = kvzalloc(array_size(hashsize,
220211 sizeof(*nn->drc_hashtbl)), GFP_KERNEL);
221212 if (!nn->drc_hashtbl)
213+ return -ENOMEM;
214+
215+ nn->nfsd_reply_cache_shrinker = shrinker_alloc(0, "nfsd-reply:%s",
216+ nn->nfsd_name);
217+ if (!nn->nfsd_reply_cache_shrinker)
222218 goto out_shrinker;
223219
220+ nn->nfsd_reply_cache_shrinker->scan_objects = nfsd_reply_cache_scan;
221+ nn->nfsd_reply_cache_shrinker->count_objects = nfsd_reply_cache_count;
222+ nn->nfsd_reply_cache_shrinker->seeks = 1;
223+ nn->nfsd_reply_cache_shrinker->private_data = nn;
224+
225+ shrinker_register(nn->nfsd_reply_cache_shrinker);
226+
224227 for (i = 0; i < hashsize; i++) {
225228 INIT_LIST_HEAD(&nn->drc_hashtbl[i].lru_head);
226229 spin_lock_init(&nn->drc_hashtbl[i].cache_lock);
@@ -229,7 +232,7 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 2 unmodified lines
229232
230233 return 0;
231234 out_shrinker:
232− unregister_shrinker(&nn->nfsd_reply_cache_shrinker);
235+ kvfree(nn->drc_hashtbl);
233236 printk(KERN_ERR "nfsd: failed to allocate reply cache\n");
234237 return -ENOMEM;
235238 }
@@ -239,7 +242,7 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn) · 3 unmodified lines
239242 struct nfsd_cacherep *rp;
240243 unsigned int i;
241244
242− unregister_shrinker(&nn->nfsd_reply_cache_shrinker);
245+ shrinker_free(nn->nfsd_reply_cache_shrinker);
243246
244247 for (i = 0; i < nn->drc_hashsize; i++) {
245248 struct list_head *head = &nn->drc_hashtbl[i].lru_head;
@@ -323,8 +326,7 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, · 77 unmodified lines
323326 static unsigned long
324327 nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)
325328 {
326− struct nfsd_net *nn = container_of(shrink,
327− struct nfsd_net, nfsd_reply_cache_shrinker);
329+ struct nfsd_net *nn = shrink->private_data;
328330
329331 return atomic_read(&nn->num_drc_entries);
330332 }
@@ -343,8 +345,7 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) · 12 unmodified lines
343345 static unsigned long
344346 nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
345347 {
346− struct nfsd_net *nn = container_of(shrink,
347− struct nfsd_net, nfsd_reply_cache_shrinker);
348+ struct nfsd_net *nn = shrink->private_data;
348349 unsigned long freed = 0;
349350 LIST_HEAD(dispose);
350351 unsigned int i;
fs/nilfs2/mdt.c+34−32
@@ -356,30 +356,28 @@ int nilfs_mdt_delete_block(struct inode *inode, unsigned long block) · 355 unmodified lines
356356 */
357357 int nilfs_mdt_forget_block(struct inode *inode, unsigned long block)
358358 {
359− pgoff_t index = (pgoff_t)block >>
360− (PAGE_SHIFT - inode->i_blkbits);
361− struct page *page;
362− unsigned long first_block;
359+ pgoff_t index = block >> (PAGE_SHIFT - inode->i_blkbits);
360+ struct folio *folio;
361+ struct buffer_head *bh;
363362 int ret = 0;
364363 int still_dirty;
365364
366− page = find_lock_page(inode->i_mapping, index);
367− if (!page)
365+ folio = filemap_lock_folio(inode->i_mapping, index);
366+ if (IS_ERR(folio))
368367 return -ENOENT;
369368
370− wait_on_page_writeback(page);
371−
372− first_block = (unsigned long)index <<
373− (PAGE_SHIFT - inode->i_blkbits);
374− if (page_has_buffers(page)) {
375− struct buffer_head *bh;
369+ folio_wait_writeback(folio);
376370
377− bh = nilfs_page_get_nth_block(page, block - first_block);
371+ bh = folio_buffers(folio);
372+ if (bh) {
373+ unsigned long first_block = index <<
374+ (PAGE_SHIFT - inode->i_blkbits);
375+ bh = get_nth_bh(bh, block - first_block);
378376 nilfs_forget_buffer(bh);
379377 }
380− still_dirty = PageDirty(page);
381− unlock_page(page);
382− put_page(page);
378+ still_dirty = folio_test_dirty(folio);
379+ folio_unlock(folio);
380+ folio_put(folio);
383381
384382 if (still_dirty ||
385383 invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0)
@@ -560,17 +558,19 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 174 unmodified lines
560558 {
561559 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;
562560 struct buffer_head *bh_frozen;
563− struct page *page;
561+ struct folio *folio;
564562 int blkbits = inode->i_blkbits;
565563
566− page = grab_cache_page(shadow->inode->i_mapping, bh->b_folio->index);
567− if (!page)
568− return -ENOMEM;
564+ folio = filemap_grab_folio(shadow->inode->i_mapping,
565+ bh->b_folio->index);
566+ if (IS_ERR(folio))
567+ return PTR_ERR(folio);
569568
570− if (!page_has_buffers(page))
571− create_empty_buffers(page, 1 << blkbits, 0);
569+ bh_frozen = folio_buffers(folio);
570+ if (!bh_frozen)
571+ bh_frozen = create_empty_buffers(folio, 1 << blkbits, 0);
572572
573− bh_frozen = nilfs_page_get_nth_block(page, bh_offset(bh) >> blkbits);
573+ bh_frozen = get_nth_bh(bh_frozen, bh_offset(bh) >> blkbits);
574574
575575 if (!buffer_uptodate(bh_frozen))
576576 nilfs_copy_buffer(bh_frozen, bh);
@@ -582,8 +582,8 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 5 unmodified lines
582582 brelse(bh_frozen); /* already frozen */
583583 }
584584
585− unlock_page(page);
586− put_page(page);
585+ folio_unlock(folio);
586+ folio_put(folio);
587587 return 0;
588588 }
589589
@@ -592,17 +592,19 @@ nilfs_mdt_get_frozen_buffer(struct inode *inode, struct buffer_head *bh) · 2 unmodified lines
592592 {
593593 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;
594594 struct buffer_head *bh_frozen = NULL;
595− struct page *page;
595+ struct folio *folio;
596596 int n;
597597
598− page = find_lock_page(shadow->inode->i_mapping, bh->b_folio->index);
599− if (page) {
600− if (page_has_buffers(page)) {
598+ folio = filemap_lock_folio(shadow->inode->i_mapping,
599+ bh->b_folio->index);
600+ if (!IS_ERR(folio)) {
601+ bh_frozen = folio_buffers(folio);
602+ if (bh_frozen) {
601603 n = bh_offset(bh) >> inode->i_blkbits;
602− bh_frozen = nilfs_page_get_nth_block(page, n);
604+ bh_frozen = get_nth_bh(bh_frozen, n);
603605 }
604− unlock_page(page);
605− put_page(page);
606+ folio_unlock(folio);
607+ folio_put(folio);
606608 }
607609 return bh_frozen;
608610 }
fs/nilfs2/page.c+39−37
@@ -25,19 +25,19 @@ · 24 unmodified lines
2525 (BIT(BH_Uptodate) | BIT(BH_Mapped) | BIT(BH_NILFS_Node) | \
2626 BIT(BH_NILFS_Volatile) | BIT(BH_NILFS_Checked))
2727
28−static struct buffer_head *
29−__nilfs_get_page_block(struct page *page, unsigned long block, pgoff_t index,
30− int blkbits, unsigned long b_state)
28+static struct buffer_head *__nilfs_get_folio_block(struct folio *folio,
29+ unsigned long block, pgoff_t index, int blkbits,
30+ unsigned long b_state)
3131
3232 {
3333 unsigned long first_block;
34− struct buffer_head *bh;
34+ struct buffer_head *bh = folio_buffers(folio);
3535
36− if (!page_has_buffers(page))
37− create_empty_buffers(page, 1 << blkbits, b_state);
36+ if (!bh)
37+ bh = create_empty_buffers(folio, 1 << blkbits, b_state);
3838
3939 first_block = (unsigned long)index << (PAGE_SHIFT - blkbits);
40− bh = nilfs_page_get_nth_block(page, block - first_block);
40+ bh = get_nth_bh(bh, block - first_block);
4141
4242 touch_buffer(bh);
4343 wait_on_buffer(bh);
@@ -51,17 +51,17 @@ struct buffer_head *nilfs_grab_buffer(struct inode *inode, · 7 unmodified lines
5151 {
5252 int blkbits = inode->i_blkbits;
5353 pgoff_t index = blkoff >> (PAGE_SHIFT - blkbits);
54− struct page *page;
54+ struct folio *folio;
5555 struct buffer_head *bh;
5656
57− page = grab_cache_page(mapping, index);
58− if (unlikely(!page))
57+ folio = filemap_grab_folio(mapping, index);
58+ if (IS_ERR(folio))
5959 return NULL;
6060
61− bh = __nilfs_get_page_block(page, blkoff, index, blkbits, b_state);
61+ bh = __nilfs_get_folio_block(folio, blkoff, index, blkbits, b_state);
6262 if (unlikely(!bh)) {
63− unlock_page(page);
64− put_page(page);
63+ folio_unlock(folio);
64+ folio_put(folio);
6565 return NULL;
6666 }
6767 return bh;
@@ -184,30 +184,32 @@ void nilfs_page_bug(struct page *page) · 116 unmodified lines
184184 }
185185
186186 /**
187− * nilfs_copy_page -- copy the page with buffers
188− * @dst: destination page
189− * @src: source page
190− * @copy_dirty: flag whether to copy dirty states on the page's buffer heads.
187+ * nilfs_copy_folio -- copy the folio with buffers
188+ * @dst: destination folio
189+ * @src: source folio
190+ * @copy_dirty: flag whether to copy dirty states on the folio's buffer heads.
191191 *
192− * This function is for both data pages and btnode pages. The dirty flag
193− * should be treated by caller. The page must not be under i/o.
194− * Both src and dst page must be locked
192+ * This function is for both data folios and btnode folios. The dirty flag
193+ * should be treated by caller. The folio must not be under i/o.
194+ * Both src and dst folio must be locked
195195 */
196−static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty)
196+static void nilfs_copy_folio(struct folio *dst, struct folio *src,
197+ bool copy_dirty)
197198 {
198199 struct buffer_head *dbh, *dbufs, *sbh;
199200 unsigned long mask = NILFS_BUFFER_INHERENT_BITS;
200201
201− BUG_ON(PageWriteback(dst));
202+ BUG_ON(folio_test_writeback(dst));
202203
203− sbh = page_buffers(src);
204− if (!page_has_buffers(dst))
205− create_empty_buffers(dst, sbh->b_size, 0);
204+ sbh = folio_buffers(src);
205+ dbh = folio_buffers(dst);
206+ if (!dbh)
207+ dbh = create_empty_buffers(dst, sbh->b_size, 0);
206208
207209 if (copy_dirty)
208210 mask |= BIT(BH_Dirty);
209211
210− dbh = dbufs = page_buffers(dst);
212+ dbufs = dbh;
211213 do {
212214 lock_buffer(sbh);
213215 lock_buffer(dbh);
@@ -218,16 +220,16 @@ static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty) · 4 unmodified lines
218220 dbh = dbh->b_this_page;
219221 } while (dbh != dbufs);
220222
221− copy_highpage(dst, src);
223+ folio_copy(dst, src);
222224
223− if (PageUptodate(src) && !PageUptodate(dst))
224− SetPageUptodate(dst);
225− else if (!PageUptodate(src) && PageUptodate(dst))
226− ClearPageUptodate(dst);
227− if (PageMappedToDisk(src) && !PageMappedToDisk(dst))
228− SetPageMappedToDisk(dst);
229− else if (!PageMappedToDisk(src) && PageMappedToDisk(dst))
230− ClearPageMappedToDisk(dst);
225+ if (folio_test_uptodate(src) && !folio_test_uptodate(dst))
226+ folio_mark_uptodate(dst);
227+ else if (!folio_test_uptodate(src) && folio_test_uptodate(dst))
228+ folio_clear_uptodate(dst);
229+ if (folio_test_mappedtodisk(src) && !folio_test_mappedtodisk(dst))
230+ folio_set_mappedtodisk(dst);
231+ else if (!folio_test_mappedtodisk(src) && folio_test_mappedtodisk(dst))
232+ folio_clear_mappedtodisk(dst);
231233
232234 do {
233235 unlock_buffer(sbh);
@@ -269,7 +271,7 @@ repeat: · 35 unmodified lines
269271 NILFS_PAGE_BUG(&folio->page,
270272 "found empty page in dat page cache");
271273
272− nilfs_copy_page(&dfolio->page, &folio->page, 1);
274+ nilfs_copy_folio(dfolio, folio, true);
273275 filemap_dirty_folio(folio_mapping(dfolio), dfolio);
274276
275277 folio_unlock(dfolio);
@@ -314,7 +316,7 @@ repeat: · 38 unmodified lines
314316 if (!IS_ERR(dfolio)) {
315317 /* overwrite existing folio in the destination cache */
316318 WARN_ON(folio_test_dirty(dfolio));
317− nilfs_copy_page(&dfolio->page, &folio->page, 0);
319+ nilfs_copy_folio(dfolio, folio, false);
318320 folio_unlock(dfolio);
319321 folio_put(dfolio);
320322 /* Do we not need to remove folio from smap here? */
fs/nilfs2/page.h+0−11
@@ -52,15 +52,4 @@ unsigned long nilfs_find_uncommitted_extent(struct inode *inode, · 51 unmodified lines
5252 #define NILFS_PAGE_BUG(page, m, a...) \
5353 do { nilfs_page_bug(page); BUG(); } while (0)
5454
55−static inline struct buffer_head *
56−nilfs_page_get_nth_block(struct page *page, unsigned int count)
57−{
58− struct buffer_head *bh = page_buffers(page);
59−
60− while (count-- > 0)
61− bh = bh->b_this_page;
62− get_bh(bh);
63− return bh;
64−}
65−
6655 #endif /* _NILFS_PAGE_H */
fs/nilfs2/segment.c+3−4
@@ -731,10 +731,9 @@ static size_t nilfs_lookup_dirty_data_buffers(struct inode *inode, · 730 unmodified lines
731731 continue;
732732 }
733733 head = folio_buffers(folio);
734− if (!head) {
735− create_empty_buffers(&folio->page, i_blocksize(inode), 0);
736− head = folio_buffers(folio);
737− }
734+ if (!head)
735+ head = create_empty_buffers(folio,
736+ i_blocksize(inode), 0);
738737 folio_unlock(folio);
739738
740739 bh = head;
fs/ntfs/aops.c+119−136
@@ -145,13 +145,12 @@ still_busy: · 144 unmodified lines
145145 }
146146
147147 /**
148− * ntfs_read_block - fill a @page of an address space with data
149− * @page: page cache page to fill with data
148+ * ntfs_read_block - fill a @folio of an address space with data
149+ * @folio: page cache folio to fill with data
150150 *
151− * Fill the page @page of the address space belonging to the @page->host inode.
152151 * We read each buffer asynchronously and when all buffers are read in, our io
153152 * completion handler ntfs_end_buffer_read_async(), if required, automatically
154− * applies the mst fixups to the page before finally marking it uptodate and
153+ * applies the mst fixups to the folio before finally marking it uptodate and
155154 * unlocking it.
156155 *
157156 * We only enforce allocated_size limit because i_size is checked for in
@@ -161,7 +160,7 @@ still_busy: · 3 unmodified lines
161160 *
162161 * Contains an adapted version of fs/buffer.c::block_read_full_folio().
163162 */
164−static int ntfs_read_block(struct page *page)
163+static int ntfs_read_block(struct folio *folio)
165164 {
166165 loff_t i_size;
167166 VCN vcn;
@@ -178,7 +177,7 @@ static int ntfs_read_block(struct page *page) · 10 unmodified lines
178177 int i, nr;
179178 unsigned char blocksize_bits;
180179
181− vi = page->mapping->host;
180+ vi = folio->mapping->host;
182181 ni = NTFS_I(vi);
183182 vol = ni->vol;
184183
@@ -188,15 +187,10 @@ static int ntfs_read_block(struct page *page) · 3 unmodified lines
188187 blocksize = vol->sb->s_blocksize;
189188 blocksize_bits = vol->sb->s_blocksize_bits;
190189
191− if (!page_has_buffers(page)) {
192− create_empty_buffers(page, blocksize, 0);
193− if (unlikely(!page_has_buffers(page))) {
194− unlock_page(page);
195− return -ENOMEM;
196− }
197− }
198− bh = head = page_buffers(page);
199− BUG_ON(!bh);
190+ head = folio_buffers(folio);
191+ if (!head)
192+ head = create_empty_buffers(folio, blocksize, 0);
193+ bh = head;
200194
201195 /*
202196 * We may be racing with truncate. To avoid some of the problems we
@@ -205,11 +199,11 @@ static int ntfs_read_block(struct page *page) · 2 unmodified lines
205199 * may leave some buffers unmapped which are now allocated. This is
206200 * not a problem since these buffers will just get mapped when a write
207201 * occurs. In case of a shrinking truncate, we will detect this later
208− * on due to the runlist being incomplete and if the page is being
202+ * on due to the runlist being incomplete and if the folio is being
209203 * fully truncated, truncate will throw it away as soon as we unlock
210204 * it so no need to worry what we do with it.
211205 */
212− iblock = (s64)page->index << (PAGE_SHIFT - blocksize_bits);
206+ iblock = (s64)folio->index << (PAGE_SHIFT - blocksize_bits);
213207 read_lock_irqsave(&ni->size_lock, flags);
214208 lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits;
215209 init_size = ni->initialized_size;
@@ -221,7 +215,7 @@ static int ntfs_read_block(struct page *page) · 5 unmodified lines
221215 }
222216 zblock = (init_size + blocksize - 1) >> blocksize_bits;
223217
224− /* Loop through all the buffers in the page. */
218+ /* Loop through all the buffers in the folio. */
225219 rl = NULL;
226220 nr = i = 0;
227221 do {
@@ -299,7 +293,7 @@ lock_retry_remap: · 71 unmodified lines
299293 if (!err)
300294 err = -EIO;
301295 bh->b_blocknr = -1;
302− SetPageError(page);
296+ folio_set_error(folio);
303297 ntfs_error(vol->sb, "Failed to read from inode 0x%lx, "
304298 "attribute type 0x%x, vcn 0x%llx, "
305299 "offset 0x%x because its location on "
@@ -312,13 +306,13 @@ lock_retry_remap: · 6 unmodified lines
312306 /*
313307 * Either iblock was outside lblock limits or
314308 * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion
315− * of the page and set the buffer uptodate.
309+ * of the folio and set the buffer uptodate.
316310 */
317311 handle_hole:
318312 bh->b_blocknr = -1UL;
319313 clear_buffer_mapped(bh);
320314 handle_zblock:
321− zero_user(page, i * blocksize, blocksize);
315+ folio_zero_range(folio, i * blocksize, blocksize);
322316 if (likely(!err))
323317 set_buffer_uptodate(bh);
324318 } while (i++, iblock++, (bh = bh->b_this_page) != head);
@@ -349,11 +343,11 @@ handle_zblock: · 24 unmodified lines
349343 return 0;
350344 }
351345 /* No i/o was scheduled on any of the buffers. */
352− if (likely(!PageError(page)))
353− SetPageUptodate(page);
346+ if (likely(!folio_test_error(folio)))
347+ folio_mark_uptodate(folio);
354348 else /* Signal synchronous i/o error. */
355349 nr = -EIO;
356− unlock_page(page);
350+ folio_unlock(folio);
357351 return nr;
358352 }
359353
@@ -433,7 +427,7 @@ retry_readpage: · 73 unmodified lines
433427 /* NInoNonResident() == NInoIndexAllocPresent() */
434428 if (NInoNonResident(ni)) {
435429 /* Normal, non-resident data stream. */
436− return ntfs_read_block(page);
430+ return ntfs_read_block(folio);
437431 }
438432 /*
439433 * Attribute is resident, implying it is not compressed or encrypted.
@@ -507,28 +501,29 @@ err_out: · 67 unmodified lines
507501 #ifdef NTFS_RW
508502
509503 /**
510− * ntfs_write_block - write a @page to the backing store
511− * @page: page cache page to write out
504+ * ntfs_write_block - write a @folio to the backing store
505+ * @folio: page cache folio to write out
512506 * @wbc: writeback control structure
513507 *
514− * This function is for writing pages belonging to non-resident, non-mst
508+ * This function is for writing folios belonging to non-resident, non-mst
515509 * protected attributes to their backing store.
516510 *
517− * For a page with buffers, map and write the dirty buffers asynchronously
518− * under page writeback. For a page without buffers, create buffers for the
519− * page, then proceed as above.
511+ * For a folio with buffers, map and write the dirty buffers asynchronously
512+ * under folio writeback. For a folio without buffers, create buffers for the
513+ * folio, then proceed as above.
520514 *
521− * If a page doesn't have buffers the page dirty state is definitive. If a page
522− * does have buffers, the page dirty state is just a hint, and the buffer dirty
523− * state is definitive. (A hint which has rules: dirty buffers against a clean
524− * page is illegal. Other combinations are legal and need to be handled. In
525− * particular a dirty page containing clean buffers for example.)
515+ * If a folio doesn't have buffers the folio dirty state is definitive. If
516+ * a folio does have buffers, the folio dirty state is just a hint,
517+ * and the buffer dirty state is definitive. (A hint which has rules:
518+ * dirty buffers against a clean folio is illegal. Other combinations are
519+ * legal and need to be handled. In particular a dirty folio containing
520+ * clean buffers for example.)
526521 *
527522 * Return 0 on success and -errno on error.
528523 *
529524 * Based on ntfs_read_block() and __block_write_full_folio().
530525 */
531−static int ntfs_write_block(struct page *page, struct writeback_control *wbc)
526+static int ntfs_write_block(struct folio *folio, struct writeback_control *wbc)
532527 {
533528 VCN vcn;
534529 LCN lcn;
@@ -546,41 +541,29 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 11 unmodified lines
546541 bool need_end_writeback;
547542 unsigned char blocksize_bits;
548543
549− vi = page->mapping->host;
544+ vi = folio->mapping->host;
550545 ni = NTFS_I(vi);
551546 vol = ni->vol;
552547
553548 ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index "
554− "0x%lx.", ni->mft_no, ni->type, page->index);
549+ "0x%lx.", ni->mft_no, ni->type, folio->index);
555550
556551 BUG_ON(!NInoNonResident(ni));
557552 BUG_ON(NInoMstProtected(ni));
558553 blocksize = vol->sb->s_blocksize;
559554 blocksize_bits = vol->sb->s_blocksize_bits;
560− if (!page_has_buffers(page)) {
561− BUG_ON(!PageUptodate(page));
562− create_empty_buffers(page, blocksize,
555+ head = folio_buffers(folio);
556+ if (!head) {
557+ BUG_ON(!folio_test_uptodate(folio));
558+ head = create_empty_buffers(folio, blocksize,
563559 (1 << BH_Uptodate) | (1 << BH_Dirty));
564− if (unlikely(!page_has_buffers(page))) {
565− ntfs_warning(vol->sb, "Error allocating page "
566− "buffers. Redirtying page so we try "
567− "again later.");
568− /*
569− * Put the page back on mapping->dirty_pages, but leave
570− * its buffers' dirty state as-is.
571− */
572− redirty_page_for_writepage(wbc, page);
573− unlock_page(page);
574− return 0;
575− }
576560 }
577− bh = head = page_buffers(page);
578− BUG_ON(!bh);
561+ bh = head;
579562
580563 /* NOTE: Different naming scheme to ntfs_read_block()! */
581564
582− /* The first block in the page. */
583− block = (s64)page->index << (PAGE_SHIFT - blocksize_bits);
565+ /* The first block in the folio. */
566+ block = (s64)folio->index << (PAGE_SHIFT - blocksize_bits);
584567
585568 read_lock_irqsave(&ni->size_lock, flags);
586569 i_size = i_size_read(vi);
@@ -597,14 +580,14 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 10 unmodified lines
597580 * Be very careful. We have no exclusion from block_dirty_folio
598581 * here, and the (potentially unmapped) buffers may become dirty at
599582 * any time. If a buffer becomes dirty here after we've inspected it
600− * then we just miss that fact, and the page stays dirty.
583+ * then we just miss that fact, and the folio stays dirty.
601584 *
602585 * Buffers outside i_size may be dirtied by block_dirty_folio;
603586 * handle that here by just cleaning them.
604587 */
605588
606589 /*
607− * Loop through all the buffers in the page, mapping all the dirty
590+ * Loop through all the buffers in the folio, mapping all the dirty
608591 * buffers to disk addresses and handling any aliases from the
609592 * underlying block device's mapping.
610593 */
@@ -616,13 +599,13 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 5 unmodified lines
616599 if (unlikely(block >= dblock)) {
617600 /*
618601 * Mapped buffers outside i_size will occur, because
619− * this page can be outside i_size when there is a
602+ * this folio can be outside i_size when there is a
620603 * truncate in progress. The contents of such buffers
621604 * were zeroed by ntfs_writepage().
622605 *
623606 * FIXME: What about the small race window where
624607 * ntfs_writepage() has not done any clearing because
625− * the page was within i_size but before we get here,
608+ * the folio was within i_size but before we get here,
626609 * vmtruncate() modifies i_size?
627610 */
628611 clear_buffer_dirty(bh);
@@ -638,38 +621,38 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 9 unmodified lines
638621 if (unlikely((block >= iblock) &&
639622 (initialized_size < i_size))) {
640623 /*
641− * If this page is fully outside initialized
642− * size, zero out all pages between the current
643− * initialized size and the current page. Just
624+ * If this folio is fully outside initialized
625+ * size, zero out all folios between the current
626+ * initialized size and the current folio. Just
644627 * use ntfs_read_folio() to do the zeroing
645628 * transparently.
646629 */
647630 if (block > iblock) {
648631 // TODO:
649− // For each page do:
650− // - read_cache_page()
651− // Again for each page do:
652− // - wait_on_page_locked()
653− // - Check (PageUptodate(page) &&
654− // !PageError(page))
632+ // For each folio do:
633+ // - read_cache_folio()
634+ // Again for each folio do:
635+ // - wait_on_folio_locked()
636+ // - Check (folio_test_uptodate(folio) &&
637+ // !folio_test_error(folio))
655638 // Update initialized size in the attribute and
656639 // in the inode.
657− // Again, for each page do:
640+ // Again, for each folio do:
658641 // block_dirty_folio();
659− // put_page()
642+ // folio_put()
660643 // We don't need to wait on the writes.
661644 // Update iblock.
662645 }
663646 /*
664− * The current page straddles initialized size. Zero
647+ * The current folio straddles initialized size. Zero
665648 * all non-uptodate buffers and set them uptodate (and
666649 * dirty?). Note, there aren't any non-uptodate buffers
667− * if the page is uptodate.
668− * FIXME: For an uptodate page, the buffers may need to
650+ * if the folio is uptodate.
651+ * FIXME: For an uptodate folio, the buffers may need to
669652 * be written out because they were not initialized on
670653 * disk before.
671654 */
672− if (!PageUptodate(page)) {
655+ if (!folio_test_uptodate(folio)) {
673656 // TODO:
674657 // Zero any non-uptodate buffers up to i_size.
675658 // Set them uptodate and dirty.
@@ -727,14 +710,14 @@ lock_retry_remap: · 51 unmodified lines
727710 unsigned long *bpos, *bend;
728711
729712 /* Check if the buffer is zero. */
730− kaddr = kmap_atomic(page);
731− bpos = (unsigned long *)(kaddr + bh_offset(bh));
732− bend = (unsigned long *)((u8*)bpos + blocksize);
713+ kaddr = kmap_local_folio(folio, bh_offset(bh));
714+ bpos = (unsigned long *)kaddr;
715+ bend = (unsigned long *)(kaddr + blocksize);
733716 do {
734717 if (unlikely(*bpos))
735718 break;
736719 } while (likely(++bpos < bend));
737− kunmap_atomic(kaddr);
720+ kunmap_local(kaddr);
738721 if (bpos == bend) {
739722 /*
740723 * Buffer is zero and sparse, no need to write
@@ -774,7 +757,7 @@ lock_retry_remap: · 33 unmodified lines
774757 if (err == -ENOENT || lcn == LCN_ENOENT) {
775758 bh->b_blocknr = -1;
776759 clear_buffer_dirty(bh);
777− zero_user(page, bh_offset(bh), blocksize);
760+ folio_zero_range(folio, bh_offset(bh), blocksize);
778761 set_buffer_uptodate(bh);
779762 err = 0;
780763 continue;
@@ -801,7 +784,7 @@ lock_retry_remap: · 20 unmodified lines
801784 bh = head;
802785
803786 /* Just an optimization, so ->read_folio() is not called later. */
804− if (unlikely(!PageUptodate(page))) {
787+ if (unlikely(!folio_test_uptodate(folio))) {
805788 int uptodate = 1;
806789 do {
807790 if (!buffer_uptodate(bh)) {
@@ -811,7 +794,7 @@ lock_retry_remap: · 3 unmodified lines
811794 }
812795 } while ((bh = bh->b_this_page) != head);
813796 if (uptodate)
814− SetPageUptodate(page);
797+ folio_mark_uptodate(folio);
815798 }
816799
817800 /* Setup all mapped, dirty buffers for async write i/o. */
@@ -826,7 +809,7 @@ lock_retry_remap: · 8 unmodified lines
826809 } else if (unlikely(err)) {
827810 /*
828811 * For the error case. The buffer may have been set
829− * dirty during attachment to a dirty page.
812+ * dirty during attachment to a dirty folio.
830813 */
831814 if (err != -ENOMEM)
832815 clear_buffer_dirty(bh);
@@ -839,20 +822,20 @@ lock_retry_remap: · 6 unmodified lines
839822 err = 0;
840823 else if (err == -ENOMEM) {
841824 ntfs_warning(vol->sb, "Error allocating memory. "
842− "Redirtying page so we try again "
825+ "Redirtying folio so we try again "
843826 "later.");
844827 /*
845− * Put the page back on mapping->dirty_pages, but
828+ * Put the folio back on mapping->dirty_pages, but
846829 * leave its buffer's dirty state as-is.
847830 */
848− redirty_page_for_writepage(wbc, page);
831+ folio_redirty_for_writepage(wbc, folio);
849832 err = 0;
850833 } else
851− SetPageError(page);
834+ folio_set_error(folio);
852835 }
853836
854− BUG_ON(PageWriteback(page));
855− set_page_writeback(page); /* Keeps try_to_free_buffers() away. */
837+ BUG_ON(folio_test_writeback(folio));
838+ folio_start_writeback(folio); /* Keeps try_to_free_buffers() away. */
856839
857840 /* Submit the prepared buffers for i/o. */
858841 need_end_writeback = true;
@@ -864,11 +847,11 @@ lock_retry_remap: · 5 unmodified lines
864847 }
865848 bh = next;
866849 } while (bh != head);
867− unlock_page(page);
850+ folio_unlock(folio);
868851
869− /* If no i/o was started, need to end_page_writeback(). */
852+ /* If no i/o was started, need to end writeback here. */
870853 if (unlikely(need_end_writeback))
871− end_page_writeback(page);
854+ folio_end_writeback(folio);
872855
873856 ntfs_debug("Done.");
874857 return err;
@@ -1337,8 +1320,9 @@ done: · 462 unmodified lines
13371320 */
13381321 static int ntfs_writepage(struct page *page, struct writeback_control *wbc)
13391322 {
1323+ struct folio *folio = page_folio(page);
13401324 loff_t i_size;
1341− struct inode *vi = page->mapping->host;
1325+ struct inode *vi = folio->mapping->host;
13421326 ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi);
13431327 char *addr;
13441328 ntfs_attr_search_ctx *ctx = NULL;
@@ -1347,14 +1331,13 @@ static int ntfs_writepage(struct page *page, struct writeback_control *wbc) · 2 unmodified lines
13471331 int err;
13481332
13491333 retry_writepage:
1350− BUG_ON(!PageLocked(page));
1334+ BUG_ON(!folio_test_locked(folio));
13511335 i_size = i_size_read(vi);
1352− /* Is the page fully outside i_size? (truncate in progress) */
1353− if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >>
1336+ /* Is the folio fully outside i_size? (truncate in progress) */
1337+ if (unlikely(folio->index >= (i_size + PAGE_SIZE - 1) >>
13541338 PAGE_SHIFT)) {
1355− struct folio *folio = page_folio(page);
13561339 /*
1357− * The page may have dirty, unmapped buffers. Make them
1340+ * The folio may have dirty, unmapped buffers. Make them
13581341 * freeable here, so the page does not leak.
13591342 */
13601343 block_invalidate_folio(folio, 0, folio_size(folio));
@@ -1373,7 +1356,7 @@ retry_writepage: · 12 unmodified lines
13731356 if (ni->type != AT_INDEX_ALLOCATION) {
13741357 /* If file is encrypted, deny access, just like NT4. */
13751358 if (NInoEncrypted(ni)) {
1376− unlock_page(page);
1359+ folio_unlock(folio);
13771360 BUG_ON(ni->type != AT_DATA);
13781361 ntfs_debug("Denying write access to encrypted file.");
13791362 return -EACCES;
@@ -1384,14 +1367,14 @@ retry_writepage: · 4 unmodified lines
13841367 BUG_ON(ni->name_len);
13851368 // TODO: Implement and replace this with
13861369 // return ntfs_write_compressed_block(page);
1387− unlock_page(page);
1370+ folio_unlock(folio);
13881371 ntfs_error(vi->i_sb, "Writing to compressed files is "
13891372 "not supported yet. Sorry.");
13901373 return -EOPNOTSUPP;
13911374 }
13921375 // TODO: Implement and remove this check.
13931376 if (NInoNonResident(ni) && NInoSparse(ni)) {
1394− unlock_page(page);
1377+ folio_unlock(folio);
13951378 ntfs_error(vi->i_sb, "Writing to sparse files is not "
13961379 "supported yet. Sorry.");
13971380 return -EOPNOTSUPP;
@@ -1400,34 +1383,34 @@ retry_writepage: · 2 unmodified lines
14001383 /* NInoNonResident() == NInoIndexAllocPresent() */
14011384 if (NInoNonResident(ni)) {
14021385 /* We have to zero every time due to mmap-at-end-of-file. */
1403− if (page->index >= (i_size >> PAGE_SHIFT)) {
1404− /* The page straddles i_size. */
1405− unsigned int ofs = i_size & ~PAGE_MASK;
1406− zero_user_segment(page, ofs, PAGE_SIZE);
1386+ if (folio->index >= (i_size >> PAGE_SHIFT)) {
1387+ /* The folio straddles i_size. */
1388+ unsigned int ofs = i_size & (folio_size(folio) - 1);
1389+ folio_zero_segment(folio, ofs, folio_size(folio));
14071390 }
14081391 /* Handle mst protected attributes. */
14091392 if (NInoMstProtected(ni))
14101393 return ntfs_write_mst_block(page, wbc);
14111394 /* Normal, non-resident data stream. */
1412− return ntfs_write_block(page, wbc);
1395+ return ntfs_write_block(folio, wbc);
14131396 }
14141397 /*
14151398 * Attribute is resident, implying it is not compressed, encrypted, or
14161399 * mst protected. This also means the attribute is smaller than an mft
1417− * record and hence smaller than a page, so can simply return error on
1418− * any pages with index above 0. Note the attribute can actually be
1400+ * record and hence smaller than a folio, so can simply return error on
1401+ * any folios with index above 0. Note the attribute can actually be
14191402 * marked compressed but if it is resident the actual data is not
14201403 * compressed so we are ok to ignore the compressed flag here.
14211404 */
1422− BUG_ON(page_has_buffers(page));
1423− BUG_ON(!PageUptodate(page));
1424− if (unlikely(page->index > 0)) {
1425− ntfs_error(vi->i_sb, "BUG()! page->index (0x%lx) > 0. "
1426− "Aborting write.", page->index);
1427− BUG_ON(PageWriteback(page));
1428− set_page_writeback(page);
1429− unlock_page(page);
1430− end_page_writeback(page);
1405+ BUG_ON(folio_buffers(folio));
1406+ BUG_ON(!folio_test_uptodate(folio));
1407+ if (unlikely(folio->index > 0)) {
1408+ ntfs_error(vi->i_sb, "BUG()! folio->index (0x%lx) > 0. "
1409+ "Aborting write.", folio->index);
1410+ BUG_ON(folio_test_writeback(folio));
1411+ folio_start_writeback(folio);
1412+ folio_unlock(folio);
1413+ folio_end_writeback(folio);
14311414 return -EIO;
14321415 }
14331416 if (!NInoAttr(ni))
@@ -1460,12 +1443,12 @@ retry_writepage: · 26 unmodified lines
14601443 if (unlikely(err))
14611444 goto err_out;
14621445 /*
1463− * Keep the VM happy. This must be done otherwise the radix-tree tag
1464− * PAGECACHE_TAG_DIRTY remains set even though the page is clean.
1446+ * Keep the VM happy. This must be done otherwise
1447+ * PAGECACHE_TAG_DIRTY remains set even though the folio is clean.
14651448 */
1466− BUG_ON(PageWriteback(page));
1467− set_page_writeback(page);
1468− unlock_page(page);
1449+ BUG_ON(folio_test_writeback(folio));
1450+ folio_start_writeback(folio);
1451+ folio_unlock(folio);
14691452 attr_len = le32_to_cpu(ctx->attr->data.resident.value_length);
14701453 i_size = i_size_read(vi);
14711454 if (unlikely(attr_len > i_size)) {
@@ -1480,18 +1463,18 @@ retry_writepage: · 8 unmodified lines
14801463 /* Shrinking cannot fail. */
14811464 BUG_ON(err);
14821465 }
1483− addr = kmap_atomic(page);
1484− /* Copy the data from the page to the mft record. */
1466+ addr = kmap_local_folio(folio, 0);
1467+ /* Copy the data from the folio to the mft record. */
14851468 memcpy((u8*)ctx->attr +
14861469 le16_to_cpu(ctx->attr->data.resident.value_offset),
14871470 addr, attr_len);
1488− /* Zero out of bounds area in the page cache page. */
1489− memset(addr + attr_len, 0, PAGE_SIZE - attr_len);
1490− kunmap_atomic(addr);
1491− flush_dcache_page(page);
1471+ /* Zero out of bounds area in the page cache folio. */
1472+ memset(addr + attr_len, 0, folio_size(folio) - attr_len);
1473+ kunmap_local(addr);
1474+ flush_dcache_folio(folio);
14921475 flush_dcache_mft_record_page(ctx->ntfs_ino);
1493− /* We are done with the page. */
1494− end_page_writeback(page);
1476+ /* We are done with the folio. */
1477+ folio_end_writeback(folio);
14951478 /* Finally, mark the mft record dirty, so it gets written back. */
14961479 mark_mft_record_dirty(ctx->ntfs_ino);
14971480 ntfs_attr_put_search_ctx(ctx);
@@ -1502,18 +1485,18 @@ err_out: · 4 unmodified lines
15021485 ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying "
15031486 "page so we try again later.");
15041487 /*
1505− * Put the page back on mapping->dirty_pages, but leave its
1488+ * Put the folio back on mapping->dirty_pages, but leave its
15061489 * buffers' dirty state as-is.
15071490 */
1508− redirty_page_for_writepage(wbc, page);
1491+ folio_redirty_for_writepage(wbc, folio);
15091492 err = 0;
15101493 } else {
15111494 ntfs_error(vi->i_sb, "Resident attribute write failed with "
15121495 "error %i.", err);
1513− SetPageError(page);
1496+ folio_set_error(folio);
15141497 NVolSetErrors(ni->vol);
15151498 }
1516− unlock_page(page);
1499+ folio_unlock(folio);
15171500 if (ctx)
15181501 ntfs_attr_put_search_ctx(ctx);
15191502 if (m)
fs/ntfs/file.c+41−48
@@ -567,7 +567,7 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 566 unmodified lines
567567 LCN lcn;
568568 s64 bh_pos, vcn_len, end, initialized_size;
569569 sector_t lcn_block;
570− struct page *page;
570+ struct folio *folio;
571571 struct inode *vi;
572572 ntfs_inode *ni, *base_ni = NULL;
573573 ntfs_volume *vol;
@@ -601,20 +601,6 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 27 unmodified lines
601601 (long long)pos, bytes);
602602 blocksize = vol->sb->s_blocksize;
603603 blocksize_bits = vol->sb->s_blocksize_bits;
604− u = 0;
605− do {
606− page = pages[u];
607− BUG_ON(!page);
608− /*
609− * create_empty_buffers() will create uptodate/dirty buffers if
610− * the page is uptodate/dirty.
611− */
612− if (!page_has_buffers(page)) {
613− create_empty_buffers(page, blocksize, 0);
614− if (unlikely(!page_has_buffers(page)))
615− return -ENOMEM;
616− }
617− } while (++u < nr_pages);
618604 rl_write_locked = false;
619605 rl = NULL;
620606 err = 0;
@@ -626,14 +612,21 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 5 unmodified lines
626612 end = pos + bytes;
627613 cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits;
628614 /*
629− * Loop over each page and for each page over each buffer. Use goto to
615+ * Loop over each buffer in each folio. Use goto to
630616 * reduce indentation.
631617 */
632618 u = 0;
633−do_next_page:
634− page = pages[u];
635− bh_pos = (s64)page->index << PAGE_SHIFT;
636− bh = head = page_buffers(page);
619+do_next_folio:
620+ folio = page_folio(pages[u]);
621+ bh_pos = folio_pos(folio);
622+ head = folio_buffers(folio);
623+ if (!head)
624+ /*
625+ * create_empty_buffers() will create uptodate/dirty
626+ * buffers if the folio is uptodate/dirty.
627+ */
628+ head = create_empty_buffers(folio, blocksize, 0);
629+ bh = head;
637630 do {
638631 VCN cdelta;
639632 s64 bh_end;
@@ -653,15 +646,15 @@ do_next_page: · 13 unmodified lines
653646 if (buffer_uptodate(bh))
654647 continue;
655648 /*
656− * The buffer is not uptodate. If the page is uptodate
649+ * The buffer is not uptodate. If the folio is uptodate
657650 * set the buffer uptodate and otherwise ignore it.
658651 */
659− if (PageUptodate(page)) {
652+ if (folio_test_uptodate(folio)) {
660653 set_buffer_uptodate(bh);
661654 continue;
662655 }
663656 /*
664− * Neither the page nor the buffer are uptodate. If
657+ * Neither the folio nor the buffer are uptodate. If
665658 * the buffer is only partially being written to, we
666659 * need to read it in before the write, i.e. now.
667660 */
@@ -679,7 +672,7 @@ do_next_page: · 11 unmodified lines
679672 ntfs_submit_bh_for_read(bh);
680673 *wait_bh++ = bh;
681674 } else {
682− zero_user(page, bh_offset(bh),
675+ folio_zero_range(folio, bh_offset(bh),
683676 blocksize);
684677 set_buffer_uptodate(bh);
685678 }
@@ -706,7 +699,7 @@ map_buffer_cached: · 20 unmodified lines
706699 (bh_cofs >> blocksize_bits);
707700 set_buffer_mapped(bh);
708701 /*
709− * If the page is uptodate so is the buffer. If the
702+ * If the folio is uptodate so is the buffer. If the
710703 * buffer is fully outside the write, we ignore it if
711704 * it was already allocated and we mark it dirty so it
712705 * gets written out if we allocated it. On the other
@@ -714,7 +707,7 @@ map_buffer_cached: · 1 unmodified line
714707 * marking it dirty we set buffer_new so we can do
715708 * error recovery.
716709 */
717− if (PageUptodate(page)) {
710+ if (folio_test_uptodate(folio)) {
718711 if (!buffer_uptodate(bh))
719712 set_buffer_uptodate(bh);
720713 if (unlikely(was_hole)) {
@@ -754,7 +747,8 @@ map_buffer_cached: · 33 unmodified lines
754747 ntfs_submit_bh_for_read(bh);
755748 *wait_bh++ = bh;
756749 } else {
757− zero_user(page, bh_offset(bh),
750+ folio_zero_range(folio,
751+ bh_offset(bh),
758752 blocksize);
759753 set_buffer_uptodate(bh);
760754 }
@@ -773,7 +767,7 @@ map_buffer_cached: · 12 unmodified lines
773767 */
774768 if (bh_end <= pos || bh_pos >= end) {
775769 if (!buffer_uptodate(bh)) {
776− zero_user(page, bh_offset(bh),
770+ folio_zero_range(folio, bh_offset(bh),
777771 blocksize);
778772 set_buffer_uptodate(bh);
779773 }
@@ -786,7 +780,7 @@ map_buffer_cached: · 6 unmodified lines
786780 u8 *kaddr;
787781 unsigned pofs;
788782
789− kaddr = kmap_atomic(page);
783+ kaddr = kmap_local_folio(folio, 0);
790784 if (bh_pos < pos) {
791785 pofs = bh_pos & ~PAGE_MASK;
792786 memset(kaddr + pofs, 0, pos - bh_pos);
@@ -795,8 +789,8 @@ map_buffer_cached: · 2 unmodified lines
795789 pofs = end & ~PAGE_MASK;
796790 memset(kaddr + pofs, 0, bh_end - end);
797791 }
798− kunmap_atomic(kaddr);
799− flush_dcache_page(page);
792+ kunmap_local(kaddr);
793+ flush_dcache_folio(folio);
800794 }
801795 continue;
802796 }
@@ -809,11 +803,12 @@ map_buffer_cached: · 6 unmodified lines
809803 initialized_size = ni->allocated_size;
810804 read_unlock_irqrestore(&ni->size_lock, flags);
811805 if (bh_pos > initialized_size) {
812− if (PageUptodate(page)) {
806+ if (folio_test_uptodate(folio)) {
813807 if (!buffer_uptodate(bh))
814808 set_buffer_uptodate(bh);
815809 } else if (!buffer_uptodate(bh)) {
816− zero_user(page, bh_offset(bh), blocksize);
810+ folio_zero_range(folio, bh_offset(bh),
811+ blocksize);
817812 set_buffer_uptodate(bh);
818813 }
819814 continue;
@@ -927,17 +922,17 @@ rl_not_mapped_enoent: · 107 unmodified lines
927922 bh->b_blocknr = -1;
928923 /*
929924 * If the buffer is uptodate we skip it. If it
930− * is not but the page is uptodate, we can set
931− * the buffer uptodate. If the page is not
925+ * is not but the folio is uptodate, we can set
926+ * the buffer uptodate. If the folio is not
932927 * uptodate, we can clear the buffer and set it
933928 * uptodate. Whether this is worthwhile is
934929 * debatable and this could be removed.
935930 */
936− if (PageUptodate(page)) {
931+ if (folio_test_uptodate(folio)) {
937932 if (!buffer_uptodate(bh))
938933 set_buffer_uptodate(bh);
939934 } else if (!buffer_uptodate(bh)) {
940− zero_user(page, bh_offset(bh),
935+ folio_zero_range(folio, bh_offset(bh),
941936 blocksize);
942937 set_buffer_uptodate(bh);
943938 }
@@ -1167,7 +1162,7 @@ rl_not_mapped_enoent: · 223 unmodified lines
11671162 } while (bh_pos += blocksize, (bh = bh->b_this_page) != head);
11681163 /* If there are no errors, do the next page. */
11691164 if (likely(!err && ++u < nr_pages))
1170− goto do_next_page;
1165+ goto do_next_folio;
11711166 /* If there are no errors, release the runlist lock if we took it. */
11721167 if (likely(!err)) {
11731168 if (unlikely(rl_write_locked)) {
@@ -1185,9 +1180,8 @@ rl_not_mapped_enoent: · 11 unmodified lines
11851180 bh = *--wait_bh;
11861181 wait_on_buffer(bh);
11871182 if (likely(buffer_uptodate(bh))) {
1188− page = bh->b_page;
1189− bh_pos = ((s64)page->index << PAGE_SHIFT) +
1190− bh_offset(bh);
1183+ folio = bh->b_folio;
1184+ bh_pos = folio_pos(folio) + bh_offset(bh);
11911185 /*
11921186 * If the buffer overflows the initialized size, need
11931187 * to zero the overflowing region.
@@ -1197,7 +1191,7 @@ rl_not_mapped_enoent: · 3 unmodified lines
11971191
11981192 if (likely(bh_pos < initialized_size))
11991193 ofs = initialized_size - bh_pos;
1200− zero_user_segment(page, bh_offset(bh) + ofs,
1194+ folio_zero_segment(folio, bh_offset(bh) + ofs,
12011195 blocksize);
12021196 }
12031197 } else /* if (unlikely(!buffer_uptodate(bh))) */
@@ -1324,21 +1318,20 @@ rl_not_mapped_enoent: · 120 unmodified lines
13241318 u = 0;
13251319 end = bh_cpos << vol->cluster_size_bits;
13261320 do {
1327− page = pages[u];
1328− bh = head = page_buffers(page);
1321+ folio = page_folio(pages[u]);
1322+ bh = head = folio_buffers(folio);
13291323 do {
13301324 if (u == nr_pages &&
1331− ((s64)page->index << PAGE_SHIFT) +
1332− bh_offset(bh) >= end)
1325+ folio_pos(folio) + bh_offset(bh) >= end)
13331326 break;
13341327 if (!buffer_new(bh))
13351328 continue;
13361329 clear_buffer_new(bh);
13371330 if (!buffer_uptodate(bh)) {
1338− if (PageUptodate(page))
1331+ if (folio_test_uptodate(folio))
13391332 set_buffer_uptodate(bh);
13401333 else {
1341− zero_user(page, bh_offset(bh),
1334+ folio_zero_range(folio, bh_offset(bh),
13421335 blocksize);
13431336 set_buffer_uptodate(bh);
13441337 }
fs/ntfs3/file.c+16−15
@@ -187,7 +187,7 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 186 unmodified lines
187187 struct buffer_head *head, *bh;
188188 u32 bh_next, bh_off, to;
189189 sector_t iblock;
190− struct page *page;
190+ struct folio *folio;
191191
192192 for (; idx < idx_end; idx += 1, from = 0) {
193193 page_off = (loff_t)idx << PAGE_SHIFT;
@@ -195,16 +195,17 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 1 unmodified line
195195 PAGE_SIZE;
196196 iblock = page_off >> inode->i_blkbits;
197197
198− page = find_or_create_page(mapping, idx,
199− mapping_gfp_constraint(mapping,
200− ~__GFP_FS));
201− if (!page)
202− return -ENOMEM;
198+ folio = __filemap_get_folio(mapping, idx,
199+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
200+ mapping_gfp_constraint(mapping, ~__GFP_FS));
201+ if (IS_ERR(folio))
202+ return PTR_ERR(folio);
203203
204− if (!page_has_buffers(page))
205− create_empty_buffers(page, blocksize, 0);
204+ head = folio_buffers(folio);
205+ if (!head)
206+ head = create_empty_buffers(folio, blocksize, 0);
206207
207− bh = head = page_buffers(page);
208+ bh = head;
208209 bh_off = 0;
209210 do {
210211 bh_next = bh_off + blocksize;
@@ -220,14 +221,14 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 9 unmodified lines
220221 }
221222
222223 /* Ok, it's mapped. Make sure it's up-to-date. */
223− if (PageUptodate(page))
224+ if (folio_test_uptodate(folio))
224225 set_buffer_uptodate(bh);
225226
226227 if (!buffer_uptodate(bh)) {
227228 err = bh_read(bh, 0);
228229 if (err < 0) {
229− unlock_page(page);
230− put_page(page);
230+ folio_unlock(folio);
231+ folio_put(folio);
231232 goto out;
232233 }
233234 }
@@ -237,10 +238,10 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 3 unmodified lines
237238 } while (bh_off = bh_next, iblock += 1,
238239 head != (bh = bh->b_this_page));
239240
240− zero_user_segment(page, from, to);
241+ folio_zero_segment(folio, from, to);
241242
242− unlock_page(page);
243− put_page(page);
243+ folio_unlock(folio);
244+ folio_put(folio);
244245 cond_resched();
245246 }
246247 out:
fs/ocfs2/aops.c+10−9
@@ -568,10 +568,10 @@ static void ocfs2_clear_page_regions(struct page *page, · 567 unmodified lines
568568 * read-in the blocks at the tail of our file. Avoid reading them by
569569 * testing i_size against each block offset.
570570 */
571−static int ocfs2_should_read_blk(struct inode *inode, struct page *page,
571+static int ocfs2_should_read_blk(struct inode *inode, struct folio *folio,
572572 unsigned int block_start)
573573 {
574− u64 offset = page_offset(page) + block_start;
574+ u64 offset = folio_pos(folio) + block_start;
575575
576576 if (ocfs2_sparse_alloc(OCFS2_SB(inode->i_sb)))
577577 return 1;
@@ -593,15 +593,16 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 15 unmodified lines
593593 struct inode *inode, unsigned int from,
594594 unsigned int to, int new)
595595 {
596+ struct folio *folio = page_folio(page);
596597 int ret = 0;
597598 struct buffer_head *head, *bh, *wait[2], **wait_bh = wait;
598599 unsigned int block_end, block_start;
599600 unsigned int bsize = i_blocksize(inode);
600601
601− if (!page_has_buffers(page))
602− create_empty_buffers(page, bsize, 0);
602+ head = folio_buffers(folio);
603+ if (!head)
604+ head = create_empty_buffers(folio, bsize, 0);
603605
604− head = page_buffers(page);
605606 for (bh = head, block_start = 0; bh != head || !block_start;
606607 bh = bh->b_this_page, block_start += bsize) {
607608 block_end = block_start + bsize;
@@ -613,7 +614,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 5 unmodified lines
613614 * they may belong to unallocated clusters.
614615 */
615616 if (block_start >= to || block_end <= from) {
616− if (PageUptodate(page))
617+ if (folio_test_uptodate(folio))
617618 set_buffer_uptodate(bh);
618619 continue;
619620 }
@@ -630,11 +631,11 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 10 unmodified lines
630631 clean_bdev_bh_alias(bh);
631632 }
632633
633− if (PageUptodate(page)) {
634+ if (folio_test_uptodate(folio)) {
634635 set_buffer_uptodate(bh);
635636 } else if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
636637 !buffer_new(bh) &&
637− ocfs2_should_read_blk(inode, page, block_start) &&
638+ ocfs2_should_read_blk(inode, folio, block_start) &&
638639 (block_start < from || block_end > to)) {
639640 bh_read_nowait(bh, 0);
640641 *wait_bh++=bh;
@@ -668,7 +669,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 27 unmodified lines
668669 if (block_start >= to)
669670 break;
670671
671− zero_user(page, block_start, bh->b_size);
672+ folio_zero_range(folio, block_start, bh->b_size);
672673 set_buffer_uptodate(bh);
673674 mark_buffer_dirty(bh);
674675
fs/proc/kcore.c+2−1
@@ -546,7 +546,8 @@ static ssize_t read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) · 545 unmodified lines
546546 * and explicitly excluded physical ranges.
547547 */
548548 if (!page || PageOffline(page) ||
549− is_page_hwpoison(page) || !pfn_is_ram(pfn)) {
549+ is_page_hwpoison(page) || !pfn_is_ram(pfn) ||
550+ pfn_is_unaccepted_memory(pfn)) {
550551 if (iov_iter_zero(tsz, iter) != tsz) {
551552 ret = -EFAULT;
552553 goto out;
fs/proc/root.c+1−1
@@ -188,7 +188,7 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc) · 187 unmodified lines
188188 s->s_stack_depth = FILESYSTEM_MAX_STACK_DEPTH;
189189
190190 /* procfs dentries and inodes don't require IO to create */
191− s->s_shrink.seeks = 0;
191+ s->s_shrink->seeks = 0;
192192
193193 pde_get(&proc_root);
194194 root_inode = proc_get_inode(s, &proc_root);
fs/proc/task_mmu.c+731−2
@@ -20,6 +20,8 @@ · 19 unmodified lines
2020 #include <linux/shmem_fs.h>
2121 #include <linux/uaccess.h>
2222 #include <linux/pkeys.h>
23+#include <linux/minmax.h>
24+#include <linux/overflow.h>
2325
2426 #include <asm/elf.h>
2527 #include <asm/tlb.h>
@@ -1761,11 +1763,737 @@ static int pagemap_release(struct inode *inode, struct file *file) · 1735 unmodified lines
17611763 return 0;
17621764 }
17631765
1766+#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \
1767+ PAGE_IS_FILE | PAGE_IS_PRESENT | \
1768+ PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \
1769+ PAGE_IS_HUGE)
1770+#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
1771+
1772+struct pagemap_scan_private {
1773+ struct pm_scan_arg arg;
1774+ unsigned long masks_of_interest, cur_vma_category;
1775+ struct page_region *vec_buf;
1776+ unsigned long vec_buf_len, vec_buf_index, found_pages;
1777+ struct page_region __user *vec_out;
1778+};
1779+
1780+static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
1781+ struct vm_area_struct *vma,
1782+ unsigned long addr, pte_t pte)
1783+{
1784+ unsigned long categories = 0;
1785+
1786+ if (pte_present(pte)) {
1787+ struct page *page;
1788+
1789+ categories |= PAGE_IS_PRESENT;
1790+ if (!pte_uffd_wp(pte))
1791+ categories |= PAGE_IS_WRITTEN;
1792+
1793+ if (p->masks_of_interest & PAGE_IS_FILE) {
1794+ page = vm_normal_page(vma, addr, pte);
1795+ if (page && !PageAnon(page))
1796+ categories |= PAGE_IS_FILE;
1797+ }
1798+
1799+ if (is_zero_pfn(pte_pfn(pte)))
1800+ categories |= PAGE_IS_PFNZERO;
1801+ } else if (is_swap_pte(pte)) {
1802+ swp_entry_t swp;
1803+
1804+ categories |= PAGE_IS_SWAPPED;
1805+ if (!pte_swp_uffd_wp_any(pte))
1806+ categories |= PAGE_IS_WRITTEN;
1807+
1808+ if (p->masks_of_interest & PAGE_IS_FILE) {
1809+ swp = pte_to_swp_entry(pte);
1810+ if (is_pfn_swap_entry(swp) &&
1811+ !PageAnon(pfn_swap_entry_to_page(swp)))
1812+ categories |= PAGE_IS_FILE;
1813+ }
1814+ }
1815+
1816+ return categories;
1817+}
1818+
1819+static void make_uffd_wp_pte(struct vm_area_struct *vma,
1820+ unsigned long addr, pte_t *pte)
1821+{
1822+ pte_t ptent = ptep_get(pte);
1823+
1824+ if (pte_present(ptent)) {
1825+ pte_t old_pte;
1826+
1827+ old_pte = ptep_modify_prot_start(vma, addr, pte);
1828+ ptent = pte_mkuffd_wp(ptent);
1829+ ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
1830+ } else if (is_swap_pte(ptent)) {
1831+ ptent = pte_swp_mkuffd_wp(ptent);
1832+ set_pte_at(vma->vm_mm, addr, pte, ptent);
1833+ } else {
1834+ set_pte_at(vma->vm_mm, addr, pte,
1835+ make_pte_marker(PTE_MARKER_UFFD_WP));
1836+ }
1837+}
1838+
1839+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1840+static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
1841+ struct vm_area_struct *vma,
1842+ unsigned long addr, pmd_t pmd)
1843+{
1844+ unsigned long categories = PAGE_IS_HUGE;
1845+
1846+ if (pmd_present(pmd)) {
1847+ struct page *page;
1848+
1849+ categories |= PAGE_IS_PRESENT;
1850+ if (!pmd_uffd_wp(pmd))
1851+ categories |= PAGE_IS_WRITTEN;
1852+
1853+ if (p->masks_of_interest & PAGE_IS_FILE) {
1854+ page = vm_normal_page_pmd(vma, addr, pmd);
1855+ if (page && !PageAnon(page))
1856+ categories |= PAGE_IS_FILE;
1857+ }
1858+
1859+ if (is_zero_pfn(pmd_pfn(pmd)))
1860+ categories |= PAGE_IS_PFNZERO;
1861+ } else if (is_swap_pmd(pmd)) {
1862+ swp_entry_t swp;
1863+
1864+ categories |= PAGE_IS_SWAPPED;
1865+ if (!pmd_swp_uffd_wp(pmd))
1866+ categories |= PAGE_IS_WRITTEN;
1867+
1868+ if (p->masks_of_interest & PAGE_IS_FILE) {
1869+ swp = pmd_to_swp_entry(pmd);
1870+ if (is_pfn_swap_entry(swp) &&
1871+ !PageAnon(pfn_swap_entry_to_page(swp)))
1872+ categories |= PAGE_IS_FILE;
1873+ }
1874+ }
1875+
1876+ return categories;
1877+}
1878+
1879+static void make_uffd_wp_pmd(struct vm_area_struct *vma,
1880+ unsigned long addr, pmd_t *pmdp)
1881+{
1882+ pmd_t old, pmd = *pmdp;
1883+
1884+ if (pmd_present(pmd)) {
1885+ old = pmdp_invalidate_ad(vma, addr, pmdp);
1886+ pmd = pmd_mkuffd_wp(old);
1887+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
1888+ } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
1889+ pmd = pmd_swp_mkuffd_wp(pmd);
1890+ set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
1891+ }
1892+}
1893+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
1894+
1895+#ifdef CONFIG_HUGETLB_PAGE
1896+static unsigned long pagemap_hugetlb_category(pte_t pte)
1897+{
1898+ unsigned long categories = PAGE_IS_HUGE;
1899+
1900+ /*
1901+ * According to pagemap_hugetlb_range(), file-backed HugeTLB
1902+ * page cannot be swapped. So PAGE_IS_FILE is not checked for
1903+ * swapped pages.
1904+ */
1905+ if (pte_present(pte)) {
1906+ categories |= PAGE_IS_PRESENT;
1907+ if (!huge_pte_uffd_wp(pte))
1908+ categories |= PAGE_IS_WRITTEN;
1909+ if (!PageAnon(pte_page(pte)))
1910+ categories |= PAGE_IS_FILE;
1911+ if (is_zero_pfn(pte_pfn(pte)))
1912+ categories |= PAGE_IS_PFNZERO;
1913+ } else if (is_swap_pte(pte)) {
1914+ categories |= PAGE_IS_SWAPPED;
1915+ if (!pte_swp_uffd_wp_any(pte))
1916+ categories |= PAGE_IS_WRITTEN;
1917+ }
1918+
1919+ return categories;
1920+}
1921+
1922+static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
1923+ unsigned long addr, pte_t *ptep,
1924+ pte_t ptent)
1925+{
1926+ unsigned long psize;
1927+
1928+ if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
1929+ return;
1930+
1931+ psize = huge_page_size(hstate_vma(vma));
1932+
1933+ if (is_hugetlb_entry_migration(ptent))
1934+ set_huge_pte_at(vma->vm_mm, addr, ptep,
1935+ pte_swp_mkuffd_wp(ptent), psize);
1936+ else if (!huge_pte_none(ptent))
1937+ huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
1938+ huge_pte_mkuffd_wp(ptent));
1939+ else
1940+ set_huge_pte_at(vma->vm_mm, addr, ptep,
1941+ make_pte_marker(PTE_MARKER_UFFD_WP), psize);
1942+}
1943+#endif /* CONFIG_HUGETLB_PAGE */
1944+
1945+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE)
1946+static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
1947+ unsigned long addr, unsigned long end)
1948+{
1949+ struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index];
1950+
1951+ if (cur_buf->start != addr)
1952+ cur_buf->end = addr;
1953+ else
1954+ cur_buf->start = cur_buf->end = 0;
1955+
1956+ p->found_pages -= (end - addr) / PAGE_SIZE;
1957+}
1958+#endif
1959+
1960+static bool pagemap_scan_is_interesting_page(unsigned long categories,
1961+ const struct pagemap_scan_private *p)
1962+{
1963+ categories ^= p->arg.category_inverted;
1964+ if ((categories & p->arg.category_mask) != p->arg.category_mask)
1965+ return false;
1966+ if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask))
1967+ return false;
1968+
1969+ return true;
1970+}
1971+
1972+static bool pagemap_scan_is_interesting_vma(unsigned long categories,
1973+ const struct pagemap_scan_private *p)
1974+{
1975+ unsigned long required = p->arg.category_mask & PAGE_IS_WPALLOWED;
1976+
1977+ categories ^= p->arg.category_inverted;
1978+ if ((categories & required) != required)
1979+ return false;
1980+
1981+ return true;
1982+}
1983+
1984+static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
1985+ struct mm_walk *walk)
1986+{
1987+ struct pagemap_scan_private *p = walk->private;
1988+ struct vm_area_struct *vma = walk->vma;
1989+ unsigned long vma_category = 0;
1990+
1991+ if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
1992+ vma_category |= PAGE_IS_WPALLOWED;
1993+ else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
1994+ return -EPERM;
1995+
1996+ if (vma->vm_flags & VM_PFNMAP)
1997+ return 1;
1998+
1999+ if (!pagemap_scan_is_interesting_vma(vma_category, p))
2000+ return 1;
2001+
2002+ p->cur_vma_category = vma_category;
2003+
2004+ return 0;
2005+}
2006+
2007+static bool pagemap_scan_push_range(unsigned long categories,
2008+ struct pagemap_scan_private *p,
2009+ unsigned long addr, unsigned long end)
2010+{
2011+ struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index];
2012+
2013+ /*
2014+ * When there is no output buffer provided at all, the sentinel values
2015+ * won't match here. There is no other way for `cur_buf->end` to be
2016+ * non-zero other than it being non-empty.
2017+ */
2018+ if (addr == cur_buf->end && categories == cur_buf->categories) {
2019+ cur_buf->end = end;
2020+ return true;
2021+ }
2022+
2023+ if (cur_buf->end) {
2024+ if (p->vec_buf_index >= p->vec_buf_len - 1)
2025+ return false;
2026+
2027+ cur_buf = &p->vec_buf[++p->vec_buf_index];
2028+ }
2029+
2030+ cur_buf->start = addr;
2031+ cur_buf->end = end;
2032+ cur_buf->categories = categories;
2033+
2034+ return true;
2035+}
2036+
2037+static int pagemap_scan_output(unsigned long categories,
2038+ struct pagemap_scan_private *p,
2039+ unsigned long addr, unsigned long *end)
2040+{
2041+ unsigned long n_pages, total_pages;
2042+ int ret = 0;
2043+
2044+ if (!p->vec_buf)
2045+ return 0;
2046+
2047+ categories &= p->arg.return_mask;
2048+
2049+ n_pages = (*end - addr) / PAGE_SIZE;
2050+ if (check_add_overflow(p->found_pages, n_pages, &total_pages) ||
2051+ total_pages > p->arg.max_pages) {
2052+ size_t n_too_much = total_pages - p->arg.max_pages;
2053+ *end -= n_too_much * PAGE_SIZE;
2054+ n_pages -= n_too_much;
2055+ ret = -ENOSPC;
2056+ }
2057+
2058+ if (!pagemap_scan_push_range(categories, p, addr, *end)) {
2059+ *end = addr;
2060+ n_pages = 0;
2061+ ret = -ENOSPC;
2062+ }
2063+
2064+ p->found_pages += n_pages;
2065+ if (ret)
2066+ p->arg.walk_end = *end;
2067+
2068+ return ret;
2069+}
2070+
2071+static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
2072+ unsigned long end, struct mm_walk *walk)
2073+{
2074+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
2075+ struct pagemap_scan_private *p = walk->private;
2076+ struct vm_area_struct *vma = walk->vma;
2077+ unsigned long categories;
2078+ spinlock_t *ptl;
2079+ int ret = 0;
2080+
2081+ ptl = pmd_trans_huge_lock(pmd, vma);
2082+ if (!ptl)
2083+ return -ENOENT;
2084+
2085+ categories = p->cur_vma_category |
2086+ pagemap_thp_category(p, vma, start, *pmd);
2087+
2088+ if (!pagemap_scan_is_interesting_page(categories, p))
2089+ goto out_unlock;
2090+
2091+ ret = pagemap_scan_output(categories, p, start, &end);
2092+ if (start == end)
2093+ goto out_unlock;
2094+
2095+ if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2096+ goto out_unlock;
2097+ if (~categories & PAGE_IS_WRITTEN)
2098+ goto out_unlock;
2099+
2100+ /*
2101+ * Break huge page into small pages if the WP operation
2102+ * needs to be performed on a portion of the huge page.
2103+ */
2104+ if (end != start + HPAGE_SIZE) {
2105+ spin_unlock(ptl);
2106+ split_huge_pmd(vma, pmd, start);
2107+ pagemap_scan_backout_range(p, start, end);
2108+ /* Report as if there was no THP */
2109+ return -ENOENT;
2110+ }
2111+
2112+ make_uffd_wp_pmd(vma, start, pmd);
2113+ flush_tlb_range(vma, start, end);
2114+out_unlock:
2115+ spin_unlock(ptl);
2116+ return ret;
2117+#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
2118+ return -ENOENT;
2119+#endif
2120+}
2121+
2122+static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
2123+ unsigned long end, struct mm_walk *walk)
2124+{
2125+ struct pagemap_scan_private *p = walk->private;
2126+ struct vm_area_struct *vma = walk->vma;
2127+ unsigned long addr, flush_end = 0;
2128+ pte_t *pte, *start_pte;
2129+ spinlock_t *ptl;
2130+ int ret;
2131+
2132+ arch_enter_lazy_mmu_mode();
2133+
2134+ ret = pagemap_scan_thp_entry(pmd, start, end, walk);
2135+ if (ret != -ENOENT) {
2136+ arch_leave_lazy_mmu_mode();
2137+ return ret;
2138+ }
2139+
2140+ ret = 0;
2141+ start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
2142+ if (!pte) {
2143+ arch_leave_lazy_mmu_mode();
2144+ walk->action = ACTION_AGAIN;
2145+ return 0;
2146+ }
2147+
2148+ if (!p->vec_out) {
2149+ /* Fast path for performing exclusive WP */
2150+ for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
2151+ if (pte_uffd_wp(ptep_get(pte)))
2152+ continue;
2153+ make_uffd_wp_pte(vma, addr, pte);
2154+ if (!flush_end)
2155+ start = addr;
2156+ flush_end = addr + PAGE_SIZE;
2157+ }
2158+ goto flush_and_return;
2159+ }
2160+
2161+ if (!p->arg.category_anyof_mask && !p->arg.category_inverted &&
2162+ p->arg.category_mask == PAGE_IS_WRITTEN &&
2163+ p->arg.return_mask == PAGE_IS_WRITTEN) {
2164+ for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
2165+ unsigned long next = addr + PAGE_SIZE;
2166+
2167+ if (pte_uffd_wp(ptep_get(pte)))
2168+ continue;
2169+ ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN,
2170+ p, addr, &next);
2171+ if (next == addr)
2172+ break;
2173+ if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2174+ continue;
2175+ make_uffd_wp_pte(vma, addr, pte);
2176+ if (!flush_end)
2177+ start = addr;
2178+ flush_end = next;
2179+ }
2180+ goto flush_and_return;
2181+ }
2182+
2183+ for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
2184+ unsigned long categories = p->cur_vma_category |
2185+ pagemap_page_category(p, vma, addr, ptep_get(pte));
2186+ unsigned long next = addr + PAGE_SIZE;
2187+
2188+ if (!pagemap_scan_is_interesting_page(categories, p))
2189+ continue;
2190+
2191+ ret = pagemap_scan_output(categories, p, addr, &next);
2192+ if (next == addr)
2193+ break;
2194+
2195+ if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2196+ continue;
2197+ if (~categories & PAGE_IS_WRITTEN)
2198+ continue;
2199+
2200+ make_uffd_wp_pte(vma, addr, pte);
2201+ if (!flush_end)
2202+ start = addr;
2203+ flush_end = next;
2204+ }
2205+
2206+flush_and_return:
2207+ if (flush_end)
2208+ flush_tlb_range(vma, start, addr);
2209+
2210+ pte_unmap_unlock(start_pte, ptl);
2211+ arch_leave_lazy_mmu_mode();
2212+
2213+ cond_resched();
2214+ return ret;
2215+}
2216+
2217+#ifdef CONFIG_HUGETLB_PAGE
2218+static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
2219+ unsigned long start, unsigned long end,
2220+ struct mm_walk *walk)
2221+{
2222+ struct pagemap_scan_private *p = walk->private;
2223+ struct vm_area_struct *vma = walk->vma;
2224+ unsigned long categories;
2225+ spinlock_t *ptl;
2226+ int ret = 0;
2227+ pte_t pte;
2228+
2229+ if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
2230+ /* Go the short route when not write-protecting pages. */
2231+
2232+ pte = huge_ptep_get(ptep);
2233+ categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
2234+
2235+ if (!pagemap_scan_is_interesting_page(categories, p))
2236+ return 0;
2237+
2238+ return pagemap_scan_output(categories, p, start, &end);
2239+ }
2240+
2241+ i_mmap_lock_write(vma->vm_file->f_mapping);
2242+ ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
2243+
2244+ pte = huge_ptep_get(ptep);
2245+ categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
2246+
2247+ if (!pagemap_scan_is_interesting_page(categories, p))
2248+ goto out_unlock;
2249+
2250+ ret = pagemap_scan_output(categories, p, start, &end);
2251+ if (start == end)
2252+ goto out_unlock;
2253+
2254+ if (~categories & PAGE_IS_WRITTEN)
2255+ goto out_unlock;
2256+
2257+ if (end != start + HPAGE_SIZE) {
2258+ /* Partial HugeTLB page WP isn't possible. */
2259+ pagemap_scan_backout_range(p, start, end);
2260+ p->arg.walk_end = start;
2261+ ret = 0;
2262+ goto out_unlock;
2263+ }
2264+
2265+ make_uffd_wp_huge_pte(vma, start, ptep, pte);
2266+ flush_hugetlb_tlb_range(vma, start, end);
2267+
2268+out_unlock:
2269+ spin_unlock(ptl);
2270+ i_mmap_unlock_write(vma->vm_file->f_mapping);
2271+
2272+ return ret;
2273+}
2274+#else
2275+#define pagemap_scan_hugetlb_entry NULL
2276+#endif
2277+
2278+static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
2279+ int depth, struct mm_walk *walk)
2280+{
2281+ struct pagemap_scan_private *p = walk->private;
2282+ struct vm_area_struct *vma = walk->vma;
2283+ int ret, err;
2284+
2285+ if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p))
2286+ return 0;
2287+
2288+ ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
2289+ if (addr == end)
2290+ return ret;
2291+
2292+ if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2293+ return ret;
2294+
2295+ err = uffd_wp_range(vma, addr, end - addr, true);
2296+ if (err < 0)
2297+ ret = err;
2298+
2299+ return ret;
2300+}
2301+
2302+static const struct mm_walk_ops pagemap_scan_ops = {
2303+ .test_walk = pagemap_scan_test_walk,
2304+ .pmd_entry = pagemap_scan_pmd_entry,
2305+ .pte_hole = pagemap_scan_pte_hole,
2306+ .hugetlb_entry = pagemap_scan_hugetlb_entry,
2307+};
2308+
2309+static int pagemap_scan_get_args(struct pm_scan_arg *arg,
2310+ unsigned long uarg)
2311+{
2312+ if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
2313+ return -EFAULT;
2314+
2315+ if (arg->size != sizeof(struct pm_scan_arg))
2316+ return -EINVAL;
2317+
2318+ /* Validate requested features */
2319+ if (arg->flags & ~PM_SCAN_FLAGS)
2320+ return -EINVAL;
2321+ if ((arg->category_inverted | arg->category_mask |
2322+ arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
2323+ return -EINVAL;
2324+
2325+ arg->start = untagged_addr((unsigned long)arg->start);
2326+ arg->end = untagged_addr((unsigned long)arg->end);
2327+ arg->vec = untagged_addr((unsigned long)arg->vec);
2328+
2329+ /* Validate memory pointers */
2330+ if (!IS_ALIGNED(arg->start, PAGE_SIZE))
2331+ return -EINVAL;
2332+ if (!access_ok((void __user *)(long)arg->start, arg->end - arg->start))
2333+ return -EFAULT;
2334+ if (!arg->vec && arg->vec_len)
2335+ return -EINVAL;
2336+ if (arg->vec && !access_ok((void __user *)(long)arg->vec,
2337+ arg->vec_len * sizeof(struct page_region)))
2338+ return -EFAULT;
2339+
2340+ /* Fixup default values */
2341+ arg->end = ALIGN(arg->end, PAGE_SIZE);
2342+ arg->walk_end = 0;
2343+ if (!arg->max_pages)
2344+ arg->max_pages = ULONG_MAX;
2345+
2346+ return 0;
2347+}
2348+
2349+static int pagemap_scan_writeback_args(struct pm_scan_arg *arg,
2350+ unsigned long uargl)
2351+{
2352+ struct pm_scan_arg __user *uarg = (void __user *)uargl;
2353+
2354+ if (copy_to_user(&uarg->walk_end, &arg->walk_end, sizeof(arg->walk_end)))
2355+ return -EFAULT;
2356+
2357+ return 0;
2358+}
2359+
2360+static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
2361+{
2362+ if (!p->arg.vec_len)
2363+ return 0;
2364+
2365+ p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
2366+ p->arg.vec_len);
2367+ p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
2368+ GFP_KERNEL);
2369+ if (!p->vec_buf)
2370+ return -ENOMEM;
2371+
2372+ p->vec_buf->start = p->vec_buf->end = 0;
2373+ p->vec_out = (struct page_region __user *)(long)p->arg.vec;
2374+
2375+ return 0;
2376+}
2377+
2378+static long pagemap_scan_flush_buffer(struct pagemap_scan_private *p)
2379+{
2380+ const struct page_region *buf = p->vec_buf;
2381+ long n = p->vec_buf_index;
2382+
2383+ if (!p->vec_buf)
2384+ return 0;
2385+
2386+ if (buf[n].end != buf[n].start)
2387+ n++;
2388+
2389+ if (!n)
2390+ return 0;
2391+
2392+ if (copy_to_user(p->vec_out, buf, n * sizeof(*buf)))
2393+ return -EFAULT;
2394+
2395+ p->arg.vec_len -= n;
2396+ p->vec_out += n;
2397+
2398+ p->vec_buf_index = 0;
2399+ p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len);
2400+ p->vec_buf->start = p->vec_buf->end = 0;
2401+
2402+ return n;
2403+}
2404+
2405+static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg)
2406+{
2407+ struct mmu_notifier_range range;
2408+ struct pagemap_scan_private p = {0};
2409+ unsigned long walk_start;
2410+ size_t n_ranges_out = 0;
2411+ int ret;
2412+
2413+ ret = pagemap_scan_get_args(&p.arg, uarg);
2414+ if (ret)
2415+ return ret;
2416+
2417+ p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask |
2418+ p.arg.return_mask;
2419+ ret = pagemap_scan_init_bounce_buffer(&p);
2420+ if (ret)
2421+ return ret;
2422+
2423+ /* Protection change for the range is going to happen. */
2424+ if (p.arg.flags & PM_SCAN_WP_MATCHING) {
2425+ mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
2426+ mm, p.arg.start, p.arg.end);
2427+ mmu_notifier_invalidate_range_start(&range);
2428+ }
2429+
2430+ for (walk_start = p.arg.start; walk_start < p.arg.end;
2431+ walk_start = p.arg.walk_end) {
2432+ long n_out;
2433+
2434+ if (fatal_signal_pending(current)) {
2435+ ret = -EINTR;
2436+ break;
2437+ }
2438+
2439+ ret = mmap_read_lock_killable(mm);
2440+ if (ret)
2441+ break;
2442+ ret = walk_page_range(mm, walk_start, p.arg.end,
2443+ &pagemap_scan_ops, &p);
2444+ mmap_read_unlock(mm);
2445+
2446+ n_out = pagemap_scan_flush_buffer(&p);
2447+ if (n_out < 0)
2448+ ret = n_out;
2449+ else
2450+ n_ranges_out += n_out;
2451+
2452+ if (ret != -ENOSPC)
2453+ break;
2454+
2455+ if (p.arg.vec_len == 0 || p.found_pages == p.arg.max_pages)
2456+ break;
2457+ }
2458+
2459+ /* ENOSPC signifies early stop (buffer full) from the walk. */
2460+ if (!ret || ret == -ENOSPC)
2461+ ret = n_ranges_out;
2462+
2463+ /* The walk_end isn't set when ret is zero */
2464+ if (!p.arg.walk_end)
2465+ p.arg.walk_end = p.arg.end;
2466+ if (pagemap_scan_writeback_args(&p.arg, uarg))
2467+ ret = -EFAULT;
2468+
2469+ if (p.arg.flags & PM_SCAN_WP_MATCHING)
2470+ mmu_notifier_invalidate_range_end(&range);
2471+
2472+ kfree(p.vec_buf);
2473+ return ret;
2474+}
2475+
2476+static long do_pagemap_cmd(struct file *file, unsigned int cmd,
2477+ unsigned long arg)
2478+{
2479+ struct mm_struct *mm = file->private_data;
2480+
2481+ switch (cmd) {
2482+ case PAGEMAP_SCAN:
2483+ return do_pagemap_scan(mm, arg);
2484+
2485+ default:
2486+ return -EINVAL;
2487+ }
2488+}
2489+
17642490 const struct file_operations proc_pagemap_operations = {
17652491 .llseek = mem_lseek, /* borrow this */
17662492 .read = pagemap_read,
17672493 .open = pagemap_open,
17682494 .release = pagemap_release,
2495+ .unlocked_ioctl = do_pagemap_cmd,
2496+ .compat_ioctl = do_pagemap_cmd,
17692497 };
17702498 #endif /* CONFIG_PROC_PAGE_MONITOR */
17712499
@@ -1945,8 +2673,9 @@ static int show_numa_map(struct seq_file *m, void *v) · 173 unmodified lines
19452673 struct numa_maps *md = &numa_priv->md;
19462674 struct file *file = vma->vm_file;
19472675 struct mm_struct *mm = vma->vm_mm;
1948− struct mempolicy *pol;
19492676 char buffer[64];
2677+ struct mempolicy *pol;
2678+ pgoff_t ilx;
19502679 int nid;
19512680
19522681 if (!mm)
@@ -1955,7 +2684,7 @@ static int show_numa_map(struct seq_file *m, void *v) · 2 unmodified lines
19552684 /* Ensure we start with an empty set of numa_maps statistics. */
19562685 memset(md, 0, sizeof(*md));
19572686
1958− pol = __get_vma_policy(vma, vma->vm_start);
2687+ pol = __get_vma_policy(vma, vma->vm_start, &ilx);
19592688 if (pol) {
19602689 mpol_to_str(buffer, sizeof(buffer), pol);
19612690 mpol_cond_put(pol);
fs/quota/dquot.c+9−8
@@ -803,12 +803,6 @@ dqcache_shrink_count(struct shrinker *shrink, struct shrink_control *sc) · 802 unmodified lines
803803 percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));
804804 }
805805
806−static struct shrinker dqcache_shrinker = {
807− .count_objects = dqcache_shrink_count,
808− .scan_objects = dqcache_shrink_scan,
809− .seeks = DEFAULT_SEEKS,
810−};
811−
812806 /*
813807 * Safely release dquot and put reference to dquot.
814808 */
@@ -2982,6 +2976,7 @@ static int __init dquot_init(void) · 2167 unmodified lines
29822976 {
29832977 int i, ret;
29842978 unsigned long nr_hash, order;
2979+ struct shrinker *dqcache_shrinker;
29852980
29862981 printk(KERN_NOTICE "VFS: Disk quotas %s\n", __DQUOT_VERSION__);
29872982
@@ -3016,8 +3011,14 @@ static int __init dquot_init(void) · 28 unmodified lines
30163011 pr_info("VFS: Dquot-cache hash table entries: %ld (order %ld,"
30173012 " %ld bytes)\n", nr_hash, order, (PAGE_SIZE << order));
30183013
3019− if (register_shrinker(&dqcache_shrinker, "dquota-cache"))
3020− panic("Cannot register dquot shrinker");
3014+ dqcache_shrinker = shrinker_alloc(0, "dquota-cache");
3015+ if (!dqcache_shrinker)
3016+ panic("Cannot allocate dquot shrinker");
3017+
3018+ dqcache_shrinker->count_objects = dqcache_shrink_count;
3019+ dqcache_shrinker->scan_objects = dqcache_shrink_scan;
3020+
3021+ shrinker_register(dqcache_shrinker);
30213022
30223023 return 0;
30233024 }
fs/reiserfs/inode.c+40−40
@@ -2503,10 +2503,10 @@ out: · 2502 unmodified lines
25032503 * start/recovery path as __block_write_full_folio, along with special
25042504 * code to handle reiserfs tails.
25052505 */
2506−static int reiserfs_write_full_page(struct page *page,
2506+static int reiserfs_write_full_folio(struct folio *folio,
25072507 struct writeback_control *wbc)
25082508 {
2509− struct inode *inode = page->mapping->host;
2509+ struct inode *inode = folio->mapping->host;
25102510 unsigned long end_index = inode->i_size >> PAGE_SHIFT;
25112511 int error = 0;
25122512 unsigned long block;
@@ -2514,7 +2514,7 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line
25142514 struct buffer_head *head, *bh;
25152515 int partial = 0;
25162516 int nr = 0;
2517− int checked = PageChecked(page);
2517+ int checked = folio_test_checked(folio);
25182518 struct reiserfs_transaction_handle th;
25192519 struct super_block *s = inode->i_sb;
25202520 int bh_per_page = PAGE_SIZE / s->s_blocksize;
@@ -2522,47 +2522,46 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line
25222522
25232523 /* no logging allowed when nonblocking or from PF_MEMALLOC */
25242524 if (checked && (current->flags & PF_MEMALLOC)) {
2525− redirty_page_for_writepage(wbc, page);
2526− unlock_page(page);
2525+ folio_redirty_for_writepage(wbc, folio);
2526+ folio_unlock(folio);
25272527 return 0;
25282528 }
25292529
25302530 /*
2531− * The page dirty bit is cleared before writepage is called, which
2531+ * The folio dirty bit is cleared before writepage is called, which
25322532 * means we have to tell create_empty_buffers to make dirty buffers
2533− * The page really should be up to date at this point, so tossing
2533+ * The folio really should be up to date at this point, so tossing
25342534 * in the BH_Uptodate is just a sanity check.
25352535 */
2536− if (!page_has_buffers(page)) {
2537− create_empty_buffers(page, s->s_blocksize,
2536+ head = folio_buffers(folio);
2537+ if (!head)
2538+ head = create_empty_buffers(folio, s->s_blocksize,
25382539 (1 << BH_Dirty) | (1 << BH_Uptodate));
2539− }
2540− head = page_buffers(page);
25412540
25422541 /*
2543− * last page in the file, zero out any contents past the
2542+ * last folio in the file, zero out any contents past the
25442543 * last byte in the file
25452544 */
2546− if (page->index >= end_index) {
2545+ if (folio->index >= end_index) {
25472546 unsigned last_offset;
25482547
25492548 last_offset = inode->i_size & (PAGE_SIZE - 1);
2550− /* no file contents in this page */
2551− if (page->index >= end_index + 1 || !last_offset) {
2552− unlock_page(page);
2549+ /* no file contents in this folio */
2550+ if (folio->index >= end_index + 1 || !last_offset) {
2551+ folio_unlock(folio);
25532552 return 0;
25542553 }
2555− zero_user_segment(page, last_offset, PAGE_SIZE);
2554+ folio_zero_segment(folio, last_offset, folio_size(folio));
25562555 }
25572556 bh = head;
2558− block = page->index << (PAGE_SHIFT - s->s_blocksize_bits);
2557+ block = folio->index << (PAGE_SHIFT - s->s_blocksize_bits);
25592558 last_block = (i_size_read(inode) - 1) >> inode->i_blkbits;
25602559 /* first map all the buffers, logging any direct items we find */
25612560 do {
25622561 if (block > last_block) {
25632562 /*
25642563 * This can happen when the block size is less than
2565− * the page size. The corresponding bytes in the page
2564+ * the folio size. The corresponding bytes in the folio
25662565 * were zero filled above
25672566 */
25682567 clear_buffer_dirty(bh);
@@ -2589,7 +2588,7 @@ static int reiserfs_write_full_page(struct page *page, · 20 unmodified lines
25892588 * blocks we're going to log
25902589 */
25912590 if (checked) {
2592− ClearPageChecked(page);
2591+ folio_clear_checked(folio);
25932592 reiserfs_write_lock(s);
25942593 error = journal_begin(&th, s, bh_per_page + 1);
25952594 if (error) {
@@ -2598,7 +2597,7 @@ static int reiserfs_write_full_page(struct page *page, · 2 unmodified lines
25982597 }
25992598 reiserfs_update_inode_transaction(inode);
26002599 }
2601− /* now go through and lock any dirty buffers on the page */
2600+ /* now go through and lock any dirty buffers on the folio */
26022601 do {
26032602 get_bh(bh);
26042603 if (!buffer_mapped(bh))
@@ -2619,7 +2618,7 @@ static int reiserfs_write_full_page(struct page *page, · 14 unmodified lines
26192618 lock_buffer(bh);
26202619 } else {
26212620 if (!trylock_buffer(bh)) {
2622− redirty_page_for_writepage(wbc, page);
2621+ folio_redirty_for_writepage(wbc, folio);
26232622 continue;
26242623 }
26252624 }
@@ -2636,13 +2635,13 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines
26362635 if (error)
26372636 goto fail;
26382637 }
2639− BUG_ON(PageWriteback(page));
2640− set_page_writeback(page);
2641− unlock_page(page);
2638+ BUG_ON(folio_test_writeback(folio));
2639+ folio_start_writeback(folio);
2640+ folio_unlock(folio);
26422641
26432642 /*
2644− * since any buffer might be the only dirty buffer on the page,
2645− * the first submit_bh can bring the page out of writeback.
2643+ * since any buffer might be the only dirty buffer on the folio,
2644+ * the first submit_bh can bring the folio out of writeback.
26462645 * be careful with the buffers.
26472646 */
26482647 do {
@@ -2659,10 +2658,10 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines
26592658 done:
26602659 if (nr == 0) {
26612660 /*
2662− * if this page only had a direct item, it is very possible for
2661+ * if this folio only had a direct item, it is very possible for
26632662 * no io to be required without there being an error. Or,
26642663 * someone else could have locked them and sent them down the
2665− * pipe without locking the page
2664+ * pipe without locking the folio
26662665 */
26672666 bh = head;
26682667 do {
@@ -2673,18 +2672,18 @@ done: · 4 unmodified lines
26732672 bh = bh->b_this_page;
26742673 } while (bh != head);
26752674 if (!partial)
2676− SetPageUptodate(page);
2677− end_page_writeback(page);
2675+ folio_mark_uptodate(folio);
2676+ folio_end_writeback(folio);
26782677 }
26792678 return error;
26802679
26812680 fail:
26822681 /*
26832682 * catches various errors, we need to make sure any valid dirty blocks
2684− * get to the media. The page is currently locked and not marked for
2683+ * get to the media. The folio is currently locked and not marked for
26852684 * writeback
26862685 */
2687− ClearPageUptodate(page);
2686+ folio_clear_uptodate(folio);
26882687 bh = head;
26892688 do {
26902689 get_bh(bh);
@@ -2694,16 +2693,16 @@ fail: · 3 unmodified lines
26942693 } else {
26952694 /*
26962695 * clear any dirty bits that might have come from
2697− * getting attached to a dirty page
2696+ * getting attached to a dirty folio
26982697 */
26992698 clear_buffer_dirty(bh);
27002699 }
27012700 bh = bh->b_this_page;
27022701 } while (bh != head);
2703− SetPageError(page);
2704− BUG_ON(PageWriteback(page));
2705− set_page_writeback(page);
2706− unlock_page(page);
2702+ folio_set_error(folio);
2703+ BUG_ON(folio_test_writeback(folio));
2704+ folio_start_writeback(folio);
2705+ folio_unlock(folio);
27072706 do {
27082707 struct buffer_head *next = bh->b_this_page;
27092708 if (buffer_async_write(bh)) {
@@ -2724,9 +2723,10 @@ static int reiserfs_read_folio(struct file *f, struct folio *folio) · 14 unmodified lines
27242723
27252724 static int reiserfs_writepage(struct page *page, struct writeback_control *wbc)
27262725 {
2727− struct inode *inode = page->mapping->host;
2726+ struct folio *folio = page_folio(page);
2727+ struct inode *inode = folio->mapping->host;
27282728 reiserfs_wait_on_write_block(inode->i_sb);
2729− return reiserfs_write_full_page(page, wbc);
2729+ return reiserfs_write_full_folio(folio, wbc);
27302730 }
27312731
27322732 static void reiserfs_truncate_failed_write(struct inode *inode)
fs/super.c+19−16
@@ -178,7 +178,7 @@ static void super_wake(struct super_block *sb, unsigned int flag) · 177 unmodified lines
178178 * One thing we have to be careful of with a per-sb shrinker is that we don't
179179 * drop the last active reference to the superblock from within the shrinker.
180180 * If that happens we could trigger unregistering the shrinker from within the
181− * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we
181+ * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we
182182 * take a passive reference to the superblock to avoid this from occurring.
183183 */
184184 static unsigned long super_cache_scan(struct shrinker *shrink,
@@ -191,7 +191,7 @@ static unsigned long super_cache_scan(struct shrinker *shrink, · 6 unmodified lines
191191 long dentries;
192192 long inodes;
193193
194− sb = container_of(shrink, struct super_block, s_shrink);
194+ sb = shrink->private_data;
195195
196196 /*
197197 * Deadlock avoidance. We may hold various FS locks, and we don't want
@@ -244,7 +244,7 @@ static unsigned long super_cache_count(struct shrinker *shrink, · 46 unmodified lines
244244 struct super_block *sb;
245245 long total_objects = 0;
246246
247− sb = container_of(shrink, struct super_block, s_shrink);
247+ sb = shrink->private_data;
248248
249249 /*
250250 * We don't call super_trylock_shared() here as it is a scalability
@@ -306,7 +306,7 @@ static void destroy_unused_super(struct super_block *s) · 55 unmodified lines
306306 security_sb_free(s);
307307 put_user_ns(s->s_user_ns);
308308 kfree(s->s_subtype);
309− free_prealloced_shrinker(&s->s_shrink);
309+ shrinker_free(s->s_shrink);
310310 /* no delays needed */
311311 destroy_super_work(&s->destroy_work);
312312 }
@@ -383,16 +383,19 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags, · 70 unmodified lines
383383 s->s_time_min = TIME64_MIN;
384384 s->s_time_max = TIME64_MAX;
385385
386− s->s_shrink.seeks = DEFAULT_SEEKS;
387− s->s_shrink.scan_objects = super_cache_scan;
388− s->s_shrink.count_objects = super_cache_count;
389− s->s_shrink.batch = 1024;
390− s->s_shrink.flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE;
391− if (prealloc_shrinker(&s->s_shrink, "sb-%s", type->name))
386+ s->s_shrink = shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE,
387+ "sb-%s", type->name);
388+ if (!s->s_shrink)
392389 goto fail;
393− if (list_lru_init_memcg(&s->s_dentry_lru, &s->s_shrink))
390+
391+ s->s_shrink->scan_objects = super_cache_scan;
392+ s->s_shrink->count_objects = super_cache_count;
393+ s->s_shrink->batch = 1024;
394+ s->s_shrink->private_data = s;
395+
396+ if (list_lru_init_memcg(&s->s_dentry_lru, s->s_shrink))
394397 goto fail;
395− if (list_lru_init_memcg(&s->s_inode_lru, &s->s_shrink))
398+ if (list_lru_init_memcg(&s->s_inode_lru, s->s_shrink))
396399 goto fail;
397400 return s;
398401
@@ -477,7 +480,7 @@ void deactivate_locked_super(struct super_block *s) · 78 unmodified lines
477480 {
478481 struct file_system_type *fs = s->s_type;
479482 if (atomic_dec_and_test(&s->s_active)) {
480− unregister_shrinker(&s->s_shrink);
483+ shrinker_free(s->s_shrink);
481484 fs->kill_sb(s);
482485
483486 kill_super_notify(s);
@@ -818,7 +821,7 @@ retry: · 334 unmodified lines
818821 hlist_add_head(&s->s_instances, &s->s_type->fs_supers);
819822 spin_unlock(&sb_lock);
820823 get_filesystem(s->s_type);
821− register_shrinker_prepared(&s->s_shrink);
824+ shrinker_register(s->s_shrink);
822825 return s;
823826
824827 share_extant_sb:
@@ -901,7 +904,7 @@ retry: · 76 unmodified lines
901904 hlist_add_head(&s->s_instances, &type->fs_supers);
902905 spin_unlock(&sb_lock);
903906 get_filesystem(type);
904− register_shrinker_prepared(&s->s_shrink);
907+ shrinker_register(s->s_shrink);
905908 return s;
906909 }
907910 EXPORT_SYMBOL(sget);
@@ -1540,7 +1543,7 @@ int setup_bdev_super(struct super_block *sb, int sb_flags, · 632 unmodified lines
15401543 mutex_unlock(&bdev->bd_fsfreeze_mutex);
15411544
15421545 snprintf(sb->s_id, sizeof(sb->s_id), "%pg", bdev);
1543− shrinker_debugfs_rename(&sb->s_shrink, "sb-%s:%s", sb->s_type->name,
1546+ shrinker_debugfs_rename(sb->s_shrink, "sb-%s:%s", sb->s_type->name,
15441547 sb->s_id);
15451548 sb_set_blocksize(sb, block_size(bdev));
15461549 return 0;
fs/ubifs/super.c+11−10
@@ -54,11 +54,7 @@ module_param_cb(default_version, &ubifs_default_version_ops, &ubifs_default_vers · 53 unmodified lines
5454 static struct kmem_cache *ubifs_inode_slab;
5555
5656 /* UBIFS TNC shrinker description */
57−static struct shrinker ubifs_shrinker_info = {
58− .scan_objects = ubifs_shrink_scan,
59− .count_objects = ubifs_shrink_count,
60− .seeks = DEFAULT_SEEKS,
61−};
57+static struct shrinker *ubifs_shrinker_info;
6258
6359 /**
6460 * validate_inode - validate inode.
@@ -2373,7 +2369,7 @@ static void inode_slab_ctor(void *obj) · 2308 unmodified lines
23732369
23742370 static int __init ubifs_init(void)
23752371 {
2376− int err;
2372+ int err = -ENOMEM;
23772373
23782374 BUILD_BUG_ON(sizeof(struct ubifs_ch) != 24);
23792375
@@ -2439,10 +2435,15 @@ static int __init ubifs_init(void) · 59 unmodified lines
24392435 if (!ubifs_inode_slab)
24402436 return -ENOMEM;
24412437
2442− err = register_shrinker(&ubifs_shrinker_info, "ubifs-slab");
2443− if (err)
2438+ ubifs_shrinker_info = shrinker_alloc(0, "ubifs-slab");
2439+ if (!ubifs_shrinker_info)
24442440 goto out_slab;
24452441
2442+ ubifs_shrinker_info->count_objects = ubifs_shrink_count;
2443+ ubifs_shrinker_info->scan_objects = ubifs_shrink_scan;
2444+
2445+ shrinker_register(ubifs_shrinker_info);
2446+
24462447 err = ubifs_compressors_init();
24472448 if (err)
24482449 goto out_shrinker;
@@ -2467,7 +2468,7 @@ out_dbg: · 18 unmodified lines
24672468 dbg_debugfs_exit();
24682469 ubifs_compressors_exit();
24692470 out_shrinker:
2470− unregister_shrinker(&ubifs_shrinker_info);
2471+ shrinker_free(ubifs_shrinker_info);
24712472 out_slab:
24722473 kmem_cache_destroy(ubifs_inode_slab);
24732474 return err;
@@ -2483,7 +2484,7 @@ static void __exit ubifs_exit(void) · 9 unmodified lines
24832484 dbg_debugfs_exit();
24842485 ubifs_sysfs_exit();
24852486 ubifs_compressors_exit();
2486− unregister_shrinker(&ubifs_shrinker_info);
2487+ shrinker_free(ubifs_shrinker_info);
24872488
24882489 /*
24892490 * Make sure all delayed rcu free inodes are flushed before we
fs/ufs/balloc.c+9−11
@@ -240,6 +240,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 239 unmodified lines
240240 unsigned int count, sector_t oldb,
241241 sector_t newb, struct page *locked_page)
242242 {
243+ struct folio *folio, *locked_folio = page_folio(locked_page);
243244 const unsigned blks_per_page =
244245 1 << (PAGE_SHIFT - inode->i_blkbits);
245246 const unsigned mask = blks_per_page - 1;
@@ -247,42 +248,39 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 1 unmodified line
247248 pgoff_t index, cur_index, last_index;
248249 unsigned pos, j, lblock;
249250 sector_t end, i;
250− struct page *page;
251251 struct buffer_head *head, *bh;
252252
253253 UFSD("ENTER, ino %lu, count %u, oldb %llu, newb %llu\n",
254254 inode->i_ino, count,
255255 (unsigned long long)oldb, (unsigned long long)newb);
256256
257− BUG_ON(!locked_page);
258− BUG_ON(!PageLocked(locked_page));
257+ BUG_ON(!folio_test_locked(locked_folio));
259258
260− cur_index = locked_page->index;
259+ cur_index = locked_folio->index;
261260 end = count + beg;
262261 last_index = end >> (PAGE_SHIFT - inode->i_blkbits);
263262 for (i = beg; i < end; i = (i | mask) + 1) {
264263 index = i >> (PAGE_SHIFT - inode->i_blkbits);
265264
266265 if (likely(cur_index != index)) {
267− page = ufs_get_locked_page(mapping, index);
268− if (!page)/* it was truncated */
266+ folio = ufs_get_locked_folio(mapping, index);
267+ if (!folio) /* it was truncated */
269268 continue;
270− if (IS_ERR(page)) {/* or EIO */
269+ if (IS_ERR(folio)) {/* or EIO */
271270 ufs_error(inode->i_sb, __func__,
272271 "read of page %llu failed\n",
273272 (unsigned long long)index);
274273 continue;
275274 }
276275 } else
277− page = locked_page;
276+ folio = locked_folio;
278277
279− head = page_buffers(page);
278+ head = folio_buffers(folio);
280279 bh = head;
281280 pos = i & mask;
282281 for (j = 0; j < pos; ++j)
283282 bh = bh->b_this_page;
284283
285−
286284 if (unlikely(index == last_index))
287285 lblock = end & mask;
288286 else
@@ -313,7 +311,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 24 unmodified lines
313311 } while (bh != head);
314312
315313 if (likely(cur_index != index))
316− ufs_put_locked_page(page);
314+ ufs_put_locked_folio(folio);
317315 }
318316 UFSD("EXIT\n");
319317 }
fs/ufs/inode.c+12−13
@@ -1063,7 +1063,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 1062 unmodified lines
10631063 struct ufs_sb_private_info *uspi = UFS_SB(sb)->s_uspi;
10641064 unsigned i, end;
10651065 sector_t lastfrag;
1066− struct page *lastpage;
1066+ struct folio *folio;
10671067 struct buffer_head *bh;
10681068 u64 phys64;
10691069
@@ -1074,18 +1074,17 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 4 unmodified lines
10741074
10751075 lastfrag--;
10761076
1077− lastpage = ufs_get_locked_page(mapping, lastfrag >>
1077+ folio = ufs_get_locked_folio(mapping, lastfrag >>
10781078 (PAGE_SHIFT - inode->i_blkbits));
1079− if (IS_ERR(lastpage)) {
1080− err = -EIO;
1081− goto out;
1082− }
1083−
1084− end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1);
1085− bh = page_buffers(lastpage);
1086− for (i = 0; i < end; ++i)
1087− bh = bh->b_this_page;
1079+ if (IS_ERR(folio)) {
1080+ err = -EIO;
1081+ goto out;
1082+ }
10881083
1084+ end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1);
1085+ bh = folio_buffers(folio);
1086+ for (i = 0; i < end; ++i)
1087+ bh = bh->b_this_page;
10891088
10901089 err = ufs_getfrag_block(inode, lastfrag, bh, 1);
10911090
@@ -1101,7 +1100,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 9 unmodified lines
11011100 */
11021101 set_buffer_uptodate(bh);
11031102 mark_buffer_dirty(bh);
1104− set_page_dirty(lastpage);
1103+ folio_mark_dirty(folio);
11051104 }
11061105
11071106 if (lastfrag >= UFS_IND_FRAGMENT) {
@@ -1119,7 +1118,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 11 unmodified lines
11191118 }
11201119 }
11211120 out_unlock:
1122− ufs_put_locked_page(lastpage);
1121+ ufs_put_locked_folio(folio);
11231122 out:
11241123 return err;
11251124 }
fs/ufs/util.c+16−18
@@ -230,42 +230,40 @@ ufs_set_inode_dev(struct super_block *sb, struct ufs_inode_info *ufsi, dev_t dev · 229 unmodified lines
230230 }
231231
232232 /**
233− * ufs_get_locked_page() - locate, pin and lock a pagecache page, if not exist
233+ * ufs_get_locked_folio() - locate, pin and lock a pagecache folio, if not exist
234234 * read it from disk.
235235 * @mapping: the address_space to search
236236 * @index: the page index
237237 *
238− * Locates the desired pagecache page, if not exist we'll read it,
238+ * Locates the desired pagecache folio, if not exist we'll read it,
239239 * locks it, increments its reference
240240 * count and returns its address.
241241 *
242242 */
243−
244−struct page *ufs_get_locked_page(struct address_space *mapping,
243+struct folio *ufs_get_locked_folio(struct address_space *mapping,
245244 pgoff_t index)
246245 {
247246 struct inode *inode = mapping->host;
248− struct page *page = find_lock_page(mapping, index);
249− if (!page) {
250− page = read_mapping_page(mapping, index, NULL);
247+ struct folio *folio = filemap_lock_folio(mapping, index);
248+ if (!folio) {
249+ folio = read_mapping_folio(mapping, index, NULL);
251250
252− if (IS_ERR(page)) {
253− printk(KERN_ERR "ufs_change_blocknr: "
254− "read_mapping_page error: ino %lu, index: %lu\n",
251+ if (IS_ERR(folio)) {
252+ printk(KERN_ERR "ufs_change_blocknr: read_mapping_folio error: ino %lu, index: %lu\n",
255253 mapping->host->i_ino, index);
256− return page;
254+ return folio;
257255 }
258256
259− lock_page(page);
257+ folio_lock(folio);
260258
261− if (unlikely(page->mapping == NULL)) {
259+ if (unlikely(folio->mapping == NULL)) {
262260 /* Truncate got there first */
263− unlock_page(page);
264− put_page(page);
261+ folio_unlock(folio);
262+ folio_put(folio);
265263 return NULL;
266264 }
267265 }
268− if (!page_has_buffers(page))
269− create_empty_buffers(page, 1 << inode->i_blkbits, 0);
270− return page;
266+ if (!folio_buffers(folio))
267+ create_empty_buffers(folio, 1 << inode->i_blkbits, 0);
268+ return folio;
271269 }
fs/ufs/util.h+4−6
@@ -273,15 +273,13 @@ extern void _ubh_ubhcpymem_(struct ufs_sb_private_info *, unsigned char *, struc · 272 unmodified lines
273273 extern void _ubh_memcpyubh_(struct ufs_sb_private_info *, struct ufs_buffer_head *, unsigned char *, unsigned);
274274
275275 /* This functions works with cache pages*/
276−extern struct page *ufs_get_locked_page(struct address_space *mapping,
277− pgoff_t index);
278−static inline void ufs_put_locked_page(struct page *page)
276+struct folio *ufs_get_locked_folio(struct address_space *mapping, pgoff_t index);
277+static inline void ufs_put_locked_folio(struct folio *folio)
279278 {
280− unlock_page(page);
281− put_page(page);
279+ folio_unlock(folio);
280+ folio_put(folio);
282281 }
283282
284−
285283 /*
286284 * macros and inline function to get important structures from ufs_sb_private_info
287285 */
fs/userfaultfd.c+40−56
@@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx) · 122 unmodified lines
123123 return ctx->features & UFFD_FEATURE_INITIALIZED;
124124 }
125125
126+static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
127+{
128+ return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
129+}
130+
126131 /*
127132 * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
128133 * meaningful when userfaultfd_wp()==true on the vma and when it's
@@ -922,20 +927,15 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 793 unmodified lines
922927 continue;
923928 }
924929 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;
925− prev = vma_merge(&vmi, mm, prev, vma->vm_start, vma->vm_end,
926− new_flags, vma->anon_vma,
927− vma->vm_file, vma->vm_pgoff,
928− vma_policy(vma),
929− NULL_VM_UFFD_CTX, anon_vma_name(vma));
930− if (prev) {
931− vma = prev;
932− } else {
933− prev = vma;
934− }
930+ vma = vma_modify_flags_uffd(&vmi, prev, vma, vma->vm_start,
931+ vma->vm_end, new_flags,
932+ NULL_VM_UFFD_CTX);
935933
936934 vma_start_write(vma);
937935 userfaultfd_set_vm_flags(vma, new_flags);
938936 vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX;
937+
938+ prev = vma;
939939 }
940940 mmap_write_unlock(mm);
941941 mmput(mm);
@@ -1325,7 +1325,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 383 unmodified lines
13251325 bool basic_ioctls;
13261326 unsigned long start, end, vma_end;
13271327 struct vma_iterator vmi;
1328− pgoff_t pgoff;
1328+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
13291329
13301330 user_uffdio_register = (struct uffdio_register __user *) arg;
13311331
@@ -1399,7 +1399,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 67 unmodified lines
13991399
14001400 /* check not compatible vmas */
14011401 ret = -EINVAL;
1402− if (!vma_can_userfault(cur, vm_flags))
1402+ if (!vma_can_userfault(cur, vm_flags, wp_async))
14031403 goto out_unlock;
14041404
14051405 /*
@@ -1460,7 +1460,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 54 unmodified lines
14601460 for_each_vma_range(vmi, vma, end) {
14611461 cond_resched();
14621462
1463− BUG_ON(!vma_can_userfault(vma, vm_flags));
1463+ BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async));
14641464 BUG_ON(vma->vm_userfaultfd_ctx.ctx &&
14651465 vma->vm_userfaultfd_ctx.ctx != ctx);
14661466 WARN_ON(!(vma->vm_flags & VM_MAYWRITE));
@@ -1478,28 +1478,14 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 11 unmodified lines
14781478 vma_end = min(end, vma->vm_end);
14791479
14801480 new_flags = (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags;
1481− pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);
1482− prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags,
1483− vma->anon_vma, vma->vm_file, pgoff,
1484− vma_policy(vma),
1485− ((struct vm_userfaultfd_ctx){ ctx }),
1486− anon_vma_name(vma));
1487− if (prev) {
1488− /* vma_merge() invalidated the mas */
1489− vma = prev;
1490− goto next;
1491− }
1492− if (vma->vm_start < start) {
1493− ret = split_vma(&vmi, vma, start, 1);
1494− if (ret)
1495− break;
1496− }
1497− if (vma->vm_end > end) {
1498− ret = split_vma(&vmi, vma, end, 0);
1499− if (ret)
1500− break;
1481+ vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end,
1482+ new_flags,
1483+ (struct vm_userfaultfd_ctx){ctx});
1484+ if (IS_ERR(vma)) {
1485+ ret = PTR_ERR(vma);
1486+ break;
15011487 }
1502− next:
1488+
15031489 /*
15041490 * In the vma_merge() successful mprotect-like case 8:
15051491 * the next vma was merged into the current one and
@@ -1561,7 +1547,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 55 unmodified lines
15611547 unsigned long start, end, vma_end;
15621548 const void __user *buf = (void __user *)arg;
15631549 struct vma_iterator vmi;
1564− pgoff_t pgoff;
1550+ bool wp_async = userfaultfd_wp_async_ctx(ctx);
15651551
15661552 ret = -EFAULT;
15671553 if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
@@ -1615,7 +1601,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 47 unmodified lines
16151601 * provides for more strict behavior to notice
16161602 * unregistration errors.
16171603 */
1618− if (!vma_can_userfault(cur, cur->vm_flags))
1604+ if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
16191605 goto out_unlock;
16201606
16211607 found = true;
@@ -1631,7 +1617,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 9 unmodified lines
16311617 for_each_vma_range(vmi, vma, end) {
16321618 cond_resched();
16331619
1634− BUG_ON(!vma_can_userfault(vma, vma->vm_flags));
1620+ BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async));
16351621
16361622 /*
16371623 * Nothing to do: this vma is already registered into this
@@ -1664,26 +1650,13 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 26 unmodified lines
16641650 uffd_wp_range(vma, start, vma_end - start, false);
16651651
16661652 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;
1667− pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);
1668− prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags,
1669− vma->anon_vma, vma->vm_file, pgoff,
1670− vma_policy(vma),
1671− NULL_VM_UFFD_CTX, anon_vma_name(vma));
1672− if (prev) {
1673− vma = prev;
1674− goto next;
1675− }
1676− if (vma->vm_start < start) {
1677− ret = split_vma(&vmi, vma, start, 1);
1678− if (ret)
1679− break;
1680− }
1681− if (vma->vm_end > end) {
1682− ret = split_vma(&vmi, vma, end, 0);
1683− if (ret)
1684− break;
1653+ vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end,
1654+ new_flags, NULL_VM_UFFD_CTX);
1655+ if (IS_ERR(vma)) {
1656+ ret = PTR_ERR(vma);
1657+ break;
16851658 }
1686− next:
1659+
16871660 /*
16881661 * In the vma_merge() successful mprotect-like case 8:
16891662 * the next vma was merged into the current one and
@@ -2018,6 +1991,11 @@ out: · 328 unmodified lines
20181991 return ret;
20191992 }
20201993
1994+bool userfaultfd_wp_async(struct vm_area_struct *vma)
1995+{
1996+ return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
1997+}
1998+
20211999 static inline unsigned int uffd_ctx_features(__u64 user_features)
20222000 {
20232001 /*
@@ -2051,6 +2029,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 27 unmodified lines
20512029 ret = -EPERM;
20522030 if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
20532031 goto err_out;
2032+
2033+ /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
2034+ if (features & UFFD_FEATURE_WP_ASYNC)
2035+ features |= UFFD_FEATURE_WP_UNPOPULATED;
2036+
20542037 /* report all available features and ioctls to userland */
20552038 uffdio_api.features = UFFD_API_FEATURES;
20562039 #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
@@ -2063,6 +2046,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 6 unmodified lines
20632046 #ifndef CONFIG_PTE_MARKER_UFFD_WP
20642047 uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
20652048 uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
2049+ uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
20662050 #endif
20672051 uffdio_api.ioctls = UFFD_API_IOCTLS;
20682052 ret = -EFAULT;
fs/xfs/xfs_buf.c+13−11
@@ -1913,8 +1913,7 @@ xfs_buftarg_shrink_scan( · 1912 unmodified lines
19131913 struct shrinker *shrink,
19141914 struct shrink_control *sc)
19151915 {
1916− struct xfs_buftarg *btp = container_of(shrink,
1917− struct xfs_buftarg, bt_shrinker);
1916+ struct xfs_buftarg *btp = shrink->private_data;
19181917 LIST_HEAD(dispose);
19191918 unsigned long freed;
19201919
@@ -1936,8 +1935,7 @@ xfs_buftarg_shrink_count( · 15 unmodified lines
19361935 struct shrinker *shrink,
19371936 struct shrink_control *sc)
19381937 {
1939− struct xfs_buftarg *btp = container_of(shrink,
1940− struct xfs_buftarg, bt_shrinker);
1938+ struct xfs_buftarg *btp = shrink->private_data;
19411939 return list_lru_shrink_count(&btp->bt_lru, sc);
19421940 }
19431941
@@ -1945,7 +1943,7 @@ void · 1 unmodified line
19451943 xfs_free_buftarg(
19461944 struct xfs_buftarg *btp)
19471945 {
1948− unregister_shrinker(&btp->bt_shrinker);
1946+ shrinker_free(btp->bt_shrinker);
19491947 ASSERT(percpu_counter_sum(&btp->bt_io_count) == 0);
19501948 percpu_counter_destroy(&btp->bt_io_count);
19511949 list_lru_destroy(&btp->bt_lru);
@@ -2029,13 +2027,17 @@ xfs_alloc_buftarg( · 77 unmodified lines
20292027 if (percpu_counter_init(&btp->bt_io_count, 0, GFP_KERNEL))
20302028 goto error_lru;
20312029
2032− btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count;
2033− btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan;
2034− btp->bt_shrinker.seeks = DEFAULT_SEEKS;
2035− btp->bt_shrinker.flags = SHRINKER_NUMA_AWARE;
2036− if (register_shrinker(&btp->bt_shrinker, "xfs-buf:%s",
2037− mp->m_super->s_id))
2030+ btp->bt_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-buf:%s",
2031+ mp->m_super->s_id);
2032+ if (!btp->bt_shrinker)
20382033 goto error_pcpu;
2034+
2035+ btp->bt_shrinker->count_objects = xfs_buftarg_shrink_count;
2036+ btp->bt_shrinker->scan_objects = xfs_buftarg_shrink_scan;
2037+ btp->bt_shrinker->private_data = btp;
2038+
2039+ shrinker_register(btp->bt_shrinker);
2040+
20392041 return btp;
20402042
20412043 error_pcpu:
fs/xfs/xfs_buf.h+1−1
@@ -109,7 +109,7 @@ typedef struct xfs_buftarg { · 108 unmodified lines
109109 size_t bt_logical_sectormask;
110110
111111 /* LRU control structures */
112− struct shrinker bt_shrinker;
112+ struct shrinker *bt_shrinker;
113113 struct list_lru bt_lru;
114114
115115 struct percpu_counter bt_io_count;
fs/xfs/xfs_icache.c+15−11
@@ -2165,8 +2165,7 @@ xfs_inodegc_shrinker_count( · 2164 unmodified lines
21652165 struct shrinker *shrink,
21662166 struct shrink_control *sc)
21672167 {
2168− struct xfs_mount *mp = container_of(shrink, struct xfs_mount,
2169− m_inodegc_shrinker);
2168+ struct xfs_mount *mp = shrink->private_data;
21702169 struct xfs_inodegc *gc;
21712170 int cpu;
21722171
@@ -2187,8 +2186,7 @@ xfs_inodegc_shrinker_scan( · 14 unmodified lines
21872186 struct shrinker *shrink,
21882187 struct shrink_control *sc)
21892188 {
2190− struct xfs_mount *mp = container_of(shrink, struct xfs_mount,
2191− m_inodegc_shrinker);
2189+ struct xfs_mount *mp = shrink->private_data;
21922190 struct xfs_inodegc *gc;
21932191 int cpu;
21942192 bool no_items = true;
@@ -2224,13 +2222,19 @@ int · 29 unmodified lines
22242222 xfs_inodegc_register_shrinker(
22252223 struct xfs_mount *mp)
22262224 {
2227− struct shrinker *shrink = &mp->m_inodegc_shrinker;
2225+ mp->m_inodegc_shrinker = shrinker_alloc(SHRINKER_NONSLAB,
2226+ "xfs-inodegc:%s",
2227+ mp->m_super->s_id);
2228+ if (!mp->m_inodegc_shrinker)
2229+ return -ENOMEM;
2230+
2231+ mp->m_inodegc_shrinker->count_objects = xfs_inodegc_shrinker_count;
2232+ mp->m_inodegc_shrinker->scan_objects = xfs_inodegc_shrinker_scan;
2233+ mp->m_inodegc_shrinker->seeks = 0;
2234+ mp->m_inodegc_shrinker->batch = XFS_INODEGC_SHRINKER_BATCH;
2235+ mp->m_inodegc_shrinker->private_data = mp;
22282236
2229− shrink->count_objects = xfs_inodegc_shrinker_count;
2230− shrink->scan_objects = xfs_inodegc_shrinker_scan;
2231− shrink->seeks = 0;
2232− shrink->flags = SHRINKER_NONSLAB;
2233− shrink->batch = XFS_INODEGC_SHRINKER_BATCH;
2237+ shrinker_register(mp->m_inodegc_shrinker);
22342238
2235− return register_shrinker(shrink, "xfs-inodegc:%s", mp->m_super->s_id);
2239+ return 0;
22362240 }
fs/xfs/xfs_mount.c+2−2
@@ -1021,7 +1021,7 @@ xfs_mountfs( · 1020 unmodified lines
10211021 out_log_dealloc:
10221022 xfs_log_mount_cancel(mp);
10231023 out_inodegc_shrinker:
1024− unregister_shrinker(&mp->m_inodegc_shrinker);
1024+ shrinker_free(mp->m_inodegc_shrinker);
10251025 out_fail_wait:
10261026 if (mp->m_logdev_targp && mp->m_logdev_targp != mp->m_ddev_targp)
10271027 xfs_buftarg_drain(mp->m_logdev_targp);
@@ -1104,7 +1104,7 @@ xfs_unmountfs( · 76 unmodified lines
11041104 #if defined(DEBUG)
11051105 xfs_errortag_clearall(mp);
11061106 #endif
1107− unregister_shrinker(&mp->m_inodegc_shrinker);
1107+ shrinker_free(mp->m_inodegc_shrinker);
11081108 xfs_free_perag(mp);
11091109
11101110 xfs_errortag_del(mp);
fs/xfs/xfs_mount.h+1−1
@@ -219,7 +219,7 @@ typedef struct xfs_mount { · 218 unmodified lines
219219 atomic_t m_agirotor; /* last ag dir inode alloced */
220220
221221 /* Memory shrinker to throttle and reprioritize inodegc */
222− struct shrinker m_inodegc_shrinker;
222+ struct shrinker *m_inodegc_shrinker;
223223 /*
224224 * Workqueue item so that we can coalesce multiple inode flush attempts
225225 * into a single flush.
fs/xfs/xfs_qm.c+14−13
@@ -504,8 +504,7 @@ xfs_qm_shrink_scan( · 503 unmodified lines
504504 struct shrinker *shrink,
505505 struct shrink_control *sc)
506506 {
507− struct xfs_quotainfo *qi = container_of(shrink,
508− struct xfs_quotainfo, qi_shrinker);
507+ struct xfs_quotainfo *qi = shrink->private_data;
509508 struct xfs_qm_isolate isol;
510509 unsigned long freed;
511510 int error;
@@ -539,8 +538,7 @@ xfs_qm_shrink_count( · 27 unmodified lines
539538 struct shrinker *shrink,
540539 struct shrink_control *sc)
541540 {
542− struct xfs_quotainfo *qi = container_of(shrink,
543− struct xfs_quotainfo, qi_shrinker);
541+ struct xfs_quotainfo *qi = shrink->private_data;
544542
545543 return list_lru_shrink_count(&qi->qi_lru, sc);
546544 }
@@ -680,15 +678,18 @@ xfs_qm_init_quotainfo( · 133 unmodified lines
680678 if (XFS_IS_PQUOTA_ON(mp))
681679 xfs_qm_set_defquota(mp, XFS_DQTYPE_PROJ, qinf);
682680
683− qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
684− qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
685− qinf->qi_shrinker.seeks = DEFAULT_SEEKS;
686− qinf->qi_shrinker.flags = SHRINKER_NUMA_AWARE;
687−
688− error = register_shrinker(&qinf->qi_shrinker, "xfs-qm:%s",
689− mp->m_super->s_id);
690− if (error)
681+ qinf->qi_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-qm:%s",
682+ mp->m_super->s_id);
683+ if (!qinf->qi_shrinker) {
684+ error = -ENOMEM;
691685 goto out_free_inos;
686+ }
687+
688+ qinf->qi_shrinker->count_objects = xfs_qm_shrink_count;
689+ qinf->qi_shrinker->scan_objects = xfs_qm_shrink_scan;
690+ qinf->qi_shrinker->private_data = qinf;
691+
692+ shrinker_register(qinf->qi_shrinker);
692693
693694 return 0;
694695
@@ -718,7 +719,7 @@ xfs_qm_destroy_quotainfo( · 23 unmodified lines
718719 qi = mp->m_quotainfo;
719720 ASSERT(qi != NULL);
720721
721− unregister_shrinker(&qi->qi_shrinker);
722+ shrinker_free(qi->qi_shrinker);
722723 list_lru_destroy(&qi->qi_lru);
723724 xfs_qm_destroy_quotainos(qi);
724725 mutex_destroy(&qi->qi_tree_lock);
fs/xfs/xfs_qm.h+1−1
@@ -63,7 +63,7 @@ struct xfs_quotainfo { · 62 unmodified lines
6363 struct xfs_def_quota qi_usr_default;
6464 struct xfs_def_quota qi_grp_default;
6565 struct xfs_def_quota qi_prj_default;
66− struct shrinker qi_shrinker;
66+ struct shrinker *qi_shrinker;
6767
6868 /* Minimum and maximum quota expiration timestamp values. */
6969 time64_t qi_expiry_min;
include/asm-generic/bitops/instrumented-lock.h+13−15
@@ -58,27 +58,25 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr) · 57 unmodified lines
5858 return arch_test_and_set_bit_lock(nr, addr);
5959 }
6060
61−#if defined(arch_clear_bit_unlock_is_negative_byte)
6261 /**
63− * clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom
64− * byte is negative, for unlock.
65− * @nr: the bit to clear
66− * @addr: the address to start counting from
62+ * xor_unlock_is_negative_byte - XOR a single byte in memory and test if
63+ * it is negative, for unlock.
64+ * @mask: Change the bits which are set in this mask.
65+ * @addr: The address of the word containing the byte to change.
6766 *
67+ * Changes some of bits 0-6 in the word pointed to by @addr.
6868 * This operation is atomic and provides release barrier semantics.
69+ * Used to optimise some folio operations which are commonly paired
70+ * with an unlock or end of writeback. Bit 7 is used as PG_waiters to
71+ * indicate whether anybody is waiting for the unlock.
6972 *
70− * This is a bit of a one-trick-pony for the filemap code, which clears
71− * PG_locked and tests PG_waiters,
73+ * Return: Whether the top bit of the byte is set.
7274 */
73−static inline bool
74−clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr)
75+static inline bool xor_unlock_is_negative_byte(unsigned long mask,
76+ volatile unsigned long *addr)
7577 {
7678 kcsan_release();
77− instrument_atomic_write(addr + BIT_WORD(nr), sizeof(long));
78− return arch_clear_bit_unlock_is_negative_byte(nr, addr);
79+ instrument_atomic_write(addr, sizeof(long));
80+ return arch_xor_unlock_is_negative_byte(mask, addr);
7981 }
80−/* Let everybody know we have it. */
81−#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte
82−#endif
83−
8482 #endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */
include/asm-generic/bitops/lock.h+4−16
@@ -66,27 +66,15 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p) · 65 unmodified lines
6666 raw_atomic_long_set_release((atomic_long_t *)p, old);
6767 }
6868
69−/**
70− * arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom
71− * byte is negative, for unlock.
72− * @nr: the bit to clear
73− * @addr: the address to start counting from
74− *
75− * This is a bit of a one-trick-pony for the filemap code, which clears
76− * PG_locked and tests PG_waiters,
77− */
78−#ifndef arch_clear_bit_unlock_is_negative_byte
79−static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr,
80− volatile unsigned long *p)
69+#ifndef arch_xor_unlock_is_negative_byte
70+static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
71+ volatile unsigned long *p)
8172 {
8273 long old;
83− unsigned long mask = BIT_MASK(nr);
8474
85− p += BIT_WORD(nr);
86− old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
75+ old = raw_atomic_long_fetch_xor_release(mask, (atomic_long_t *)p);
8776 return !!(old & BIT(7));
8877 }
89−#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte
9078 #endif
9179
9280 #include <asm-generic/bitops/instrumented-lock.h>
include/asm-generic/pgalloc.h+6−1
@@ -169,6 +169,8 @@ static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 168 unmodified lines
169169 ptdesc = pagetable_alloc(gfp, 0);
170170 if (!ptdesc)
171171 return NULL;
172+
173+ pagetable_pud_ctor(ptdesc);
172174 return ptdesc_address(ptdesc);
173175 }
174176
@@ -190,8 +192,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 15 unmodified lines
190192
191193 static inline void __pud_free(struct mm_struct *mm, pud_t *pud)
192194 {
195+ struct ptdesc *ptdesc = virt_to_ptdesc(pud);
196+
193197 BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
194− pagetable_free(virt_to_ptdesc(pud));
198+ pagetable_pud_dtor(ptdesc);
199+ pagetable_free(ptdesc);
195200 }
196201
197202 #ifndef __HAVE_ARCH_PUD_FREE
include/linux/bootmem_info.h+1−1
@@ -60,7 +60,7 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 59 unmodified lines
6060
6161 static inline void free_bootmem_page(struct page *page)
6262 {
63− kmemleak_free_part(page_to_virt(page), PAGE_SIZE);
63+ kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
6464 free_reserved_page(page);
6565 }
6666 #endif
include/linux/buffer_head.h+54−27
@@ -198,13 +198,11 @@ void touch_buffer(struct buffer_head *bh); · 197 unmodified lines
198198 void folio_set_bh(struct buffer_head *bh, struct folio *folio,
199199 unsigned long offset);
200200 struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
201− bool retry);
201+ gfp_t gfp);
202202 struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
203203 bool retry);
204−void create_empty_buffers(struct page *, unsigned long,
205− unsigned long b_state);
206−void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize,
207− unsigned long b_state);
204+struct buffer_head *create_empty_buffers(struct folio *folio,
205+ unsigned long blocksize, unsigned long b_state);
208206 void end_buffer_read_sync(struct buffer_head *bh, int uptodate);
209207 void end_buffer_write_sync(struct buffer_head *bh, int uptodate);
210208 void end_buffer_async_write(struct buffer_head *bh, int uptodate);
@@ -227,8 +225,8 @@ void __wait_on_buffer(struct buffer_head *); · 16 unmodified lines
227225 wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
228226 struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
229227 unsigned size);
230−struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block,
231− unsigned size, gfp_t gfp);
228+struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
229+ unsigned size, gfp_t gfp);
232230 void __brelse(struct buffer_head *);
233231 void __bforget(struct buffer_head *);
234232 void __breadahead(struct block_device *, sector_t block, unsigned int size);
@@ -338,17 +336,38 @@ sb_breadahead(struct super_block *sb, sector_t block) · 103 unmodified lines
338336 __breadahead(sb->s_bdev, block, sb->s_blocksize);
339337 }
340338
341−static inline struct buffer_head *
342−sb_getblk(struct super_block *sb, sector_t block)
339+static inline struct buffer_head *getblk_unmovable(struct block_device *bdev,
340+ sector_t block, unsigned size)
343341 {
344− return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, __GFP_MOVABLE);
342+ gfp_t gfp;
343+
344+ gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
345+ gfp |= __GFP_NOFAIL;
346+
347+ return bdev_getblk(bdev, block, size, gfp);
345348 }
346349
350+static inline struct buffer_head *__getblk(struct block_device *bdev,
351+ sector_t block, unsigned size)
352+{
353+ gfp_t gfp;
347354
348−static inline struct buffer_head *
349−sb_getblk_gfp(struct super_block *sb, sector_t block, gfp_t gfp)
355+ gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
356+ gfp |= __GFP_MOVABLE | __GFP_NOFAIL;
357+
358+ return bdev_getblk(bdev, block, size, gfp);
359+}
360+
361+static inline struct buffer_head *sb_getblk(struct super_block *sb,
362+ sector_t block)
350363 {
351− return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, gfp);
364+ return __getblk(sb->s_bdev, block, sb->s_blocksize);
365+}
366+
367+static inline struct buffer_head *sb_getblk_gfp(struct super_block *sb,
368+ sector_t block, gfp_t gfp)
369+{
370+ return bdev_getblk(sb->s_bdev, block, sb->s_blocksize, gfp);
352371 }
353372
354373 static inline struct buffer_head *
@@ -385,20 +404,6 @@ static inline void lock_buffer(struct buffer_head *bh) · 30 unmodified lines
385404 __lock_buffer(bh);
386405 }
387406
388−static inline struct buffer_head *getblk_unmovable(struct block_device *bdev,
389− sector_t block,
390− unsigned size)
391−{
392− return __getblk_gfp(bdev, block, size, 0);
393−}
394−
395−static inline struct buffer_head *__getblk(struct block_device *bdev,
396− sector_t block,
397− unsigned size)
398−{
399− return __getblk_gfp(bdev, block, size, __GFP_MOVABLE);
400−}
401−
402407 static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags)
403408 {
404409 if (!buffer_uptodate(bh) && trylock_buffer(bh)) {
@@ -450,6 +455,28 @@ __bread(struct block_device *bdev, sector_t block, unsigned size) · 45 unmodified lines
450455 return __bread_gfp(bdev, block, size, __GFP_MOVABLE);
451456 }
452457
458+/**
459+ * get_nth_bh - Get a reference on the n'th buffer after this one.
460+ * @bh: The buffer to start counting from.
461+ * @count: How many buffers to skip.
462+ *
463+ * This is primarily useful for finding the nth buffer in a folio; in
464+ * that case you pass the head buffer and the byte offset in the folio
465+ * divided by the block size. It can be used for other purposes, but
466+ * it will wrap at the end of the folio rather than returning NULL or
467+ * proceeding to the next folio for you.
468+ *
469+ * Return: The requested buffer with an elevated refcount.
470+ */
471+static inline __must_check
472+struct buffer_head *get_nth_bh(struct buffer_head *bh, unsigned int count)
473+{
474+ while (count--)
475+ bh = bh->b_this_page;
476+ get_bh(bh);
477+ return bh;
478+}
479+
453480 bool block_dirty_folio(struct address_space *mapping, struct folio *folio);
454481
455482 #ifdef CONFIG_BUFFER_HEAD
include/linux/cacheinfo.h+1−0
@@ -73,6 +73,7 @@ struct cacheinfo { · 72 unmodified lines
7373
7474 struct cpu_cacheinfo {
7575 struct cacheinfo *info_list;
76+ unsigned int per_cpu_data_slice_size;
7677 unsigned int num_levels;
7778 unsigned int num_leaves;
7879 bool cpu_map_populated;
include/linux/cgroup-defs.h+5−0
@@ -115,6 +115,11 @@ enum { · 114 unmodified lines
115115 * Enable recursive subtree protection
116116 */
117117 CGRP_ROOT_MEMORY_RECURSIVE_PROT = (1 << 18),
118+
119+ /*
120+ * Enable hugetlb accounting for the memory controller.
121+ */
122+ CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING = (1 << 19),
118123 };
119124
120125 /* cftype->flags */
include/linux/damon.h+60−14
@@ -40,9 +40,24 @@ struct damon_addr_range { · 39 unmodified lines
4040 * @ar: The address range of the region.
4141 * @sampling_addr: Address of the sample for the next access check.
4242 * @nr_accesses: Access frequency of this region.
43+ * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
44+ * each sampling interval.
4345 * @list: List head for siblings.
4446 * @age: Age of this region.
4547 *
48+ * @nr_accesses is reset to zero for every &damon_attrs->aggr_interval and be
49+ * increased for every &damon_attrs->sample_interval if an access to the region
50+ * during the last sampling interval is found. The update of this field should
51+ * not be done with direct access but with the helper function,
52+ * damon_update_region_access_rate().
53+ *
54+ * @nr_accesses_bp is another representation of @nr_accesses in basis point
55+ * (1 in 10,000) that updated for every &damon_attrs->sample_interval in a
56+ * manner similar to moving sum. By the algorithm, this value becomes
57+ * @nr_accesses * 10000 for every &struct damon_attrs->aggr_interval. This can
58+ * be used when the aggregation interval is too huge and therefore cannot wait
59+ * for it before getting the access monitoring results.
60+ *
4661 * @age is initially zero, increased for each aggregation interval, and reset
4762 * to zero again if the access frequency is significantly changed. If two
4863 * regions are merged into a new region, both @nr_accesses and @age of the new
@@ -52,6 +67,7 @@ struct damon_region { · 3 unmodified lines
5267 struct damon_addr_range ar;
5368 unsigned long sampling_addr;
5469 unsigned int nr_accesses;
70+ unsigned int nr_accesses_bp;
5571 struct list_head list;
5672
5773 unsigned int age;
@@ -298,24 +314,24 @@ struct damos_access_pattern { · 240 unmodified lines
298314 * struct damos - Represents a Data Access Monitoring-based Operation Scheme.
299315 * @pattern: Access pattern of target regions.
300316 * @action: &damo_action to be applied to the target regions.
317+ * @apply_interval_us: The time between applying the @action.
301318 * @quota: Control the aggressiveness of this scheme.
302319 * @wmarks: Watermarks for automated (in)activation of this scheme.
303320 * @filters: Additional set of &struct damos_filter for &action.
304321 * @stat: Statistics of this scheme.
305322 * @list: List head for siblings.
306323 *
307− * For each aggregation interval, DAMON finds regions which fit in the
324+ * For each @apply_interval_us, DAMON finds regions which fit in the
308325 * &pattern and applies &action to those. To avoid consuming too much
309326 * CPU time or IO resources for the &action, &quota is used.
310327 *
328+ * If @apply_interval_us is zero, &damon_attrs->aggr_interval is used instead.
329+ *
311330 * To do the work only when needed, schemes can be activated for specific
312331 * system situations using &wmarks. If all schemes that registered to the
313332 * monitoring context are inactive, DAMON stops monitoring either, and just
314333 * repeatedly checks the watermarks.
315334 *
316− * If all schemes that registered to a &struct damon_ctx are inactive, DAMON
317− * stops monitoring and just repeatedly checks the watermarks.
318− *
319335 * Before applying the &action to a memory region, &struct damon_operations
320336 * implementation could check pages of the region and skip &action to respect
321337 * &filters
@@ -327,6 +343,14 @@ struct damos_access_pattern { · 5 unmodified lines
327343 struct damos {
328344 struct damos_access_pattern pattern;
329345 enum damos_action action;
346+ unsigned long apply_interval_us;
347+/* private: internal use only */
348+ /*
349+ * number of sample intervals that should be passed before applying
350+ * @action
351+ */
352+ unsigned long next_apply_sis;
353+/* public: */
330354 struct damos_quota quota;
331355 struct damos_watermarks wmarks;
332356 struct list_head filters;
@@ -472,13 +496,14 @@ struct damon_callback { · 139 unmodified lines
472496 * regions.
473497 *
474498 * For each @sample_interval, DAMON checks whether each region is accessed or
475− * not. It aggregates and keeps the access information (number of accesses to
476− * each region) for @aggr_interval time. DAMON also checks whether the target
477− * memory regions need update (e.g., by ``mmap()`` calls from the application,
478− * in case of virtual memory monitoring) and applies the changes for each
479− * @ops_update_interval. All time intervals are in micro-seconds.
480− * Please refer to &struct damon_operations and &struct damon_callback for more
481− * detail.
499+ * not during the last @sample_interval. If such access is found, DAMON
500+ * aggregates the information by increasing &damon_region->nr_accesses for
501+ * @aggr_interval time. For each @aggr_interval, the count is reset. DAMON
502+ * also checks whether the target memory regions need update (e.g., by
503+ * ``mmap()`` calls from the application, in case of virtual memory monitoring)
504+ * and applies the changes for each @ops_update_interval. All time intervals
505+ * are in micro-seconds. Please refer to &struct damon_operations and &struct
506+ * damon_callback for more detail.
482507 */
483508 struct damon_attrs {
484509 unsigned long sample_interval;
@@ -522,8 +547,18 @@ struct damon_ctx { · 37 unmodified lines
522547 struct damon_attrs attrs;
523548
524549 /* private: internal use only */
525− struct timespec64 last_aggregation;
526− struct timespec64 last_ops_update;
550+ /* number of sample intervals that passed since this context started */
551+ unsigned long passed_sample_intervals;
552+ /*
553+ * number of sample intervals that should be passed before next
554+ * aggregation
555+ */
556+ unsigned long next_aggregation_sis;
557+ /*
558+ * number of sample intervals that should be passed before next ops
559+ * update
560+ */
561+ unsigned long next_ops_update_sis;
527562
528563 /* public: */
529564 struct task_struct *kdamond;
@@ -608,6 +643,8 @@ void damon_add_region(struct damon_region *r, struct damon_target *t); · 78 unmodified lines
608643 void damon_destroy_region(struct damon_region *r, struct damon_target *t);
609644 int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
610645 unsigned int nr_ranges);
646+void damon_update_region_access_rate(struct damon_region *r, bool accessed,
647+ struct damon_attrs *attrs);
611648
612649 struct damos_filter *damos_new_filter(enum damos_filter_type type,
613650 bool matching);
@@ -615,7 +652,9 @@ void damos_add_filter(struct damos *s, struct damos_filter *f); · 1 unmodified line
615652 void damos_destroy_filter(struct damos_filter *f);
616653
617654 struct damos *damon_new_scheme(struct damos_access_pattern *pattern,
618− enum damos_action action, struct damos_quota *quota,
655+ enum damos_action action,
656+ unsigned long apply_interval_us,
657+ struct damos_quota *quota,
619658 struct damos_watermarks *wmarks);
620659 void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);
621660 void damon_destroy_scheme(struct damos *s);
@@ -642,6 +681,13 @@ static inline bool damon_target_has_pid(const struct damon_ctx *ctx) · 20 unmodified lines
642681 return ctx->ops.id == DAMON_OPS_VADDR || ctx->ops.id == DAMON_OPS_FVADDR;
643682 }
644683
684+static inline unsigned int damon_max_nr_accesses(const struct damon_attrs *attrs)
685+{
686+ /* {aggr,sample}_interval are unsigned long, hence could overflow */
687+ return min(attrs->aggr_interval / attrs->sample_interval,
688+ (unsigned long)UINT_MAX);
689+}
690+
645691
646692 int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive);
647693 int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
include/linux/dax.h+5−5
@@ -159,8 +159,8 @@ int dax_writeback_mapping_range(struct address_space *mapping, · 158 unmodified lines
159159
160160 struct page *dax_layout_busy_page(struct address_space *mapping);
161161 struct page *dax_layout_busy_page_range(struct address_space *mapping, loff_t start, loff_t end);
162−dax_entry_t dax_lock_page(struct page *page);
163−void dax_unlock_page(struct page *page, dax_entry_t cookie);
162+dax_entry_t dax_lock_folio(struct folio *folio);
163+void dax_unlock_folio(struct folio *folio, dax_entry_t cookie);
164164 dax_entry_t dax_lock_mapping_entry(struct address_space *mapping,
165165 unsigned long index, struct page **page);
166166 void dax_unlock_mapping_entry(struct address_space *mapping,
@@ -182,14 +182,14 @@ static inline int dax_writeback_mapping_range(struct address_space *mapping, · 15 unmodified lines
182182 return -EOPNOTSUPP;
183183 }
184184
185−static inline dax_entry_t dax_lock_page(struct page *page)
185+static inline dax_entry_t dax_lock_folio(struct folio *folio)
186186 {
187− if (IS_DAX(page->mapping->host))
187+ if (IS_DAX(folio->mapping->host))
188188 return ~0UL;
189189 return 0;
190190 }
191191
192−static inline void dax_unlock_page(struct page *page, dax_entry_t cookie)
192+static inline void dax_unlock_folio(struct folio *folio, dax_entry_t cookie)
193193 {
194194 }
195195
include/linux/fs.h+3−3
@@ -454,7 +454,7 @@ extern const struct address_space_operations empty_aops; · 453 unmodified lines
454454 * It is also used to block modification of page cache contents through
455455 * memory mappings.
456456 * @gfp_mask: Memory allocation flags to use for allocating pages.
457− * @i_mmap_writable: Number of VM_SHARED mappings.
457+ * @i_mmap_writable: Number of VM_SHARED, VM_MAYWRITE mappings.
458458 * @nr_thps: Number of THPs in the pagecache (non-shmem only).
459459 * @i_mmap: Tree of private and shared mappings.
460460 * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable.
@@ -557,7 +557,7 @@ static inline int mapping_mapped(struct address_space *mapping) · 96 unmodified lines
557557
558558 /*
559559 * Might pages of this file have been modified in userspace?
560− * Note that i_mmap_writable counts all VM_SHARED vmas: do_mmap
560+ * Note that i_mmap_writable counts all VM_SHARED, VM_MAYWRITE vmas: do_mmap
561561 * marks vma as VM_SHARED if it is shared, and the file was opened for
562562 * writing i.e. vma may be mprotected writable even if now readonly.
563563 *
@@ -1270,7 +1270,7 @@ struct super_block { · 706 unmodified lines
12701270
12711271 const struct dentry_operations *s_d_op; /* default d_op for dentries */
12721272
1273− struct shrinker s_shrink; /* per-sb shrinker handle */
1273+ struct shrinker *s_shrink; /* per-sb shrinker handle */
12741274
12751275 /* Number of inodes with nlink == 0 but still referenced */
12761276 atomic_long_t s_remove_count;
include/linux/gfp.h+11−1
@@ -8,6 +8,7 @@ · 7 unmodified lines
88 #include <linux/topology.h>
99
1010 struct vm_area_struct;
11+struct mempolicy;
1112
1213 /* Convert GFP flags to their corresponding migrate type */
1314 #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
@@ -262,7 +263,9 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, · 248 unmodified lines
262263
263264 #ifdef CONFIG_NUMA
264265 struct page *alloc_pages(gfp_t gfp, unsigned int order);
265−struct folio *folio_alloc(gfp_t gfp, unsigned order);
266+struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
267+ struct mempolicy *mpol, pgoff_t ilx, int nid);
268+struct folio *folio_alloc(gfp_t gfp, unsigned int order);
266269 struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,
267270 unsigned long addr, bool hugepage);
268271 #else
@@ -270,6 +273,11 @@ static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) · 1 unmodified line
270273 {
271274 return alloc_pages_node(numa_node_id(), gfp_mask, order);
272275 }
276+static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
277+ struct mempolicy *mpol, pgoff_t ilx, int nid)
278+{
279+ return alloc_pages(gfp, order);
280+}
273281 static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order)
274282 {
275283 return __folio_alloc_node(gfp, order, numa_node_id());
@@ -320,11 +328,13 @@ extern void page_frag_free(void *addr); · 44 unmodified lines
320328 #define free_page(addr) free_pages((addr), 0)
321329
322330 void page_alloc_init_cpuhp(void);
331+int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp);
323332 void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp);
324333 void drain_all_pages(struct zone *zone);
325334 void drain_local_pages(struct zone *zone);
326335
327336 void page_alloc_init_late(void);
337+void setup_pcp_cacheinfo(void);
328338
329339 /*
330340 * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
include/linux/hugetlb.h+14−11
@@ -30,7 +30,7 @@ void free_huge_folio(struct folio *folio); · 29 unmodified lines
3030
3131 #ifdef CONFIG_HUGETLB_PAGE
3232
33−#include <linux/mempolicy.h>
33+#include <linux/pagemap.h>
3434 #include <linux/shm.h>
3535 #include <asm/tlbflush.h>
3636
@@ -280,6 +280,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, · 243 unmodified lines
280280 unsigned long cp_flags);
281281
282282 bool is_hugetlb_entry_migration(pte_t pte);
283+bool is_hugetlb_entry_hwpoisoned(pte_t pte);
283284 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
284285
285286 #else /* !CONFIG_HUGETLB_PAGE */
@@ -544,7 +545,6 @@ static inline struct hugetlbfs_sb_info *HUGETLBFS_SB(struct super_block *sb) · 258 unmodified lines
544545 }
545546
546547 struct hugetlbfs_inode_info {
547− struct shared_policy policy;
548548 struct inode vfs_inode;
549549 unsigned int seals;
550550 };
@@ -748,8 +748,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 197 unmodified lines
748748 unsigned long addr, int avoid_reserve);
749749 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
750750 nodemask_t *nmask, gfp_t gfp_mask);
751−struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma,
752− unsigned long address);
753751 int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping,
754752 pgoff_t idx);
755753 void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
@@ -844,6 +842,12 @@ static inline unsigned int blocks_per_huge_page(struct hstate *h) · 88 unmodified lines
844842 return huge_page_size(h) / 512;
845843 }
846844
845+static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h,
846+ struct address_space *mapping, pgoff_t idx)
847+{
848+ return filemap_lock_folio(mapping, idx << huge_page_order(h));
849+}
850+
847851 #include <asm/hugetlb.h>
848852
849853 #ifndef is_hugepage_only_range
@@ -1040,6 +1044,12 @@ static inline struct hugepage_subpool *hugetlb_folio_subpool(struct folio *folio · 190 unmodified lines
10401044 return NULL;
10411045 }
10421046
1047+static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h,
1048+ struct address_space *mapping, pgoff_t idx)
1049+{
1050+ return NULL;
1051+}
1052+
10431053 static inline int isolate_or_dissolve_huge_page(struct page *page,
10441054 struct list_head *list)
10451055 {
@@ -1060,13 +1070,6 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 14 unmodified lines
10601070 return NULL;
10611071 }
10621072
1063−static inline struct folio *alloc_hugetlb_folio_vma(struct hstate *h,
1064− struct vm_area_struct *vma,
1065− unsigned long address)
1066−{
1067− return NULL;
1068−}
1069−
10701073 static inline int __alloc_bootmem_huge_page(struct hstate *h)
10711074 {
10721075 return 0;
include/linux/hugetlb_cgroup.h+0−11
@@ -22,13 +22,6 @@ struct resv_map; · 21 unmodified lines
2222 struct file_region;
2323
2424 #ifdef CONFIG_CGROUP_HUGETLB
25−/*
26− * Minimum page order trackable by hugetlb cgroup.
27− * At least 3 pages are necessary for all the tracking information.
28− * The second tail page contains all of the hugetlb-specific fields.
29− */
30−#define HUGETLB_CGROUP_MIN_ORDER order_base_2(__NR_USED_SUBPAGE)
31−
3225 enum hugetlb_memory_event {
3326 HUGETLB_MAX,
3427 HUGETLB_NR_MEMORY_EVENTS,
@@ -68,8 +61,6 @@ static inline struct hugetlb_cgroup * · 33 unmodified lines
6861 __hugetlb_cgroup_from_folio(struct folio *folio, bool rsvd)
6962 {
7063 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);
71− if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER)
72− return NULL;
7364 if (rsvd)
7465 return folio->_hugetlb_cgroup_rsvd;
7566 else
@@ -91,8 +82,6 @@ static inline void __set_hugetlb_cgroup(struct folio *folio, · 15 unmodified lines
9182 struct hugetlb_cgroup *h_cg, bool rsvd)
9283 {
9384 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);
94− if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER)
95− return;
9685 if (rsvd)
9786 folio->_hugetlb_cgroup_rsvd = h_cg;
9887 else
include/linux/jbd2.h+1−1
@@ -886,7 +886,7 @@ struct journal_s · 885 unmodified lines
886886 * Journal head shrinker, reclaim buffer's journal head which
887887 * has been written back.
888888 */
889− struct shrinker j_shrinker;
889+ struct shrinker *j_shrinker;
890890
891891 /**
892892 * @j_checkpoint_jh_count:
include/linux/memblock.h+9−0
@@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; · 39 unmodified lines
4040 * via a driver, and never indicated in the firmware-provided memory map as
4141 * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the
4242 * kernel resource tree.
43+ * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are
44+ * not initialized (only for reserved regions).
4345 */
4446 enum memblock_flags {
4547 MEMBLOCK_NONE = 0x0, /* No special request */
@@ -47,6 +49,7 @@ enum memblock_flags { · 1 unmodified line
4749 MEMBLOCK_MIRROR = 0x2, /* mirrored region */
4850 MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */
4951 MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */
52+ MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */
5053 };
5154
5255 /**
@@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); · 72 unmodified lines
125128 int memblock_mark_mirror(phys_addr_t base, phys_addr_t size);
126129 int memblock_mark_nomap(phys_addr_t base, phys_addr_t size);
127130 int memblock_clear_nomap(phys_addr_t base, phys_addr_t size);
131+int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size);
128132
129133 void memblock_free_all(void);
130134 void memblock_free(void *ptr, size_t size);
@@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) · 128 unmodified lines
259263 return m->flags & MEMBLOCK_NOMAP;
260264 }
261265
266+static inline bool memblock_is_reserved_noinit(struct memblock_region *m)
267+{
268+ return m->flags & MEMBLOCK_RSRV_NOINIT;
269+}
270+
262271 static inline bool memblock_is_driver_managed(struct memblock_region *m)
263272 {
264273 return m->flags & MEMBLOCK_DRIVER_MANAGED;
include/linux/memcontrol.h+65−28
@@ -21,6 +21,7 @@ · 20 unmodified lines
2121 #include <linux/vmstat.h>
2222 #include <linux/writeback.h>
2323 #include <linux/page-flags.h>
24+#include <linux/shrinker.h>
2425
2526 struct mem_cgroup;
2627 struct obj_cgroup;
@@ -88,17 +89,6 @@ struct mem_cgroup_reclaim_iter { · 61 unmodified lines
8889 unsigned int generation;
8990 };
9091
91−/*
92− * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
93− * shrinkers, which have elements charged to this memcg.
94− */
95−struct shrinker_info {
96− struct rcu_head rcu;
97− atomic_long_t *nr_deferred;
98− unsigned long *map;
99− int map_nr_max;
100−};
101−
10292 struct lruvec_stats_percpu {
10393 /* Local (CPU and cgroup) state */
10494 long state[NR_VM_NODE_STAT_ITEMS];
@@ -153,7 +143,7 @@ struct mem_cgroup_threshold_ary { · 48 unmodified lines
153143 /* Size of entries[] */
154144 unsigned int size;
155145 /* Array of thresholds */
156− struct mem_cgroup_threshold entries[];
146+ struct mem_cgroup_threshold entries[] __counted_by(size);
157147 };
158148
159149 struct mem_cgroup_thresholds {
@@ -299,7 +289,13 @@ struct mem_cgroup { · 139 unmodified lines
299289
300290 #ifdef CONFIG_MEMCG_KMEM
301291 int kmemcg_id;
302− struct obj_cgroup __rcu *objcg;
292+ /*
293+ * memcg->objcg is wiped out as a part of the objcg repaprenting
294+ * process. memcg->orig_objcg preserves a pointer (and a reference)
295+ * to the original objcg until the end of live of memcg.
296+ */
297+ struct obj_cgroup __rcu *objcg;
298+ struct obj_cgroup *orig_objcg;
303299 /* list of inherited objcgs, protected by objcg_lock */
304300 struct list_head objcg_list;
305301 #endif
@@ -662,6 +658,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 356 unmodified lines
662658 page_counter_read(&memcg->memory);
663659 }
664660
661+void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg);
662+
665663 int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);
666664
667665 /**
@@ -686,6 +684,9 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 18 unmodified lines
686684 return __mem_cgroup_charge(folio, mm, gfp);
687685 }
688686
687+int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
688+ long nr_pages);
689+
689690 int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
690691 gfp_t gfp, swp_entry_t entry);
691692 void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
@@ -713,6 +714,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 21 unmodified lines
713714 __mem_cgroup_uncharge_list(page_list);
714715 }
715716
717+void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages);
718+
719+void mem_cgroup_replace_folio(struct folio *old, struct folio *new);
720+
716721 void mem_cgroup_migrate(struct folio *old, struct folio *new);
717722
718723 /**
@@ -769,6 +774,8 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); · 50 unmodified lines
769774
770775 struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);
771776
777+struct mem_cgroup *get_mem_cgroup_from_current(void);
778+
772779 struct lruvec *folio_lruvec_lock(struct folio *folio);
773780 struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
774781 struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
@@ -1080,15 +1087,6 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, · 305 unmodified lines
10801087 local_irq_restore(flags);
10811088 }
10821089
1083−static inline void count_memcg_page_event(struct page *page,
1084− enum vm_event_item idx)
1085−{
1086− struct mem_cgroup *memcg = page_memcg(page);
1087−
1088− if (memcg)
1089− count_memcg_events(memcg, idx, 1);
1090−}
1091−
10921090 static inline void count_memcg_folio_events(struct folio *folio,
10931091 enum vm_event_item idx, unsigned long nr)
10941092 {
@@ -1249,12 +1247,23 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 154 unmodified lines
12491247 return false;
12501248 }
12511249
1250+static inline void mem_cgroup_commit_charge(struct folio *folio,
1251+ struct mem_cgroup *memcg)
1252+{
1253+}
1254+
12521255 static inline int mem_cgroup_charge(struct folio *folio,
12531256 struct mm_struct *mm, gfp_t gfp)
12541257 {
12551258 return 0;
12561259 }
12571260
1261+static inline int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg,
1262+ gfp_t gfp, long nr_pages)
1263+{
1264+ return 0;
1265+}
1266+
12581267 static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
12591268 struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
12601269 {
@@ -1273,6 +1282,16 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 12 unmodified lines
12731282 {
12741283 }
12751284
1285+static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg,
1286+ unsigned int nr_pages)
1287+{
1288+}
1289+
1290+static inline void mem_cgroup_replace_folio(struct folio *old,
1291+ struct folio *new)
1292+{
1293+}
1294+
12761295 static inline void mem_cgroup_migrate(struct folio *old, struct folio *new)
12771296 {
12781297 }
@@ -1310,6 +1329,11 @@ static inline struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 31 unmodified lines
13101329 return NULL;
13111330 }
13121331
1332+static inline struct mem_cgroup *get_mem_cgroup_from_current(void)
1333+{
1334+ return NULL;
1335+}
1336+
13131337 static inline
13141338 struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)
13151339 {
@@ -1565,11 +1589,6 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, · 249 unmodified lines
15651589 {
15661590 }
15671591
1568−static inline void count_memcg_page_event(struct page *page,
1569− int idx)
1570−{
1571−}
1572−
15731592 static inline void count_memcg_folio_events(struct folio *folio,
15741593 enum vm_event_item idx, unsigned long nr)
15751594 {
@@ -1763,9 +1782,27 @@ bool mem_cgroup_kmem_disabled(void); · 187 unmodified lines
17631782 int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
17641783 void __memcg_kmem_uncharge_page(struct page *page, int order);
17651784
1766−struct obj_cgroup *get_obj_cgroup_from_current(void);
1785+/*
1786+ * The returned objcg pointer is safe to use without additional
1787+ * protection within a scope. The scope is defined either by
1788+ * the current task (similar to the "current" global variable)
1789+ * or by set_active_memcg() pair.
1790+ * Please, use obj_cgroup_get() to get a reference if the pointer
1791+ * needs to be used outside of the local scope.
1792+ */
1793+struct obj_cgroup *current_obj_cgroup(void);
17671794 struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio);
17681795
1796+static inline struct obj_cgroup *get_obj_cgroup_from_current(void)
1797+{
1798+ struct obj_cgroup *objcg = current_obj_cgroup();
1799+
1800+ if (objcg)
1801+ obj_cgroup_get(objcg);
1802+
1803+ return objcg;
1804+}
1805+
17691806 int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size);
17701807 void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size);
17711808
include/linux/memory-tiers.h+40−1
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 #include <linux/nodemask.h>
77 #include <linux/kref.h>
88 #include <linux/mmzone.h>
9+#include <linux/notifier.h>
910 /*
1011 * Each tier cover a abstrace distance chunk size of 128
1112 */
@@ -22,7 +23,9 @@ · 10 unmodified lines
2223 struct memory_tier;
2324 struct memory_dev_type {
2425 /* list of memory types that are part of same tier as this type */
25− struct list_head tier_sibiling;
26+ struct list_head tier_sibling;
27+ /* list of memory types that are managed by one driver */
28+ struct list_head list;
2629 /* abstract distance for this specific memory type */
2730 int adistance;
2831 /* Nodes of same abstract distance */
@@ -30,12 +33,21 @@ struct memory_dev_type { · 1 unmodified line
3033 struct kref kref;
3134 };
3235
36+struct node_hmem_attrs;
37+
3338 #ifdef CONFIG_NUMA
3439 extern bool numa_demotion_enabled;
40+extern struct memory_dev_type *default_dram_type;
3541 struct memory_dev_type *alloc_memory_type(int adistance);
3642 void put_memory_type(struct memory_dev_type *memtype);
3743 void init_node_memory_type(int node, struct memory_dev_type *default_type);
3844 void clear_node_memory_type(int node, struct memory_dev_type *memtype);
45+int register_mt_adistance_algorithm(struct notifier_block *nb);
46+int unregister_mt_adistance_algorithm(struct notifier_block *nb);
47+int mt_calc_adistance(int node, int *adist);
48+int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
49+ const char *source);
50+int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist);
3951 #ifdef CONFIG_MIGRATION
4052 int next_demotion_node(int node);
4153 void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
@@ -60,6 +72,7 @@ static inline bool node_is_toptier(int node) · 18 unmodified lines
6072 #else
6173
6274 #define numa_demotion_enabled false
75+#define default_dram_type NULL
6376 /*
6477 * CONFIG_NUMA implementation returns non NULL error.
6578 */
@@ -97,5 +110,31 @@ static inline bool node_is_toptier(int node) · 31 unmodified lines
97110 {
98111 return true;
99112 }
113+
114+static inline int register_mt_adistance_algorithm(struct notifier_block *nb)
115+{
116+ return 0;
117+}
118+
119+static inline int unregister_mt_adistance_algorithm(struct notifier_block *nb)
120+{
121+ return 0;
122+}
123+
124+static inline int mt_calc_adistance(int node, int *adist)
125+{
126+ return NOTIFY_DONE;
127+}
128+
129+static inline int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
130+ const char *source)
131+{
132+ return -EIO;
133+}
134+
135+static inline int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist)
136+{
137+ return -EIO;
138+}
100139 #endif /* CONFIG_NUMA */
101140 #endif /* _LINUX_MEMORY_TIERS_H */
include/linux/mempolicy.h+31−27
@@ -17,6 +17,8 @@ · 16 unmodified lines
1717
1818 struct mm_struct;
1919
20+#define NO_INTERLEAVE_INDEX (-1UL) /* use task il_prev for interleaving */
21+
2022 #ifdef CONFIG_NUMA
2123
2224 /*
@@ -89,8 +91,6 @@ static inline struct mempolicy *mpol_dup(struct mempolicy *pol) · 66 unmodified lines
8991 return pol;
9092 }
9193
92−#define vma_policy(vma) ((vma)->vm_policy)
93−
9494 static inline void mpol_get(struct mempolicy *pol)
9595 {
9696 if (pol)
@@ -107,35 +107,30 @@ static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) · 10 unmodified lines
107107
108108 /*
109109 * Tree of shared policies for a shared memory region.
110− * Maintain the policies in a pseudo mm that contains vmas. The vmas
111− * carry the policy. As a special twist the pseudo mm is indexed in pages, not
112− * bytes, so that we can work with shared memory segments bigger than
113− * unsigned long.
114110 */
115−
116−struct sp_node {
117− struct rb_node nd;
118− unsigned long start, end;
119− struct mempolicy *policy;
120−};
121−
122111 struct shared_policy {
123112 struct rb_root root;
124113 rwlock_t lock;
125114 };
115+struct sp_node {
116+ struct rb_node nd;
117+ pgoff_t start, end;
118+ struct mempolicy *policy;
119+};
126120
127121 int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst);
128122 void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol);
129−int mpol_set_shared_policy(struct shared_policy *info,
130− struct vm_area_struct *vma,
131− struct mempolicy *new);
132−void mpol_free_shared_policy(struct shared_policy *p);
123+int mpol_set_shared_policy(struct shared_policy *sp,
124+ struct vm_area_struct *vma, struct mempolicy *mpol);
125+void mpol_free_shared_policy(struct shared_policy *sp);
133126 struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
134− unsigned long idx);
127+ pgoff_t idx);
135128
136129 struct mempolicy *get_task_policy(struct task_struct *p);
137130 struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
138− unsigned long addr);
131+ unsigned long addr, pgoff_t *ilx);
132+struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
133+ unsigned long addr, int order, pgoff_t *ilx);
139134 bool vma_policy_mof(struct vm_area_struct *vma);
140135
141136 extern void numa_default_policy(void);
@@ -149,8 +144,6 @@ extern int huge_node(struct vm_area_struct *vma, · 7 unmodified lines
149144 extern bool init_nodemask_of_mempolicy(nodemask_t *mask);
150145 extern bool mempolicy_in_oom_domain(struct task_struct *tsk,
151146 const nodemask_t *mask);
152−extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy);
153−
154147 extern unsigned int mempolicy_slab_node(void);
155148
156149 extern enum zone_type policy_zone;
@@ -174,7 +167,7 @@ extern void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol); · 17 unmodified lines
174167 /* Check if a vma is migratable */
175168 extern bool vma_migratable(struct vm_area_struct *vma);
176169
177−extern int mpol_misplaced(struct page *, struct vm_area_struct *, unsigned long);
170+int mpol_misplaced(struct folio *, struct vm_area_struct *, unsigned long);
178171 extern void mpol_put_task_policy(struct task_struct *);
179172
180173 static inline bool mpol_is_preferred_many(struct mempolicy *pol)
@@ -188,12 +181,17 @@ extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone); · 7 unmodified lines
188181
189182 struct mempolicy {};
190183
184+static inline struct mempolicy *get_task_policy(struct task_struct *p)
185+{
186+ return NULL;
187+}
188+
191189 static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b)
192190 {
193191 return true;
194192 }
195193
196−static inline void mpol_put(struct mempolicy *p)
194+static inline void mpol_put(struct mempolicy *pol)
197195 {
198196 }
199197
@@ -212,17 +210,22 @@ static inline void mpol_shared_policy_init(struct shared_policy *sp, · 12 unmodified lines
212210 {
213211 }
214212
215−static inline void mpol_free_shared_policy(struct shared_policy *p)
213+static inline void mpol_free_shared_policy(struct shared_policy *sp)
216214 {
217215 }
218216
219217 static inline struct mempolicy *
220−mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx)
218+mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx)
221219 {
222220 return NULL;
223221 }
224222
225−#define vma_policy(vma) NULL
223+static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
224+ unsigned long addr, int order, pgoff_t *ilx)
225+{
226+ *ilx = 0;
227+ return NULL;
228+}
226229
227230 static inline int
228231 vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
@@ -278,7 +281,8 @@ static inline int mpol_parse_str(char *str, struct mempolicy **mpol) · 49 unmodified lines
278281 }
279282 #endif
280283
281−static inline int mpol_misplaced(struct page *page, struct vm_area_struct *vma,
284+static inline int mpol_misplaced(struct folio *folio,
285+ struct vm_area_struct *vma,
282286 unsigned long address)
283287 {
284288 return -1; /* no node preference */
include/linux/migrate.h+2−2
@@ -142,10 +142,10 @@ const struct movable_operations *page_movable_ops(struct page *page) · 141 unmodified lines
142142 }
143143
144144 #ifdef CONFIG_NUMA_BALANCING
145−int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
145+int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
146146 int node);
147147 #else
148−static inline int migrate_misplaced_page(struct page *page,
148+static inline int migrate_misplaced_folio(struct folio *folio,
149149 struct vm_area_struct *vma, int node)
150150 {
151151 return -EAGAIN; /* can't migrate now */
include/linux/mm.h+130−46
@@ -617,7 +617,7 @@ struct vm_operations_struct { · 616 unmodified lines
617617 * policy.
618618 */
619619 struct mempolicy *(*get_policy)(struct vm_area_struct *vma,
620− unsigned long addr);
620+ unsigned long addr, pgoff_t *ilx);
621621 #endif
622622 /*
623623 * Called by vm_normal_page() for special PTEs to find the
@@ -935,6 +935,17 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) · 311 unmodified lines
935935 return vma->vm_flags & VM_ACCESS_FLAGS;
936936 }
937937
938+static inline bool is_shared_maywrite(vm_flags_t vm_flags)
939+{
940+ return (vm_flags & (VM_SHARED | VM_MAYWRITE)) ==
941+ (VM_SHARED | VM_MAYWRITE);
942+}
943+
944+static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma)
945+{
946+ return is_shared_maywrite(vma->vm_flags);
947+}
948+
938949 static inline
939950 struct vm_area_struct *vma_find(struct vma_iterator *vmi, unsigned long max)
940951 {
@@ -1335,7 +1346,6 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, · 394 unmodified lines
13351346 struct page *page, unsigned int nr, unsigned long addr);
13361347
13371348 vm_fault_t finish_fault(struct vm_fault *vmf);
1338−vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);
13391349 #endif
13401350
13411351 /*
@@ -1684,26 +1694,26 @@ static inline bool __cpupid_match_pid(pid_t task_pid, int cpupid) · 342 unmodified lines
16841694
16851695 #define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid)
16861696 #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
1687−static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
1697+static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
16881698 {
1689− return xchg(&page->_last_cpupid, cpupid & LAST_CPUPID_MASK);
1699+ return xchg(&folio->_last_cpupid, cpupid & LAST_CPUPID_MASK);
16901700 }
16911701
1692−static inline int page_cpupid_last(struct page *page)
1702+static inline int folio_last_cpupid(struct folio *folio)
16931703 {
1694− return page->_last_cpupid;
1704+ return folio->_last_cpupid;
16951705 }
16961706 static inline void page_cpupid_reset_last(struct page *page)
16971707 {
16981708 page->_last_cpupid = -1 & LAST_CPUPID_MASK;
16991709 }
17001710 #else
1701−static inline int page_cpupid_last(struct page *page)
1711+static inline int folio_last_cpupid(struct folio *folio)
17021712 {
1703− return (page->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
1713+ return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
17041714 }
17051715
1706−extern int page_cpupid_xchg_last(struct page *page, int cpupid);
1716+int folio_xchg_last_cpupid(struct folio *folio, int cpupid);
17071717
17081718 static inline void page_cpupid_reset_last(struct page *page)
17091719 {
@@ -1711,11 +1721,12 @@ static inline void page_cpupid_reset_last(struct page *page) · 1 unmodified line
17111721 }
17121722 #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
17131723
1714−static inline int xchg_page_access_time(struct page *page, int time)
1724+static inline int folio_xchg_access_time(struct folio *folio, int time)
17151725 {
17161726 int last_time;
17171727
1718− last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS);
1728+ last_time = folio_xchg_last_cpupid(folio,
1729+ time >> PAGE_ACCESS_TIME_BUCKETS);
17191730 return last_time << PAGE_ACCESS_TIME_BUCKETS;
17201731 }
17211732
@@ -1729,19 +1740,19 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) · 7 unmodified lines
17291740 }
17301741 }
17311742 #else /* !CONFIG_NUMA_BALANCING */
1732−static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
1743+static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
17331744 {
1734− return page_to_nid(page); /* XXX */
1745+ return folio_nid(folio); /* XXX */
17351746 }
17361747
1737−static inline int xchg_page_access_time(struct page *page, int time)
1748+static inline int folio_xchg_access_time(struct folio *folio, int time)
17381749 {
17391750 return 0;
17401751 }
17411752
1742−static inline int page_cpupid_last(struct page *page)
1753+static inline int folio_last_cpupid(struct folio *folio)
17431754 {
1744− return page_to_nid(page); /* XXX */
1755+ return folio_nid(folio); /* XXX */
17451756 }
17461757
17471758 static inline int cpupid_to_nid(int cpupid)
@@ -2325,6 +2336,8 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, · 577 unmodified lines
23252336 pte_t pte);
23262337 struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
23272338 pte_t pte);
2339+struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
2340+ unsigned long addr, pmd_t pmd);
23282341 struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
23292342 pmd_t pmd);
23302343
@@ -2411,8 +2424,6 @@ extern int access_process_vm(struct task_struct *tsk, unsigned long addr, · 80 unmodified lines
24112424 void *buf, int len, unsigned int gup_flags);
24122425 extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,
24132426 void *buf, int len, unsigned int gup_flags);
2414−extern int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
2415− void *buf, int len, unsigned int gup_flags);
24162427
24172428 long get_user_pages_remote(struct mm_struct *mm,
24182429 unsigned long start, unsigned long nr_pages,
@@ -2423,6 +2434,9 @@ long pin_user_pages_remote(struct mm_struct *mm, · 4 unmodified lines
24232434 unsigned int gup_flags, struct page **pages,
24242435 int *locked);
24252436
2437+/*
2438+ * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
2439+ */
24262440 static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,
24272441 unsigned long addr,
24282442 int gup_flags,
@@ -2430,12 +2444,15 @@ static inline struct page *get_user_page_vma_remote(struct mm_struct *mm, · 1 unmodified line
24302444 {
24312445 struct page *page;
24322446 struct vm_area_struct *vma;
2433− int got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);
2447+ int got;
2448+
2449+ if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT)))
2450+ return ERR_PTR(-EINVAL);
2451+
2452+ got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);
24342453
24352454 if (got < 0)
24362455 return ERR_PTR(got);
2437− if (got == 0)
2438− return NULL;
24392456
24402457 vma = vma_lookup(mm, addr);
24412458 if (WARN_ON_ONCE(!vma)) {
@@ -2478,7 +2495,7 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen); · 36 unmodified lines
24782495 extern unsigned long move_page_tables(struct vm_area_struct *vma,
24792496 unsigned long old_addr, struct vm_area_struct *new_vma,
24802497 unsigned long new_addr, unsigned long len,
2481− bool need_rmap_locks);
2498+ bool need_rmap_locks, bool for_stack);
24822499
24832500 /*
24842501 * Flags used by change_protection(). For now we make it a bitmap so
@@ -2626,14 +2643,6 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, · 141 unmodified lines
26262643 *maxrss = hiwater_rss;
26272644 }
26282645
2629−#if defined(SPLIT_RSS_COUNTING)
2630−void sync_mm_rss(struct mm_struct *mm);
2631−#else
2632−static inline void sync_mm_rss(struct mm_struct *mm)
2633−{
2634−}
2635−#endif
2636−
26372646 #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL
26382647 static inline int pte_special(pte_t pte)
26392648 {
@@ -3055,6 +3064,22 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) · 415 unmodified lines
30553064 return ptl;
30563065 }
30573066
3067+static inline void pagetable_pud_ctor(struct ptdesc *ptdesc)
3068+{
3069+ struct folio *folio = ptdesc_folio(ptdesc);
3070+
3071+ __folio_set_pgtable(folio);
3072+ lruvec_stat_add_folio(folio, NR_PAGETABLE);
3073+}
3074+
3075+static inline void pagetable_pud_dtor(struct ptdesc *ptdesc)
3076+{
3077+ struct folio *folio = ptdesc_folio(ptdesc);
3078+
3079+ __folio_clear_pgtable(folio);
3080+ lruvec_stat_sub_folio(folio, NR_PAGETABLE);
3081+}
3082+
30583083 extern void __init pagecache_init(void);
30593084 extern void free_initmem(void);
30603085
@@ -3219,22 +3244,73 @@ extern int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, · 158 unmodified lines
32193244 struct vm_area_struct *next);
32203245 extern int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
32213246 unsigned long start, unsigned long end, pgoff_t pgoff);
3222−extern struct vm_area_struct *vma_merge(struct vma_iterator *vmi,
3223− struct mm_struct *, struct vm_area_struct *prev, unsigned long addr,
3224− unsigned long end, unsigned long vm_flags, struct anon_vma *,
3225− struct file *, pgoff_t, struct mempolicy *, struct vm_userfaultfd_ctx,
3226− struct anon_vma_name *);
32273247 extern struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *);
3228−extern int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *,
3229− unsigned long addr, int new_below);
3230−extern int split_vma(struct vma_iterator *vmi, struct vm_area_struct *,
3231− unsigned long addr, int new_below);
32323248 extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);
32333249 extern void unlink_file_vma(struct vm_area_struct *);
32343250 extern struct vm_area_struct *copy_vma(struct vm_area_struct **,
32353251 unsigned long addr, unsigned long len, pgoff_t pgoff,
32363252 bool *need_rmap_locks);
32373253 extern void exit_mmap(struct mm_struct *);
3254+struct vm_area_struct *vma_modify(struct vma_iterator *vmi,
3255+ struct vm_area_struct *prev,
3256+ struct vm_area_struct *vma,
3257+ unsigned long start, unsigned long end,
3258+ unsigned long vm_flags,
3259+ struct mempolicy *policy,
3260+ struct vm_userfaultfd_ctx uffd_ctx,
3261+ struct anon_vma_name *anon_name);
3262+
3263+/* We are about to modify the VMA's flags. */
3264+static inline struct vm_area_struct
3265+*vma_modify_flags(struct vma_iterator *vmi,
3266+ struct vm_area_struct *prev,
3267+ struct vm_area_struct *vma,
3268+ unsigned long start, unsigned long end,
3269+ unsigned long new_flags)
3270+{
3271+ return vma_modify(vmi, prev, vma, start, end, new_flags,
3272+ vma_policy(vma), vma->vm_userfaultfd_ctx,
3273+ anon_vma_name(vma));
3274+}
3275+
3276+/* We are about to modify the VMA's flags and/or anon_name. */
3277+static inline struct vm_area_struct
3278+*vma_modify_flags_name(struct vma_iterator *vmi,
3279+ struct vm_area_struct *prev,
3280+ struct vm_area_struct *vma,
3281+ unsigned long start,
3282+ unsigned long end,
3283+ unsigned long new_flags,
3284+ struct anon_vma_name *new_name)
3285+{
3286+ return vma_modify(vmi, prev, vma, start, end, new_flags,
3287+ vma_policy(vma), vma->vm_userfaultfd_ctx, new_name);
3288+}
3289+
3290+/* We are about to modify the VMA's memory policy. */
3291+static inline struct vm_area_struct
3292+*vma_modify_policy(struct vma_iterator *vmi,
3293+ struct vm_area_struct *prev,
3294+ struct vm_area_struct *vma,
3295+ unsigned long start, unsigned long end,
3296+ struct mempolicy *new_pol)
3297+{
3298+ return vma_modify(vmi, prev, vma, start, end, vma->vm_flags,
3299+ new_pol, vma->vm_userfaultfd_ctx, anon_vma_name(vma));
3300+}
3301+
3302+/* We are about to modify the VMA's flags and/or uffd context. */
3303+static inline struct vm_area_struct
3304+*vma_modify_flags_uffd(struct vma_iterator *vmi,
3305+ struct vm_area_struct *prev,
3306+ struct vm_area_struct *vma,
3307+ unsigned long start, unsigned long end,
3308+ unsigned long new_flags,
3309+ struct vm_userfaultfd_ctx new_ctx)
3310+{
3311+ return vma_modify(vmi, prev, vma, start, end, new_flags,
3312+ vma_policy(vma), new_ctx, anon_vma_name(vma));
3313+}
32383314
32393315 static inline int check_data_rlimit(unsigned long rlim,
32403316 unsigned long new,
@@ -3997,25 +4073,26 @@ static inline void mem_dump_obj(void *object) {} · 756 unmodified lines
39974073 #endif
39984074
39994075 /**
4000− * seal_check_future_write - Check for F_SEAL_FUTURE_WRITE flag and handle it
4076+ * seal_check_write - Check for F_SEAL_WRITE or F_SEAL_FUTURE_WRITE flags and
4077+ * handle them.
40014078 * @seals: the seals to check
40024079 * @vma: the vma to operate on
40034080 *
4004− * Check whether F_SEAL_FUTURE_WRITE is set; if so, do proper check/handling on
4005− * the vma flags. Return 0 if check pass, or <0 for errors.
4081+ * Check whether F_SEAL_WRITE or F_SEAL_FUTURE_WRITE are set; if so, do proper
4082+ * check/handling on the vma flags. Return 0 if check pass, or <0 for errors.
40064083 */
4007−static inline int seal_check_future_write(int seals, struct vm_area_struct *vma)
4084+static inline int seal_check_write(int seals, struct vm_area_struct *vma)
40084085 {
4009− if (seals & F_SEAL_FUTURE_WRITE) {
4086+ if (seals & (F_SEAL_WRITE | F_SEAL_FUTURE_WRITE)) {
40104087 /*
40114088 * New PROT_WRITE and MAP_SHARED mmaps are not allowed when
4012− * "future write" seal active.
4089+ * write seals are active.
40134090 */
40144091 if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE))
40154092 return -EPERM;
40164093
40174094 /*
4018− * Since an F_SEAL_FUTURE_WRITE sealed memfd can be mapped as
4095+ * Since an F_SEAL_[FUTURE_]WRITE sealed memfd can be mapped as
40194096 * MAP_SHARED and read-only, take care to not allow mprotect to
40204097 * revert protections on such mappings. Do this only for shared
40214098 * mappings. For private mappings, don't need to mask
@@ -4059,4 +4136,11 @@ static inline void accept_memory(phys_addr_t start, phys_addr_t end) · 37 unmodified lines
40594136
40604137 #endif
40614138
4139+static inline bool pfn_is_unaccepted_memory(unsigned long pfn)
4140+{
4141+ phys_addr_t paddr = pfn << PAGE_SHIFT;
4142+
4143+ return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE);
4144+}
4145+
40624146 #endif /* _LINUX_MM_H */
include/linux/mm_inline.h+1−19
@@ -4,6 +4,7 @@ · 3 unmodified lines
44
55 #include <linux/atomic.h>
66 #include <linux/huge_mm.h>
7+#include <linux/mm_types.h>
78 #include <linux/swap.h>
89 #include <linux/string.h>
910 #include <linux/userfaultfd_k.h>
@@ -352,15 +353,6 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) · 342 unmodified lines
352353 }
353354
354355 #ifdef CONFIG_ANON_VMA_NAME
355−/*
356− * mmap_lock should be read-locked when calling anon_vma_name(). Caller should
357− * either keep holding the lock while using the returned pointer or it should
358− * raise anon_vma_name refcount before releasing the lock.
359− */
360−extern struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma);
361−extern struct anon_vma_name *anon_vma_name_alloc(const char *name);
362−extern void anon_vma_name_free(struct kref *kref);
363−
364356 /* mmap_lock should be read-locked */
365357 static inline void anon_vma_name_get(struct anon_vma_name *anon_name)
366358 {
@@ -415,16 +407,6 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, · 48 unmodified lines
415407 }
416408
417409 #else /* CONFIG_ANON_VMA_NAME */
418−static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma)
419−{
420− return NULL;
421−}
422−
423−static inline struct anon_vma_name *anon_vma_name_alloc(const char *name)
424−{
425− return NULL;
426−}
427−
428410 static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {}
429411 static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {}
430412 static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma,
include/linux/mm_types.h+45−4
@@ -188,6 +188,10 @@ struct page { · 187 unmodified lines
188188 not kmapped, ie. highmem) */
189189 #endif /* WANT_PAGE_VIRTUAL */
190190
191+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
192+ int _last_cpupid;
193+#endif
194+
191195 #ifdef CONFIG_KMSAN
192196 /*
193197 * KMSAN metadata for this page:
@@ -199,10 +203,6 @@ struct page { · 5 unmodified lines
199203 struct page *kmsan_shadow;
200204 struct page *kmsan_origin;
201205 #endif
202−
203−#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
204− int _last_cpupid;
205−#endif
206206 } _struct_page_alignment;
207207
208208 /*
@@ -261,6 +261,8 @@ typedef struct { · 52 unmodified lines
261261 * @_refcount: Do not access this member directly. Use folio_ref_count()
262262 * to find how many references there are to this folio.
263263 * @memcg_data: Memory Control Group data.
264+ * @virtual: Virtual address in the kernel direct map.
265+ * @_last_cpupid: IDs of last CPU and last process that accessed the folio.
264266 * @_entire_mapcount: Do not use directly, call folio_entire_mapcount().
265267 * @_nr_pages_mapped: Do not use directly, call folio_mapcount().
266268 * @_pincount: Do not use directly, call folio_maybe_dma_pinned().
@@ -306,6 +308,12 @@ struct folio { · 39 unmodified lines
306308 atomic_t _refcount;
307309 #ifdef CONFIG_MEMCG
308310 unsigned long memcg_data;
311+#endif
312+#if defined(WANT_PAGE_VIRTUAL)
313+ void *virtual;
314+#endif
315+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
316+ int _last_cpupid;
309317 #endif
310318 /* private: the union with struct page is transitional */
311319 };
@@ -362,6 +370,12 @@ FOLIO_MATCH(_refcount, _refcount); · 50 unmodified lines
362370 #ifdef CONFIG_MEMCG
363371 FOLIO_MATCH(memcg_data, memcg_data);
364372 #endif
373+#if defined(WANT_PAGE_VIRTUAL)
374+FOLIO_MATCH(virtual, virtual);
375+#endif
376+#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
377+FOLIO_MATCH(_last_cpupid, _last_cpupid);
378+#endif
365379 #undef FOLIO_MATCH
366380 #define FOLIO_MATCH(pg, fl) \
367381 static_assert(offsetof(struct folio, fl) == \
@@ -535,6 +549,27 @@ struct anon_vma_name { · 167 unmodified lines
535549 char name[];
536550 };
537551
552+#ifdef CONFIG_ANON_VMA_NAME
553+/*
554+ * mmap_lock should be read-locked when calling anon_vma_name(). Caller should
555+ * either keep holding the lock while using the returned pointer or it should
556+ * raise anon_vma_name refcount before releasing the lock.
557+ */
558+struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma);
559+struct anon_vma_name *anon_vma_name_alloc(const char *name);
560+void anon_vma_name_free(struct kref *kref);
561+#else /* CONFIG_ANON_VMA_NAME */
562+static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma)
563+{
564+ return NULL;
565+}
566+
567+static inline struct anon_vma_name *anon_vma_name_alloc(const char *name)
568+{
569+ return NULL;
570+}
571+#endif
572+
538573 struct vma_lock {
539574 struct rw_semaphore lock;
540575 };
@@ -678,6 +713,12 @@ struct vm_area_struct { · 137 unmodified lines
678713 struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
679714 } __randomize_layout;
680715
716+#ifdef CONFIG_NUMA
717+#define vma_policy(vma) ((vma)->vm_policy)
718+#else
719+#define vma_policy(vma) NULL
720+#endif
721+
681722 #ifdef CONFIG_SCHED_MM_CID
682723 struct mm_cid {
683724 u64 time;
include/linux/mmu_notifier.h+8−1
@@ -459,7 +459,14 @@ mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) · 458 unmodified lines
459459 lock_map_release(&__mmu_notifier_invalidate_range_start_map);
460460 }
461461
462−static inline int
462+/*
463+ * This version of mmu_notifier_invalidate_range_start() avoids blocking, but it
464+ * can return an error if a notifier can't proceed without blocking, in which
465+ * case you're not allowed to modify PTEs in the specified range.
466+ *
467+ * This is mainly intended for OOM handling.
468+ */
469+static inline int __must_check
463470 mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range)
464471 {
465472 int ret = 0;
include/linux/mmzone.h+24−5
@@ -639,8 +639,6 @@ struct lruvec { · 638 unmodified lines
639639 #endif
640640 };
641641
642−/* Isolate unmapped pages */
643−#define ISOLATE_UNMAPPED ((__force isolate_mode_t)0x2)
644642 /* Isolate for asynchronous migration */
645643 #define ISOLATE_ASYNC_MIGRATE ((__force isolate_mode_t)0x4)
646644 /* Isolate unevictable pages */
@@ -676,15 +674,34 @@ enum zone_watermarks { · 29 unmodified lines
676674 #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
677675 #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)
678676
677+/*
678+ * Flags used in pcp->flags field.
679+ *
680+ * PCPF_PREV_FREE_HIGH_ORDER: a high-order page is freed in the
681+ * previous page freeing. To avoid to drain PCP for an accident
682+ * high-order page freeing.
683+ *
684+ * PCPF_FREE_HIGH_BATCH: preserve "pcp->batch" pages in PCP before
685+ * draining PCP for consecutive high-order pages freeing without
686+ * allocation if data cache slice of CPU is large enough. To reduce
687+ * zone lock contention and keep cache-hot pages reusing.
688+ */
689+#define PCPF_PREV_FREE_HIGH_ORDER BIT(0)
690+#define PCPF_FREE_HIGH_BATCH BIT(1)
691+
679692 struct per_cpu_pages {
680693 spinlock_t lock; /* Protects lists field */
681694 int count; /* number of pages in the list */
682695 int high; /* high watermark, emptying needed */
696+ int high_min; /* min high watermark */
697+ int high_max; /* max high watermark */
683698 int batch; /* chunk size for buddy add/remove */
684− short free_factor; /* batch scaling factor during free */
699+ u8 flags; /* protected by pcp->lock */
700+ u8 alloc_factor; /* batch scaling factor during allocate */
685701 #ifdef CONFIG_NUMA
686− short expire; /* When 0, remote pagesets are drained */
702+ u8 expire; /* When 0, remote pagesets are drained */
687703 #endif
704+ short free_count; /* consecutive free count */
688705
689706 /* Lists of pages, one per migrate type stored on the pcp-lists */
690707 struct list_head lists[NR_PCP_LISTS];
@@ -837,7 +854,8 @@ struct zone { · 146 unmodified lines
837854 * the high and batch values are copied to individual pagesets for
838855 * faster access
839856 */
840− int pageset_high;
857+ int pageset_high_min;
858+ int pageset_high_max;
841859 int pageset_batch;
842860
843861 #ifndef CONFIG_SPARSEMEM
@@ -998,6 +1016,7 @@ enum zone_flags { · 154 unmodified lines
9981016 * Cleared when kswapd is woken.
9991017 */
10001018 ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */
1019+ ZONE_BELOW_HIGH, /* zone is below high watermark. */
10011020 };
10021021
10031022 static inline unsigned long zone_managed_pages(struct zone *zone)
include/linux/page-flags.h+19−0
@@ -692,6 +692,25 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) · 691 unmodified lines
692692
693693 u64 stable_page_flags(struct page *page);
694694
695+/**
696+ * folio_xor_flags_has_waiters - Change some folio flags.
697+ * @folio: The folio.
698+ * @mask: Bits set in this word will be changed.
699+ *
700+ * This must only be used for flags which are changed with the folio
701+ * lock held. For example, it is unsafe to use for PG_dirty as that
702+ * can be set without the folio lock held. It can also only be used
703+ * on flags which are in the range 0-6 as some of the implementations
704+ * only affect those bits.
705+ *
706+ * Return: Whether there are tasks waiting on the folio.
707+ */
708+static inline bool folio_xor_flags_has_waiters(struct folio *folio,
709+ unsigned long mask)
710+{
711+ return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0));
712+}
713+
695714 /**
696715 * folio_test_uptodate - Is this folio up to date?
697716 * @folio: The folio.
include/linux/pagemap.h+3−30
@@ -789,9 +789,6 @@ static inline pgoff_t folio_next_index(struct folio *folio) · 788 unmodified lines
789789 */
790790 static inline struct page *folio_file_page(struct folio *folio, pgoff_t index)
791791 {
792− /* HugeTLBfs indexes the page cache in units of hpage_size */
793− if (folio_test_hugetlb(folio))
794− return &folio->page;
795792 return folio_page(folio, index & (folio_nr_pages(folio) - 1));
796793 }
797794
@@ -807,9 +804,6 @@ static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) · 9 unmodified lines
807804 */
808805 static inline bool folio_contains(struct folio *folio, pgoff_t index)
809806 {
810− /* HugeTLBfs indexes the page cache in units of hpage_size */
811− if (folio_test_hugetlb(folio))
812− return folio->index == index;
813807 return index - folio_index(folio) < folio_nr_pages(folio);
814808 }
815809
@@ -867,10 +861,9 @@ static inline struct folio *read_mapping_folio(struct address_space *mapping, · 51 unmodified lines
867861 }
868862
869863 /*
870− * Get index of the page within radix-tree (but not for hugetlb pages).
871− * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE)
864+ * Get the offset in PAGE_SIZE (even for hugetlb pages).
872865 */
873−static inline pgoff_t page_to_index(struct page *page)
866+static inline pgoff_t page_to_pgoff(struct page *page)
874867 {
875868 struct page *head;
876869
@@ -885,19 +878,6 @@ static inline pgoff_t page_to_index(struct page *page) · 8 unmodified lines
885878 return head->index + page - head;
886879 }
887880
888−extern pgoff_t hugetlb_basepage_index(struct page *page);
889−
890−/*
891− * Get the offset in PAGE_SIZE (even for hugetlb pages).
892− * (TODO: hugetlb pages should have ->index in PAGE_SIZE)
893− */
894−static inline pgoff_t page_to_pgoff(struct page *page)
895−{
896− if (unlikely(PageHuge(page)))
897− return hugetlb_basepage_index(page);
898− return page_to_index(page);
899−}
900−
901881 /*
902882 * Return byte-offset into filesystem object for page.
903883 */
@@ -934,24 +914,16 @@ static inline loff_t folio_file_pos(struct folio *folio) · 30 unmodified lines
934914
935915 /*
936916 * Get the offset in PAGE_SIZE (even for hugetlb folios).
937− * (TODO: hugetlb folios should have ->index in PAGE_SIZE)
938917 */
939918 static inline pgoff_t folio_pgoff(struct folio *folio)
940919 {
941− if (unlikely(folio_test_hugetlb(folio)))
942− return hugetlb_basepage_index(&folio->page);
943920 return folio->index;
944921 }
945922
946−extern pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
947− unsigned long address);
948−
949923 static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
950924 unsigned long address)
951925 {
952926 pgoff_t pgoff;
953− if (unlikely(is_vm_hugetlb_page(vma)))
954− return linear_hugepage_index(vma, address);
955927 pgoff = (address - vma->vm_start) >> PAGE_SHIFT;
956928 pgoff += vma->vm_pgoff;
957929 return pgoff;
@@ -1129,6 +1101,7 @@ static inline void wait_on_page_locked(struct page *page) · 171 unmodified lines
11291101 folio_wait_locked(page_folio(page));
11301102 }
11311103
1104+void folio_end_read(struct folio *folio, bool success);
11321105 void wait_on_page_writeback(struct page *page);
11331106 void folio_wait_writeback(struct folio *folio);
11341107 int folio_wait_writeback_killable(struct folio *folio);
include/linux/percpu_counter.h+30−0
@@ -57,6 +57,8 @@ void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, · 56 unmodified lines
5757 s32 batch);
5858 s64 __percpu_counter_sum(struct percpu_counter *fbc);
5959 int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch);
60+bool __percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit,
61+ s64 amount, s32 batch);
6062 void percpu_counter_sync(struct percpu_counter *fbc);
6163
6264 static inline int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs)
@@ -69,6 +71,13 @@ static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 6 unmodified lines
6971 percpu_counter_add_batch(fbc, amount, percpu_counter_batch);
7072 }
7173
74+static inline bool
75+percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount)
76+{
77+ return __percpu_counter_limited_add(fbc, limit, amount,
78+ percpu_counter_batch);
79+}
80+
7281 /*
7382 * With percpu_counter_add_local() and percpu_counter_sub_local(), counts
7483 * are accumulated in local per cpu counter and not in fbc->count until
@@ -185,6 +194,27 @@ percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 110 unmodified lines
185194 local_irq_restore(flags);
186195 }
187196
197+static inline bool
198+percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount)
199+{
200+ unsigned long flags;
201+ bool good = false;
202+ s64 count;
203+
204+ if (amount == 0)
205+ return true;
206+
207+ local_irq_save(flags);
208+ count = fbc->count + amount;
209+ if ((amount > 0 && count <= limit) ||
210+ (amount < 0 && count >= limit)) {
211+ fbc->count = count;
212+ good = true;
213+ }
214+ local_irq_restore(flags);
215+ return good;
216+}
217+
188218 /* non-SMP percpu_counter_add_local is the same with percpu_counter_add */
189219 static inline void
190220 percpu_counter_add_local(struct percpu_counter *fbc, s64 amount)
include/linux/rmap.h+2−2
@@ -189,7 +189,7 @@ typedef int __bitwise rmap_t; · 188 unmodified lines
189189 /*
190190 * rmap interfaces called when adding or removing pte of page
191191 */
192−void page_move_anon_rmap(struct page *, struct vm_area_struct *);
192+void folio_move_anon_rmap(struct folio *, struct vm_area_struct *);
193193 void page_add_anon_rmap(struct page *, struct vm_area_struct *,
194194 unsigned long address, rmap_t flags);
195195 void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
@@ -203,7 +203,7 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, · 7 unmodified lines
203203 void page_remove_rmap(struct page *, struct vm_area_struct *,
204204 bool compound);
205205
206−void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *,
206+void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *,
207207 unsigned long address, rmap_t flags);
208208 void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
209209 unsigned long address);
include/linux/sched.h+4−0
@@ -1448,6 +1448,10 @@ struct task_struct { · 1447 unmodified lines
14481448 struct mem_cgroup *active_memcg;
14491449 #endif
14501450
1451+#ifdef CONFIG_MEMCG_KMEM
1452+ struct obj_cgroup *objcg;
1453+#endif
1454+
14511455 #ifdef CONFIG_BLK_CGROUP
14521456 struct gendisk *throttle_disk;
14531457 #endif
include/linux/sched/coredump.h+16−3
@@ -71,6 +71,7 @@ static inline int get_dumpable(struct mm_struct *mm) · 70 unmodified lines
7171 #define MMF_UNSTABLE 22 /* mm is unstable for copy_from_user */
7272 #define MMF_HUGE_ZERO_PAGE 23 /* mm has ever used the global huge zero page */
7373 #define MMF_DISABLE_THP 24 /* disable THP for all VMAs */
74+#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)
7475 #define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */
7576 #define MMF_MULTIPROCESS 26 /* mm is shared between processes */
7677 /*
@@ -85,10 +86,22 @@ static inline int get_dumpable(struct mm_struct *mm) · 8 unmodified lines
8586 #define MMF_HAS_MDWE 28
8687 #define MMF_HAS_MDWE_MASK (1 << MMF_HAS_MDWE)
8788
88−#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)
89+
90+#define MMF_HAS_MDWE_NO_INHERIT 29
91+
92+#define MMF_VM_MERGE_ANY 30
93+#define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY)
8994
9095 #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\
91− MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK)
96+ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\
97+ MMF_VM_MERGE_ANY_MASK)
98+
99+static inline unsigned long mmf_init_flags(unsigned long flags)
100+{
101+ if (flags & (1UL << MMF_HAS_MDWE_NO_INHERIT))
102+ flags &= ~((1UL << MMF_HAS_MDWE) |
103+ (1UL << MMF_HAS_MDWE_NO_INHERIT));
104+ return flags & MMF_INIT_MASK;
105+}
92106
93−#define MMF_VM_MERGE_ANY 29
94107 #endif /* _LINUX_SCHED_COREDUMP_H */
include/linux/sched/mm.h+4−0
@@ -403,6 +403,10 @@ DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); · 402 unmodified lines
403403 * __GFP_ACCOUNT allocations till the end of the scope will be charged to the
404404 * given memcg.
405405 *
406+ * Please, make sure that caller has a reference to the passed memcg structure,
407+ * so its lifetime is guaranteed to exceed the scope between two
408+ * set_active_memcg() calls.
409+ *
406410 * NOTE: This function can nest. Users must save the return value and
407411 * reset the previous value after their own charging scope is over.
408412 */
include/linux/sched/numa_balancing.h+3−3
@@ -30,8 +30,8 @@ extern void task_numa_fault(int last_node, int node, int pages, int flags); · 29 unmodified lines
3030 extern pid_t task_numa_group_id(struct task_struct *p);
3131 extern void set_numabalancing_state(bool enabled);
3232 extern void task_numa_free(struct task_struct *p, bool final);
33−extern bool should_numa_migrate_memory(struct task_struct *p, struct page *page,
34− int src_nid, int dst_cpu);
33+bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
34+ int src_nid, int dst_cpu);
3535 #else
3636 static inline void task_numa_fault(int last_node, int node, int pages,
3737 int flags)
@@ -48,7 +48,7 @@ static inline void task_numa_free(struct task_struct *p, bool final) · 10 unmodified lines
4848 {
4949 }
5050 static inline bool should_numa_migrate_memory(struct task_struct *p,
51− struct page *page, int src_nid, int dst_cpu)
51+ struct folio *folio, int src_nid, int dst_cpu)
5252 {
5353 return true;
5454 }
include/linux/shmem_fs.h+10−6
@@ -23,18 +23,22 @@ struct shmem_inode_info { · 22 unmodified lines
2323 unsigned long flags;
2424 unsigned long alloced; /* data pages alloced to file */
2525 unsigned long swapped; /* subtotal assigned to swap */
26− pgoff_t fallocend; /* highest fallocate endindex */
27− struct list_head shrinklist; /* shrinkable hpage inodes */
28− struct list_head swaplist; /* chain of maybes on swap */
26+ union {
27+ struct offset_ctx dir_offsets; /* stable directory offsets */
28+ struct {
29+ struct list_head shrinklist; /* shrinkable hpage inodes */
30+ struct list_head swaplist; /* chain of maybes on swap */
31+ };
32+ };
33+ struct timespec64 i_crtime; /* file creation time */
2934 struct shared_policy policy; /* NUMA memory alloc policy */
3035 struct simple_xattrs xattrs; /* list of xattrs */
36+ pgoff_t fallocend; /* highest fallocate endindex */
37+ unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */
3138 atomic_t stop_eviction; /* hold when working on inode */
32− struct timespec64 i_crtime; /* file creation time */
33− unsigned int fsflags; /* flags for FS_IOC_[SG]ETFLAGS */
3439 #ifdef CONFIG_TMPFS_QUOTA
3540 struct dquot *i_dquot[MAXQUOTAS];
3641 #endif
37− struct offset_ctx dir_offsets; /* stable entry offsets */
3842 struct inode vfs_inode;
3943 };
4044
include/linux/shrinker.h+55−32
@@ -4,6 +4,25 @@ · 3 unmodified lines
44
55 #include <linux/atomic.h>
66 #include <linux/types.h>
7+#include <linux/refcount.h>
8+#include <linux/completion.h>
9+
10+#define SHRINKER_UNIT_BITS BITS_PER_LONG
11+
12+/*
13+ * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
14+ * shrinkers, which have elements charged to the memcg.
15+ */
16+struct shrinker_info_unit {
17+ atomic_long_t nr_deferred[SHRINKER_UNIT_BITS];
18+ DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
19+};
20+
21+struct shrinker_info {
22+ struct rcu_head rcu;
23+ int map_nr_max;
24+ struct shrinker_info_unit *unit[];
25+};
726
827 /*
928 * This struct is used to pass information from page reclaim to the shrinkers.
@@ -70,6 +89,19 @@ struct shrinker { · 60 unmodified lines
7089 int seeks; /* seeks to recreate an obj */
7190 unsigned flags;
7291
92+ /*
93+ * The reference count of this shrinker. Registered shrinker have an
94+ * initial refcount of 1, then the lookup operations are now allowed
95+ * to use it via shrinker_try_get(). Later in the unregistration step,
96+ * the initial refcount will be discarded, and will free the shrinker
97+ * asynchronously via RCU after its refcount reaches 0.
98+ */
99+ refcount_t refcount;
100+ struct completion done; /* use to wait for refcount to reach 0 */
101+ struct rcu_head rcu;
102+
103+ void *private_data;
104+
73105 /* These are for internal use */
74106 struct list_head list;
75107 #ifdef CONFIG_MEMCG
@@ -86,48 +118,39 @@ struct shrinker { · 10 unmodified lines
86118 };
87119 #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */
88120
89−/* Flags */
90−#define SHRINKER_REGISTERED (1 << 0)
91−#define SHRINKER_NUMA_AWARE (1 << 1)
92−#define SHRINKER_MEMCG_AWARE (1 << 2)
121+/* Internal flags */
122+#define SHRINKER_REGISTERED BIT(0)
123+#define SHRINKER_ALLOCATED BIT(1)
124+
125+/* Flags for users to use */
126+#define SHRINKER_NUMA_AWARE BIT(2)
127+#define SHRINKER_MEMCG_AWARE BIT(3)
93128 /*
94129 * It just makes sense when the shrinker is also MEMCG_AWARE for now,
95130 * non-MEMCG_AWARE shrinker should not have this flag set.
96131 */
97−#define SHRINKER_NONSLAB (1 << 3)
132+#define SHRINKER_NONSLAB BIT(4)
98133
99−extern int __printf(2, 3) prealloc_shrinker(struct shrinker *shrinker,
100− const char *fmt, ...);
101−extern void register_shrinker_prepared(struct shrinker *shrinker);
102−extern int __printf(2, 3) register_shrinker(struct shrinker *shrinker,
103− const char *fmt, ...);
104−extern void unregister_shrinker(struct shrinker *shrinker);
105−extern void free_prealloced_shrinker(struct shrinker *shrinker);
106−extern void synchronize_shrinkers(void);
134+__printf(2, 3)
135+struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...);
136+void shrinker_register(struct shrinker *shrinker);
137+void shrinker_free(struct shrinker *shrinker);
107138
108−#ifdef CONFIG_SHRINKER_DEBUG
109−extern int shrinker_debugfs_add(struct shrinker *shrinker);
110−extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
111− int *debugfs_id);
112−extern void shrinker_debugfs_remove(struct dentry *debugfs_entry,
113− int debugfs_id);
114−extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker,
115− const char *fmt, ...);
116−#else /* CONFIG_SHRINKER_DEBUG */
117−static inline int shrinker_debugfs_add(struct shrinker *shrinker)
118−{
119− return 0;
120−}
121−static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
122− int *debugfs_id)
139+static inline bool shrinker_try_get(struct shrinker *shrinker)
123140 {
124− *debugfs_id = -1;
125− return NULL;
141+ return refcount_inc_not_zero(&shrinker->refcount);
126142 }
127−static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,
128− int debugfs_id)
143+
144+static inline void shrinker_put(struct shrinker *shrinker)
129145 {
146+ if (refcount_dec_and_test(&shrinker->refcount))
147+ complete(&shrinker->done);
130148 }
149+
150+#ifdef CONFIG_SHRINKER_DEBUG
151+extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker,
152+ const char *fmt, ...);
153+#else /* CONFIG_SHRINKER_DEBUG */
131154 static inline __printf(2, 3)
132155 int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...)
133156 {
include/linux/userfaultfd_k.h+27−1
@@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 160 unmodified lines
161161 }
162162
163163 static inline bool vma_can_userfault(struct vm_area_struct *vma,
164− unsigned long vm_flags)
164+ unsigned long vm_flags,
165+ bool wp_async)
165166 {
167+ vm_flags &= __VM_UFFD_FLAGS;
168+
166169 if ((vm_flags & VM_UFFD_MINOR) &&
167170 (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
168171 return false;
172+
173+ /*
174+ * If wp async enabled, and WP is the only mode enabled, allow any
175+ * memory type.
176+ */
177+ if (wp_async && (vm_flags == VM_UFFD_WP))
178+ return true;
179+
169180 #ifndef CONFIG_PTE_MARKER_UFFD_WP
170181 /*
171182 * If user requested uffd-wp but not enabled pte markers for
@@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, · 3 unmodified lines
175186 if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))
176187 return false;
177188 #endif
189+
190+ /* By default, allow any of anon|shmem|hugetlb */
178191 return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
179192 vma_is_shmem(vma);
180193 }
@@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, · 16 unmodified lines
197210 extern void userfaultfd_unmap_complete(struct mm_struct *mm,
198211 struct list_head *uf);
199212 extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
213+extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
200214
201215 #else /* CONFIG_USERFAULTFD */
202216
@@ -207,6 +221,13 @@ static inline vm_fault_t handle_userfault(struct vm_fault *vmf, · 4 unmodified lines
207221 return VM_FAULT_SIGBUS;
208222 }
209223
224+static inline long uffd_wp_range(struct vm_area_struct *vma,
225+ unsigned long start, unsigned long len,
226+ bool enable_wp)
227+{
228+ return false;
229+}
230+
210231 static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
211232 struct vm_userfaultfd_ctx vm_ctx)
212233 {
@@ -297,6 +318,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) · 84 unmodified lines
297318 return false;
298319 }
299320
321+static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
322+{
323+ return false;
324+}
325+
300326 #endif /* CONFIG_USERFAULTFD */
301327
302328 static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
include/linux/wait.h+3−6
@@ -19,10 +19,9 @@ int default_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int · 18 unmodified lines
1919 /* wait_queue_entry::flags */
2020 #define WQ_FLAG_EXCLUSIVE 0x01
2121 #define WQ_FLAG_WOKEN 0x02
22−#define WQ_FLAG_BOOKMARK 0x04
23−#define WQ_FLAG_CUSTOM 0x08
24−#define WQ_FLAG_DONE 0x10
25−#define WQ_FLAG_PRIORITY 0x20
22+#define WQ_FLAG_CUSTOM 0x04
23+#define WQ_FLAG_DONE 0x08
24+#define WQ_FLAG_PRIORITY 0x10
2625
2726 /*
2827 * A single wait-queue entry structure:
@@ -212,8 +211,6 @@ __remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq · 183 unmodified lines
212211 int __wake_up(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key);
213212 void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode, void *key);
214213 void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
215−void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head,
216− unsigned int mode, void *key, wait_queue_entry_t *bookmark);
217214 void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
218215 void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
219216 void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr);
include/trace/events/damon.h+42−3
@@ -9,12 +9,51 @@ · 8 unmodified lines
99 #include <linux/types.h>
1010 #include <linux/tracepoint.h>
1111
12+TRACE_EVENT_CONDITION(damos_before_apply,
13+
14+ TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,
15+ unsigned int target_idx, struct damon_region *r,
16+ unsigned int nr_regions, bool do_trace),
17+
18+ TP_ARGS(context_idx, target_idx, scheme_idx, r, nr_regions, do_trace),
19+
20+ TP_CONDITION(do_trace),
21+
22+ TP_STRUCT__entry(
23+ __field(unsigned int, context_idx)
24+ __field(unsigned int, scheme_idx)
25+ __field(unsigned long, target_idx)
26+ __field(unsigned long, start)
27+ __field(unsigned long, end)
28+ __field(unsigned int, nr_accesses)
29+ __field(unsigned int, age)
30+ __field(unsigned int, nr_regions)
31+ ),
32+
33+ TP_fast_assign(
34+ __entry->context_idx = context_idx;
35+ __entry->scheme_idx = scheme_idx;
36+ __entry->target_idx = target_idx;
37+ __entry->start = r->ar.start;
38+ __entry->end = r->ar.end;
39+ __entry->nr_accesses = r->nr_accesses_bp / 10000;
40+ __entry->age = r->age;
41+ __entry->nr_regions = nr_regions;
42+ ),
43+
44+ TP_printk("ctx_idx=%u scheme_idx=%u target_idx=%lu nr_regions=%u %lu-%lu: %u %u",
45+ __entry->context_idx, __entry->scheme_idx,
46+ __entry->target_idx, __entry->nr_regions,
47+ __entry->start, __entry->end,
48+ __entry->nr_accesses, __entry->age)
49+);
50+
1251 TRACE_EVENT(damon_aggregated,
1352
14− TP_PROTO(struct damon_target *t, unsigned int target_id,
15− struct damon_region *r, unsigned int nr_regions),
53+ TP_PROTO(unsigned int target_id, struct damon_region *r,
54+ unsigned int nr_regions),
1655
17− TP_ARGS(t, target_id, r, nr_regions),
56+ TP_ARGS(target_id, r, nr_regions),
1857
1958 TP_STRUCT__entry(
2059 __field(unsigned long, target_id)
include/trace/events/migrate.h+14−10
@@ -49,10 +49,11 @@ TRACE_EVENT(mm_migrate_pages, · 48 unmodified lines
4949
5050 TP_PROTO(unsigned long succeeded, unsigned long failed,
5151 unsigned long thp_succeeded, unsigned long thp_failed,
52− unsigned long thp_split, enum migrate_mode mode, int reason),
52+ unsigned long thp_split, unsigned long large_folio_split,
53+ enum migrate_mode mode, int reason),
5354
5455 TP_ARGS(succeeded, failed, thp_succeeded, thp_failed,
55− thp_split, mode, reason),
56+ thp_split, large_folio_split, mode, reason),
5657
5758 TP_STRUCT__entry(
5859 __field( unsigned long, succeeded)
@@ -60,26 +61,29 @@ TRACE_EVENT(mm_migrate_pages, · 1 unmodified line
6061 __field( unsigned long, thp_succeeded)
6162 __field( unsigned long, thp_failed)
6263 __field( unsigned long, thp_split)
64+ __field( unsigned long, large_folio_split)
6365 __field( enum migrate_mode, mode)
6466 __field( int, reason)
6567 ),
6668
6769 TP_fast_assign(
68− __entry->succeeded = succeeded;
69− __entry->failed = failed;
70− __entry->thp_succeeded = thp_succeeded;
71− __entry->thp_failed = thp_failed;
72− __entry->thp_split = thp_split;
73− __entry->mode = mode;
74− __entry->reason = reason;
70+ __entry->succeeded = succeeded;
71+ __entry->failed = failed;
72+ __entry->thp_succeeded = thp_succeeded;
73+ __entry->thp_failed = thp_failed;
74+ __entry->thp_split = thp_split;
75+ __entry->large_folio_split = large_folio_split;
76+ __entry->mode = mode;
77+ __entry->reason = reason;
7578 ),
7679
77− TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu mode=%s reason=%s",
80+ TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu nr_split=%lu mode=%s reason=%s",
7881 __entry->succeeded,
7982 __entry->failed,
8083 __entry->thp_succeeded,
8184 __entry->thp_failed,
8285 __entry->thp_split,
86+ __entry->large_folio_split,
8387 __print_symbolic(__entry->mode, MIGRATE_MODE),
8488 __print_symbolic(__entry->reason, MIGRATE_REASON))
8589 );
include/trace/events/vmscan.h+2−6
@@ -285,10 +285,9 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 284 unmodified lines
285285 unsigned long nr_scanned,
286286 unsigned long nr_skipped,
287287 unsigned long nr_taken,
288− isolate_mode_t isolate_mode,
289288 int lru),
290289
291− TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, isolate_mode, lru),
290+ TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, lru),
292291
293292 TP_STRUCT__entry(
294293 __field(int, highest_zoneidx)
@@ -297,7 +296,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 2 unmodified lines
297296 __field(unsigned long, nr_scanned)
298297 __field(unsigned long, nr_skipped)
299298 __field(unsigned long, nr_taken)
300− __field(unsigned int, isolate_mode)
301299 __field(int, lru)
302300 ),
303301
@@ -308,7 +306,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 4 unmodified lines
308306 __entry->nr_scanned = nr_scanned;
309307 __entry->nr_skipped = nr_skipped;
310308 __entry->nr_taken = nr_taken;
311− __entry->isolate_mode = (__force unsigned int)isolate_mode;
312309 __entry->lru = lru;
313310 ),
314311
@@ -316,8 +313,7 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 1 unmodified line
316313 * classzone is previous name of the highest_zoneidx.
317314 * Reason not to change it is the ABI requirement of the tracepoint.
318315 */
319− TP_printk("isolate_mode=%d classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s",
320− __entry->isolate_mode,
316+ TP_printk("classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s",
321317 __entry->highest_zoneidx,
322318 __entry->order,
323319 __entry->nr_requested,
include/uapi/linux/fs.h+59−0
@@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; · 304 unmodified lines
305305 #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
306306 RWF_APPEND)
307307
308+/* Pagemap ioctl */
309+#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
310+
311+/* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */
312+#define PAGE_IS_WPALLOWED (1 << 0)
313+#define PAGE_IS_WRITTEN (1 << 1)
314+#define PAGE_IS_FILE (1 << 2)
315+#define PAGE_IS_PRESENT (1 << 3)
316+#define PAGE_IS_SWAPPED (1 << 4)
317+#define PAGE_IS_PFNZERO (1 << 5)
318+#define PAGE_IS_HUGE (1 << 6)
319+
320+/*
321+ * struct page_region - Page region with flags
322+ * @start: Start of the region
323+ * @end: End of the region (exclusive)
324+ * @categories: PAGE_IS_* category bitmask for the region
325+ */
326+struct page_region {
327+ __u64 start;
328+ __u64 end;
329+ __u64 categories;
330+};
331+
332+/* Flags for PAGEMAP_SCAN ioctl */
333+#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */
334+#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */
335+
336+/*
337+ * struct pm_scan_arg - Pagemap ioctl argument
338+ * @size: Size of the structure
339+ * @flags: Flags for the IOCTL
340+ * @start: Starting address of the region
341+ * @end: Ending address of the region
342+ * @walk_end Address where the scan stopped (written by kernel).
343+ * walk_end == end (address tags cleared) informs that the scan completed on entire range.
344+ * @vec: Address of page_region struct array for output
345+ * @vec_len: Length of the page_region struct array
346+ * @max_pages: Optional limit for number of returned pages (0 = disabled)
347+ * @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1
348+ * @category_mask: Skip pages for which any category doesn't match
349+ * @category_anyof_mask: Skip pages for which no category matches
350+ * @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned
351+ */
352+struct pm_scan_arg {
353+ __u64 size;
354+ __u64 flags;
355+ __u64 start;
356+ __u64 end;
357+ __u64 walk_end;
358+ __u64 vec;
359+ __u64 vec_len;
360+ __u64 max_pages;
361+ __u64 category_inverted;
362+ __u64 category_mask;
363+ __u64 category_anyof_mask;
364+ __u64 return_mask;
365+};
366+
308367 #endif /* _UAPI_LINUX_FS_H */
include/uapi/linux/mempolicy.h+1−1
@@ -48,7 +48,7 @@ enum { · 47 unmodified lines
4848 #define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform
4949 to policy */
5050 #define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */
51−#define MPOL_MF_LAZY (1<<3) /* Modifies '_MOVE: lazy migrate on fault */
51+#define MPOL_MF_LAZY (1<<3) /* UNSUPPORTED FLAG: Lazy migrate on fault */
5252 #define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */
5353
5454 #define MPOL_MF_VALID (MPOL_MF_STRICT | \
include/uapi/linux/prctl.h+2−1
@@ -283,7 +283,8 @@ struct prctl_mm_map { · 282 unmodified lines
283283
284284 /* Memory deny write / execute */
285285 #define PR_SET_MDWE 65
286−# define PR_MDWE_REFUSE_EXEC_GAIN 1
286+# define PR_MDWE_REFUSE_EXEC_GAIN (1UL << 0)
287+# define PR_MDWE_NO_INHERIT (1UL << 1)
287288
288289 #define PR_GET_MDWE 66
289290
include/uapi/linux/userfaultfd.h+8−1
@@ -40,7 +40,8 @@ · 39 unmodified lines
4040 UFFD_FEATURE_EXACT_ADDRESS | \
4141 UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
4242 UFFD_FEATURE_WP_UNPOPULATED | \
43− UFFD_FEATURE_POISON)
43+ UFFD_FEATURE_POISON | \
44+ UFFD_FEATURE_WP_ASYNC)
4445 #define UFFD_API_IOCTLS \
4546 ((__u64)1 << _UFFDIO_REGISTER | \
4647 (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -216,6 +217,11 @@ struct uffdio_api { · 169 unmodified lines
216217 * (i.e. empty ptes). This will be the default behavior for shmem
217218 * & hugetlbfs, so this flag only affects anonymous memory behavior
218219 * when userfault write-protection mode is registered.
220+ *
221+ * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
222+ * asynchronous mode is supported in which the write fault is
223+ * automatically resolved and write-protection is un-set.
224+ * It implies UFFD_FEATURE_WP_UNPOPULATED.
219225 */
220226 #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
221227 #define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -232,6 +238,7 @@ struct uffdio_api { · 10 unmodified lines
232238 #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
233239 #define UFFD_FEATURE_WP_UNPOPULATED (1<<13)
234240 #define UFFD_FEATURE_POISON (1<<14)
241+#define UFFD_FEATURE_WP_ASYNC (1<<15)
235242 __u64 features;
236243
237244 __u64 ioctls;
ipc/shm.c+8−13
@@ -562,30 +562,25 @@ static unsigned long shm_pagesize(struct vm_area_struct *vma) · 561 unmodified lines
562562 }
563563
564564 #ifdef CONFIG_NUMA
565−static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *new)
565+static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol)
566566 {
567− struct file *file = vma->vm_file;
568− struct shm_file_data *sfd = shm_file_data(file);
567+ struct shm_file_data *sfd = shm_file_data(vma->vm_file);
569568 int err = 0;
570569
571570 if (sfd->vm_ops->set_policy)
572− err = sfd->vm_ops->set_policy(vma, new);
571+ err = sfd->vm_ops->set_policy(vma, mpol);
573572 return err;
574573 }
575574
576575 static struct mempolicy *shm_get_policy(struct vm_area_struct *vma,
577− unsigned long addr)
576+ unsigned long addr, pgoff_t *ilx)
578577 {
579− struct file *file = vma->vm_file;
580− struct shm_file_data *sfd = shm_file_data(file);
581− struct mempolicy *pol = NULL;
578+ struct shm_file_data *sfd = shm_file_data(vma->vm_file);
579+ struct mempolicy *mpol = vma->vm_policy;
582580
583581 if (sfd->vm_ops->get_policy)
584− pol = sfd->vm_ops->get_policy(vma, addr);
585− else if (vma->vm_policy)
586− pol = vma->vm_policy;
587−
588− return pol;
582+ mpol = sfd->vm_ops->get_policy(vma, addr, ilx);
583+ return mpol;
589584 }
590585 #endif
591586
kernel/cgroup/cgroup.c+14−1
@@ -1908,6 +1908,7 @@ enum cgroup2_param { · 1907 unmodified lines
19081908 Opt_favordynmods,
19091909 Opt_memory_localevents,
19101910 Opt_memory_recursiveprot,
1911+ Opt_memory_hugetlb_accounting,
19111912 nr__cgroup2_params
19121913 };
19131914
@@ -1916,6 +1917,7 @@ static const struct fs_parameter_spec cgroup2_fs_parameters[] = { · 2 unmodified lines
19161917 fsparam_flag("favordynmods", Opt_favordynmods),
19171918 fsparam_flag("memory_localevents", Opt_memory_localevents),
19181919 fsparam_flag("memory_recursiveprot", Opt_memory_recursiveprot),
1920+ fsparam_flag("memory_hugetlb_accounting", Opt_memory_hugetlb_accounting),
19191921 {}
19201922 };
19211923
@@ -1942,6 +1944,9 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *param · 20 unmodified lines
19421944 case Opt_memory_recursiveprot:
19431945 ctx->flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
19441946 return 0;
1947+ case Opt_memory_hugetlb_accounting:
1948+ ctx->flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
1949+ return 0;
19451950 }
19461951 return -EINVAL;
19471952 }
@@ -1966,6 +1971,11 @@ static void apply_cgroup_root_flags(unsigned int root_flags) · 18 unmodified lines
19661971 cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
19671972 else
19681973 cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_RECURSIVE_PROT;
1974+
1975+ if (root_flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING)
1976+ cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
1977+ else
1978+ cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
19691979 }
19701980 }
19711981
@@ -1979,6 +1989,8 @@ static int cgroup_show_options(struct seq_file *seq, struct kernfs_root *kf_root · 7 unmodified lines
19791989 seq_puts(seq, ",memory_localevents");
19801990 if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT)
19811991 seq_puts(seq, ",memory_recursiveprot");
1992+ if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING)
1993+ seq_puts(seq, ",memory_hugetlb_accounting");
19821994 return 0;
19831995 }
19841996
@@ -7068,7 +7080,8 @@ static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, · 5083 unmodified lines
70687080 "nsdelegate\n"
70697081 "favordynmods\n"
70707082 "memory_localevents\n"
7071− "memory_recursiveprot\n");
7083+ "memory_recursiveprot\n"
7084+ "memory_hugetlb_accounting\n");
70727085 }
70737086 static struct kobj_attribute cgroup_features_attr = __ATTR_RO(features);
70747087
kernel/events/uprobes.c+2−2
@@ -474,8 +474,8 @@ retry: · 473 unmodified lines
474474 gup_flags |= FOLL_SPLIT_PMD;
475475 /* Read the page with vaddr into memory */
476476 old_page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);
477− if (IS_ERR_OR_NULL(old_page))
478− return old_page ? PTR_ERR(old_page) : 0;
477+ if (IS_ERR(old_page))
478+ return PTR_ERR(old_page);
479479
480480 ret = verify_opcode(old_page, vaddr, &opcode);
481481 if (ret <= 0)
kernel/exit.c+0−4
@@ -541,7 +541,6 @@ static void exit_mm(void) · 540 unmodified lines
541541 exit_mm_release(current, mm);
542542 if (!mm)
543543 return;
544− sync_mm_rss(mm);
545544 mmap_read_lock(mm);
546545 mmgrab_lazy_tlb(mm);
547546 BUG_ON(mm != current->active_mm);
@@ -831,9 +830,6 @@ void __noreturn do_exit(long code) · 283 unmodified lines
831830 io_uring_files_cancel();
832831 exit_signals(tsk); /* sets PF_EXITING */
833832
834− /* sync mm's RSS info before statistics gathering */
835− if (tsk->mm)
836− sync_mm_rss(tsk->mm);
837833 acct_update_integrals(tsk);
838834 group_dead = atomic_dec_and_test(&tsk->signal->live);
839835 if (group_dead) {
kernel/fork.c+2−6
@@ -733,7 +733,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 732 unmodified lines
733733
734734 get_file(file);
735735 i_mmap_lock_write(mapping);
736− if (tmp->vm_flags & VM_SHARED)
736+ if (vma_is_shared_maywrite(tmp))
737737 mapping_allow_writable(mapping);
738738 flush_dcache_mmap_lock(mapping);
739739 /* insert tmp into the share list, just after mpnt */
@@ -1288,7 +1288,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, · 548 unmodified lines
12881288 hugetlb_count_init(mm);
12891289
12901290 if (current->mm) {
1291− mm->flags = current->mm->flags & MMF_INIT_MASK;
1291+ mm->flags = mmf_init_flags(current->mm->flags);
12921292 mm->def_flags = current->mm->def_flags & VM_INIT_DEF_MASK;
12931293 } else {
12941294 mm->flags = default_dump_filter;
@@ -2411,10 +2411,6 @@ __latent_entropy struct task_struct *copy_process( · 1116 unmodified lines
24112411 p->io_uring = NULL;
24122412 #endif
24132413
2414−#if defined(SPLIT_RSS_COUNTING)
2415− memset(&p->rss_stat, 0, sizeof(p->rss_stat));
2416−#endif
2417−
24182414 p->default_timer_slack_ns = current->timer_slack_ns;
24192415
24202416 #ifdef CONFIG_PSI
kernel/kcsan/kcsan_test.c+3−6
@@ -699,12 +699,9 @@ static void test_barrier_nothreads(struct kunit *test) · 698 unmodified lines
699699 KCSAN_EXPECT_RW_BARRIER(spin_unlock(&test_spinlock), true);
700700 KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false);
701701 KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true);
702−
703−#ifdef clear_bit_unlock_is_negative_byte
704− KCSAN_EXPECT_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);
705− KCSAN_EXPECT_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);
706− KCSAN_EXPECT_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);
707−#endif
702+ KCSAN_EXPECT_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
703+ KCSAN_EXPECT_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
704+ KCSAN_EXPECT_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
708705 kcsan_nestable_atomic_end();
709706 }
710707
kernel/kcsan/selftest.c+3−6
@@ -227,12 +227,9 @@ static bool __init test_barrier(void) · 226 unmodified lines
227227 KCSAN_CHECK_RW_BARRIER(arch_spin_unlock(&arch_spinlock));
228228 spin_lock(&test_spinlock);
229229 KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock));
230−
231−#ifdef clear_bit_unlock_is_negative_byte
232− KCSAN_CHECK_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));
233− KCSAN_CHECK_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));
234− KCSAN_CHECK_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));
235−#endif
230+ KCSAN_CHECK_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
231+ KCSAN_CHECK_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
232+ KCSAN_CHECK_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
236233 kcsan_nestable_atomic_end();
237234
238235 return ret;
kernel/kthread.c+0−1
@@ -1469,7 +1469,6 @@ void kthread_unuse_mm(struct mm_struct *mm) · 1468 unmodified lines
14691469 * clearing tsk->mm.
14701470 */
14711471 smp_mb__after_spinlock();
1472− sync_mm_rss(mm);
14731472 local_irq_disable();
14741473 tsk->mm = NULL;
14751474 membarrier_update_current_mm(NULL);
kernel/ptrace.c+1−1
@@ -59,7 +59,7 @@ int ptrace_access_vm(struct task_struct *tsk, unsigned long addr, · 58 unmodified lines
5959 return 0;
6060 }
6161
62− ret = __access_remote_vm(mm, addr, buf, len, gup_flags);
62+ ret = access_remote_vm(mm, addr, buf, len, gup_flags);
6363 mmput(mm);
6464
6565 return ret;
kernel/rcu/tree.c+12−9
@@ -3471,13 +3471,6 @@ kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 3470 unmodified lines
34713471 return freed == 0 ? SHRINK_STOP : freed;
34723472 }
34733473
3474−static struct shrinker kfree_rcu_shrinker = {
3475− .count_objects = kfree_rcu_shrink_count,
3476− .scan_objects = kfree_rcu_shrink_scan,
3477− .batch = 0,
3478− .seeks = DEFAULT_SEEKS,
3479−};
3480−
34813474 void __init kfree_rcu_scheduler_running(void)
34823475 {
34833476 int cpu;
@@ -5029,6 +5022,7 @@ static void __init kfree_rcu_batch_init(void) · 1545 unmodified lines
50295022 {
50305023 int cpu;
50315024 int i, j;
5025+ struct shrinker *kfree_rcu_shrinker;
50325026
50335027 /* Clamp it to [0:100] seconds interval. */
50345028 if (rcu_delay_page_cache_fill_msec < 0 ||
@@ -5060,8 +5054,17 @@ static void __init kfree_rcu_batch_init(void) · 25 unmodified lines
50605054 INIT_DELAYED_WORK(&krcp->page_cache_work, fill_page_cache_func);
50615055 krcp->initialized = true;
50625056 }
5063− if (register_shrinker(&kfree_rcu_shrinker, "rcu-kfree"))
5064− pr_err("Failed to register kfree_rcu() shrinker!\n");
5057+
5058+ kfree_rcu_shrinker = shrinker_alloc(0, "rcu-kfree");
5059+ if (!kfree_rcu_shrinker) {
5060+ pr_err("Failed to allocate kfree_rcu() shrinker!\n");
5061+ return;
5062+ }
5063+
5064+ kfree_rcu_shrinker->count_objects = kfree_rcu_shrink_count;
5065+ kfree_rcu_shrinker->scan_objects = kfree_rcu_shrink_scan;
5066+
5067+ shrinker_register(kfree_rcu_shrinker);
50655068 }
50665069
50675070 void __init rcu_init(void)
kernel/rcu/tree_nocb.h+10−9
@@ -1396,13 +1396,6 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1395 unmodified lines
13961396
13971397 return count ? count : SHRINK_STOP;
13981398 }
1399−
1400−static struct shrinker lazy_rcu_shrinker = {
1401− .count_objects = lazy_rcu_shrink_count,
1402− .scan_objects = lazy_rcu_shrink_scan,
1403− .batch = 0,
1404− .seeks = DEFAULT_SEEKS,
1405−};
14061399 #endif // #ifdef CONFIG_RCU_LAZY
14071400
14081401 void __init rcu_init_nohz(void)
@@ -1410,6 +1403,7 @@ void __init rcu_init_nohz(void) · 1 unmodified line
14101403 int cpu;
14111404 struct rcu_data *rdp;
14121405 const struct cpumask *cpumask = NULL;
1406+ struct shrinker * __maybe_unused lazy_rcu_shrinker;
14131407
14141408 #if defined(CONFIG_NO_HZ_FULL)
14151409 if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))
@@ -1436,8 +1430,15 @@ void __init rcu_init_nohz(void) · 20 unmodified lines
14361430 return;
14371431
14381432 #ifdef CONFIG_RCU_LAZY
1439− if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy"))
1440− pr_err("Failed to register lazy_rcu shrinker!\n");
1433+ lazy_rcu_shrinker = shrinker_alloc(0, "rcu-lazy");
1434+ if (!lazy_rcu_shrinker) {
1435+ pr_err("Failed to allocate lazy_rcu shrinker!\n");
1436+ } else {
1437+ lazy_rcu_shrinker->count_objects = lazy_rcu_shrink_count;
1438+ lazy_rcu_shrinker->scan_objects = lazy_rcu_shrink_scan;
1439+
1440+ shrinker_register(lazy_rcu_shrinker);
1441+ }
14411442 #endif // #ifdef CONFIG_RCU_LAZY
14421443
14431444 if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {
kernel/sched/fair.c+6−6
@@ -1759,12 +1759,12 @@ static bool pgdat_free_space_enough(struct pglist_data *pgdat) · 1758 unmodified lines
17591759 * The smaller the hint page fault latency, the higher the possibility
17601760 * for the page to be hot.
17611761 */
1762−static int numa_hint_fault_latency(struct page *page)
1762+static int numa_hint_fault_latency(struct folio *folio)
17631763 {
17641764 int last_time, time;
17651765
17661766 time = jiffies_to_msecs(jiffies);
1767− last_time = xchg_page_access_time(page, time);
1767+ last_time = folio_xchg_access_time(folio, time);
17681768
17691769 return (time - last_time) & PAGE_ACCESS_TIME_MASK;
17701770 }
@@ -1821,7 +1821,7 @@ static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, · 50 unmodified lines
18211821 }
18221822 }
18231823
1824−bool should_numa_migrate_memory(struct task_struct *p, struct page * page,
1824+bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
18251825 int src_nid, int dst_cpu)
18261826 {
18271827 struct numa_group *ng = deref_curr_numa_group(p);
@@ -1851,16 +1851,16 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, · 23 unmodified lines
18511851 numa_promotion_adjust_threshold(pgdat, rate_limit, def_th);
18521852
18531853 th = pgdat->nbp_threshold ? : def_th;
1854− latency = numa_hint_fault_latency(page);
1854+ latency = numa_hint_fault_latency(folio);
18551855 if (latency >= th)
18561856 return false;
18571857
18581858 return !numa_promotion_rate_limit(pgdat, rate_limit,
1859− thp_nr_pages(page));
1859+ folio_nr_pages(folio));
18601860 }
18611861
18621862 this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
1863− last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
1863+ last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid);
18641864
18651865 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
18661866 !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))
kernel/sched/wait.c+10−50
@@ -57,13 +57,6 @@ void remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry · 56 unmodified lines
5757 }
5858 EXPORT_SYMBOL(remove_wait_queue);
5959
60−/*
61− * Scan threshold to break wait queue walk.
62− * This allows a waker to take a break from holding the
63− * wait queue lock during the wait queue walk.
64− */
65−#define WAITQUEUE_WALK_BREAK_CNT 64
66−
6760 /*
6861 * The core wakeup function. Non-exclusive wakeups (nr_exclusive == 0) just
6962 * wake everything up. If it's an exclusive wakeup (nr_exclusive == small +ve
@@ -78,21 +71,13 @@ EXPORT_SYMBOL(remove_wait_queue); · 8 unmodified lines
7871 * zero in this (rare) case, and we handle it by continuing to scan the queue.
7972 */
8073 static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode,
81− int nr_exclusive, int wake_flags, void *key,
82− wait_queue_entry_t *bookmark)
74+ int nr_exclusive, int wake_flags, void *key)
8375 {
8476 wait_queue_entry_t *curr, *next;
85− int cnt = 0;
8677
8778 lockdep_assert_held(&wq_head->lock);
8879
89− if (bookmark && (bookmark->flags & WQ_FLAG_BOOKMARK)) {
90− curr = list_next_entry(bookmark, entry);
91−
92− list_del(&bookmark->entry);
93− bookmark->flags = 0;
94− } else
95− curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry);
80+ curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry);
9681
9782 if (&curr->entry == &wq_head->head)
9883 return nr_exclusive;
@@ -101,21 +86,11 @@ static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode, · 2 unmodified lines
10186 unsigned flags = curr->flags;
10287 int ret;
10388
104− if (flags & WQ_FLAG_BOOKMARK)
105− continue;
106−
10789 ret = curr->func(curr, mode, wake_flags, key);
10890 if (ret < 0)
10991 break;
11092 if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
11193 break;
112−
113− if (bookmark && (++cnt > WAITQUEUE_WALK_BREAK_CNT) &&
114− (&next->entry != &wq_head->head)) {
115− bookmark->flags = WQ_FLAG_BOOKMARK;
116− list_add_tail(&bookmark->entry, &next->entry);
117− break;
118− }
11994 }
12095
12196 return nr_exclusive;
@@ -125,20 +100,12 @@ static int __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int m · 3 unmodified lines
125100 int nr_exclusive, int wake_flags, void *key)
126101 {
127102 unsigned long flags;
128− wait_queue_entry_t bookmark;
129− int remaining = nr_exclusive;
103+ int remaining;
130104
131− bookmark.flags = 0;
132− bookmark.private = NULL;
133− bookmark.func = NULL;
134− INIT_LIST_HEAD(&bookmark.entry);
135−
136− do {
137− spin_lock_irqsave(&wq_head->lock, flags);
138− remaining = __wake_up_common(wq_head, mode, remaining,
139− wake_flags, key, &bookmark);
140− spin_unlock_irqrestore(&wq_head->lock, flags);
141− } while (bookmark.flags & WQ_FLAG_BOOKMARK);
105+ spin_lock_irqsave(&wq_head->lock, flags);
106+ remaining = __wake_up_common(wq_head, mode, nr_exclusive, wake_flags,
107+ key);
108+ spin_unlock_irqrestore(&wq_head->lock, flags);
142109
143110 return nr_exclusive - remaining;
144111 }
@@ -171,23 +138,16 @@ void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode · 26 unmodified lines
171138 */
172139 void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr)
173140 {
174− __wake_up_common(wq_head, mode, nr, 0, NULL, NULL);
141+ __wake_up_common(wq_head, mode, nr, 0, NULL);
175142 }
176143 EXPORT_SYMBOL_GPL(__wake_up_locked);
177144
178145 void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key)
179146 {
180− __wake_up_common(wq_head, mode, 1, 0, key, NULL);
147+ __wake_up_common(wq_head, mode, 1, 0, key);
181148 }
182149 EXPORT_SYMBOL_GPL(__wake_up_locked_key);
183150
184−void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head,
185− unsigned int mode, void *key, wait_queue_entry_t *bookmark)
186−{
187− __wake_up_common(wq_head, mode, 1, 0, key, bookmark);
188−}
189−EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark);
190−
191151 /**
192152 * __wake_up_sync_key - wake up threads blocked on a waitqueue.
193153 * @wq_head: the waitqueue
@@ -233,7 +193,7 @@ EXPORT_SYMBOL_GPL(__wake_up_sync_key); · 39 unmodified lines
233193 void __wake_up_locked_sync_key(struct wait_queue_head *wq_head,
234194 unsigned int mode, void *key)
235195 {
236− __wake_up_common(wq_head, mode, 1, WF_SYNC, key, NULL);
196+ __wake_up_common(wq_head, mode, 1, WF_SYNC, key);
237197 }
238198 EXPORT_SYMBOL_GPL(__wake_up_locked_sync_key);
239199
kernel/sys.c+26−6
@@ -2368,19 +2368,41 @@ static int prctl_set_vma(unsigned long opt, unsigned long start, · 2367 unmodified lines
23682368 }
23692369 #endif /* CONFIG_ANON_VMA_NAME */
23702370
2371+static inline unsigned long get_current_mdwe(void)
2372+{
2373+ unsigned long ret = 0;
2374+
2375+ if (test_bit(MMF_HAS_MDWE, &current->mm->flags))
2376+ ret |= PR_MDWE_REFUSE_EXEC_GAIN;
2377+ if (test_bit(MMF_HAS_MDWE_NO_INHERIT, &current->mm->flags))
2378+ ret |= PR_MDWE_NO_INHERIT;
2379+
2380+ return ret;
2381+}
2382+
23712383 static inline int prctl_set_mdwe(unsigned long bits, unsigned long arg3,
23722384 unsigned long arg4, unsigned long arg5)
23732385 {
2386+ unsigned long current_bits;
2387+
23742388 if (arg3 || arg4 || arg5)
23752389 return -EINVAL;
23762390
2377− if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN))
2391+ if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN | PR_MDWE_NO_INHERIT))
2392+ return -EINVAL;
2393+
2394+ /* NO_INHERIT only makes sense with REFUSE_EXEC_GAIN */
2395+ if (bits & PR_MDWE_NO_INHERIT && !(bits & PR_MDWE_REFUSE_EXEC_GAIN))
23782396 return -EINVAL;
23792397
2398+ current_bits = get_current_mdwe();
2399+ if (current_bits && current_bits != bits)
2400+ return -EPERM; /* Cannot unset the flags */
2401+
2402+ if (bits & PR_MDWE_NO_INHERIT)
2403+ set_bit(MMF_HAS_MDWE_NO_INHERIT, &current->mm->flags);
23802404 if (bits & PR_MDWE_REFUSE_EXEC_GAIN)
23812405 set_bit(MMF_HAS_MDWE, &current->mm->flags);
2382− else if (test_bit(MMF_HAS_MDWE, &current->mm->flags))
2383− return -EPERM; /* Cannot unset the flag */
23842406
23852407 return 0;
23862408 }
@@ -2390,9 +2412,7 @@ static inline int prctl_get_mdwe(unsigned long arg2, unsigned long arg3, · 3 unmodified lines
23902412 {
23912413 if (arg2 || arg3 || arg4 || arg5)
23922414 return -EINVAL;
2393−
2394− return test_bit(MMF_HAS_MDWE, &current->mm->flags) ?
2395− PR_MDWE_REFUSE_EXEC_GAIN : 0;
2415+ return get_current_mdwe();
23962416 }
23972417
23982418 static int prctl_get_auxv(void __user *addr, unsigned long len)
lib/percpu_counter.c+79−0
@@ -278,6 +278,85 @@ int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch) · 277 unmodified lines
278278 }
279279 EXPORT_SYMBOL(__percpu_counter_compare);
280280
281+/*
282+ * Compare counter, and add amount if total is: less than or equal to limit if
283+ * amount is positive, or greater than or equal to limit if amount is negative.
284+ * Return true if amount is added, or false if total would be beyond the limit.
285+ *
286+ * Negative limit is allowed, but unusual.
287+ * When negative amounts (subs) are given to percpu_counter_limited_add(),
288+ * the limit would most naturally be 0 - but other limits are also allowed.
289+ *
290+ * Overflow beyond S64_MAX is not allowed for: counter, limit and amount
291+ * are all assumed to be sane (far from S64_MIN and S64_MAX).
292+ */
293+bool __percpu_counter_limited_add(struct percpu_counter *fbc,
294+ s64 limit, s64 amount, s32 batch)
295+{
296+ s64 count;
297+ s64 unknown;
298+ unsigned long flags;
299+ bool good = false;
300+
301+ if (amount == 0)
302+ return true;
303+
304+ local_irq_save(flags);
305+ unknown = batch * num_online_cpus();
306+ count = __this_cpu_read(*fbc->counters);
307+
308+ /* Skip taking the lock when safe */
309+ if (abs(count + amount) <= batch &&
310+ ((amount > 0 && fbc->count + unknown <= limit) ||
311+ (amount < 0 && fbc->count - unknown >= limit))) {
312+ this_cpu_add(*fbc->counters, amount);
313+ local_irq_restore(flags);
314+ return true;
315+ }
316+
317+ raw_spin_lock(&fbc->lock);
318+ count = fbc->count + amount;
319+
320+ /* Skip percpu_counter_sum() when safe */
321+ if (amount > 0) {
322+ if (count - unknown > limit)
323+ goto out;
324+ if (count + unknown <= limit)
325+ good = true;
326+ } else {
327+ if (count + unknown < limit)
328+ goto out;
329+ if (count - unknown >= limit)
330+ good = true;
331+ }
332+
333+ if (!good) {
334+ s32 *pcount;
335+ int cpu;
336+
337+ for_each_cpu_or(cpu, cpu_online_mask, cpu_dying_mask) {
338+ pcount = per_cpu_ptr(fbc->counters, cpu);
339+ count += *pcount;
340+ }
341+ if (amount > 0) {
342+ if (count > limit)
343+ goto out;
344+ } else {
345+ if (count < limit)
346+ goto out;
347+ }
348+ good = true;
349+ }
350+
351+ count = __this_cpu_read(*fbc->counters);
352+ fbc->count += count + amount;
353+ __this_cpu_sub(*fbc->counters, count);
354+out:
355+ raw_spin_unlock(&fbc->lock);
356+ local_irq_restore(flags);
357+ return good;
358+}
359+
281360 static int __init percpu_counter_startup(void)
282361 {
283362 int ret;
mm/Kconfig+12−0
@@ -130,6 +130,7 @@ config ZSWAP_COMPRESSOR_DEFAULT · 129 unmodified lines
130130 choice
131131 prompt "Default allocator"
132132 depends on ZSWAP
133+ default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if MMU
133134 default ZSWAP_ZPOOL_DEFAULT_ZBUD
134135 help
135136 Selects the default allocator for the compressed cache for
@@ -704,6 +705,17 @@ config HUGETLB_PAGE_SIZE_VARIABLE · 568 unmodified lines
704705 config CONTIG_ALLOC
705706 def_bool (MEMORY_ISOLATION && COMPACTION) || CMA
706707
708+config PCP_BATCH_SCALE_MAX
709+ int "Maximum scale factor of PCP (Per-CPU pageset) batch allocate/free"
710+ default 5
711+ range 0 6
712+ help
713+ In page allocator, PCP (Per-CPU pageset) is refilled and drained in
714+ batches. The batch number is scaled automatically to improve page
715+ allocation/free throughput. But too large scale factor may hurt
716+ latency. This option sets the upper limit of scale factor to limit
717+ the maximum latency.
718+
707719 config PHYS_ADDR_T_64BIT
708720 def_bool 64BIT
709721
mm/Makefile+2−2
@@ -48,8 +48,8 @@ endif · 47 unmodified lines
4848
4949 obj-y := filemap.o mempool.o oom_kill.o fadvise.o \
5050 maccess.o page-writeback.o folio-compat.o \
51− readahead.o swap.o truncate.o vmscan.o shmem.o \
52− util.o mmzone.o vmstat.o backing-dev.o \
51+ readahead.o swap.o truncate.o vmscan.o shrinker.o \
52+ shmem.o util.o mmzone.o vmstat.o backing-dev.o \
5353 mm_init.o percpu.o slab_common.o \
5454 compaction.o show_mem.o shmem_quota.o\
5555 interval_tree.o list_lru.o workingset.o \
mm/bootmem_info.c+1−1
@@ -34,7 +34,7 @@ void put_page_bootmem(struct page *page) · 33 unmodified lines
3434 ClearPagePrivate(page);
3535 set_page_private(page, 0);
3636 INIT_LIST_HEAD(&page->lru);
37− kmemleak_free_part(page_to_virt(page), PAGE_SIZE);
37+ kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
3838 free_reserved_page(page);
3939 }
4040 }
mm/cma.c+1−1
@@ -505,7 +505,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, · 504 unmodified lines
505505 */
506506 if (page) {
507507 for (i = 0; i < count; i++)
508− page_kasan_tag_reset(page + i);
508+ page_kasan_tag_reset(nth_page(page, i));
509509 }
510510
511511 if (ret && !no_warn) {
mm/compaction.c+52−39
@@ -382,6 +382,7 @@ static void __reset_isolation_suitable(struct zone *zone) · 381 unmodified lines
382382 bool source_set = false;
383383 bool free_set = false;
384384
385+ /* Only flush if a full compaction finished recently */
385386 if (!zone->compact_blockskip_flush)
386387 return;
387388
@@ -434,9 +435,7 @@ void reset_isolation_suitable(pg_data_t *pgdat) · 46 unmodified lines
434435 if (!populated_zone(zone))
435436 continue;
436437
437− /* Only flush if a full compaction finished recently */
438− if (zone->compact_blockskip_flush)
439− __reset_isolation_suitable(zone);
438+ __reset_isolation_suitable(zone);
440439 }
441440 }
442441
@@ -626,11 +625,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, · 183 unmodified lines
626625 if (PageCompound(page)) {
627626 const unsigned int order = compound_order(page);
628627
629− if (likely(order <= MAX_ORDER)) {
628+ if (blockpfn + (1UL << order) <= end_pfn) {
630629 blockpfn += (1UL << order) - 1;
631630 page += (1UL << order) - 1;
632631 nr_scanned += (1UL << order) - 1;
633632 }
633+
634634 goto isolate_fail;
635635 }
636636
@@ -678,8 +678,7 @@ isolate_fail: · 41 unmodified lines
678678 spin_unlock_irqrestore(&cc->zone->lock, flags);
679679
680680 /*
681− * There is a tiny chance that we have read bogus compound_order(),
682− * so be careful to not go outside of the pageblock.
681+ * Be careful to not go outside of the pageblock.
683682 */
684683 if (unlikely(blockpfn > end_pfn))
685684 blockpfn = end_pfn;
@@ -1395,8 +1394,8 @@ move_freelist_head(struct list_head *freelist, struct page *freepage) · 709 unmodified lines
13951394 {
13961395 LIST_HEAD(sublist);
13971396
1398− if (!list_is_last(freelist, &freepage->lru)) {
1399− list_cut_before(&sublist, freelist, &freepage->lru);
1397+ if (!list_is_first(&freepage->buddy_list, freelist)) {
1398+ list_cut_before(&sublist, freelist, &freepage->buddy_list);
14001399 list_splice_tail(&sublist, freelist);
14011400 }
14021401 }
@@ -1412,8 +1411,8 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) · 9 unmodified lines
14121411 {
14131412 LIST_HEAD(sublist);
14141413
1415− if (!list_is_first(freelist, &freepage->lru)) {
1416− list_cut_position(&sublist, freelist, &freepage->lru);
1414+ if (!list_is_last(&freepage->buddy_list, freelist)) {
1415+ list_cut_position(&sublist, freelist, &freepage->buddy_list);
14171416 list_splice_tail(&sublist, freelist);
14181417 }
14191418 }
@@ -2066,8 +2065,10 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) · 646 unmodified lines
20662065 }
20672066
20682067 /*
2069− * order == -1 is expected when compacting via
2070− * /proc/sys/vm/compact_memory
2068+ * order == -1 is expected when compacting proactively via
2069+ * 1. /proc/sys/vm/compact_memory
2070+ * 2. /sys/devices/system/node/nodex/compact
2071+ * 3. /proc/sys/vm/compaction_proactiveness
20712072 */
20722073 static inline bool is_via_compact_memory(int order)
20732074 {
@@ -2377,6 +2378,30 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, · 303 unmodified lines
23772378 return false;
23782379 }
23792380
2381+/*
2382+ * Should we do compaction for target allocation order.
2383+ * Return COMPACT_SUCCESS if allocation for target order can be already
2384+ * satisfied
2385+ * Return COMPACT_SKIPPED if compaction for target order is likely to fail
2386+ * Return COMPACT_CONTINUE if compaction for target order should be ran
2387+ */
2388+static enum compact_result
2389+compaction_suit_allocation_order(struct zone *zone, unsigned int order,
2390+ int highest_zoneidx, unsigned int alloc_flags)
2391+{
2392+ unsigned long watermark;
2393+
2394+ watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
2395+ if (zone_watermark_ok(zone, order, watermark, highest_zoneidx,
2396+ alloc_flags))
2397+ return COMPACT_SUCCESS;
2398+
2399+ if (!compaction_suitable(zone, order, highest_zoneidx))
2400+ return COMPACT_SKIPPED;
2401+
2402+ return COMPACT_CONTINUE;
2403+}
2404+
23802405 static enum compact_result
23812406 compact_zone(struct compact_control *cc, struct capture_control *capc)
23822407 {
@@ -2402,19 +2427,11 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) · 19 unmodified lines
24022427 cc->migratetype = gfp_migratetype(cc->gfp_mask);
24032428
24042429 if (!is_via_compact_memory(cc->order)) {
2405− unsigned long watermark;
2406−
2407− /* Allocation can already succeed, nothing to do */
2408− watermark = wmark_pages(cc->zone,
2409− cc->alloc_flags & ALLOC_WMARK_MASK);
2410− if (zone_watermark_ok(cc->zone, cc->order, watermark,
2411− cc->highest_zoneidx, cc->alloc_flags))
2412− return COMPACT_SUCCESS;
2413−
2414− /* Compaction is likely to fail */
2415− if (!compaction_suitable(cc->zone, cc->order,
2416− cc->highest_zoneidx))
2417− return COMPACT_SKIPPED;
2430+ ret = compaction_suit_allocation_order(cc->zone, cc->order,
2431+ cc->highest_zoneidx,
2432+ cc->alloc_flags);
2433+ if (ret != COMPACT_CONTINUE)
2434+ return ret;
24182435 }
24192436
24202437 /*
@@ -2913,6 +2930,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 492 unmodified lines
29132930 int zoneid;
29142931 struct zone *zone;
29152932 enum zone_type highest_zoneidx = pgdat->kcompactd_highest_zoneidx;
2933+ enum compact_result ret;
29162934
29172935 for (zoneid = 0; zoneid <= highest_zoneidx; zoneid++) {
29182936 zone = &pgdat->node_zones[zoneid];
@@ -2920,14 +2938,10 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 1 unmodified line
29202938 if (!populated_zone(zone))
29212939 continue;
29222940
2923− /* Allocation can already succeed, check other zones */
2924− if (zone_watermark_ok(zone, pgdat->kcompactd_max_order,
2925− min_wmark_pages(zone),
2926− highest_zoneidx, 0))
2927− continue;
2928−
2929− if (compaction_suitable(zone, pgdat->kcompactd_max_order,
2930− highest_zoneidx))
2941+ ret = compaction_suit_allocation_order(zone,
2942+ pgdat->kcompactd_max_order,
2943+ highest_zoneidx, ALLOC_WMARK_MIN);
2944+ if (ret == COMPACT_CONTINUE)
29312945 return true;
29322946 }
29332947
@@ -2950,6 +2964,8 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 16 unmodified lines
29502964 .ignore_skip_hint = false,
29512965 .gfp_mask = GFP_KERNEL,
29522966 };
2967+ enum compact_result ret;
2968+
29532969 trace_mm_compaction_kcompactd_wake(pgdat->node_id, cc.order,
29542970 cc.highest_zoneidx);
29552971 count_compact_event(KCOMPACTD_WAKE);
@@ -2964,12 +2980,9 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 8 unmodified lines
29642980 if (compaction_deferred(zone, cc.order))
29652981 continue;
29662982
2967− /* Allocation can already succeed, nothing to do */
2968− if (zone_watermark_ok(zone, cc.order,
2969− min_wmark_pages(zone), zoneid, 0))
2970− continue;
2971−
2972− if (!compaction_suitable(zone, cc.order, zoneid))
2983+ ret = compaction_suit_allocation_order(zone,
2984+ cc.order, zoneid, ALLOC_WMARK_MIN);
2985+ if (ret != COMPACT_CONTINUE)
29732986 continue;
29742987
29752988 if (kthread_should_stop())
mm/damon/Kconfig+12−0
@@ -59,6 +59,18 @@ config DAMON_SYSFS · 58 unmodified lines
5959 This builds the sysfs interface for DAMON. The user space can use
6060 the interface for arbitrary data access monitoring.
6161
62+config DAMON_SYSFS_KUNIT_TEST
63+ bool "Test for damon debugfs interface" if !KUNIT_ALL_TESTS
64+ depends on DAMON_SYSFS && KUNIT=y
65+ default KUNIT_ALL_TESTS
66+ help
67+ This builds the DAMON sysfs interface Kunit test suite.
68+
69+ For more information on KUnit and unit tests in general, please refer
70+ to the KUnit documentation.
71+
72+ If unsure, say N.
73+
6274 config DAMON_DBGFS
6375 bool "DAMON debugfs interface (DEPRECATED!)"
6476 depends on DAMON_VADDR && DAMON_PADDR && DEBUG_FS
mm/damon/core-test.h+28−1
@@ -30,7 +30,7 @@ static void damon_test_regions(struct kunit *test) · 29 unmodified lines
3030 damon_add_region(r, t);
3131 KUNIT_EXPECT_EQ(test, 1u, damon_nr_regions(t));
3232
33− damon_del_region(r, t);
33+ damon_destroy_region(r, t);
3434 KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t));
3535
3636 damon_free_target(t);
@@ -94,6 +94,7 @@ static void damon_test_aggregate(struct kunit *test) · 57 unmodified lines
9494 for (ir = 0; ir < 3; ir++) {
9595 r = damon_new_region(saddr[it][ir], eaddr[it][ir]);
9696 r->nr_accesses = accesses[it][ir];
97+ r->nr_accesses_bp = accesses[it][ir] * 10000;
9798 damon_add_region(r, t);
9899 }
99100 it++;
@@ -147,9 +148,11 @@ static void damon_test_merge_two(struct kunit *test) · 47 unmodified lines
147148 t = damon_new_target();
148149 r = damon_new_region(0, 100);
149150 r->nr_accesses = 10;
151+ r->nr_accesses_bp = 100000;
150152 damon_add_region(r, t);
151153 r2 = damon_new_region(100, 300);
152154 r2->nr_accesses = 20;
155+ r2->nr_accesses_bp = 200000;
153156 damon_add_region(r2, t);
154157
155158 damon_merge_two_regions(t, r, r2);
@@ -196,6 +199,7 @@ static void damon_test_merge_regions_of(struct kunit *test) · 40 unmodified lines
196199 for (i = 0; i < ARRAY_SIZE(sa); i++) {
197200 r = damon_new_region(sa[i], ea[i]);
198201 r->nr_accesses = nrs[i];
202+ r->nr_accesses_bp = nrs[i] * 10000;
199203 damon_add_region(r, t);
200204 }
201205
@@ -265,6 +269,8 @@ static void damon_test_ops_registration(struct kunit *test) · 63 unmodified lines
265269
266270 /* Check double-registration failure again */
267271 KUNIT_EXPECT_EQ(test, damon_register_ops(&ops), -EINVAL);
272+
273+ damon_destroy_ctx(c);
268274 }
269275
270276 static void damon_test_set_regions(struct kunit *test)
@@ -297,6 +303,7 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 26 unmodified lines
297303 struct damon_region *r = damon_new_region(3, 7);
298304
299305 r->nr_accesses = 15;
306+ r->nr_accesses_bp = 150000;
300307 r->age = 20;
301308
302309 new_attrs = (struct damon_attrs){
@@ -316,6 +323,8 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 13 unmodified lines
316323 damon_update_monitoring_result(r, &old_attrs, &new_attrs);
317324 KUNIT_EXPECT_EQ(test, r->nr_accesses, 150);
318325 KUNIT_EXPECT_EQ(test, r->age, 20);
326+
327+ damon_free_region(r);
319328 }
320329
321330 static void damon_test_set_attrs(struct kunit *test)
@@ -339,6 +348,23 @@ static void damon_test_set_attrs(struct kunit *test) · 17 unmodified lines
339348 invalid_attrs = valid_attrs;
340349 invalid_attrs.aggr_interval = 4999;
341350 KUNIT_EXPECT_EQ(test, damon_set_attrs(c, &invalid_attrs), -EINVAL);
351+
352+ damon_destroy_ctx(c);
353+}
354+
355+static void damon_test_moving_sum(struct kunit *test)
356+{
357+ unsigned int mvsum = 50000, nomvsum = 50000, len_window = 10;
358+ unsigned int new_values[] = {10000, 0, 10000, 0, 0, 0, 10000, 0, 0, 0};
359+ unsigned int expects[] = {55000, 50000, 55000, 50000, 45000, 40000,
360+ 45000, 40000, 35000, 30000};
361+ int i;
362+
363+ for (i = 0; i < ARRAY_SIZE(new_values); i++) {
364+ mvsum = damon_moving_sum(mvsum, nomvsum, len_window,
365+ new_values[i]);
366+ KUNIT_EXPECT_EQ(test, mvsum, expects[i]);
367+ }
342368 }
343369
344370 static void damos_test_new_filter(struct kunit *test)
@@ -425,6 +451,7 @@ static struct kunit_case damon_test_cases[] = { · 80 unmodified lines
425451 KUNIT_CASE(damon_test_set_regions),
426452 KUNIT_CASE(damon_test_update_monitoring_result),
427453 KUNIT_CASE(damon_test_set_attrs),
454+ KUNIT_CASE(damon_test_moving_sum),
428455 KUNIT_CASE(damos_test_new_filter),
429456 KUNIT_CASE(damos_test_filter_out),
430457 {},
mm/damon/core.c+226−69
@@ -128,6 +128,7 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 127 unmodified lines
128128 region->ar.start = start;
129129 region->ar.end = end;
130130 region->nr_accesses = 0;
131+ region->nr_accesses_bp = 0;
131132 INIT_LIST_HEAD(&region->list);
132133
133134 region->age = 0;
@@ -312,7 +313,9 @@ static struct damos_quota *damos_quota_init_priv(struct damos_quota *quota) · 178 unmodified lines
312313 }
313314
314315 struct damos *damon_new_scheme(struct damos_access_pattern *pattern,
315− enum damos_action action, struct damos_quota *quota,
316+ enum damos_action action,
317+ unsigned long apply_interval_us,
318+ struct damos_quota *quota,
316319 struct damos_watermarks *wmarks)
317320 {
318321 struct damos *scheme;
@@ -322,6 +325,13 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 3 unmodified lines
322325 return NULL;
323326 scheme->pattern = *pattern;
324327 scheme->action = action;
328+ scheme->apply_interval_us = apply_interval_us;
329+ /*
330+ * next_apply_sis will be set when kdamond starts. While kdamond is
331+ * running, it will also updated when it is added to the DAMON context,
332+ * or damon_attrs are updated.
333+ */
334+ scheme->next_apply_sis = 0;
325335 INIT_LIST_HEAD(&scheme->filters);
326336 scheme->stat = (struct damos_stat){};
327337 INIT_LIST_HEAD(&scheme->list);
@@ -334,9 +344,21 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 6 unmodified lines
334344 return scheme;
335345 }
336346
347+static void damos_set_next_apply_sis(struct damos *s, struct damon_ctx *ctx)
348+{
349+ unsigned long sample_interval = ctx->attrs.sample_interval ?
350+ ctx->attrs.sample_interval : 1;
351+ unsigned long apply_interval = s->apply_interval_us ?
352+ s->apply_interval_us : ctx->attrs.aggr_interval;
353+
354+ s->next_apply_sis = ctx->passed_sample_intervals +
355+ apply_interval / sample_interval;
356+}
357+
337358 void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)
338359 {
339360 list_add_tail(&s->list, &ctx->schemes);
361+ damos_set_next_apply_sis(s, ctx);
340362 }
341363
342364 static void damon_del_scheme(struct damos *s)
@@ -427,8 +449,10 @@ struct damon_ctx *damon_new_ctx(void) · 84 unmodified lines
427449 ctx->attrs.aggr_interval = 100 * 1000;
428450 ctx->attrs.ops_update_interval = 60 * 1000 * 1000;
429451
430− ktime_get_coarse_ts64(&ctx->last_aggregation);
431− ctx->last_ops_update = ctx->last_aggregation;
452+ ctx->passed_sample_intervals = 0;
453+ /* These will be set from kdamond_init_intervals_sis() */
454+ ctx->next_aggregation_sis = 0;
455+ ctx->next_ops_update_sis = 0;
432456
433457 mutex_init(&ctx->kdamond_lock);
434458
@@ -476,20 +500,14 @@ static unsigned int damon_age_for_new_attrs(unsigned int age, · 41 unmodified lines
476500 static unsigned int damon_accesses_bp_to_nr_accesses(
477501 unsigned int accesses_bp, struct damon_attrs *attrs)
478502 {
479− unsigned int max_nr_accesses =
480− attrs->aggr_interval / attrs->sample_interval;
481−
482− return accesses_bp * max_nr_accesses / 10000;
503+ return accesses_bp * damon_max_nr_accesses(attrs) / 10000;
483504 }
484505
485506 /* convert nr_accesses to access ratio in bp (per 10,000) */
486507 static unsigned int damon_nr_accesses_to_accesses_bp(
487508 unsigned int nr_accesses, struct damon_attrs *attrs)
488509 {
489− unsigned int max_nr_accesses =
490− attrs->aggr_interval / attrs->sample_interval;
491−
492− return nr_accesses * 10000 / max_nr_accesses;
510+ return nr_accesses * 10000 / damon_max_nr_accesses(attrs);
493511 }
494512
495513 static unsigned int damon_nr_accesses_for_new_attrs(unsigned int nr_accesses,
@@ -506,6 +524,7 @@ static void damon_update_monitoring_result(struct damon_region *r, · 10 unmodified lines
506524 {
507525 r->nr_accesses = damon_nr_accesses_for_new_attrs(r->nr_accesses,
508526 old_attrs, new_attrs);
527+ r->nr_accesses_bp = r->nr_accesses * 10000;
509528 r->age = damon_age_for_new_attrs(r->age, old_attrs, new_attrs);
510529 }
511530
@@ -541,13 +560,21 @@ static void damon_update_monitoring_results(struct damon_ctx *ctx, · 29 unmodified lines
541560 * @ctx: monitoring context
542561 * @attrs: monitoring attributes
543562 *
544− * This function should not be called while the kdamond is running.
563+ * This function should be called while the kdamond is not running, or an
564+ * access check results aggregation is not ongoing (e.g., from
565+ * &struct damon_callback->after_aggregation or
566+ * &struct damon_callback->after_wmarks_check callbacks).
567+ *
545568 * Every time interval is in micro-seconds.
546569 *
547570 * Return: 0 on success, negative error code otherwise.
548571 */
549572 int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs)
550573 {
574+ unsigned long sample_interval = attrs->sample_interval ?
575+ attrs->sample_interval : 1;
576+ struct damos *s;
577+
551578 if (attrs->min_nr_regions < 3)
552579 return -EINVAL;
553580 if (attrs->min_nr_regions > attrs->max_nr_regions)
@@ -555,8 +582,17 @@ int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) · 1 unmodified line
555582 if (attrs->sample_interval > attrs->aggr_interval)
556583 return -EINVAL;
557584
585+ ctx->next_aggregation_sis = ctx->passed_sample_intervals +
586+ attrs->aggr_interval / sample_interval;
587+ ctx->next_ops_update_sis = ctx->passed_sample_intervals +
588+ attrs->ops_update_interval / sample_interval;
589+
558590 damon_update_monitoring_results(ctx, attrs);
559591 ctx->attrs = *attrs;
592+
593+ damon_for_each_scheme(s, ctx)
594+ damos_set_next_apply_sis(s, ctx);
595+
560596 return 0;
561597 }
562598
@@ -728,38 +764,6 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs) · 165 unmodified lines
728764 return err;
729765 }
730766
731−/*
732− * damon_check_reset_time_interval() - Check if a time interval is elapsed.
733− * @baseline: the time to check whether the interval has elapsed since
734− * @interval: the time interval (microseconds)
735− *
736− * See whether the given time interval has passed since the given baseline
737− * time. If so, it also updates the baseline to current time for next check.
738− *
739− * Return: true if the time interval has passed, or false otherwise.
740− */
741−static bool damon_check_reset_time_interval(struct timespec64 *baseline,
742− unsigned long interval)
743−{
744− struct timespec64 now;
745−
746− ktime_get_coarse_ts64(&now);
747− if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) <
748− interval * 1000)
749− return false;
750− *baseline = now;
751− return true;
752−}
753−
754−/*
755− * Check whether it is time to flush the aggregated information
756− */
757−static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
758−{
759− return damon_check_reset_time_interval(&ctx->last_aggregation,
760− ctx->attrs.aggr_interval);
761−}
762−
763767 /*
764768 * Reset the aggregated monitoring results ('nr_accesses' of each region).
765769 */
@@ -772,7 +776,7 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) · 6 unmodified lines
772776 struct damon_region *r;
773777
774778 damon_for_each_region(r, t) {
775− trace_damon_aggregated(t, ti, r, damon_nr_regions(t));
779+ trace_damon_aggregated(ti, r, damon_nr_regions(t));
776780 r->last_nr_accesses = r->nr_accesses;
777781 r->nr_accesses = 0;
778782 }
@@ -786,12 +790,13 @@ static void damon_split_region_at(struct damon_target *t, · 7 unmodified lines
786790 static bool __damos_valid_target(struct damon_region *r, struct damos *s)
787791 {
788792 unsigned long sz;
793+ unsigned int nr_accesses = r->nr_accesses_bp / 10000;
789794
790795 sz = damon_sz_region(r);
791796 return s->pattern.min_sz_region <= sz &&
792797 sz <= s->pattern.max_sz_region &&
793− s->pattern.min_nr_accesses <= r->nr_accesses &&
794− r->nr_accesses <= s->pattern.max_nr_accesses &&
798+ s->pattern.min_nr_accesses <= nr_accesses &&
799+ nr_accesses <= s->pattern.max_nr_accesses &&
795800 s->pattern.min_age_region <= r->age &&
796801 r->age <= s->pattern.max_age_region;
797802 }
@@ -946,6 +951,33 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 148 unmodified lines
946951 struct timespec64 begin, end;
947952 unsigned long sz_applied = 0;
948953 int err = 0;
954+ /*
955+ * We plan to support multiple context per kdamond, as DAMON sysfs
956+ * implies with 'nr_contexts' file. Nevertheless, only single context
957+ * per kdamond is supported for now. So, we can simply use '0' context
958+ * index here.
959+ */
960+ unsigned int cidx = 0;
961+ struct damos *siter; /* schemes iterator */
962+ unsigned int sidx = 0;
963+ struct damon_target *titer; /* targets iterator */
964+ unsigned int tidx = 0;
965+ bool do_trace = false;
966+
967+ /* get indices for trace_damos_before_apply() */
968+ if (trace_damos_before_apply_enabled()) {
969+ damon_for_each_scheme(siter, c) {
970+ if (siter == s)
971+ break;
972+ sidx++;
973+ }
974+ damon_for_each_target(titer, c) {
975+ if (titer == t)
976+ break;
977+ tidx++;
978+ }
979+ do_trace = true;
980+ }
949981
950982 if (c->ops.apply_scheme) {
951983 if (quota->esz && quota->charged_sz + sz > quota->esz) {
@@ -960,8 +992,11 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 8 unmodified lines
960992 ktime_get_coarse_ts64(&begin);
961993 if (c->callback.before_damos_apply)
962994 err = c->callback.before_damos_apply(c, t, r, s);
963− if (!err)
995+ if (!err) {
996+ trace_damos_before_apply(cidx, sidx, tidx, r,
997+ damon_nr_regions(t), do_trace);
964998 sz_applied = c->ops.apply_scheme(c, t, r, s);
999+ }
9651000 ktime_get_coarse_ts64(&end);
9661001 quota->total_charged_ns += timespec64_to_ns(&end) -
9671002 timespec64_to_ns(&begin);
@@ -1079,14 +1114,29 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 111 unmodified lines
10791114 struct damon_target *t;
10801115 struct damon_region *r, *next_r;
10811116 struct damos *s;
1117+ unsigned long sample_interval = c->attrs.sample_interval ?
1118+ c->attrs.sample_interval : 1;
1119+ bool has_schemes_to_apply = false;
10821120
10831121 damon_for_each_scheme(s, c) {
1122+ if (c->passed_sample_intervals != s->next_apply_sis)
1123+ continue;
1124+
1125+ s->next_apply_sis +=
1126+ (s->apply_interval_us ? s->apply_interval_us :
1127+ c->attrs.aggr_interval) / sample_interval;
1128+
10841129 if (!s->wmarks.activated)
10851130 continue;
10861131
1132+ has_schemes_to_apply = true;
1133+
10871134 damos_adjust_quota(c, s);
10881135 }
10891136
1137+ if (!has_schemes_to_apply)
1138+ return;
1139+
10901140 damon_for_each_target(t, c) {
10911141 damon_for_each_region_safe(r, next_r, t)
10921142 damon_do_apply_schemes(c, t, r);
@@ -1103,6 +1153,7 @@ static void damon_merge_two_regions(struct damon_target *t, · 10 unmodified lines
11031153
11041154 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
11051155 (sz_l + sz_r);
1156+ l->nr_accesses_bp = l->nr_accesses * 10000;
11061157 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
11071158 l->ar.end = r->ar.end;
11081159 damon_destroy_region(r, t);
@@ -1174,6 +1225,7 @@ static void damon_split_region_at(struct damon_target *t, · 65 unmodified lines
11741225
11751226 new->age = r->age;
11761227 new->last_nr_accesses = r->last_nr_accesses;
1228+ new->nr_accesses_bp = r->nr_accesses_bp;
11771229
11781230 damon_insert_region(new, r, damon_next_region(r), t);
11791231 }
@@ -1240,18 +1292,6 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 60 unmodified lines
12401292 last_nr_regions = nr_regions;
12411293 }
12421294
1243−/*
1244− * Check whether it is time to check and apply the operations-related data
1245− * structures.
1246− *
1247− * Returns true if it is.
1248− */
1249−static bool kdamond_need_update_operations(struct damon_ctx *ctx)
1250−{
1251− return damon_check_reset_time_interval(&ctx->last_ops_update,
1252− ctx->attrs.ops_update_interval);
1253−}
1254−
12551295 /*
12561296 * Check whether current monitoring should be stopped
12571297 *
@@ -1280,12 +1320,10 @@ static bool kdamond_need_stop(struct damon_ctx *ctx) · 22 unmodified lines
12801320
12811321 static unsigned long damos_wmark_metric_value(enum damos_wmark_metric metric)
12821322 {
1283− struct sysinfo i;
1284−
12851323 switch (metric) {
12861324 case DAMOS_WMARK_FREE_MEM_RATE:
1287− si_meminfo(&i);
1288− return i.freeram * 1000 / i.totalram;
1325+ return global_zone_page_state(NR_FREE_PAGES) * 1000 /
1326+ totalram_pages();
12891327 default:
12901328 break;
12911329 }
@@ -1363,6 +1401,25 @@ static int kdamond_wait_activation(struct damon_ctx *ctx) · 71 unmodified lines
13631401 return -EBUSY;
13641402 }
13651403
1404+static void kdamond_init_intervals_sis(struct damon_ctx *ctx)
1405+{
1406+ unsigned long sample_interval = ctx->attrs.sample_interval ?
1407+ ctx->attrs.sample_interval : 1;
1408+ unsigned long apply_interval;
1409+ struct damos *scheme;
1410+
1411+ ctx->passed_sample_intervals = 0;
1412+ ctx->next_aggregation_sis = ctx->attrs.aggr_interval / sample_interval;
1413+ ctx->next_ops_update_sis = ctx->attrs.ops_update_interval /
1414+ sample_interval;
1415+
1416+ damon_for_each_scheme(scheme, ctx) {
1417+ apply_interval = scheme->apply_interval_us ?
1418+ scheme->apply_interval_us : ctx->attrs.aggr_interval;
1419+ scheme->next_apply_sis = apply_interval / sample_interval;
1420+ }
1421+}
1422+
13661423 /*
13671424 * The monitoring daemon that runs as a kernel thread
13681425 */
@@ -1376,6 +1433,8 @@ static int kdamond_fn(void *data) · 7 unmodified lines
13761433
13771434 pr_debug("kdamond (%d) starts\n", current->pid);
13781435
1436+ kdamond_init_intervals_sis(ctx);
1437+
13791438 if (ctx->ops.init)
13801439 ctx->ops.init(ctx);
13811440 if (ctx->callback.before_start && ctx->callback.before_start(ctx))
@@ -1384,6 +1443,17 @@ static int kdamond_fn(void *data) · 2 unmodified lines
13841443 sz_limit = damon_region_sz_limit(ctx);
13851444
13861445 while (!kdamond_need_stop(ctx)) {
1446+ /*
1447+ * ctx->attrs and ctx->next_{aggregation,ops_update}_sis could
1448+ * be changed from after_wmarks_check() or after_aggregation()
1449+ * callbacks. Read the values here, and use those for this
1450+ * iteration. That is, damon_set_attrs() updated new values
1451+ * are respected from next iteration.
1452+ */
1453+ unsigned long next_aggregation_sis = ctx->next_aggregation_sis;
1454+ unsigned long next_ops_update_sis = ctx->next_ops_update_sis;
1455+ unsigned long sample_interval = ctx->attrs.sample_interval;
1456+
13871457 if (kdamond_wait_activation(ctx))
13881458 break;
13891459
@@ -1393,27 +1463,44 @@ static int kdamond_fn(void *data) · 3 unmodified lines
13931463 ctx->callback.after_sampling(ctx))
13941464 break;
13951465
1396− kdamond_usleep(ctx->attrs.sample_interval);
1466+ kdamond_usleep(sample_interval);
1467+ ctx->passed_sample_intervals++;
13971468
13981469 if (ctx->ops.check_accesses)
13991470 max_nr_accesses = ctx->ops.check_accesses(ctx);
14001471
1401− if (kdamond_aggregate_interval_passed(ctx)) {
1472+ if (ctx->passed_sample_intervals == next_aggregation_sis) {
14021473 kdamond_merge_regions(ctx,
14031474 max_nr_accesses / 10,
14041475 sz_limit);
14051476 if (ctx->callback.after_aggregation &&
14061477 ctx->callback.after_aggregation(ctx))
14071478 break;
1408− if (!list_empty(&ctx->schemes))
1409− kdamond_apply_schemes(ctx);
1479+ }
1480+
1481+ /*
1482+ * do kdamond_apply_schemes() after kdamond_merge_regions() if
1483+ * possible, to reduce overhead
1484+ */
1485+ if (!list_empty(&ctx->schemes))
1486+ kdamond_apply_schemes(ctx);
1487+
1488+ sample_interval = ctx->attrs.sample_interval ?
1489+ ctx->attrs.sample_interval : 1;
1490+ if (ctx->passed_sample_intervals == next_aggregation_sis) {
1491+ ctx->next_aggregation_sis = next_aggregation_sis +
1492+ ctx->attrs.aggr_interval / sample_interval;
1493+
14101494 kdamond_reset_aggregated(ctx);
14111495 kdamond_split_regions(ctx);
14121496 if (ctx->ops.reset_aggregated)
14131497 ctx->ops.reset_aggregated(ctx);
14141498 }
14151499
1416− if (kdamond_need_update_operations(ctx)) {
1500+ if (ctx->passed_sample_intervals == next_ops_update_sis) {
1501+ ctx->next_ops_update_sis = next_ops_update_sis +
1502+ ctx->attrs.ops_update_interval /
1503+ sample_interval;
14171504 if (ctx->ops.update)
14181505 ctx->ops.update(ctx);
14191506 sz_limit = damon_region_sz_limit(ctx);
@@ -1517,6 +1604,76 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 97 unmodified lines
15171604 return damon_set_regions(t, &addr_range, 1);
15181605 }
15191606
1607+/*
1608+ * damon_moving_sum() - Calculate an inferred moving sum value.
1609+ * @mvsum: Inferred sum of the last @len_window values.
1610+ * @nomvsum: Non-moving sum of the last discrete @len_window window values.
1611+ * @len_window: The number of last values to take care of.
1612+ * @new_value: New value that will be added to the pseudo moving sum.
1613+ *
1614+ * Moving sum (moving average * window size) is good for handling noise, but
1615+ * the cost of keeping past values can be high for arbitrary window size. This
1616+ * function implements a lightweight pseudo moving sum function that doesn't
1617+ * keep the past window values.
1618+ *
1619+ * It simply assumes there was no noise in the past, and get the no-noise
1620+ * assumed past value to drop from @nomvsum and @len_window. @nomvsum is a
1621+ * non-moving sum of the last window. For example, if @len_window is 10 and we
1622+ * have 25 values, @nomvsum is the sum of the 11th to 20th values of the 25
1623+ * values. Hence, this function simply drops @nomvsum / @len_window from
1624+ * given @mvsum and add @new_value.
1625+ *
1626+ * For example, if @len_window is 10 and @nomvsum is 50, the last 10 values for
1627+ * the last window could be vary, e.g., 0, 10, 0, 10, 0, 10, 0, 0, 0, 20. For
1628+ * calculating next moving sum with a new value, we should drop 0 from 50 and
1629+ * add the new value. However, this function assumes it got value 5 for each
1630+ * of the last ten times. Based on the assumption, when the next value is
1631+ * measured, it drops the assumed past value, 5 from the current sum, and add
1632+ * the new value to get the updated pseduo-moving average.
1633+ *
1634+ * This means the value could have errors, but the errors will be disappeared
1635+ * for every @len_window aligned calls. For example, if @len_window is 10, the
1636+ * pseudo moving sum with 11th value to 19th value would have an error. But
1637+ * the sum with 20th value will not have the error.
1638+ *
1639+ * Return: Pseudo-moving average after getting the @new_value.
1640+ */
1641+static unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
1642+ unsigned int len_window, unsigned int new_value)
1643+{
1644+ return mvsum - nomvsum / len_window + new_value;
1645+}
1646+
1647+/**
1648+ * damon_update_region_access_rate() - Update the access rate of a region.
1649+ * @r: The DAMON region to update for its access check result.
1650+ * @accessed: Whether the region has accessed during last sampling interval.
1651+ * @attrs: The damon_attrs of the DAMON context.
1652+ *
1653+ * Update the access rate of a region with the region's last sampling interval
1654+ * access check result.
1655+ *
1656+ * Usually this will be called by &damon_operations->check_accesses callback.
1657+ */
1658+void damon_update_region_access_rate(struct damon_region *r, bool accessed,
1659+ struct damon_attrs *attrs)
1660+{
1661+ unsigned int len_window = 1;
1662+
1663+ /*
1664+ * sample_interval can be zero, but cannot be larger than
1665+ * aggr_interval, owing to validation of damon_set_attrs().
1666+ */
1667+ if (attrs->sample_interval)
1668+ len_window = damon_max_nr_accesses(attrs);
1669+ r->nr_accesses_bp = damon_moving_sum(r->nr_accesses_bp,
1670+ r->last_nr_accesses * 10000, len_window,
1671+ accessed ? 10000 : 0);
1672+
1673+ if (accessed)
1674+ r->nr_accesses++;
1675+}
1676+
15201677 static int __init damon_init(void)
15211678 {
15221679 damon_region_cache = KMEM_CACHE(damon_region, 0);
mm/damon/dbgfs.c+2−1
@@ -278,7 +278,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len, · 277 unmodified lines
278278 goto fail;
279279
280280 pos += parsed;
281− scheme = damon_new_scheme(&pattern, action, &quota, &wmarks);
281+ scheme = damon_new_scheme(&pattern, action, 0, &quota,
282+ &wmarks);
282283 if (!scheme)
283284 goto fail;
284285
mm/damon/lru_sort.c+3−3
@@ -158,6 +158,8 @@ static struct damos *damon_lru_sort_new_scheme( · 157 unmodified lines
158158 pattern,
159159 /* (de)prioritize on LRU-lists */
160160 action,
161+ /* for each aggregation interval */
162+ 0,
161163 /* under the quota. */
162164 &quota,
163165 /* (De)activate this according to the watermarks. */
@@ -193,9 +195,7 @@ static int damon_lru_sort_apply_parameters(void) · 29 unmodified lines
193195 if (err)
194196 return err;
195197
196− /* aggr_interval / sample_interval is the maximum nr_accesses */
197− hot_thres = damon_lru_sort_mon_attrs.aggr_interval /
198− damon_lru_sort_mon_attrs.sample_interval *
198+ hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) *
199199 hot_thres_access_freq / 1000;
200200 scheme = damon_lru_sort_new_hot_scheme(hot_thres);
201201 if (!scheme)
mm/damon/ops-common.c+2−3
@@ -73,7 +73,6 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr · 72 unmodified lines
7373 int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
7474 struct damos *s)
7575 {
76− unsigned int max_nr_accesses;
7776 int freq_subscore;
7877 unsigned int age_in_sec;
7978 int age_in_log, age_subscore;
@@ -81,8 +80,8 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, · 1 unmodified line
8180 unsigned int age_weight = s->quota.weight_age;
8281 int hotness;
8382
84− max_nr_accesses = c->attrs.aggr_interval / c->attrs.sample_interval;
85− freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / max_nr_accesses;
83+ freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE /
84+ damon_max_nr_accesses(&c->attrs);
8685
8786 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
8887 for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
mm/damon/paddr.c+5−6
@@ -148,7 +148,8 @@ out: · 147 unmodified lines
148148 return accessed;
149149 }
150150
151−static void __damon_pa_check_access(struct damon_region *r)
151+static void __damon_pa_check_access(struct damon_region *r,
152+ struct damon_attrs *attrs)
152153 {
153154 static unsigned long last_addr;
154155 static unsigned long last_folio_sz = PAGE_SIZE;
@@ -157,14 +158,12 @@ static void __damon_pa_check_access(struct damon_region *r) · 2 unmodified lines
157158 /* If the region is in the last checked page, reuse the result */
158159 if (ALIGN_DOWN(last_addr, last_folio_sz) ==
159160 ALIGN_DOWN(r->sampling_addr, last_folio_sz)) {
160− if (last_accessed)
161− r->nr_accesses++;
161+ damon_update_region_access_rate(r, last_accessed, attrs);
162162 return;
163163 }
164164
165165 last_accessed = damon_pa_young(r->sampling_addr, &last_folio_sz);
166− if (last_accessed)
167− r->nr_accesses++;
166+ damon_update_region_access_rate(r, last_accessed, attrs);
168167
169168 last_addr = r->sampling_addr;
170169 }
@@ -177,7 +176,7 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx) · 6 unmodified lines
177176
178177 damon_for_each_target(t, ctx) {
179178 damon_for_each_region(r, t) {
180− __damon_pa_check_access(r);
179+ __damon_pa_check_access(r, &ctx->attrs);
181180 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
182181 }
183182 }
mm/damon/reclaim.c+2−0
@@ -142,6 +142,8 @@ static struct damos *damon_reclaim_new_scheme(void) · 141 unmodified lines
142142 &pattern,
143143 /* page out those, as soon as found */
144144 DAMOS_PAGEOUT,
145+ /* for each aggregation interval */
146+ 0,
145147 /* under the quota. */
146148 &damon_reclaim_quota,
147149 /* (De)activate this according to the watermarks. */
mm/damon/sysfs-common.h+2−0
@@ -49,6 +49,8 @@ int damon_sysfs_schemes_update_regions_start( · 48 unmodified lines
4949 struct damon_sysfs_schemes *sysfs_schemes,
5050 struct damon_ctx *ctx, bool total_bytes_only);
5151
52+bool damos_sysfs_regions_upd_done(void);
53+
5254 int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx);
5355
5456 int damon_sysfs_schemes_clear_regions(
mm/damon/sysfs-schemes.c+128−5
@@ -31,7 +31,7 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc( · 30 unmodified lines
3131 return NULL;
3232 sysfs_region->kobj = (struct kobject){};
3333 sysfs_region->ar = region->ar;
34− sysfs_region->nr_accesses = region->nr_accesses;
34+ sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;
3535 sysfs_region->age = region->age;
3636 INIT_LIST_HEAD(&sysfs_region->list);
3737 return sysfs_region;
@@ -113,11 +113,47 @@ static const struct kobj_type damon_sysfs_scheme_region_ktype = { · 75 unmodified lines
113113 * scheme regions directory
114114 */
115115
116+/*
117+ * enum damos_sysfs_regions_upd_status - Represent DAMOS tried regions update
118+ * status
119+ * @DAMOS_TRIED_REGIONS_UPD_IDLE: Waiting for next request.
120+ * @DAMOS_TRIED_REGIONS_UPD_STARTED: Update started.
121+ * @DAMOS_TRIED_REGIONS_UPD_FINISHED: Update finished.
122+ *
123+ * Each DAMON-based operation scheme (&struct damos) has its own apply
124+ * interval, and we need to expose the scheme tried regions based on only
125+ * single snapshot. For this, we keep the tried regions update status for each
126+ * scheme. The status becomes 'idle' at the beginning.
127+ *
128+ * Once the tried regions update request is received, the request handling
129+ * start function (damon_sysfs_scheme_update_regions_start()) sets the status
130+ * of all schemes as 'idle' again, and register ->before_damos_apply() and
131+ * ->after_sampling() callbacks.
132+ *
133+ * Then, the first followup ->before_damos_apply() callback
134+ * (damon_sysfs_before_damos_apply()) sets the status 'started'. The first
135+ * ->after_sampling() callback (damon_sysfs_after_sampling()) after the call
136+ * is called only after the scheme is completely applied
137+ * to the given snapshot. Hence the callback knows the situation by showing
138+ * 'started' status, and sets the status as 'finished'. Then,
139+ * damon_sysfs_before_damos_apply() understands the situation by showing the
140+ * 'finished' status and do nothing.
141+ *
142+ * Finally, the tried regions request handling finisher function
143+ * (damon_sysfs_schemes_update_regions_stop()) unregisters the callbacks.
144+ */
145+enum damos_sysfs_regions_upd_status {
146+ DAMOS_TRIED_REGIONS_UPD_IDLE,
147+ DAMOS_TRIED_REGIONS_UPD_STARTED,
148+ DAMOS_TRIED_REGIONS_UPD_FINISHED,
149+};
150+
116151 struct damon_sysfs_scheme_regions {
117152 struct kobject kobj;
118153 struct list_head regions_list;
119154 int nr_regions;
120155 unsigned long total_bytes;
156+ enum damos_sysfs_regions_upd_status upd_status;
121157 };
122158
123159 static struct damon_sysfs_scheme_regions *
@@ -130,6 +166,7 @@ damon_sysfs_scheme_regions_alloc(void) · 6 unmodified lines
130166 INIT_LIST_HEAD(&regions->regions_list);
131167 regions->nr_regions = 0;
132168 regions->total_bytes = 0;
169+ regions->upd_status = DAMOS_TRIED_REGIONS_UPD_IDLE;
133170 return regions;
134171 }
135172
@@ -1121,6 +1158,7 @@ struct damon_sysfs_scheme { · 985 unmodified lines
11211158 struct kobject kobj;
11221159 enum damos_action action;
11231160 struct damon_sysfs_access_pattern *access_pattern;
1161+ unsigned long apply_interval_us;
11241162 struct damon_sysfs_quotas *quotas;
11251163 struct damon_sysfs_watermarks *watermarks;
11261164 struct damon_sysfs_scheme_filters *filters;
@@ -1141,7 +1179,7 @@ static const char * const damon_sysfs_damos_action_strs[] = { · 14 unmodified lines
11411179 };
11421180
11431181 static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc(
1144− enum damos_action action)
1182+ enum damos_action action, unsigned long apply_interval_us)
11451183 {
11461184 struct damon_sysfs_scheme *scheme = kmalloc(sizeof(*scheme),
11471185 GFP_KERNEL);
@@ -1150,6 +1188,7 @@ static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc( · 2 unmodified lines
11501188 return NULL;
11511189 scheme->kobj = (struct kobject){};
11521190 scheme->action = action;
1191+ scheme->apply_interval_us = apply_interval_us;
11531192 return scheme;
11541193 }
11551194
@@ -1353,6 +1392,25 @@ static ssize_t action_store(struct kobject *kobj, struct kobj_attribute *attr, · 197 unmodified lines
13531392 return -EINVAL;
13541393 }
13551394
1395+static ssize_t apply_interval_us_show(struct kobject *kobj,
1396+ struct kobj_attribute *attr, char *buf)
1397+{
1398+ struct damon_sysfs_scheme *scheme = container_of(kobj,
1399+ struct damon_sysfs_scheme, kobj);
1400+
1401+ return sysfs_emit(buf, "%lu\n", scheme->apply_interval_us);
1402+}
1403+
1404+static ssize_t apply_interval_us_store(struct kobject *kobj,
1405+ struct kobj_attribute *attr, const char *buf, size_t count)
1406+{
1407+ struct damon_sysfs_scheme *scheme = container_of(kobj,
1408+ struct damon_sysfs_scheme, kobj);
1409+ int err = kstrtoul(buf, 0, &scheme->apply_interval_us);
1410+
1411+ return err ? err : count;
1412+}
1413+
13561414 static void damon_sysfs_scheme_release(struct kobject *kobj)
13571415 {
13581416 kfree(container_of(kobj, struct damon_sysfs_scheme, kobj));
@@ -1361,8 +1419,12 @@ static void damon_sysfs_scheme_release(struct kobject *kobj) · 2 unmodified lines
13611419 static struct kobj_attribute damon_sysfs_scheme_action_attr =
13621420 __ATTR_RW_MODE(action, 0600);
13631421
1422+static struct kobj_attribute damon_sysfs_scheme_apply_interval_us_attr =
1423+ __ATTR_RW_MODE(apply_interval_us, 0600);
1424+
13641425 static struct attribute *damon_sysfs_scheme_attrs[] = {
13651426 &damon_sysfs_scheme_action_attr.attr,
1427+ &damon_sysfs_scheme_apply_interval_us_attr.attr,
13661428 NULL,
13671429 };
13681430 ATTRIBUTE_GROUPS(damon_sysfs_scheme);
@@ -1413,7 +1475,11 @@ static int damon_sysfs_schemes_add_dirs(struct damon_sysfs_schemes *schemes, · 44 unmodified lines
14131475 schemes->schemes_arr = schemes_arr;
14141476
14151477 for (i = 0; i < nr_schemes; i++) {
1416− scheme = damon_sysfs_scheme_alloc(DAMOS_STAT);
1478+ /*
1479+ * apply_interval_us as 0 means same to aggregation interval
1480+ * (same to before-apply_interval behavior)
1481+ */
1482+ scheme = damon_sysfs_scheme_alloc(DAMOS_STAT, 0);
14171483 if (!scheme) {
14181484 damon_sysfs_schemes_rm_dirs(schemes);
14191485 return -ENOMEM;
@@ -1610,8 +1676,8 @@ static struct damos *damon_sysfs_mk_scheme( · 190 unmodified lines
16101676 .low = sysfs_wmarks->low,
16111677 };
16121678
1613− scheme = damon_new_scheme(&pattern, sysfs_scheme->action, &quota,
1614− &wmarks);
1679+ scheme = damon_new_scheme(&pattern, sysfs_scheme->action,
1680+ sysfs_scheme->apply_interval_us, &quota, &wmarks);
16151681 if (!scheme)
16161682 return NULL;
16171683
@@ -1641,6 +1707,7 @@ static void damon_sysfs_update_scheme(struct damos *scheme, · 23 unmodified lines
16411707 scheme->pattern.max_age_region = access_pattern->age->max;
16421708
16431709 scheme->action = sysfs_scheme->action;
1710+ scheme->apply_interval_us = sysfs_scheme->apply_interval_us;
16441711
16451712 scheme->quota.ms = sysfs_quotas->ms;
16461713 scheme->quota.sz = sysfs_quotas->sz;
@@ -1747,6 +1814,10 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 100 unmodified lines
17471814 return 0;
17481815
17491816 sysfs_regions = sysfs_schemes->schemes_arr[schemes_idx]->tried_regions;
1817+ if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_FINISHED)
1818+ return 0;
1819+ if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_IDLE)
1820+ sysfs_regions->upd_status = DAMOS_TRIED_REGIONS_UPD_STARTED;
17501821 sysfs_regions->total_bytes += r->ar.end - r->ar.start;
17511822 if (damos_regions_upd_total_bytes_only)
17521823 return 0;
@@ -1763,6 +1834,29 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 10 unmodified lines
17631834 return 0;
17641835 }
17651836
1837+/*
1838+ * DAMON callback that called after each accesses sampling. While this
1839+ * callback is registered, damon_sysfs_lock should be held to ensure the
1840+ * regions directories exist.
1841+ */
1842+static int damon_sysfs_after_sampling(struct damon_ctx *ctx)
1843+{
1844+ struct damon_sysfs_schemes *sysfs_schemes =
1845+ damon_sysfs_schemes_for_damos_callback;
1846+ struct damon_sysfs_scheme_regions *sysfs_regions;
1847+ int i;
1848+
1849+ for (i = 0; i < sysfs_schemes->nr; i++) {
1850+ sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions;
1851+ if (sysfs_regions->upd_status ==
1852+ DAMOS_TRIED_REGIONS_UPD_STARTED)
1853+ sysfs_regions->upd_status =
1854+ DAMOS_TRIED_REGIONS_UPD_FINISHED;
1855+ }
1856+
1857+ return 0;
1858+}
1859+
17661860 /* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */
17671861 int damon_sysfs_schemes_clear_regions(
17681862 struct damon_sysfs_schemes *sysfs_schemes,
@@ -1786,6 +1880,16 @@ int damon_sysfs_schemes_clear_regions( · 17 unmodified lines
17861880 return 0;
17871881 }
17881882
1883+static void damos_tried_regions_init_upd_status(
1884+ struct damon_sysfs_schemes *sysfs_schemes)
1885+{
1886+ int i;
1887+
1888+ for (i = 0; i < sysfs_schemes->nr; i++)
1889+ sysfs_schemes->schemes_arr[i]->tried_regions->upd_status =
1890+ DAMOS_TRIED_REGIONS_UPD_IDLE;
1891+}
1892+
17891893 /* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */
17901894 int damon_sysfs_schemes_update_regions_start(
17911895 struct damon_sysfs_schemes *sysfs_schemes,
@@ -1793,11 +1897,29 @@ int damon_sysfs_schemes_update_regions_start( · 1 unmodified line
17931897 {
17941898 damon_sysfs_schemes_clear_regions(sysfs_schemes, ctx);
17951899 damon_sysfs_schemes_for_damos_callback = sysfs_schemes;
1900+ damos_tried_regions_init_upd_status(sysfs_schemes);
17961901 damos_regions_upd_total_bytes_only = total_bytes_only;
17971902 ctx->callback.before_damos_apply = damon_sysfs_before_damos_apply;
1903+ ctx->callback.after_sampling = damon_sysfs_after_sampling;
17981904 return 0;
17991905 }
18001906
1907+bool damos_sysfs_regions_upd_done(void)
1908+{
1909+ struct damon_sysfs_schemes *sysfs_schemes =
1910+ damon_sysfs_schemes_for_damos_callback;
1911+ struct damon_sysfs_scheme_regions *sysfs_regions;
1912+ int i;
1913+
1914+ for (i = 0; i < sysfs_schemes->nr; i++) {
1915+ sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions;
1916+ if (sysfs_regions->upd_status !=
1917+ DAMOS_TRIED_REGIONS_UPD_FINISHED)
1918+ return false;
1919+ }
1920+ return true;
1921+}
1922+
18011923 /*
18021924 * Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock. Caller
18031925 * should unlock damon_sysfs_lock which held before
@@ -1807,6 +1929,7 @@ int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx) · 3 unmodified lines
18071929 {
18081930 damon_sysfs_schemes_for_damos_callback = NULL;
18091931 ctx->callback.before_damos_apply = NULL;
1932+ ctx->callback.after_sampling = NULL;
18101933 damon_sysfs_schemes_region_idx = 0;
18111934 return 0;
18121935 }
mm/damon/sysfs-test.h+86−0
@@ -0,0 +1,86 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+/*
3+ * Data Access Monitor Unit Tests
4+ *
5+ * Author: SeongJae Park <sj@kernel.org>
6+ */
7+
8+#ifdef CONFIG_DAMON_SYSFS_KUNIT_TEST
9+
10+#ifndef _DAMON_SYSFS_TEST_H
11+#define _DAMON_SYSFS_TEST_H
12+
13+#include <kunit/test.h>
14+
15+static unsigned int nr_damon_targets(struct damon_ctx *ctx)
16+{
17+ struct damon_target *t;
18+ unsigned int nr_targets = 0;
19+
20+ damon_for_each_target(t, ctx)
21+ nr_targets++;
22+
23+ return nr_targets;
24+}
25+
26+static int __damon_sysfs_test_get_any_pid(int min, int max)
27+{
28+ struct pid *pid;
29+ int i;
30+
31+ for (i = min; i <= max; i++) {
32+ pid = find_get_pid(i);
33+ if (pid) {
34+ put_pid(pid);
35+ return i;
36+ }
37+ }
38+ return -1;
39+}
40+
41+static void damon_sysfs_test_set_targets(struct kunit *test)
42+{
43+ struct damon_sysfs_targets *sysfs_targets;
44+ struct damon_sysfs_target *sysfs_target;
45+ struct damon_ctx *ctx;
46+
47+ sysfs_targets = damon_sysfs_targets_alloc();
48+ sysfs_targets->nr = 1;
49+ sysfs_targets->targets_arr = kmalloc_array(1,
50+ sizeof(*sysfs_targets->targets_arr), GFP_KERNEL);
51+
52+ sysfs_target = damon_sysfs_target_alloc();
53+ sysfs_target->pid = __damon_sysfs_test_get_any_pid(12, 100);
54+ sysfs_target->regions = damon_sysfs_regions_alloc();
55+ sysfs_targets->targets_arr[0] = sysfs_target;
56+
57+ ctx = damon_new_ctx();
58+
59+ damon_sysfs_set_targets(ctx, sysfs_targets);
60+ KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));
61+
62+ sysfs_target->pid = __damon_sysfs_test_get_any_pid(
63+ sysfs_target->pid + 1, 200);
64+ damon_sysfs_set_targets(ctx, sysfs_targets);
65+ KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));
66+
67+ damon_destroy_ctx(ctx);
68+ kfree(sysfs_targets->targets_arr);
69+ kfree(sysfs_targets);
70+ kfree(sysfs_target);
71+}
72+
73+static struct kunit_case damon_sysfs_test_cases[] = {
74+ KUNIT_CASE(damon_sysfs_test_set_targets),
75+ {},
76+};
77+
78+static struct kunit_suite damon_sysfs_test_suite = {
79+ .name = "damon-sysfs",
80+ .test_cases = damon_sysfs_test_cases,
81+};
82+kunit_test_suite(damon_sysfs_test_suite);
83+
84+#endif /* _DAMON_SYSFS_TEST_H */
85+
86+#endif /* CONFIG_DAMON_SYSFS_KUNIT_TEST */
mm/damon/sysfs.c+83−40
@@ -1150,58 +1150,73 @@ destroy_targets_out: · 1149 unmodified lines
11501150 return err;
11511151 }
11521152
1153−/*
1154− * Search a target in a context that corresponds to the sysfs target input.
1155− *
1156− * Return: pointer to the target if found, NULL if not found, or negative
1157− * error code if the search failed.
1158− */
1159−static struct damon_target *damon_sysfs_existing_target(
1160− struct damon_sysfs_target *sys_target, struct damon_ctx *ctx)
1153+static int damon_sysfs_update_target_pid(struct damon_target *target, int pid)
11611154 {
1162− struct pid *pid;
1163− struct damon_target *t;
1155+ struct pid *pid_new;
11641156
1165− if (!damon_target_has_pid(ctx)) {
1166− /* Up to only one target for paddr could exist */
1167− damon_for_each_target(t, ctx)
1168− return t;
1169− return NULL;
1157+ pid_new = find_get_pid(pid);
1158+ if (!pid_new)
1159+ return -EINVAL;
1160+
1161+ if (pid_new == target->pid) {
1162+ put_pid(pid_new);
1163+ return 0;
11701164 }
11711165
1172− /* ops.id should be DAMON_OPS_VADDR or DAMON_OPS_FVADDR */
1173− pid = find_get_pid(sys_target->pid);
1174− if (!pid)
1175− return ERR_PTR(-EINVAL);
1176− damon_for_each_target(t, ctx) {
1177− if (t->pid == pid) {
1178− put_pid(pid);
1179− return t;
1180− }
1166+ put_pid(target->pid);
1167+ target->pid = pid_new;
1168+ return 0;
1169+}
1170+
1171+static int damon_sysfs_update_target(struct damon_target *target,
1172+ struct damon_ctx *ctx,
1173+ struct damon_sysfs_target *sys_target)
1174+{
1175+ int err;
1176+
1177+ if (damon_target_has_pid(ctx)) {
1178+ err = damon_sysfs_update_target_pid(target, sys_target->pid);
1179+ if (err)
1180+ return err;
11811181 }
1182− put_pid(pid);
1183− return NULL;
1182+
1183+ /*
1184+ * Do monitoring target region boundary update only if one or more
1185+ * regions are set by the user. This is for keeping current monitoring
1186+ * target results and range easier, especially for dynamic monitoring
1187+ * target regions update ops like 'vaddr'.
1188+ */
1189+ if (sys_target->regions->nr)
1190+ err = damon_sysfs_set_regions(target, sys_target->regions);
1191+ return err;
11841192 }
11851193
11861194 static int damon_sysfs_set_targets(struct damon_ctx *ctx,
11871195 struct damon_sysfs_targets *sysfs_targets)
11881196 {
1189− int i, err;
1197+ struct damon_target *t, *next;
1198+ int i = 0, err;
11901199
11911200 /* Multiple physical address space monitoring targets makes no sense */
11921201 if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1)
11931202 return -EINVAL;
11941203
1195− for (i = 0; i < sysfs_targets->nr; i++) {
1204+ damon_for_each_target_safe(t, next, ctx) {
1205+ if (i < sysfs_targets->nr) {
1206+ damon_sysfs_update_target(t, ctx,
1207+ sysfs_targets->targets_arr[i]);
1208+ } else {
1209+ if (damon_target_has_pid(ctx))
1210+ put_pid(t->pid);
1211+ damon_destroy_target(t);
1212+ }
1213+ i++;
1214+ }
1215+
1216+ for (; i < sysfs_targets->nr; i++) {
11961217 struct damon_sysfs_target *st = sysfs_targets->targets_arr[i];
1197− struct damon_target *t = damon_sysfs_existing_target(st, ctx);
1198−
1199− if (IS_ERR(t))
1200− return PTR_ERR(t);
1201− if (!t)
1202− err = damon_sysfs_add_target(st, ctx);
1203− else
1204− err = damon_sysfs_set_regions(t, st->regions);
1218+
1219+ err = damon_sysfs_add_target(st, ctx);
12051220 if (err)
12061221 return err;
12071222 }
@@ -1336,12 +1351,13 @@ static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond) · 128 unmodified lines
13361351
13371352 /*
13381353 * damon_sysfs_cmd_request_callback() - DAMON callback for handling requests.
1339− * @c: The DAMON context of the callback.
1354+ * @c: The DAMON context of the callback.
1355+ * @active: Whether @c is not deactivated due to watermarks.
13401356 *
13411357 * This function is periodically called back from the kdamond thread for @c.
13421358 * Then, it checks if there is a waiting DAMON sysfs request and handles it.
13431359 */
1344−static int damon_sysfs_cmd_request_callback(struct damon_ctx *c)
1360+static int damon_sysfs_cmd_request_callback(struct damon_ctx *c, bool active)
13451361 {
13461362 struct damon_sysfs_kdamond *kdamond;
13471363 bool total_bytes_only = false;
@@ -1373,6 +1389,13 @@ static int damon_sysfs_cmd_request_callback(struct damon_ctx *c) · 25 unmodified lines
13731389 goto keep_lock_out;
13741390 }
13751391 } else {
1392+ /*
1393+ * Continue regions updating if DAMON is till
1394+ * active and the update for all schemes is not
1395+ * finished.
1396+ */
1397+ if (active && !damos_sysfs_regions_upd_done())
1398+ goto keep_lock_out;
13761399 err = damon_sysfs_upd_schemes_regions_stop(kdamond);
13771400 damon_sysfs_schemes_regions_updating = false;
13781401 }
@@ -1392,6 +1415,24 @@ keep_lock_out: · 13 unmodified lines
13921415 return err;
13931416 }
13941417
1418+static int damon_sysfs_after_wmarks_check(struct damon_ctx *c)
1419+{
1420+ /*
1421+ * after_wmarks_check() is called back while the context is deactivated
1422+ * by watermarks.
1423+ */
1424+ return damon_sysfs_cmd_request_callback(c, false);
1425+}
1426+
1427+static int damon_sysfs_after_aggregation(struct damon_ctx *c)
1428+{
1429+ /*
1430+ * after_aggregation() is called back only while the context is not
1431+ * deactivated by watermarks.
1432+ */
1433+ return damon_sysfs_cmd_request_callback(c, true);
1434+}
1435+
13951436 static struct damon_ctx *damon_sysfs_build_ctx(
13961437 struct damon_sysfs_context *sys_ctx)
13971438 {
@@ -1407,8 +1448,8 @@ static struct damon_ctx *damon_sysfs_build_ctx( · 9 unmodified lines
14071448 return ERR_PTR(err);
14081449 }
14091450
1410− ctx->callback.after_wmarks_check = damon_sysfs_cmd_request_callback;
1411− ctx->callback.after_aggregation = damon_sysfs_cmd_request_callback;
1451+ ctx->callback.after_wmarks_check = damon_sysfs_after_wmarks_check;
1452+ ctx->callback.after_aggregation = damon_sysfs_after_aggregation;
14121453 ctx->callback.before_terminate = damon_sysfs_before_terminate;
14131454 return ctx;
14141455 }
@@ -1810,3 +1851,5 @@ out: · 395 unmodified lines
18101851 return err;
18111852 }
18121853 subsys_initcall(damon_sysfs_init);
1854+
1855+#include "sysfs-test.h"
mm/damon/vaddr.c+13−9
@@ -558,23 +558,27 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, · 557 unmodified lines
558558 * r the region to be checked
559559 */
560560 static void __damon_va_check_access(struct mm_struct *mm,
561− struct damon_region *r, bool same_target)
561+ struct damon_region *r, bool same_target,
562+ struct damon_attrs *attrs)
562563 {
563564 static unsigned long last_addr;
564565 static unsigned long last_folio_sz = PAGE_SIZE;
565566 static bool last_accessed;
566567
568+ if (!mm) {
569+ damon_update_region_access_rate(r, false, attrs);
570+ return;
571+ }
572+
567573 /* If the region is in the last checked page, reuse the result */
568574 if (same_target && (ALIGN_DOWN(last_addr, last_folio_sz) ==
569575 ALIGN_DOWN(r->sampling_addr, last_folio_sz))) {
570− if (last_accessed)
571− r->nr_accesses++;
576+ damon_update_region_access_rate(r, last_accessed, attrs);
572577 return;
573578 }
574579
575580 last_accessed = damon_va_young(mm, r->sampling_addr, &last_folio_sz);
576− if (last_accessed)
577− r->nr_accesses++;
581+ damon_update_region_access_rate(r, last_accessed, attrs);
578582
579583 last_addr = r->sampling_addr;
580584 }
@@ -589,15 +593,15 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx) · 8 unmodified lines
589593
590594 damon_for_each_target(t, ctx) {
591595 mm = damon_get_mm(t);
592− if (!mm)
593− continue;
594596 same_target = false;
595597 damon_for_each_region(r, t) {
596− __damon_va_check_access(mm, r, same_target);
598+ __damon_va_check_access(mm, r, same_target,
599+ &ctx->attrs);
597600 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
598601 same_target = true;
599602 }
600− mmput(mm);
603+ if (mm)
604+ mmput(mm);
601605 }
602606
603607 return max_nr_accesses;
mm/debug_vm_pgtable.c+2−2
@@ -1322,8 +1322,8 @@ static int __init debug_vm_pgtable(void) · 1321 unmodified lines
13221322 * true irrespective of the starting protection value for a
13231323 * given page table entry.
13241324 *
1325− * Protection based vm_flags combinatins are always linear
1326− * and increasing i.e starting from VM_NONE and going upto
1325+ * Protection based vm_flags combinations are always linear
1326+ * and increasing i.e starting from VM_NONE and going up to
13271327 * (VM_SHARED | READ | WRITE | EXEC).
13281328 */
13291329 #define VM_FLAGS_START (VM_NONE)
mm/filemap.c+82−133
@@ -131,11 +131,8 @@ static void page_cache_delete(struct address_space *mapping, · 130 unmodified lines
131131
132132 mapping_set_update(&xas, mapping);
133133
134− /* hugetlb pages are represented by a single entry in the xarray */
135− if (!folio_test_hugetlb(folio)) {
136− xas_set_order(&xas, folio->index, folio_order(folio));
137− nr = folio_nr_pages(folio);
138− }
134+ xas_set_order(&xas, folio->index, folio_order(folio));
135+ nr = folio_nr_pages(folio);
139136
140137 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
141138
@@ -234,7 +231,7 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio) · 92 unmodified lines
234231 if (free_folio)
235232 free_folio(folio);
236233
237− if (folio_test_large(folio) && !folio_test_hugetlb(folio))
234+ if (folio_test_large(folio))
238235 refs = folio_nr_pages(folio);
239236 folio_put_refs(folio, refs);
240237 }
@@ -819,7 +816,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) · 578 unmodified lines
819816 new->mapping = mapping;
820817 new->index = offset;
821818
822− mem_cgroup_migrate(old, new);
819+ mem_cgroup_replace_folio(old, new);
823820
824821 xas_lock_irq(&xas);
825822 xas_store(&xas, new);
@@ -855,14 +852,15 @@ noinline int __filemap_add_folio(struct address_space *mapping, · 29 unmodified lines
855852
856853 if (!huge) {
857854 int error = mem_cgroup_charge(folio, NULL, gfp);
858− VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio);
859855 if (error)
860856 return error;
861857 charged = true;
862− xas_set_order(&xas, index, folio_order(folio));
863− nr = folio_nr_pages(folio);
864858 }
865859
860+ VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio);
861+ xas_set_order(&xas, index, folio_order(folio));
862+ nr = folio_nr_pages(folio);
863+
866864 gfp &= GFP_RECLAIM_MASK;
867865 folio_ref_add(folio, nr);
868866 folio->mapping = mapping;
@@ -1135,32 +1133,13 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 266 unmodified lines
11351133 wait_queue_head_t *q = folio_waitqueue(folio);
11361134 struct wait_page_key key;
11371135 unsigned long flags;
1138− wait_queue_entry_t bookmark;
11391136
11401137 key.folio = folio;
11411138 key.bit_nr = bit_nr;
11421139 key.page_match = 0;
11431140
1144− bookmark.flags = 0;
1145− bookmark.private = NULL;
1146− bookmark.func = NULL;
1147− INIT_LIST_HEAD(&bookmark.entry);
1148−
11491141 spin_lock_irqsave(&q->lock, flags);
1150− __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark);
1151−
1152− while (bookmark.flags & WQ_FLAG_BOOKMARK) {
1153− /*
1154− * Take a breather from holding the lock,
1155− * allow pages that finish wake up asynchronously
1156− * to acquire the lock and remove themselves
1157− * from wait queue
1158− */
1159− spin_unlock_irqrestore(&q->lock, flags);
1160− cpu_relax();
1161− spin_lock_irqsave(&q->lock, flags);
1162− __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark);
1163− }
1142+ __wake_up_locked_key(q, TASK_NORMAL, &key);
11641143
11651144 /*
11661145 * It's possible to miss clearing waiters here, when we woke our page
@@ -1177,13 +1156,6 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 10 unmodified lines
11771156 spin_unlock_irqrestore(&q->lock, flags);
11781157 }
11791158
1180−static void folio_wake(struct folio *folio, int bit)
1181−{
1182− if (!folio_test_waiters(folio))
1183− return;
1184− folio_wake_bit(folio, bit);
1185−}
1186−
11871159 /*
11881160 * A choice of three behaviors for folio_wait_bit_common():
11891161 */
@@ -1484,29 +1456,6 @@ void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) · 294 unmodified lines
14841456 }
14851457 EXPORT_SYMBOL_GPL(folio_add_wait_queue);
14861458
1487−#ifndef clear_bit_unlock_is_negative_byte
1488−
1489−/*
1490− * PG_waiters is the high bit in the same byte as PG_lock.
1491− *
1492− * On x86 (and on many other architectures), we can clear PG_lock and
1493− * test the sign bit at the same time. But if the architecture does
1494− * not support that special operation, we just do this all by hand
1495− * instead.
1496− *
1497− * The read of PG_waiters has to be after (or concurrently with) PG_locked
1498− * being cleared, but a memory barrier should be unnecessary since it is
1499− * in the same byte as PG_locked.
1500− */
1501−static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem)
1502−{
1503− clear_bit_unlock(nr, mem);
1504− /* smp_mb__after_atomic(); */
1505− return test_bit(PG_waiters, mem);
1506−}
1507−
1508−#endif
1509−
15101459 /**
15111460 * folio_unlock - Unlock a locked folio.
15121461 * @folio: The folio.
@@ -1522,11 +1471,41 @@ void folio_unlock(struct folio *folio) · 9 unmodified lines
15221471 BUILD_BUG_ON(PG_waiters != 7);
15231472 BUILD_BUG_ON(PG_locked > 7);
15241473 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
1525− if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0)))
1474+ if (folio_xor_flags_has_waiters(folio, 1 << PG_locked))
15261475 folio_wake_bit(folio, PG_locked);
15271476 }
15281477 EXPORT_SYMBOL(folio_unlock);
15291478
1479+/**
1480+ * folio_end_read - End read on a folio.
1481+ * @folio: The folio.
1482+ * @success: True if all reads completed successfully.
1483+ *
1484+ * When all reads against a folio have completed, filesystems should
1485+ * call this function to let the pagecache know that no more reads
1486+ * are outstanding. This will unlock the folio and wake up any thread
1487+ * sleeping on the lock. The folio will also be marked uptodate if all
1488+ * reads succeeded.
1489+ *
1490+ * Context: May be called from interrupt or process context. May not be
1491+ * called from NMI context.
1492+ */
1493+void folio_end_read(struct folio *folio, bool success)
1494+{
1495+ unsigned long mask = 1 << PG_locked;
1496+
1497+ /* Must be in bottom byte for x86 to work */
1498+ BUILD_BUG_ON(PG_uptodate > 7);
1499+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
1500+ VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio);
1501+
1502+ if (likely(success))
1503+ mask |= 1 << PG_uptodate;
1504+ if (folio_xor_flags_has_waiters(folio, mask))
1505+ folio_wake_bit(folio, PG_locked);
1506+}
1507+EXPORT_SYMBOL(folio_end_read);
1508+
15301509 /**
15311510 * folio_end_private_2 - Clear PG_private_2 and wake any waiters.
15321511 * @folio: The folio.
@@ -1588,9 +1567,15 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); · 55 unmodified lines
15881567 /**
15891568 * folio_end_writeback - End writeback against a folio.
15901569 * @folio: The folio.
1570+ *
1571+ * The folio must actually be under writeback.
1572+ *
1573+ * Context: May be called from process or interrupt context.
15911574 */
15921575 void folio_end_writeback(struct folio *folio)
15931576 {
1577+ VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio);
1578+
15941579 /*
15951580 * folio_test_clear_reclaim() could be used here but it is an
15961581 * atomic operation and overkill in this particular case. Failing
@@ -1607,14 +1592,11 @@ void folio_end_writeback(struct folio *folio) · 10 unmodified lines
16071592 * Writeback does not hold a folio reference of its own, relying
16081593 * on truncation to wait for the clearing of PG_writeback.
16091594 * But here we must make sure that the folio is not freed and
1610− * reused before the folio_wake().
1595+ * reused before the folio_wake_bit().
16111596 */
16121597 folio_get(folio);
1613− if (!__folio_end_writeback(folio))
1614− BUG();
1615−
1616− smp_mb__after_atomic();
1617− folio_wake(folio, PG_writeback);
1598+ if (__folio_end_writeback(folio))
1599+ folio_wake_bit(folio, PG_writeback);
16181600 acct_reclaim_writeback(folio);
16191601 folio_put(folio);
16201602 }
@@ -2040,7 +2022,7 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, · 419 unmodified lines
20402022 int idx = folio_batch_count(fbatch) - 1;
20412023
20422024 folio = fbatch->folios[idx];
2043− if (!xa_is_value(folio) && !folio_test_hugetlb(folio))
2025+ if (!xa_is_value(folio))
20442026 nr = folio_nr_pages(folio);
20452027 *start = indices[idx] + nr;
20462028 }
@@ -2104,7 +2086,7 @@ put: · 57 unmodified lines
21042086 int idx = folio_batch_count(fbatch) - 1;
21052087
21062088 folio = fbatch->folios[idx];
2107− if (!xa_is_value(folio) && !folio_test_hugetlb(folio))
2089+ if (!xa_is_value(folio))
21082090 nr = folio_nr_pages(folio);
21092091 *start = indices[idx] + nr;
21102092 }
@@ -2122,51 +2104,13 @@ put: · 11 unmodified lines
21222104 * index @start and up to index @end (inclusive). The folios are returned
21232105 * in @fbatch with an elevated reference count.
21242106 *
2125− * The first folio may start before @start; if it does, it will contain
2126− * @start. The final folio may extend beyond @end; if it does, it will
2127− * contain @end. The folios have ascending indices. There may be gaps
2128− * between the folios if there are indices which have no folio in the
2129− * page cache. If folios are added to or removed from the page cache
2130− * while this is running, they may or may not be found by this call.
2131− *
21322107 * Return: The number of folios which were found.
21332108 * We also update @start to index the next folio for the traversal.
21342109 */
21352110 unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start,
21362111 pgoff_t end, struct folio_batch *fbatch)
21372112 {
2138− XA_STATE(xas, &mapping->i_pages, *start);
2139− struct folio *folio;
2140−
2141− rcu_read_lock();
2142− while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) {
2143− /* Skip over shadow, swap and DAX entries */
2144− if (xa_is_value(folio))
2145− continue;
2146− if (!folio_batch_add(fbatch, folio)) {
2147− unsigned long nr = folio_nr_pages(folio);
2148−
2149− if (folio_test_hugetlb(folio))
2150− nr = 1;
2151− *start = folio->index + nr;
2152− goto out;
2153− }
2154− }
2155−
2156− /*
2157− * We come here when there is no page beyond @end. We take care to not
2158− * overflow the index @start as it confuses some of the callers. This
2159− * breaks the iteration when there is a page at index -1 but that is
2160− * already broken anyway.
2161− */
2162− if (end == (pgoff_t)-1)
2163− *start = (pgoff_t)-1;
2164− else
2165− *start = end + 1;
2166−out:
2167− rcu_read_unlock();
2168−
2169− return folio_batch_count(fbatch);
2113+ return filemap_get_folios_tag(mapping, start, end, XA_PRESENT, fbatch);
21702114 }
21712115 EXPORT_SYMBOL(filemap_get_folios);
21722116
@@ -2213,9 +2157,6 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, · 40 unmodified lines
22132157
22142158 if (!folio_batch_add(fbatch, folio)) {
22152159 nr = folio_nr_pages(folio);
2216−
2217− if (folio_test_hugetlb(folio))
2218− nr = 1;
22192160 *start = folio->index + nr;
22202161 goto out;
22212162 }
@@ -2232,10 +2173,7 @@ update_start: · 10 unmodified lines
22322173
22332174 if (nr) {
22342175 folio = fbatch->folios[nr - 1];
2235− if (folio_test_hugetlb(folio))
2236− *start = folio->index + 1;
2237− else
2238− *start = folio_next_index(folio);
2176+ *start = folio->index + folio_nr_pages(folio);
22392177 }
22402178 out:
22412179 rcu_read_unlock();
@@ -2251,7 +2189,13 @@ EXPORT_SYMBOL(filemap_get_folios_contig); · 9 unmodified lines
22512189 * @tag: The tag index
22522190 * @fbatch: The batch to fill
22532191 *
2254− * Same as filemap_get_folios(), but only returning folios tagged with @tag.
2192+ * The first folio may start before @start; if it does, it will contain
2193+ * @start. The final folio may extend beyond @end; if it does, it will
2194+ * contain @end. The folios have ascending indices. There may be gaps
2195+ * between the folios if there are indices which have no folio in the
2196+ * page cache. If folios are added to or removed from the page cache
2197+ * while this is running, they may or may not be found by this call.
2198+ * Only returns folios that are tagged with @tag.
22552199 *
22562200 * Return: The number of folios found.
22572201 * Also update @start to index the next folio for traversal.
@@ -2273,9 +2217,6 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, · 15 unmodified lines
22732217 continue;
22742218 if (!folio_batch_add(fbatch, folio)) {
22752219 unsigned long nr = folio_nr_pages(folio);
2276−
2277− if (folio_test_hugetlb(folio))
2278− nr = 1;
22792220 *start = folio->index + nr;
22802221 goto out;
22812222 }
@@ -3104,7 +3045,7 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 822 unmodified lines
31043045
31053046 /*
31063047 * NOTE! This will make us return with VM_FAULT_RETRY, but with
3107− * the mmap_lock still held. That's how FAULT_FLAG_RETRY_NOWAIT
3048+ * the fault lock still held. That's how FAULT_FLAG_RETRY_NOWAIT
31083049 * is supposed to work. We have way too many special cases..
31093050 */
31103051 if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
@@ -3114,13 +3055,14 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 3 unmodified lines
31143055 if (vmf->flags & FAULT_FLAG_KILLABLE) {
31153056 if (__folio_lock_killable(folio)) {
31163057 /*
3117− * We didn't have the right flags to drop the mmap_lock,
3118− * but all fault_handlers only check for fatal signals
3119− * if we return VM_FAULT_RETRY, so we need to drop the
3120− * mmap_lock here and return 0 if we don't have a fpin.
3058+ * We didn't have the right flags to drop the
3059+ * fault lock, but all fault_handlers only check
3060+ * for fatal signals if we return VM_FAULT_RETRY,
3061+ * so we need to drop the fault lock here and
3062+ * return 0 if we don't have a fpin.
31213063 */
31223064 if (*fpin == NULL)
3123− mmap_read_unlock(vmf->vma->vm_mm);
3065+ release_fault_lock(vmf);
31243066 return 0;
31253067 }
31263068 } else
@@ -3321,21 +3263,28 @@ retry_find: · 194 unmodified lines
33213263 VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
33223264
33233265 /*
3324− * We have a locked page in the page cache, now we need to check
3325− * that it's up-to-date. If not, it is going to be due to an error.
3266+ * We have a locked folio in the page cache, now we need to check
3267+ * that it's up-to-date. If not, it is going to be due to an error,
3268+ * or because readahead was otherwise unable to retrieve it.
33263269 */
33273270 if (unlikely(!folio_test_uptodate(folio))) {
33283271 /*
3329− * The page was in cache and uptodate and now it is not.
3330− * Strange but possible since we didn't hold the page lock all
3331− * the time. Let's drop everything get the invalidate lock and
3332− * try again.
3272+ * If the invalidate lock is not held, the folio was in cache
3273+ * and uptodate and now it is not. Strange but possible since we
3274+ * didn't hold the page lock all the time. Let's drop
3275+ * everything, get the invalidate lock and try again.
33333276 */
33343277 if (!mapping_locked) {
33353278 folio_unlock(folio);
33363279 folio_put(folio);
33373280 goto retry_find;
33383281 }
3282+
3283+ /*
3284+ * OK, the folio is really not uptodate. This can be because the
3285+ * VMA has the VM_RAND_READ flag set, or because an error
3286+ * arose. Let's read it in directly.
3287+ */
33393288 goto page_not_uptodate;
33403289 }
33413290
@@ -3591,7 +3540,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 249 unmodified lines
35913540 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
35923541 vmf->pte += xas.xa_index - last_pgoff;
35933542 last_pgoff = xas.xa_index;
3594− end = folio->index + folio_nr_pages(folio) - 1;
3543+ end = folio_next_index(folio) - 1;
35953544 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;
35963545
35973546 if (!folio_test_large(folio))
@@ -3669,7 +3618,7 @@ int generic_file_mmap(struct file *file, struct vm_area_struct *vma) · 71 unmodified lines
36693618 */
36703619 int generic_file_readonly_mmap(struct file *file, struct vm_area_struct *vma)
36713620 {
3672− if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_MAYWRITE))
3621+ if (vma_is_shared_maywrite(vma))
36733622 return -EINVAL;
36743623 return generic_file_mmap(file, vma);
36753624 }
mm/gup.c+13−3
@@ -1471,6 +1471,9 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, · 1470 unmodified lines
14711471 long ret, pages_done;
14721472 bool must_unlock = false;
14731473
1474+ if (!nr_pages)
1475+ return 0;
1476+
14741477 /*
14751478 * The internal caller expects GUP to manage the lock internally and the
14761479 * lock must be released when this returns.
@@ -1595,6 +1598,14 @@ retry: · 118 unmodified lines
15951598 mmap_read_unlock(mm);
15961599 *locked = 0;
15971600 }
1601+
1602+ /*
1603+ * Failing to pin anything implies something has gone wrong (except when
1604+ * FOLL_NOWAIT is specified).
1605+ */
1606+ if (WARN_ON_ONCE(pages_done == 0 && !(flags & FOLL_NOWAIT)))
1607+ return -EFAULT;
1608+
15981609 return pages_done;
15991610 }
16001611
@@ -2227,12 +2238,11 @@ static bool is_valid_gup_args(struct page **pages, int *locked, · 626 unmodified lines
22272238 /*
22282239 * These flags not allowed to be specified externally to the gup
22292240 * interfaces:
2230− * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only
2241+ * - FOLL_TOUCH/FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only
22312242 * - FOLL_REMOTE is internal only and used on follow_page()
22322243 * - FOLL_UNLOCKABLE is internal only and used if locked is !NULL
22332244 */
2234− if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCKABLE |
2235− FOLL_REMOTE | FOLL_FAST_ONLY)))
2245+ if (WARN_ON_ONCE(gup_flags & INTERNAL_GUP_FLAGS))
22362246 return false;
22372247
22382248 gup_flags |= to_set;
mm/huge_memory.c+79−55
@@ -65,7 +65,11 @@ unsigned long transparent_hugepage_flags __read_mostly = · 64 unmodified lines
6565 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
6666 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
6767
68−static struct shrinker deferred_split_shrinker;
68+static struct shrinker *deferred_split_shrinker;
69+static unsigned long deferred_split_count(struct shrinker *shrink,
70+ struct shrink_control *sc);
71+static unsigned long deferred_split_scan(struct shrinker *shrink,
72+ struct shrink_control *sc);
6973
7074 static atomic_t huge_zero_refcount;
7175 struct page *huge_zero_page __read_mostly;
@@ -96,11 +100,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 24 unmodified lines
96100 return in_pf;
97101
98102 /*
99− * Special VMA and hugetlb VMA.
103+ * khugepaged special VMA and hugetlb VMA.
100104 * Must be checked after dax since some dax mappings may have
101105 * VM_MIXEDMAP set.
102106 */
103− if (vm_flags & VM_NO_KHUGEPAGED)
107+ if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED))
104108 return false;
105109
106110 /*
@@ -128,12 +132,18 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 21 unmodified lines
128132 !hugepage_flags_always())))
129133 return false;
130134
131− /* Only regular file is valid */
132− if (!in_pf && file_thp_enabled(vma))
133− return true;
134−
135− if (!vma_is_anonymous(vma))
135+ if (!vma_is_anonymous(vma)) {
136+ /*
137+ * Trust that ->huge_fault() handlers know what they are doing
138+ * in fault path.
139+ */
140+ if (((in_pf || smaps)) && vma->vm_ops->huge_fault)
141+ return true;
142+ /* Only regular file is valid in collapse path */
143+ if (((!in_pf || smaps)) && file_thp_enabled(vma))
144+ return true;
136145 return false;
146+ }
137147
138148 if (vma_is_temporary_stack(vma))
139149 return false;
@@ -229,11 +239,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, · 89 unmodified lines
229239 return 0;
230240 }
231241
232−static struct shrinker huge_zero_page_shrinker = {
233− .count_objects = shrink_huge_zero_page_count,
234− .scan_objects = shrink_huge_zero_page_scan,
235− .seeks = DEFAULT_SEEKS,
236−};
242+static struct shrinker *huge_zero_page_shrinker;
237243
238244 #ifdef CONFIG_SYSFS
239245 static ssize_t enabled_show(struct kobject *kobj,
@@ -454,6 +460,38 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · 214 unmodified lines
454460 }
455461 #endif /* CONFIG_SYSFS */
456462
463+static int __init thp_shrinker_init(void)
464+{
465+ huge_zero_page_shrinker = shrinker_alloc(0, "thp-zero");
466+ if (!huge_zero_page_shrinker)
467+ return -ENOMEM;
468+
469+ deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
470+ SHRINKER_MEMCG_AWARE |
471+ SHRINKER_NONSLAB,
472+ "thp-deferred_split");
473+ if (!deferred_split_shrinker) {
474+ shrinker_free(huge_zero_page_shrinker);
475+ return -ENOMEM;
476+ }
477+
478+ huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count;
479+ huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan;
480+ shrinker_register(huge_zero_page_shrinker);
481+
482+ deferred_split_shrinker->count_objects = deferred_split_count;
483+ deferred_split_shrinker->scan_objects = deferred_split_scan;
484+ shrinker_register(deferred_split_shrinker);
485+
486+ return 0;
487+}
488+
489+static void __init thp_shrinker_exit(void)
490+{
491+ shrinker_free(huge_zero_page_shrinker);
492+ shrinker_free(deferred_split_shrinker);
493+}
494+
457495 static int __init hugepage_init(void)
458496 {
459497 int err;
@@ -482,12 +520,9 @@ static int __init hugepage_init(void) · 22 unmodified lines
482520 if (err)
483521 goto err_slab;
484522
485− err = register_shrinker(&huge_zero_page_shrinker, "thp-zero");
523+ err = thp_shrinker_init();
486524 if (err)
487− goto err_hzp_shrinker;
488− err = register_shrinker(&deferred_split_shrinker, "thp-deferred_split");
489− if (err)
490− goto err_split_shrinker;
525+ goto err_shrinker;
491526
492527 /*
493528 * By default disable transparent hugepages on smaller systems,
@@ -505,10 +540,8 @@ static int __init hugepage_init(void) · 11 unmodified lines
505540
506541 return 0;
507542 err_khugepaged:
508− unregister_shrinker(&deferred_split_shrinker);
509−err_split_shrinker:
510− unregister_shrinker(&huge_zero_page_shrinker);
511−err_hzp_shrinker:
543+ thp_shrinker_exit();
544+err_shrinker:
512545 khugepaged_destroy();
513546 err_slab:
514547 hugepage_exit_sysfs(hugepage_kobj);
@@ -1349,7 +1382,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) · 834 unmodified lines
13491382 if (folio_ref_count(folio) == 1) {
13501383 pmd_t entry;
13511384
1352− page_move_anon_rmap(page, vma);
1385+ folio_move_anon_rmap(folio, vma);
1386+ SetPageAnonExclusive(page);
13531387 folio_unlock(folio);
13541388 reuse:
13551389 if (unlikely(unshare)) {
@@ -1490,9 +1524,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 134 unmodified lines
14901524 struct vm_area_struct *vma = vmf->vma;
14911525 pmd_t oldpmd = vmf->orig_pmd;
14921526 pmd_t pmd;
1493− struct page *page;
1527+ struct folio *folio;
14941528 unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
1495− int page_nid = NUMA_NO_NODE;
1529+ int nid = NUMA_NO_NODE;
14961530 int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
14971531 bool migrated = false, writable = false;
14981532 int flags = 0;
@@ -1514,36 +1548,34 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 15 unmodified lines
15141548 can_change_pmd_writable(vma, vmf->address, pmd))
15151549 writable = true;
15161550
1517− page = vm_normal_page_pmd(vma, haddr, pmd);
1518− if (!page)
1551+ folio = vm_normal_folio_pmd(vma, haddr, pmd);
1552+ if (!folio)
15191553 goto out_map;
15201554
15211555 /* See similar comment in do_numa_page for explanation */
15221556 if (!writable)
15231557 flags |= TNF_NO_GROUP;
15241558
1525− page_nid = page_to_nid(page);
1559+ nid = folio_nid(folio);
15261560 /*
15271561 * For memory tiering mode, cpupid of slow memory page is used
15281562 * to record page access time. So use default value.
15291563 */
1530− if (node_is_toptier(page_nid))
1531− last_cpupid = page_cpupid_last(page);
1532− target_nid = numa_migrate_prep(page, vma, haddr, page_nid,
1533− &flags);
1534−
1564+ if (node_is_toptier(nid))
1565+ last_cpupid = folio_last_cpupid(folio);
1566+ target_nid = numa_migrate_prep(folio, vma, haddr, nid, &flags);
15351567 if (target_nid == NUMA_NO_NODE) {
1536− put_page(page);
1568+ folio_put(folio);
15371569 goto out_map;
15381570 }
15391571
15401572 spin_unlock(vmf->ptl);
15411573 writable = false;
15421574
1543− migrated = migrate_misplaced_page(page, vma, target_nid);
1575+ migrated = migrate_misplaced_folio(folio, vma, target_nid);
15441576 if (migrated) {
15451577 flags |= TNF_MIGRATED;
1546− page_nid = target_nid;
1578+ nid = target_nid;
15471579 } else {
15481580 flags |= TNF_MIGRATE_FAIL;
15491581 vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -1555,9 +1587,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 5 unmodified lines
15551587 }
15561588
15571589 out:
1558− if (page_nid != NUMA_NO_NODE)
1559− task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
1560− flags);
1590+ if (nid != NUMA_NO_NODE)
1591+ task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
15611592
15621593 return 0;
15631594
@@ -1825,7 +1856,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 261 unmodified lines
18251856 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
18261857 if (is_swap_pmd(*pmd)) {
18271858 swp_entry_t entry = pmd_to_swp_entry(*pmd);
1828− struct page *page = pfn_swap_entry_to_page(entry);
1859+ struct folio *folio = page_folio(pfn_swap_entry_to_page(entry));
18291860 pmd_t newpmd;
18301861
18311862 VM_BUG_ON(!is_pmd_migration_entry(*pmd));
@@ -1834,7 +1865,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 2 unmodified lines
18341865 * A protection check is difficult so
18351866 * just be safe and disable write
18361867 */
1837− if (PageAnon(page))
1868+ if (folio_test_anon(folio))
18381869 entry = make_readable_exclusive_migration_entry(swp_offset(entry));
18391870 else
18401871 entry = make_readable_migration_entry(swp_offset(entry));
@@ -1856,7 +1887,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 15 unmodified lines
18561887 #endif
18571888
18581889 if (prot_numa) {
1859− struct page *page;
1890+ struct folio *folio;
18601891 bool toptier;
18611892 /*
18621893 * Avoid trapping faults against the zero page. The read-only
@@ -1869,8 +1900,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 6 unmodified lines
18691900 if (pmd_protnone(*pmd))
18701901 goto unlock;
18711902
1872− page = pmd_page(*pmd);
1873− toptier = node_is_toptier(page_to_nid(page));
1903+ folio = page_folio(pmd_page(*pmd));
1904+ toptier = node_is_toptier(folio_nid(folio));
18741905 /*
18751906 * Skip scanning top tier node if normal numa
18761907 * balancing is disabled
@@ -1881,7 +1912,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 4 unmodified lines
18811912
18821913 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
18831914 !toptier)
1884− xchg_page_access_time(page, jiffies_to_msecs(jiffies));
1915+ folio_xchg_access_time(folio,
1916+ jiffies_to_msecs(jiffies));
18851917 }
18861918 /*
18871919 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
@@ -2483,7 +2515,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, · 595 unmodified lines
24832515 if (page_is_idle(head))
24842516 set_page_idle(page_tail);
24852517
2486− page_cpupid_xchg_last(page_tail, page_cpupid_last(head));
2518+ folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio));
24872519
24882520 /*
24892521 * always add to the tail because some iterators expect new
@@ -2828,7 +2860,7 @@ void deferred_split_folio(struct folio *folio) · 338 unmodified lines
28282860 #ifdef CONFIG_MEMCG
28292861 if (memcg)
28302862 set_shrinker_bit(memcg, folio_nid(folio),
2831− deferred_split_shrinker.id);
2863+ deferred_split_shrinker->id);
28322864 #endif
28332865 }
28342866 spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
@@ -2902,14 +2934,6 @@ next: · 67 unmodified lines
29022934 return split;
29032935 }
29042936
2905−static struct shrinker deferred_split_shrinker = {
2906− .count_objects = deferred_split_count,
2907− .scan_objects = deferred_split_scan,
2908− .seeks = DEFAULT_SEEKS,
2909− .flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE |
2910− SHRINKER_NONSLAB,
2911−};
2912−
29132937 #ifdef CONFIG_DEBUG_FS
29142938 static void split_huge_pages_all(void)
29152939 {
mm/hugetlb.c+458−167
@@ -984,7 +984,7 @@ static long region_count(struct resv_map *resv, long f, long t) · 983 unmodified lines
984984
985985 /*
986986 * Convert the address within this vma to the page offset within
987− * the mapping, in pagecache page units; huge pages here.
987+ * the mapping, huge page units here.
988988 */
989989 static pgoff_t vma_hugecache_offset(struct hstate *h,
990990 struct vm_area_struct *vma, unsigned long address)
@@ -993,13 +993,6 @@ static pgoff_t vma_hugecache_offset(struct hstate *h, · 2 unmodified lines
993993 (vma->vm_pgoff >> huge_page_order(h));
994994 }
995995
996−pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
997− unsigned long address)
998−{
999− return vma_hugecache_offset(hstate_vma(vma), vma, address);
1000−}
1001−EXPORT_SYMBOL_GPL(linear_hugepage_index);
1002−
1003996 /**
1004997 * vma_kernel_pagesize - Page size granularity for this VMA.
1005998 * @vma: The user mapping.
@@ -1478,7 +1471,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, · 472 unmodified lines
14781471 }
14791472
14801473 /*
1481− * helper for remove_pool_huge_page() - return the previously saved
1474+ * helper for remove_pool_hugetlb_folio() - return the previously saved
14821475 * node ["this node"] from which to free a huge page. Advance the
14831476 * next node id whether or not we find a free huge page to free so
14841477 * that the next attempt to free addresses the next node.
@@ -1752,7 +1745,12 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, · 267 unmodified lines
17521745 if (folio_test_hugetlb_raw_hwp_unreliable(folio))
17531746 return;
17541747
1755− if (hugetlb_vmemmap_restore(h, &folio->page)) {
1748+ /*
1749+ * If folio is not vmemmap optimized (!clear_dtor), then the folio
1750+ * is no longer identified as a hugetlb page. hugetlb_vmemmap_restore_folio
1751+ * can only be passed hugetlb pages and will BUG otherwise.
1752+ */
1753+ if (clear_dtor && hugetlb_vmemmap_restore_folio(h, folio)) {
17561754 spin_lock_irq(&hugetlb_lock);
17571755 /*
17581756 * If we cannot allocate vmemmap pages, just refuse to free the
@@ -1814,22 +1812,22 @@ static void free_hpage_workfn(struct work_struct *work) · 55 unmodified lines
18141812 node = llist_del_all(&hpage_freelist);
18151813
18161814 while (node) {
1817− struct page *page;
1815+ struct folio *folio;
18181816 struct hstate *h;
18191817
1820− page = container_of((struct address_space **)node,
1821− struct page, mapping);
1818+ folio = container_of((struct address_space **)node,
1819+ struct folio, mapping);
18221820 node = node->next;
1823− page->mapping = NULL;
1821+ folio->mapping = NULL;
18241822 /*
18251823 * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in
18261824 * folio_hstate() is going to trigger because a previous call to
18271825 * remove_hugetlb_folio() will clear the hugetlb bit, so do
18281826 * not use folio_hstate() directly.
18291827 */
1830− h = size_to_hstate(page_size(page));
1828+ h = size_to_hstate(folio_size(folio));
18311829
1832− __update_and_free_hugetlb_folio(h, page_folio(page));
1830+ __update_and_free_hugetlb_folio(h, folio);
18331831
18341832 cond_resched();
18351833 }
@@ -1861,13 +1859,93 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, · 25 unmodified lines
18611859 schedule_work(&free_hpage_work);
18621860 }
18631861
1864−static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list)
1862+static void bulk_vmemmap_restore_error(struct hstate *h,
1863+ struct list_head *folio_list,
1864+ struct list_head *non_hvo_folios)
18651865 {
1866− struct page *page, *t_page;
1867− struct folio *folio;
1866+ struct folio *folio, *t_folio;
1867+
1868+ if (!list_empty(non_hvo_folios)) {
1869+ /*
1870+ * Free any restored hugetlb pages so that restore of the
1871+ * entire list can be retried.
1872+ * The idea is that in the common case of ENOMEM errors freeing
1873+ * hugetlb pages with vmemmap we will free up memory so that we
1874+ * can allocate vmemmap for more hugetlb pages.
1875+ */
1876+ list_for_each_entry_safe(folio, t_folio, non_hvo_folios, lru) {
1877+ list_del(&folio->lru);
1878+ spin_lock_irq(&hugetlb_lock);
1879+ __clear_hugetlb_destructor(h, folio);
1880+ spin_unlock_irq(&hugetlb_lock);
1881+ update_and_free_hugetlb_folio(h, folio, false);
1882+ cond_resched();
1883+ }
1884+ } else {
1885+ /*
1886+ * In the case where there are no folios which can be
1887+ * immediately freed, we loop through the list trying to restore
1888+ * vmemmap individually in the hope that someone elsewhere may
1889+ * have done something to cause success (such as freeing some
1890+ * memory). If unable to restore a hugetlb page, the hugetlb
1891+ * page is made a surplus page and removed from the list.
1892+ * If are able to restore vmemmap and free one hugetlb page, we
1893+ * quit processing the list to retry the bulk operation.
1894+ */
1895+ list_for_each_entry_safe(folio, t_folio, folio_list, lru)
1896+ if (hugetlb_vmemmap_restore_folio(h, folio)) {
1897+ list_del(&folio->lru);
1898+ spin_lock_irq(&hugetlb_lock);
1899+ add_hugetlb_folio(h, folio, true);
1900+ spin_unlock_irq(&hugetlb_lock);
1901+ } else {
1902+ list_del(&folio->lru);
1903+ spin_lock_irq(&hugetlb_lock);
1904+ __clear_hugetlb_destructor(h, folio);
1905+ spin_unlock_irq(&hugetlb_lock);
1906+ update_and_free_hugetlb_folio(h, folio, false);
1907+ cond_resched();
1908+ break;
1909+ }
1910+ }
1911+}
1912+
1913+static void update_and_free_pages_bulk(struct hstate *h,
1914+ struct list_head *folio_list)
1915+{
1916+ long ret;
1917+ struct folio *folio, *t_folio;
1918+ LIST_HEAD(non_hvo_folios);
1919+
1920+ /*
1921+ * First allocate required vmemmmap (if necessary) for all folios.
1922+ * Carefully handle errors and free up any available hugetlb pages
1923+ * in an effort to make forward progress.
1924+ */
1925+retry:
1926+ ret = hugetlb_vmemmap_restore_folios(h, folio_list, &non_hvo_folios);
1927+ if (ret < 0) {
1928+ bulk_vmemmap_restore_error(h, folio_list, &non_hvo_folios);
1929+ goto retry;
1930+ }
1931+
1932+ /*
1933+ * At this point, list should be empty, ret should be >= 0 and there
1934+ * should only be pages on the non_hvo_folios list.
1935+ * Do note that the non_hvo_folios list could be empty.
1936+ * Without HVO enabled, ret will be 0 and there is no need to call
1937+ * __clear_hugetlb_destructor as this was done previously.
1938+ */
1939+ VM_WARN_ON(!list_empty(folio_list));
1940+ VM_WARN_ON(ret < 0);
1941+ if (!list_empty(&non_hvo_folios) && ret) {
1942+ spin_lock_irq(&hugetlb_lock);
1943+ list_for_each_entry(folio, &non_hvo_folios, lru)
1944+ __clear_hugetlb_destructor(h, folio);
1945+ spin_unlock_irq(&hugetlb_lock);
1946+ }
18681947
1869− list_for_each_entry_safe(page, t_page, list, lru) {
1870− folio = page_folio(page);
1948+ list_for_each_entry_safe(folio, t_folio, &non_hvo_folios, lru) {
18711949 update_and_free_hugetlb_folio(h, folio, false);
18721950 cond_resched();
18731951 }
@@ -1931,6 +2009,7 @@ void free_huge_folio(struct folio *folio) · 57 unmodified lines
19312009 pages_per_huge_page(h), folio);
19322010 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
19332011 pages_per_huge_page(h), folio);
2012+ mem_cgroup_uncharge(folio);
19342013 if (restore_reserve)
19352014 h->resv_huge_pages++;
19362015
@@ -1960,16 +2039,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) · 23 unmodified lines
19602039 h->nr_huge_pages_node[nid]++;
19612040 }
19622041
1963−static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio)
2042+static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio)
19642043 {
1965− hugetlb_vmemmap_optimize(h, &folio->page);
1966− INIT_LIST_HEAD(&folio->lru);
19672044 folio_set_hugetlb(folio);
2045+ INIT_LIST_HEAD(&folio->lru);
19682046 hugetlb_set_folio_subpool(folio, NULL);
19692047 set_hugetlb_cgroup(folio, NULL);
19702048 set_hugetlb_cgroup_rsvd(folio, NULL);
19712049 }
19722050
2051+static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio)
2052+{
2053+ init_new_hugetlb_folio(h, folio);
2054+ hugetlb_vmemmap_optimize_folio(h, folio);
2055+}
2056+
19732057 static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid)
19742058 {
19752059 __prep_new_hugetlb_folio(h, folio);
@@ -2103,20 +2187,6 @@ struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage) · 127 unmodified lines
21032187 return NULL;
21042188 }
21052189
2106−pgoff_t hugetlb_basepage_index(struct page *page)
2107−{
2108− struct page *page_head = compound_head(page);
2109− pgoff_t index = page_index(page_head);
2110− unsigned long compound_idx;
2111−
2112− if (compound_order(page_head) > MAX_ORDER)
2113− compound_idx = page_to_pfn(page) - page_to_pfn(page_head);
2114− else
2115− compound_idx = page - page_head;
2116−
2117− return (index << compound_order(page_head)) + compound_idx;
2118−}
2119−
21202190 static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h,
21212191 gfp_t gfp_mask, int nid, nodemask_t *nmask,
21222192 nodemask_t *node_alloc_noretry)
@@ -2180,16 +2250,9 @@ retry: · 57 unmodified lines
21802250 return page_folio(page);
21812251 }
21822252
2183−/*
2184− * Common helper to allocate a fresh hugetlb page. All specific allocators
2185− * should use this function to get new hugetlb pages
2186− *
2187− * Note that returned page is 'frozen': ref count of head page and all tail
2188− * pages is zero.
2189− */
2190−static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
2191− gfp_t gfp_mask, int nid, nodemask_t *nmask,
2192− nodemask_t *node_alloc_noretry)
2253+static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h,
2254+ gfp_t gfp_mask, int nid, nodemask_t *nmask,
2255+ nodemask_t *node_alloc_noretry)
21932256 {
21942257 struct folio *folio;
21952258 bool retry = false;
@@ -2202,6 +2265,7 @@ retry: · 6 unmodified lines
22022265 nid, nmask, node_alloc_noretry);
22032266 if (!folio)
22042267 return NULL;
2268+
22052269 if (hstate_is_gigantic(h)) {
22062270 if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) {
22072271 /*
@@ -2216,32 +2280,84 @@ retry: · 8 unmodified lines
22162280 return NULL;
22172281 }
22182282 }
2219− prep_new_hugetlb_folio(h, folio, folio_nid(folio));
22202283
22212284 return folio;
22222285 }
22232286
2287+static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
2288+ gfp_t gfp_mask, int nid, nodemask_t *nmask,
2289+ nodemask_t *node_alloc_noretry)
2290+{
2291+ struct folio *folio;
2292+
2293+ folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask,
2294+ node_alloc_noretry);
2295+ if (folio)
2296+ init_new_hugetlb_folio(h, folio);
2297+ return folio;
2298+}
2299+
22242300 /*
2225− * Allocates a fresh page to the hugetlb allocator pool in the node interleaved
2226− * manner.
2301+ * Common helper to allocate a fresh hugetlb page. All specific allocators
2302+ * should use this function to get new hugetlb pages
2303+ *
2304+ * Note that returned page is 'frozen': ref count of head page and all tail
2305+ * pages is zero.
22272306 */
2228−static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
2229− nodemask_t *node_alloc_noretry)
2307+static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
2308+ gfp_t gfp_mask, int nid, nodemask_t *nmask,
2309+ nodemask_t *node_alloc_noretry)
22302310 {
22312311 struct folio *folio;
2232− int nr_nodes, node;
2312+
2313+ folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask,
2314+ node_alloc_noretry);
2315+ if (!folio)
2316+ return NULL;
2317+
2318+ prep_new_hugetlb_folio(h, folio, folio_nid(folio));
2319+ return folio;
2320+}
2321+
2322+static void prep_and_add_allocated_folios(struct hstate *h,
2323+ struct list_head *folio_list)
2324+{
2325+ unsigned long flags;
2326+ struct folio *folio, *tmp_f;
2327+
2328+ /* Send list for bulk vmemmap optimization processing */
2329+ hugetlb_vmemmap_optimize_folios(h, folio_list);
2330+
2331+ /* Add all new pool pages to free lists in one lock cycle */
2332+ spin_lock_irqsave(&hugetlb_lock, flags);
2333+ list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
2334+ __prep_account_new_huge_page(h, folio_nid(folio));
2335+ enqueue_hugetlb_folio(h, folio);
2336+ }
2337+ spin_unlock_irqrestore(&hugetlb_lock, flags);
2338+}
2339+
2340+/*
2341+ * Allocates a fresh hugetlb page in a node interleaved manner. The page
2342+ * will later be added to the appropriate hugetlb pool.
2343+ */
2344+static struct folio *alloc_pool_huge_folio(struct hstate *h,
2345+ nodemask_t *nodes_allowed,
2346+ nodemask_t *node_alloc_noretry)
2347+{
22332348 gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
2349+ int nr_nodes, node;
22342350
22352351 for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
2236− folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node,
2352+ struct folio *folio;
2353+
2354+ folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node,
22372355 nodes_allowed, node_alloc_noretry);
2238− if (folio) {
2239− free_huge_folio(folio); /* free it into the hugepage allocator */
2240− return 1;
2241− }
2356+ if (folio)
2357+ return folio;
22422358 }
22432359
2244− return 0;
2360+ return NULL;
22452361 }
22462362
22472363 /*
@@ -2251,13 +2367,11 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, · 3 unmodified lines
22512367 * an additional call to free the page to low level allocators.
22522368 * Called with hugetlb_lock locked.
22532369 */
2254−static struct page *remove_pool_huge_page(struct hstate *h,
2255− nodemask_t *nodes_allowed,
2256− bool acct_surplus)
2370+static struct folio *remove_pool_hugetlb_folio(struct hstate *h,
2371+ nodemask_t *nodes_allowed, bool acct_surplus)
22572372 {
22582373 int nr_nodes, node;
2259− struct page *page = NULL;
2260− struct folio *folio;
2374+ struct folio *folio = NULL;
22612375
22622376 lockdep_assert_held(&hugetlb_lock);
22632377 for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
@@ -2267,15 +2381,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, · 3 unmodified lines
22672381 */
22682382 if ((!acct_surplus || h->surplus_huge_pages_node[node]) &&
22692383 !list_empty(&h->hugepage_freelists[node])) {
2270− page = list_entry(h->hugepage_freelists[node].next,
2271− struct page, lru);
2272− folio = page_folio(page);
2384+ folio = list_entry(h->hugepage_freelists[node].next,
2385+ struct folio, lru);
22732386 remove_hugetlb_folio(h, folio, acct_surplus);
22742387 break;
22752388 }
22762389 }
22772390
2278− return page;
2391+ return folio;
22792392 }
22802393
22812394 /*
@@ -2343,17 +2456,23 @@ retry: · 61 unmodified lines
23432456 * need to adjust max_huge_pages if the page is not freed.
23442457 * Attempt to allocate vmemmmap here so that we can take
23452458 * appropriate action on failure.
2459+ *
2460+ * The folio_test_hugetlb check here is because
2461+ * remove_hugetlb_folio will clear hugetlb folio flag for
2462+ * non-vmemmap optimized hugetlb folios.
23462463 */
2347− rc = hugetlb_vmemmap_restore(h, &folio->page);
2348− if (!rc) {
2349− update_and_free_hugetlb_folio(h, folio, false);
2350− } else {
2351− spin_lock_irq(&hugetlb_lock);
2352− add_hugetlb_folio(h, folio, false);
2353− h->max_huge_pages++;
2354− spin_unlock_irq(&hugetlb_lock);
2355− }
2464+ if (folio_test_hugetlb(folio)) {
2465+ rc = hugetlb_vmemmap_restore_folio(h, folio);
2466+ if (rc) {
2467+ spin_lock_irq(&hugetlb_lock);
2468+ add_hugetlb_folio(h, folio, false);
2469+ h->max_huge_pages++;
2470+ goto out;
2471+ }
2472+ } else
2473+ rc = 0;
23562474
2475+ update_and_free_hugetlb_folio(h, folio, false);
23572476 return rc;
23582477 }
23592478 out:
@@ -2511,24 +2630,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 151 unmodified lines
25112630 return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask);
25122631 }
25132632
2514−/* mempolicy aware migration callback */
2515−struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma,
2516− unsigned long address)
2517−{
2518− struct mempolicy *mpol;
2519− nodemask_t *nodemask;
2520− struct folio *folio;
2521− gfp_t gfp_mask;
2522− int node;
2523−
2524− gfp_mask = htlb_alloc_mask(h);
2525− node = huge_node(vma, address, gfp_mask, &mpol, &nodemask);
2526− folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask);
2527− mpol_cond_put(mpol);
2528−
2529− return folio;
2530−}
2531−
25322633 /*
25332634 * Increase the hugetlb pool such that it can accommodate a reservation
25342635 * of size 'delta'.
@@ -2629,7 +2730,6 @@ static void return_unused_surplus_pages(struct hstate *h, · 94 unmodified lines
26292730 unsigned long unused_resv_pages)
26302731 {
26312732 unsigned long nr_pages;
2632− struct page *page;
26332733 LIST_HEAD(page_list);
26342734
26352735 lockdep_assert_held(&hugetlb_lock);
@@ -2650,15 +2750,17 @@ static void return_unused_surplus_pages(struct hstate *h, · 14 unmodified lines
26502750 * evenly across all nodes with memory. Iterate across these nodes
26512751 * until we can no longer free unreserved surplus pages. This occurs
26522752 * when the nodes with surplus pages have no free pages.
2653− * remove_pool_huge_page() will balance the freed pages across the
2753+ * remove_pool_hugetlb_folio() will balance the freed pages across the
26542754 * on-line nodes with memory and will handle the hstate accounting.
26552755 */
26562756 while (nr_pages--) {
2657− page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1);
2658− if (!page)
2757+ struct folio *folio;
2758+
2759+ folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1);
2760+ if (!folio)
26592761 goto out;
26602762
2661− list_add(&page->lru, &page_list);
2763+ list_add(&folio->lru, &page_list);
26622764 }
26632765
26642766 out:
@@ -3040,11 +3142,20 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 375 unmodified lines
30403142 struct hugepage_subpool *spool = subpool_vma(vma);
30413143 struct hstate *h = hstate_vma(vma);
30423144 struct folio *folio;
3043− long map_chg, map_commit;
3145+ long map_chg, map_commit, nr_pages = pages_per_huge_page(h);
30443146 long gbl_chg;
3045− int ret, idx;
3147+ int memcg_charge_ret, ret, idx;
30463148 struct hugetlb_cgroup *h_cg = NULL;
3149+ struct mem_cgroup *memcg;
30473150 bool deferred_reserve;
3151+ gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
3152+
3153+ memcg = get_mem_cgroup_from_current();
3154+ memcg_charge_ret = mem_cgroup_hugetlb_try_charge(memcg, gfp, nr_pages);
3155+ if (memcg_charge_ret == -ENOMEM) {
3156+ mem_cgroup_put(memcg);
3157+ return ERR_PTR(-ENOMEM);
3158+ }
30483159
30493160 idx = hstate_index(h);
30503161 /*
@@ -3053,8 +3164,12 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 2 unmodified lines
30533164 * code of zero indicates a reservation exists (no change).
30543165 */
30553166 map_chg = gbl_chg = vma_needs_reservation(h, vma, addr);
3056− if (map_chg < 0)
3167+ if (map_chg < 0) {
3168+ if (!memcg_charge_ret)
3169+ mem_cgroup_cancel_charge(memcg, nr_pages);
3170+ mem_cgroup_put(memcg);
30573171 return ERR_PTR(-ENOMEM);
3172+ }
30583173
30593174 /*
30603175 * Processes that did not create the mapping will have no
@@ -3065,10 +3180,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 4 unmodified lines
30653180 */
30663181 if (map_chg || avoid_reserve) {
30673182 gbl_chg = hugepage_subpool_get_pages(spool, 1);
3068− if (gbl_chg < 0) {
3069− vma_end_reservation(h, vma, addr);
3070− return ERR_PTR(-ENOSPC);
3071− }
3183+ if (gbl_chg < 0)
3184+ goto out_end_reservation;
30723185
30733186 /*
30743187 * Even though there was no reservation in the region/reserve
@@ -3150,6 +3263,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 75 unmodified lines
31503263 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
31513264 pages_per_huge_page(h), folio);
31523265 }
3266+
3267+ if (!memcg_charge_ret)
3268+ mem_cgroup_commit_charge(folio, memcg);
3269+ mem_cgroup_put(memcg);
3270+
31533271 return folio;
31543272
31553273 out_uncharge_cgroup:
@@ -3161,7 +3279,11 @@ out_uncharge_cgroup_reservation: · 5 unmodified lines
31613279 out_subpool_put:
31623280 if (map_chg || avoid_reserve)
31633281 hugepage_subpool_put_pages(spool, 1);
3282+out_end_reservation:
31643283 vma_end_reservation(h, vma, addr);
3284+ if (!memcg_charge_ret)
3285+ mem_cgroup_cancel_charge(memcg, nr_pages);
3286+ mem_cgroup_put(memcg);
31653287 return ERR_PTR(-ENOSPC);
31663288 }
31673289
@@ -3196,6 +3318,16 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) · 28 unmodified lines
31963318 }
31973319
31983320 found:
3321+
3322+ /*
3323+ * Only initialize the head struct page in memmap_init_reserved_pages,
3324+ * rest of the struct pages will be initialized by the HugeTLB
3325+ * subsystem itself.
3326+ * The head struct page is used to get folio information by the HugeTLB
3327+ * subsystem like zone id and node id.
3328+ */
3329+ memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE),
3330+ huge_page_size(h) - PAGE_SIZE);
31993331 /* Put them into a private list first because mem_map is not up yet */
32003332 INIT_LIST_HEAD(&m->list);
32013333 list_add(&m->list, &huge_boot_pages);
@@ -3203,29 +3335,102 @@ found: · 1 unmodified line
32033335 return 1;
32043336 }
32053337
3338+/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */
3339+static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
3340+ unsigned long start_page_number,
3341+ unsigned long end_page_number)
3342+{
3343+ enum zone_type zone = zone_idx(folio_zone(folio));
3344+ int nid = folio_nid(folio);
3345+ unsigned long head_pfn = folio_pfn(folio);
3346+ unsigned long pfn, end_pfn = head_pfn + end_page_number;
3347+ int ret;
3348+
3349+ for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) {
3350+ struct page *page = pfn_to_page(pfn);
3351+
3352+ __init_single_page(page, pfn, zone, nid);
3353+ prep_compound_tail((struct page *)folio, pfn - head_pfn);
3354+ ret = page_ref_freeze(page, 1);
3355+ VM_BUG_ON(!ret);
3356+ }
3357+}
3358+
3359+static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
3360+ struct hstate *h,
3361+ unsigned long nr_pages)
3362+{
3363+ int ret;
3364+
3365+ /* Prepare folio head */
3366+ __folio_clear_reserved(folio);
3367+ __folio_set_head(folio);
3368+ ret = folio_ref_freeze(folio, 1);
3369+ VM_BUG_ON(!ret);
3370+ /* Initialize the necessary tail struct pages */
3371+ hugetlb_folio_init_tail_vmemmap(folio, 1, nr_pages);
3372+ prep_compound_head((struct page *)folio, huge_page_order(h));
3373+}
3374+
3375+static void __init prep_and_add_bootmem_folios(struct hstate *h,
3376+ struct list_head *folio_list)
3377+{
3378+ unsigned long flags;
3379+ struct folio *folio, *tmp_f;
3380+
3381+ /* Send list for bulk vmemmap optimization processing */
3382+ hugetlb_vmemmap_optimize_folios(h, folio_list);
3383+
3384+ /* Add all new pool pages to free lists in one lock cycle */
3385+ spin_lock_irqsave(&hugetlb_lock, flags);
3386+ list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
3387+ if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
3388+ /*
3389+ * If HVO fails, initialize all tail struct pages
3390+ * We do not worry about potential long lock hold
3391+ * time as this is early in boot and there should
3392+ * be no contention.
3393+ */
3394+ hugetlb_folio_init_tail_vmemmap(folio,
3395+ HUGETLB_VMEMMAP_RESERVE_PAGES,
3396+ pages_per_huge_page(h));
3397+ }
3398+ __prep_account_new_huge_page(h, folio_nid(folio));
3399+ enqueue_hugetlb_folio(h, folio);
3400+ }
3401+ spin_unlock_irqrestore(&hugetlb_lock, flags);
3402+}
3403+
32063404 /*
32073405 * Put bootmem huge pages into the standard lists after mem_map is up.
32083406 * Note: This only applies to gigantic (order > MAX_ORDER) pages.
32093407 */
32103408 static void __init gather_bootmem_prealloc(void)
32113409 {
3410+ LIST_HEAD(folio_list);
32123411 struct huge_bootmem_page *m;
3412+ struct hstate *h = NULL, *prev_h = NULL;
32133413
32143414 list_for_each_entry(m, &huge_boot_pages, list) {
32153415 struct page *page = virt_to_page(m);
3216− struct folio *folio = page_folio(page);
3217− struct hstate *h = m->hstate;
3416+ struct folio *folio = (void *)page;
3417+
3418+ h = m->hstate;
3419+ /*
3420+ * It is possible to have multiple huge page sizes (hstates)
3421+ * in this list. If so, process each size separately.
3422+ */
3423+ if (h != prev_h && prev_h != NULL)
3424+ prep_and_add_bootmem_folios(prev_h, &folio_list);
3425+ prev_h = h;
32183426
32193427 VM_BUG_ON(!hstate_is_gigantic(h));
32203428 WARN_ON(folio_ref_count(folio) != 1);
3221− if (prep_compound_gigantic_folio(folio, huge_page_order(h))) {
3222− WARN_ON(folio_test_reserved(folio));
3223− prep_new_hugetlb_folio(h, folio, folio_nid(folio));
3224− free_huge_folio(folio); /* add to the hugepage allocator */
3225− } else {
3226− /* VERY unlikely inflated ref count on a tail page */
3227− free_gigantic_folio(folio, huge_page_order(h));
3228− }
3429+
3430+ hugetlb_folio_init_vmemmap(folio, h,
3431+ HUGETLB_VMEMMAP_RESERVE_PAGES);
3432+ init_new_hugetlb_folio(h, folio);
3433+ list_add(&folio->lru, &folio_list);
32293434
32303435 /*
32313436 * We need to restore the 'stolen' pages to totalram_pages
@@ -3235,7 +3440,10 @@ static void __init gather_bootmem_prealloc(void) · 3 unmodified lines
32353440 adjust_managed_page_count(page, pages_per_huge_page(h));
32363441 cond_resched();
32373442 }
3443+
3444+ prep_and_add_bootmem_folios(h, &folio_list);
32383445 }
3446+
32393447 static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
32403448 {
32413449 unsigned long i;
@@ -3267,9 +3475,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 25 unmodified lines
32673475 h->max_huge_pages_node[nid] = i;
32683476 }
32693477
3478+/*
3479+ * NOTE: this routine is called in different contexts for gigantic and
3480+ * non-gigantic pages.
3481+ * - For gigantic pages, this is called early in the boot process and
3482+ * pages are allocated from memblock allocated or something similar.
3483+ * Gigantic pages are actually added to pools later with the routine
3484+ * gather_bootmem_prealloc.
3485+ * - For non-gigantic pages, this is called later in the boot process after
3486+ * all of mm is up and functional. Pages are allocated from buddy and
3487+ * then added to hugetlb pools.
3488+ */
32703489 static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
32713490 {
32723491 unsigned long i;
3492+ struct folio *folio;
3493+ LIST_HEAD(folio_list);
32733494 nodemask_t *node_alloc_noretry;
32743495 bool node_specific_alloc = false;
32753496
@@ -3311,14 +3532,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · 35 unmodified lines
33113532
33123533 for (i = 0; i < h->max_huge_pages; ++i) {
33133534 if (hstate_is_gigantic(h)) {
3535+ /*
3536+ * gigantic pages not added to list as they are not
3537+ * added to pools now.
3538+ */
33143539 if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
33153540 break;
3316− } else if (!alloc_pool_huge_page(h,
3317− &node_states[N_MEMORY],
3318− node_alloc_noretry))
3319− break;
3541+ } else {
3542+ folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
3543+ node_alloc_noretry);
3544+ if (!folio)
3545+ break;
3546+ list_add(&folio->lru, &folio_list);
3547+ }
33203548 cond_resched();
33213549 }
3550+
3551+ /* list will be empty if hstate_is_gigantic */
3552+ prep_and_add_allocated_folios(h, &folio_list);
3553+
33223554 if (i < h->max_huge_pages) {
33233555 char buf[32];
33243556
@@ -3391,15 +3623,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, · 66 unmodified lines
33913623 * Collect pages to be freed on a list, and free after dropping lock
33923624 */
33933625 for_each_node_mask(i, *nodes_allowed) {
3394− struct page *page, *next;
3626+ struct folio *folio, *next;
33953627 struct list_head *freel = &h->hugepage_freelists[i];
3396− list_for_each_entry_safe(page, next, freel, lru) {
3628+ list_for_each_entry_safe(folio, next, freel, lru) {
33973629 if (count >= h->nr_huge_pages)
33983630 goto out;
3399− if (PageHighMem(page))
3631+ if (folio_test_highmem(folio))
34003632 continue;
3401− remove_hugetlb_folio(h, page_folio(page), false);
3402− list_add(&page->lru, &page_list);
3633+ remove_hugetlb_folio(h, folio, false);
3634+ list_add(&folio->lru, &page_list);
34033635 }
34043636 }
34053637
@@ -3452,8 +3684,9 @@ found: · 46 unmodified lines
34523684 static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
34533685 nodemask_t *nodes_allowed)
34543686 {
3455− unsigned long min_count, ret;
3456− struct page *page;
3687+ unsigned long min_count;
3688+ unsigned long allocated;
3689+ struct folio *folio;
34573690 LIST_HEAD(page_list);
34583691 NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL);
34593692
@@ -3484,7 +3717,9 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 24 unmodified lines
34843717 if (nid != NUMA_NO_NODE) {
34853718 unsigned long old_count = count;
34863719
3487− count += h->nr_huge_pages - h->nr_huge_pages_node[nid];
3720+ count += persistent_huge_pages(h) -
3721+ (h->nr_huge_pages_node[nid] -
3722+ h->surplus_huge_pages_node[nid]);
34883723 /*
34893724 * User may have specified a large count value which caused the
34903725 * above calculation to overflow. In this case, they wanted
@@ -3528,7 +3763,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 37 unmodified lines
35283763 break;
35293764 }
35303765
3531− while (count > persistent_huge_pages(h)) {
3766+ allocated = 0;
3767+ while (count > (persistent_huge_pages(h) + allocated)) {
35323768 /*
35333769 * If this allocation races such that we no longer need the
35343770 * page, free_huge_folio will handle it by freeing the page
@@ -3539,15 +3775,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 4 unmodified lines
35393775 /* yield cpu to avoid soft lockup */
35403776 cond_resched();
35413777
3542− ret = alloc_pool_huge_page(h, nodes_allowed,
3778+ folio = alloc_pool_huge_folio(h, nodes_allowed,
35433779 node_alloc_noretry);
3544− spin_lock_irq(&hugetlb_lock);
3545− if (!ret)
3780+ if (!folio) {
3781+ prep_and_add_allocated_folios(h, &page_list);
3782+ spin_lock_irq(&hugetlb_lock);
35463783 goto out;
3784+ }
3785+
3786+ list_add(&folio->lru, &page_list);
3787+ allocated++;
35473788
35483789 /* Bail for signals. Probably ctrl-c from user */
3549− if (signal_pending(current))
3790+ if (signal_pending(current)) {
3791+ prep_and_add_allocated_folios(h, &page_list);
3792+ spin_lock_irq(&hugetlb_lock);
35503793 goto out;
3794+ }
3795+
3796+ spin_lock_irq(&hugetlb_lock);
3797+ }
3798+
3799+ /* Add allocated pages to the pool */
3800+ if (!list_empty(&page_list)) {
3801+ spin_unlock_irq(&hugetlb_lock);
3802+ prep_and_add_allocated_folios(h, &page_list);
3803+ spin_lock_irq(&hugetlb_lock);
35513804 }
35523805
35533806 /*
@@ -3573,11 +3826,11 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 19 unmodified lines
35733826 * Collect pages to be removed on list without dropping lock
35743827 */
35753828 while (min_count < persistent_huge_pages(h)) {
3576− page = remove_pool_huge_page(h, nodes_allowed, 0);
3577− if (!page)
3829+ folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0);
3830+ if (!folio)
35783831 break;
35793832
3580− list_add(&page->lru, &page_list);
3833+ list_add(&folio->lru, &page_list);
35813834 }
35823835 /* free the pages after dropping lock */
35833836 spin_unlock_irq(&hugetlb_lock);
@@ -3612,13 +3865,21 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) · 28 unmodified lines
36123865 remove_hugetlb_folio_for_demote(h, folio, false);
36133866 spin_unlock_irq(&hugetlb_lock);
36143867
3615− rc = hugetlb_vmemmap_restore(h, &folio->page);
3616− if (rc) {
3617− /* Allocation of vmemmmap failed, we can not demote folio */
3618− spin_lock_irq(&hugetlb_lock);
3619− folio_ref_unfreeze(folio, 1);
3620− add_hugetlb_folio(h, folio, false);
3621− return rc;
3868+ /*
3869+ * If vmemmap already existed for folio, the remove routine above would
3870+ * have cleared the hugetlb folio flag. Hence the folio is technically
3871+ * no longer a hugetlb folio. hugetlb_vmemmap_restore_folio can only be
3872+ * passed hugetlb folios and will BUG otherwise.
3873+ */
3874+ if (folio_test_hugetlb(folio)) {
3875+ rc = hugetlb_vmemmap_restore_folio(h, folio);
3876+ if (rc) {
3877+ /* Allocation of vmemmmap failed, we can not demote folio */
3878+ spin_lock_irq(&hugetlb_lock);
3879+ folio_ref_unfreeze(folio, 1);
3880+ add_hugetlb_folio(h, folio, false);
3881+ return rc;
3882+ }
36223883 }
36233884
36243885 /*
@@ -4314,7 +4575,7 @@ void __init hugetlb_add_hstate(unsigned int order) · 689 unmodified lines
43144575 return;
43154576 }
43164577 BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE);
4317− BUG_ON(order == 0);
4578+ BUG_ON(order < order_base_2(__NR_USED_SUBPAGE));
43184579 h = &hstates[hugetlb_max_hstate++];
43194580 mutex_init(&h->resize_lock);
43204581 h->order = order;
@@ -4997,7 +5258,7 @@ bool is_hugetlb_entry_migration(pte_t pte) · 676 unmodified lines
49975258 return false;
49985259 }
49995260
5000−static bool is_hugetlb_entry_hwpoisoned(pte_t pte)
5261+bool is_hugetlb_entry_hwpoisoned(pte_t pte)
50015262 {
50025263 swp_entry_t swp;
50035264
@@ -5605,8 +5866,10 @@ retry_avoidcopy: · 601 unmodified lines
56055866 * owner and can reuse this page.
56065867 */
56075868 if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) {
5608− if (!PageAnonExclusive(&old_folio->page))
5609− page_move_anon_rmap(&old_folio->page, vma);
5869+ if (!PageAnonExclusive(&old_folio->page)) {
5870+ folio_move_anon_rmap(old_folio, vma);
5871+ SetPageAnonExclusive(&old_folio->page);
5872+ }
56105873 if (likely(!unshare))
56115874 set_huge_ptep_writable(vma, haddr, ptep);
56125875
@@ -5752,7 +6015,7 @@ static bool hugetlbfs_pagecache_present(struct hstate *h, · 139 unmodified lines
57526015 struct vm_area_struct *vma, unsigned long address)
57536016 {
57546017 struct address_space *mapping = vma->vm_file->f_mapping;
5755− pgoff_t idx = vma_hugecache_offset(h, vma, address);
6018+ pgoff_t idx = linear_page_index(vma, address);
57566019 struct folio *folio;
57576020
57586021 folio = filemap_get_folio(mapping, idx);
@@ -5769,6 +6032,7 @@ int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping · 10 unmodified lines
57696032 struct hstate *h = hstate_inode(inode);
57706033 int err;
57716034
6035+ idx <<= huge_page_order(h);
57726036 __folio_set_locked(folio);
57736037 err = __filemap_add_folio(mapping, folio, idx, GFP_KERNEL, NULL);
57746038
@@ -5876,7 +6140,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, · 101 unmodified lines
58766140 * before we get page_table_lock.
58776141 */
58786142 new_folio = false;
5879− folio = filemap_lock_folio(mapping, idx);
6143+ folio = filemap_lock_hugetlb_folio(h, mapping, idx);
58806144 if (IS_ERR(folio)) {
58816145 size = i_size_read(mapping->host) >> huge_page_shift(h);
58826146 if (idx >= size)
@@ -6185,7 +6449,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 302 unmodified lines
61856449 /* Just decrements count, does not deallocate */
61866450 vma_end_reservation(h, vma, haddr);
61876451
6188− pagecache_folio = filemap_lock_folio(mapping, idx);
6452+ pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, idx);
61896453 if (IS_ERR(pagecache_folio))
61906454 pagecache_folio = NULL;
61916455 }
@@ -6199,21 +6463,28 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 7 unmodified lines
61996463 /* Handle userfault-wp first, before trying to lock more pages */
62006464 if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
62016465 (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
6202− struct vm_fault vmf = {
6203− .vma = vma,
6204− .address = haddr,
6205− .real_address = address,
6206− .flags = flags,
6207− };
6466+ if (!userfaultfd_wp_async(vma)) {
6467+ struct vm_fault vmf = {
6468+ .vma = vma,
6469+ .address = haddr,
6470+ .real_address = address,
6471+ .flags = flags,
6472+ };
62086473
6209− spin_unlock(ptl);
6210− if (pagecache_folio) {
6211− folio_unlock(pagecache_folio);
6212− folio_put(pagecache_folio);
6474+ spin_unlock(ptl);
6475+ if (pagecache_folio) {
6476+ folio_unlock(pagecache_folio);
6477+ folio_put(pagecache_folio);
6478+ }
6479+ hugetlb_vma_unlock_read(vma);
6480+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6481+ return handle_userfault(&vmf, VM_UFFD_WP);
62136482 }
6214− hugetlb_vma_unlock_read(vma);
6215− mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6216− return handle_userfault(&vmf, VM_UFFD_WP);
6483+
6484+ entry = huge_pte_clear_uffd_wp(entry);
6485+ set_huge_pte_at(mm, haddr, ptep, entry,
6486+ huge_page_size(hstate_vma(vma)));
6487+ /* Fallthrough to CoW */
62176488 }
62186489
62196490 /*
@@ -6270,6 +6541,26 @@ out_mutex: · 50 unmodified lines
62706541 }
62716542
62726543 #ifdef CONFIG_USERFAULTFD
6544+/*
6545+ * Can probably be eliminated, but still used by hugetlb_mfill_atomic_pte().
6546+ */
6547+static struct folio *alloc_hugetlb_folio_vma(struct hstate *h,
6548+ struct vm_area_struct *vma, unsigned long address)
6549+{
6550+ struct mempolicy *mpol;
6551+ nodemask_t *nodemask;
6552+ struct folio *folio;
6553+ gfp_t gfp_mask;
6554+ int node;
6555+
6556+ gfp_mask = htlb_alloc_mask(h);
6557+ node = huge_node(vma, address, gfp_mask, &mpol, &nodemask);
6558+ folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask);
6559+ mpol_cond_put(mpol);
6560+
6561+ return folio;
6562+}
6563+
62736564 /*
62746565 * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte
62756566 * with modifications for hugetlb pages.
@@ -6318,7 +6609,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 42 unmodified lines
63186609
63196610 if (is_continue) {
63206611 ret = -EFAULT;
6321− folio = filemap_lock_folio(mapping, idx);
6612+ folio = filemap_lock_hugetlb_folio(h, mapping, idx);
63226613 if (IS_ERR(folio))
63236614 goto out;
63246615 folio_in_pagecache = true;
@@ -6520,7 +6811,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, · 195 unmodified lines
65206811 }
65216812 }
65226813
6523− page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT);
6814+ page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT));
65246815
65256816 /*
65266817 * Note that page may be a sub-page, and with vmemmap
mm/hugetlb_cgroup.c+2−18
@@ -262,12 +262,6 @@ static int __hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages, · 261 unmodified lines
262262
263263 if (hugetlb_cgroup_disabled())
264264 goto done;
265− /*
266− * We don't charge any cgroup if the compound page have less
267− * than 3 pages.
268− */
269− if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER)
270− goto done;
271265 again:
272266 rcu_read_lock();
273267 h_cg = hugetlb_cgroup_from_task(current);
@@ -397,9 +391,6 @@ static void __hugetlb_cgroup_uncharge_cgroup(int idx, unsigned long nr_pages, · 123 unmodified lines
397391 if (hugetlb_cgroup_disabled() || !h_cg)
398392 return;
399393
400− if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER)
401− return;
402−
403394 page_counter_uncharge(__hugetlb_cgroup_counter_from_cgroup(h_cg, idx,
404395 rsvd),
405396 nr_pages);
@@ -869,15 +860,8 @@ void __init hugetlb_cgroup_file_init(void) · 463 unmodified lines
869860 {
870861 struct hstate *h;
871862
872− for_each_hstate(h) {
873− /*
874− * Add cgroup control files only if the huge page consists
875− * of more than two normal pages. This is because we use
876− * page[2].private for storing cgroup details.
877− */
878− if (huge_page_order(h) >= HUGETLB_CGROUP_MIN_ORDER)
879− __hugetlb_cgroup_file_init(hstate_index(h));
880− }
863+ for_each_hstate(h)
864+ __hugetlb_cgroup_file_init(hstate_index(h));
881865 }
882866
883867 /*
mm/hugetlb_vmemmap.c+252−49
@@ -13,6 +13,7 @@ · 12 unmodified lines
1313 #include <linux/pgtable.h>
1414 #include <linux/moduleparam.h>
1515 #include <linux/bootmem_info.h>
16+#include <linux/mmdebug.h>
1617 #include <asm/pgalloc.h>
1718 #include <asm/tlbflush.h>
1819 #include "hugetlb_vmemmap.h"
@@ -26,6 +27,8 @@ · 7 unmodified lines
2627 * @reuse_addr: the virtual address of the @reuse_page page.
2728 * @vmemmap_pages: the list head of the vmemmap pages that can be freed
2829 * or is mapped from.
30+ * @flags: used to modify behavior in vmemmap page table walking
31+ * operations.
2932 */
3033 struct vmemmap_remap_walk {
3134 void (*remap_pte)(pte_t *pte, unsigned long addr,
@@ -34,9 +37,15 @@ struct vmemmap_remap_walk { · 2 unmodified lines
3437 struct page *reuse_page;
3538 unsigned long reuse_addr;
3639 struct list_head *vmemmap_pages;
40+
41+/* Skip the TLB flush when we split the PMD */
42+#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0)
43+/* Skip the TLB flush when we remap the PTE */
44+#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1)
45+ unsigned long flags;
3746 };
3847
39−static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start)
48+static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush)
4049 {
4150 pmd_t __pmd;
4251 int i;
@@ -79,7 +88,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) · 36 unmodified lines
7988 /* Make pte visible before pmd. See comment in pmd_install(). */
8089 smp_wmb();
8190 pmd_populate_kernel(&init_mm, pmd, pgtable);
82− flush_tlb_kernel_range(start, start + PMD_SIZE);
91+ if (flush)
92+ flush_tlb_kernel_range(start, start + PMD_SIZE);
8393 } else {
8494 pte_free_kernel(&init_mm, pgtable);
8595 }
@@ -126,11 +136,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, · 40 unmodified lines
126136 do {
127137 int ret;
128138
129− ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK);
139+ ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK,
140+ !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH));
130141 if (ret)
131142 return ret;
132143
133144 next = pmd_addr_end(addr, end);
145+
146+ /*
147+ * We are only splitting, not remapping the hugetlb vmemmap
148+ * pages.
149+ */
150+ if (!walk->remap_pte)
151+ continue;
152+
134153 vmemmap_pte_range(pmd, addr, next, walk);
135154 } while (pmd++, addr = next, addr != end);
136155
@@ -197,7 +216,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, · 60 unmodified lines
197216 return ret;
198217 } while (pgd++, addr = next, addr != end);
199218
200− flush_tlb_kernel_range(start, end);
219+ if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH))
220+ flush_tlb_kernel_range(start, end);
201221
202222 return 0;
203223 }
@@ -250,7 +270,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, · 46 unmodified lines
250270 }
251271
252272 entry = mk_pte(walk->reuse_page, pgprot);
253− list_add_tail(&page->lru, walk->vmemmap_pages);
273+ list_add(&page->lru, walk->vmemmap_pages);
254274 set_pte_at(&init_mm, addr, pte, entry);
255275 }
256276
@@ -296,6 +316,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 39 unmodified lines
296316 set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));
297317 }
298318
319+/**
320+ * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end)
321+ * backing PMDs of the directmap into PTEs
322+ * @start: start address of the vmemmap virtual address range that we want
323+ * to remap.
324+ * @end: end address of the vmemmap virtual address range that we want to
325+ * remap.
326+ * @reuse: reuse address.
327+ *
328+ * Return: %0 on success, negative error code otherwise.
329+ */
330+static int vmemmap_remap_split(unsigned long start, unsigned long end,
331+ unsigned long reuse)
332+{
333+ int ret;
334+ struct vmemmap_remap_walk walk = {
335+ .remap_pte = NULL,
336+ .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH,
337+ };
338+
339+ /* See the comment in the vmemmap_remap_free(). */
340+ BUG_ON(start - reuse != PAGE_SIZE);
341+
342+ mmap_read_lock(&init_mm);
343+ ret = vmemmap_remap_range(reuse, end, &walk);
344+ mmap_read_unlock(&init_mm);
345+
346+ return ret;
347+}
348+
299349 /**
300350 * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end)
301351 * to the page which @reuse is mapped to, then free vmemmap
@@ -305,22 +355,26 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 3 unmodified lines
305355 * @end: end address of the vmemmap virtual address range that we want to
306356 * remap.
307357 * @reuse: reuse address.
358+ * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers
359+ * responsibility to free pages.
360+ * @flags: modifications to vmemmap_remap_walk flags
308361 *
309362 * Return: %0 on success, negative error code otherwise.
310363 */
311364 static int vmemmap_remap_free(unsigned long start, unsigned long end,
312− unsigned long reuse)
365+ unsigned long reuse,
366+ struct list_head *vmemmap_pages,
367+ unsigned long flags)
313368 {
314369 int ret;
315− LIST_HEAD(vmemmap_pages);
316370 struct vmemmap_remap_walk walk = {
317371 .remap_pte = vmemmap_remap_pte,
318372 .reuse_addr = reuse,
319− .vmemmap_pages = &vmemmap_pages,
373+ .vmemmap_pages = vmemmap_pages,
374+ .flags = flags,
320375 };
321− int nid = page_to_nid((struct page *)start);
322− gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY |
323− __GFP_NOWARN;
376+ int nid = page_to_nid((struct page *)reuse);
377+ gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN;
324378
325379 /*
326380 * Allocate a new head vmemmap page to avoid breaking a contiguous
@@ -334,7 +388,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 7 unmodified lines
334388 if (walk.reuse_page) {
335389 copy_page(page_to_virt(walk.reuse_page),
336390 (void *)walk.reuse_addr);
337− list_add(&walk.reuse_page->lru, &vmemmap_pages);
391+ list_add(&walk.reuse_page->lru, vmemmap_pages);
338392 }
339393
340394 /*
@@ -365,22 +419,21 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 24 unmodified lines
365419 walk = (struct vmemmap_remap_walk) {
366420 .remap_pte = vmemmap_restore_pte,
367421 .reuse_addr = reuse,
368− .vmemmap_pages = &vmemmap_pages,
422+ .vmemmap_pages = vmemmap_pages,
423+ .flags = 0,
369424 };
370425
371426 vmemmap_remap_range(reuse, end, &walk);
372427 }
373428 mmap_read_unlock(&init_mm);
374429
375− free_vmemmap_page_list(&vmemmap_pages);
376−
377430 return ret;
378431 }
379432
380433 static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
381434 struct list_head *list)
382435 {
383− gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_THISNODE;
436+ gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL;
384437 unsigned long nr_pages = (end - start) >> PAGE_SHIFT;
385438 int nid = page_to_nid((struct page *)start);
386439 struct page *page, *next;
@@ -389,7 +442,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, · 2 unmodified lines
389442 page = alloc_pages_node(nid, gfp_mask, 0);
390443 if (!page)
391444 goto out;
392− list_add_tail(&page->lru, list);
445+ list_add(&page->lru, list);
393446 }
394447
395448 return 0;
@@ -408,17 +461,19 @@ out: · 12 unmodified lines
408461 * @end: end address of the vmemmap virtual address range that we want to
409462 * remap.
410463 * @reuse: reuse address.
464+ * @flags: modifications to vmemmap_remap_walk flags
411465 *
412466 * Return: %0 on success, negative error code otherwise.
413467 */
414468 static int vmemmap_remap_alloc(unsigned long start, unsigned long end,
415− unsigned long reuse)
469+ unsigned long reuse, unsigned long flags)
416470 {
417471 LIST_HEAD(vmemmap_pages);
418472 struct vmemmap_remap_walk walk = {
419473 .remap_pte = vmemmap_restore_pte,
420474 .reuse_addr = reuse,
421475 .vmemmap_pages = &vmemmap_pages,
476+ .flags = flags,
422477 };
423478
424479 /* See the comment in the vmemmap_remap_free(). */
@@ -440,23 +495,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); · 15 unmodified lines
440495 static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
441496 core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0);
442497
443−/**
444− * hugetlb_vmemmap_restore - restore previously optimized (by
445− * hugetlb_vmemmap_optimize()) vmemmap pages which
446− * will be reallocated and remapped.
447− * @h: struct hstate.
448− * @head: the head page whose vmemmap pages will be restored.
449− *
450− * Return: %0 if @head's vmemmap pages have been reallocated and remapped,
451− * negative error code otherwise.
452− */
453−int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
498+static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio, unsigned long flags)
454499 {
455500 int ret;
501+ struct page *head = &folio->page;
456502 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
457503 unsigned long vmemmap_reuse;
458504
459− if (!HPageVmemmapOptimized(head))
505+ VM_WARN_ON_ONCE(!PageHuge(head));
506+ if (!folio_test_hugetlb_vmemmap_optimized(folio))
460507 return 0;
461508
462509 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
@@ -470,18 +517,77 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) · 7 unmodified lines
470517 * When a HugeTLB page is freed to the buddy allocator, previously
471518 * discarded vmemmap pages must be allocated and remapping.
472519 */
473− ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse);
520+ ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags);
474521 if (!ret) {
475− ClearHPageVmemmapOptimized(head);
522+ folio_clear_hugetlb_vmemmap_optimized(folio);
476523 static_branch_dec(&hugetlb_optimize_vmemmap_key);
477524 }
478525
479526 return ret;
480527 }
481528
529+/**
530+ * hugetlb_vmemmap_restore_folio - restore previously optimized (by
531+ * hugetlb_vmemmap_optimize_folio()) vmemmap pages which
532+ * will be reallocated and remapped.
533+ * @h: struct hstate.
534+ * @folio: the folio whose vmemmap pages will be restored.
535+ *
536+ * Return: %0 if @folio's vmemmap pages have been reallocated and remapped,
537+ * negative error code otherwise.
538+ */
539+int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio)
540+{
541+ return __hugetlb_vmemmap_restore_folio(h, folio, 0);
542+}
543+
544+/**
545+ * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list.
546+ * @h: hstate.
547+ * @folio_list: list of folios.
548+ * @non_hvo_folios: Output list of folios for which vmemmap exists.
549+ *
550+ * Return: number of folios for which vmemmap was restored, or an error code
551+ * if an error was encountered restoring vmemmap for a folio.
552+ * Folios that have vmemmap are moved to the non_hvo_folios
553+ * list. Processing of entries stops when the first error is
554+ * encountered. The folio that experienced the error and all
555+ * non-processed folios will remain on folio_list.
556+ */
557+long hugetlb_vmemmap_restore_folios(const struct hstate *h,
558+ struct list_head *folio_list,
559+ struct list_head *non_hvo_folios)
560+{
561+ struct folio *folio, *t_folio;
562+ long restored = 0;
563+ long ret = 0;
564+
565+ list_for_each_entry_safe(folio, t_folio, folio_list, lru) {
566+ if (folio_test_hugetlb_vmemmap_optimized(folio)) {
567+ ret = __hugetlb_vmemmap_restore_folio(h, folio,
568+ VMEMMAP_REMAP_NO_TLB_FLUSH);
569+ if (ret)
570+ break;
571+ restored++;
572+ }
573+
574+ /* Add non-optimized folios to output list */
575+ list_move(&folio->lru, non_hvo_folios);
576+ }
577+
578+ if (restored)
579+ flush_tlb_all();
580+ if (!ret)
581+ ret = restored;
582+ return ret;
583+}
584+
482585 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */
483586 static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head)
484587 {
588+ if (HPageVmemmapOptimized((struct page *)head))
589+ return false;
590+
485591 if (!READ_ONCE(vmemmap_optimize_enabled))
486592 return false;
487593
@@ -535,39 +641,136 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h · 47 unmodified lines
535641 return true;
536642 }
537643
644+static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
645+ struct folio *folio,
646+ struct list_head *vmemmap_pages,
647+ unsigned long flags)
648+{
649+ int ret = 0;
650+ struct page *head = &folio->page;
651+ unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
652+ unsigned long vmemmap_reuse;
653+
654+ VM_WARN_ON_ONCE(!PageHuge(head));
655+ if (!vmemmap_should_optimize(h, head))
656+ return ret;
657+
658+ static_branch_inc(&hugetlb_optimize_vmemmap_key);
659+ /*
660+ * Very Subtle
661+ * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed
662+ * immediately after remapping. As a result, subsequent accesses
663+ * and modifications to struct pages associated with the hugetlb
664+ * page could be to the OLD struct pages. Set the vmemmap optimized
665+ * flag here so that it is copied to the new head page. This keeps
666+ * the old and new struct pages in sync.
667+ * If there is an error during optimization, we will immediately FLUSH
668+ * the TLB and clear the flag below.
669+ */
670+ folio_set_hugetlb_vmemmap_optimized(folio);
671+
672+ vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
673+ vmemmap_reuse = vmemmap_start;
674+ vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
675+
676+ /*
677+ * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
678+ * to the page which @vmemmap_reuse is mapped to. Add pages previously
679+ * mapping the range to vmemmap_pages list so that they can be freed by
680+ * the caller.
681+ */
682+ ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse,
683+ vmemmap_pages, flags);
684+ if (ret) {
685+ static_branch_dec(&hugetlb_optimize_vmemmap_key);
686+ folio_clear_hugetlb_vmemmap_optimized(folio);
687+ }
688+
689+ return ret;
690+}
691+
538692 /**
539− * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages.
693+ * hugetlb_vmemmap_optimize_folio - optimize @folio's vmemmap pages.
540694 * @h: struct hstate.
541− * @head: the head page whose vmemmap pages will be optimized.
695+ * @folio: the folio whose vmemmap pages will be optimized.
542696 *
543− * This function only tries to optimize @head's vmemmap pages and does not
697+ * This function only tries to optimize @folio's vmemmap pages and does not
544698 * guarantee that the optimization will succeed after it returns. The caller
545− * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages
546− * have been optimized.
699+ * can use folio_test_hugetlb_vmemmap_optimized(@folio) to detect if @folio's
700+ * vmemmap pages have been optimized.
547701 */
548−void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
702+void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
703+{
704+ LIST_HEAD(vmemmap_pages);
705+
706+ __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0);
707+ free_vmemmap_page_list(&vmemmap_pages);
708+}
709+
710+static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head)
549711 {
550712 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
551713 unsigned long vmemmap_reuse;
552714
553715 if (!vmemmap_should_optimize(h, head))
554− return;
555−
556− static_branch_inc(&hugetlb_optimize_vmemmap_key);
716+ return 0;
557717
558718 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
559719 vmemmap_reuse = vmemmap_start;
560720 vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
561721
562722 /*
563− * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
564− * to the page which @vmemmap_reuse is mapped to, then free the pages
565− * which the range [@vmemmap_start, @vmemmap_end] is mapped to.
723+ * Split PMDs on the vmemmap virtual address range [@vmemmap_start,
724+ * @vmemmap_end]
566725 */
567− if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse))
568− static_branch_dec(&hugetlb_optimize_vmemmap_key);
569− else
570− SetHPageVmemmapOptimized(head);
726+ return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse);
727+}
728+
729+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
730+{
731+ struct folio *folio;
732+ LIST_HEAD(vmemmap_pages);
733+
734+ list_for_each_entry(folio, folio_list, lru) {
735+ int ret = hugetlb_vmemmap_split(h, &folio->page);
736+
737+ /*
738+ * Spliting the PMD requires allocating a page, thus lets fail
739+ * early once we encounter the first OOM. No point in retrying
740+ * as it can be dynamically done on remap with the memory
741+ * we get back from the vmemmap deduplication.
742+ */
743+ if (ret == -ENOMEM)
744+ break;
745+ }
746+
747+ flush_tlb_all();
748+
749+ list_for_each_entry(folio, folio_list, lru) {
750+ int ret = __hugetlb_vmemmap_optimize_folio(h, folio,
751+ &vmemmap_pages,
752+ VMEMMAP_REMAP_NO_TLB_FLUSH);
753+
754+ /*
755+ * Pages to be freed may have been accumulated. If we
756+ * encounter an ENOMEM, free what we have and try again.
757+ * This can occur in the case that both spliting fails
758+ * halfway and head page allocation also failed. In this
759+ * case __hugetlb_vmemmap_optimize_folio() would free memory
760+ * allowing more vmemmap remaps to occur.
761+ */
762+ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) {
763+ flush_tlb_all();
764+ free_vmemmap_page_list(&vmemmap_pages);
765+ INIT_LIST_HEAD(&vmemmap_pages);
766+ __hugetlb_vmemmap_optimize_folio(h, folio,
767+ &vmemmap_pages,
768+ VMEMMAP_REMAP_NO_TLB_FLUSH);
769+ }
770+ }
771+
772+ flush_tlb_all();
773+ free_vmemmap_page_list(&vmemmap_pages);
571774 }
572775
573776 static struct ctl_table hugetlb_vmemmap_sysctls[] = {
@@ -586,7 +789,7 @@ static int __init hugetlb_vmemmap_init(void) · 12 unmodified lines
586789 const struct hstate *h;
587790
588791 /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */
589− BUILD_BUG_ON(__NR_USED_SUBPAGE * sizeof(struct page) > HUGETLB_VMEMMAP_RESERVE_SIZE);
792+ BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES);
590793
591794 for_each_hstate(h) {
592795 if (hugetlb_vmemmap_optimizable(h)) {
mm/hugetlb_vmemmap.h+24−7
@@ -10,15 +10,20 @@ · 9 unmodified lines
1010 #define _LINUX_HUGETLB_VMEMMAP_H
1111 #include <linux/hugetlb.h>
1212
13−#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
14−int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head);
15−void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head);
16−
1713 /*
1814 * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See
19− * Documentation/vm/vmemmap_dedup.rst.
15+ * Documentation/mm/vmemmap_dedup.rst.
2016 */
2117 #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE
18+#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page))
19+
20+#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
21+int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio);
22+long hugetlb_vmemmap_restore_folios(const struct hstate *h,
23+ struct list_head *folio_list,
24+ struct list_head *non_hvo_folios);
25+void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio);
26+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
2227
2328 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)
2429 {
@@ -38,12 +43,24 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate · 13 unmodified lines
3843 return size > 0 ? size : 0;
3944 }
4045 #else
41−static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
46+static inline int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio)
4247 {
4348 return 0;
4449 }
4550
46−static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
51+static long hugetlb_vmemmap_restore_folios(const struct hstate *h,
52+ struct list_head *folio_list,
53+ struct list_head *non_hvo_folios)
54+{
55+ list_splice_init(folio_list, non_hvo_folios);
56+ return 0;
57+}
58+
59+static inline void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
60+{
61+}
62+
63+static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
4764 {
4865 }
4966
mm/internal.h+136−11
@@ -415,6 +415,15 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) · 414 unmodified lines
415415
416416 void folio_undo_large_rmappable(struct folio *folio);
417417
418+static inline struct folio *page_rmappable_folio(struct page *page)
419+{
420+ struct folio *folio = (struct folio *)page;
421+
422+ if (folio && folio_order(folio) > 1)
423+ folio_prep_large_rmappable(folio);
424+ return folio;
425+}
426+
418427 static inline void prep_compound_head(struct page *page, unsigned int order)
419428 {
420429 struct folio *folio = (struct folio *)page;
@@ -586,6 +595,56 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, · 165 unmodified lines
586595 bool write, int *locked);
587596 extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags,
588597 unsigned long bytes);
598+
599+/*
600+ * NOTE: This function can't tell whether the folio is "fully mapped" in the
601+ * range.
602+ * "fully mapped" means all the pages of folio is associated with the page
603+ * table of range while this function just check whether the folio range is
604+ * within the range [start, end). Function caller needs to do page table
605+ * check if it cares about the page table association.
606+ *
607+ * Typical usage (like mlock or madvise) is:
608+ * Caller knows at least 1 page of folio is associated with page table of VMA
609+ * and the range [start, end) is intersect with the VMA range. Caller wants
610+ * to know whether the folio is fully associated with the range. It calls
611+ * this function to check whether the folio is in the range first. Then checks
612+ * the page table to know whether the folio is fully mapped to the range.
613+ */
614+static inline bool
615+folio_within_range(struct folio *folio, struct vm_area_struct *vma,
616+ unsigned long start, unsigned long end)
617+{
618+ pgoff_t pgoff, addr;
619+ unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
620+
621+ VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio);
622+ if (start > end)
623+ return false;
624+
625+ if (start < vma->vm_start)
626+ start = vma->vm_start;
627+
628+ if (end > vma->vm_end)
629+ end = vma->vm_end;
630+
631+ pgoff = folio_pgoff(folio);
632+
633+ /* if folio start address is not in vma range */
634+ if (!in_range(pgoff, vma->vm_pgoff, vma_pglen))
635+ return false;
636+
637+ addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
638+
639+ return !(addr < start || end - addr < folio_size(folio));
640+}
641+
642+static inline bool
643+folio_within_vma(struct folio *folio, struct vm_area_struct *vma)
644+{
645+ return folio_within_range(folio, vma, vma->vm_start, vma->vm_end);
646+}
647+
589648 /*
590649 * mlock_vma_folio() and munlock_vma_folio():
591650 * should be called with vma's mmap_lock held for read or write,
@@ -594,14 +653,10 @@ extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, · 2 unmodified lines
594653 * mlock is usually called at the end of page_add_*_rmap(), munlock at
595654 * the end of page_remove_rmap(); but new anon folios are managed by
596655 * folio_add_lru_vma() calling mlock_new_folio().
597− *
598− * @compound is used to include pmd mappings of THPs, but filter out
599− * pte mappings of THPs, which cannot be consistently counted: a pte
600− * mapping of the THP head cannot be distinguished by the page alone.
601656 */
602657 void mlock_folio(struct folio *folio);
603658 static inline void mlock_vma_folio(struct folio *folio,
604− struct vm_area_struct *vma, bool compound)
659+ struct vm_area_struct *vma)
605660 {
606661 /*
607662 * The VM_SPECIAL check here serves two purposes.
@@ -611,17 +666,24 @@ static inline void mlock_vma_folio(struct folio *folio, · 3 unmodified lines
611666 * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may
612667 * still be set while VM_SPECIAL bits are added: so ignore it then.
613668 */
614− if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) &&
615− (compound || !folio_test_large(folio)))
669+ if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED))
616670 mlock_folio(folio);
617671 }
618672
619673 void munlock_folio(struct folio *folio);
620674 static inline void munlock_vma_folio(struct folio *folio,
621− struct vm_area_struct *vma, bool compound)
675+ struct vm_area_struct *vma)
622676 {
623− if (unlikely(vma->vm_flags & VM_LOCKED) &&
624− (compound || !folio_test_large(folio)))
677+ /*
678+ * munlock if the function is called. Ideally, we should only
679+ * do munlock if any page of folio is unmapped from VMA and
680+ * cause folio not fully mapped to VMA.
681+ *
682+ * But it's not easy to confirm that's the situation. So we
683+ * always munlock the folio and page reclaim will correct it
684+ * if it's wrong.
685+ */
686+ if (unlikely(vma->vm_flags & VM_LOCKED))
625687 munlock_folio(folio);
626688 }
627689
@@ -930,7 +992,7 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); · 302 unmodified lines
930992
931993 void __vunmap_range_noflush(unsigned long start, unsigned long end);
932994
933−int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
995+int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma,
934996 unsigned long addr, int page_nid, int *flags);
935997
936998 void free_zone_device_page(struct page *page);
@@ -949,6 +1011,13 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · 12 unmodified lines
9491011 unsigned long addr, pmd_t *pmd,
9501012 unsigned int flags);
9511013
1014+/*
1015+ * mm/mmap.c
1016+ */
1017+struct vm_area_struct *vma_merge_extend(struct vma_iterator *vmi,
1018+ struct vm_area_struct *vma,
1019+ unsigned long delta);
1020+
9521021 enum {
9531022 /* mark page accessed */
9541023 FOLL_TOUCH = 1 << 16,
@@ -964,6 +1033,9 @@ enum { · 9 unmodified lines
9641033 FOLL_UNLOCKABLE = 1 << 21,
9651034 };
9661035
1036+#define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
1037+ FOLL_FAST_ONLY | FOLL_UNLOCKABLE)
1038+
9671039 /*
9681040 * Indicates for which pages that are write-protected in the page table,
9691041 * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the
@@ -1154,4 +1226,57 @@ struct vma_prepare { · 184 unmodified lines
11541226 struct vm_area_struct *remove;
11551227 struct vm_area_struct *remove2;
11561228 };
1229+
1230+void __meminit __init_single_page(struct page *page, unsigned long pfn,
1231+ unsigned long zone, int nid);
1232+
1233+/* shrinker related functions */
1234+unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg,
1235+ int priority);
1236+
1237+#ifdef CONFIG_SHRINKER_DEBUG
1238+static inline __printf(2, 0) int shrinker_debugfs_name_alloc(
1239+ struct shrinker *shrinker, const char *fmt, va_list ap)
1240+{
1241+ shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap);
1242+
1243+ return shrinker->name ? 0 : -ENOMEM;
1244+}
1245+
1246+static inline void shrinker_debugfs_name_free(struct shrinker *shrinker)
1247+{
1248+ kfree_const(shrinker->name);
1249+ shrinker->name = NULL;
1250+}
1251+
1252+extern int shrinker_debugfs_add(struct shrinker *shrinker);
1253+extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
1254+ int *debugfs_id);
1255+extern void shrinker_debugfs_remove(struct dentry *debugfs_entry,
1256+ int debugfs_id);
1257+#else /* CONFIG_SHRINKER_DEBUG */
1258+static inline int shrinker_debugfs_add(struct shrinker *shrinker)
1259+{
1260+ return 0;
1261+}
1262+static inline int shrinker_debugfs_name_alloc(struct shrinker *shrinker,
1263+ const char *fmt, va_list ap)
1264+{
1265+ return 0;
1266+}
1267+static inline void shrinker_debugfs_name_free(struct shrinker *shrinker)
1268+{
1269+}
1270+static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
1271+ int *debugfs_id)
1272+{
1273+ *debugfs_id = -1;
1274+ return NULL;
1275+}
1276+static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,
1277+ int debugfs_id)
1278+{
1279+}
1280+#endif /* CONFIG_SHRINKER_DEBUG */
1281+
11571282 #endif /* __MM_INTERNAL_H */
mm/kasan/kasan_test.c+9−10
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>
66 */
77
8−#define pr_fmt(fmt) "kasan_test: " fmt
8+#define pr_fmt(fmt) "kasan: test: " fmt
99
1010 #include <kunit/test.h>
1111 #include <linux/bitops.h>
@@ -91,10 +91,11 @@ static void kasan_test_exit(struct kunit *test) · 79 unmodified lines
9191 }
9292
9393 /**
94− * KUNIT_EXPECT_KASAN_FAIL() - check that the executed expression produces a
95− * KASAN report; causes a test failure otherwise. This relies on a KUnit
96− * resource named "kasan_status". Do not use this name for KUnit resources
97− * outside of KASAN tests.
94+ * KUNIT_EXPECT_KASAN_FAIL - check that the executed expression produces a
95+ * KASAN report; causes a KUnit test failure otherwise.
96+ *
97+ * @test: Currently executing KUnit test.
98+ * @expression: Expression that must produce a KASAN report.
9899 *
99100 * For hardware tag-based KASAN, when a synchronous tag fault happens, tag
100101 * checking is auto-disabled. When this happens, this test handler reenables
@@ -1097,11 +1098,9 @@ static void kasan_bitops_test_and_modify(struct kunit *test, int nr, void *addr) · 996 unmodified lines
10971098 KUNIT_EXPECT_KASAN_FAIL(test, test_and_change_bit(nr, addr));
10981099 KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr));
10991100 KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr));
1100−
1101−#if defined(clear_bit_unlock_is_negative_byte)
1102− KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result =
1103− clear_bit_unlock_is_negative_byte(nr, addr));
1104−#endif
1101+ if (nr < 7)
1102+ KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result =
1103+ xor_unlock_is_negative_byte(1 << nr, addr));
11051104 }
11061105
11071106 static void kasan_bitops_generic(struct kunit *test)
mm/kasan/kasan_test_module.c+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
55 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>
66 */
77
8−#define pr_fmt(fmt) "kasan test: %s " fmt, __func__
8+#define pr_fmt(fmt) "kasan: test: " fmt
99
1010 #include <linux/mman.h>
1111 #include <linux/module.h>
mm/kasan/quarantine.c+3−1
@@ -8,6 +8,8 @@ · 7 unmodified lines
88 * Based on code by Dmitry Chernenkov.
99 */
1010
11+#define pr_fmt(fmt) "kasan: " fmt
12+
1113 #include <linux/gfp.h>
1214 #include <linux/hash.h>
1315 #include <linux/kernel.h>
@@ -414,7 +416,7 @@ static int __init kasan_cpu_quarantine_init(void) · 400 unmodified lines
414416 ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "mm/kasan:online",
415417 kasan_cpu_online, kasan_cpu_offline);
416418 if (ret < 0)
417− pr_err("kasan cpu quarantine register failed [%d]\n", ret);
419+ pr_err("cpu quarantine register failed [%d]\n", ret);
418420 return ret;
419421 }
420422 late_initcall(kasan_cpu_quarantine_init);
mm/kasan/report.c+2−2
@@ -538,7 +538,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty · 537 unmodified lines
538538
539539 start_report(&flags, true);
540540
541− memset(&info, 0, sizeof(info));
541+ __memset(&info, 0, sizeof(info));
542542 info.type = type;
543543 info.access_addr = ptr;
544544 info.access_size = 0;
@@ -576,7 +576,7 @@ bool kasan_report(const void *addr, size_t size, bool is_write, · 31 unmodified lines
576576
577577 start_report(&irq_flags, true);
578578
579− memset(&info, 0, sizeof(info));
579+ __memset(&info, 0, sizeof(info));
580580 info.type = KASAN_REPORT_ACCESS;
581581 info.access_addr = addr;
582582 info.access_size = size;
mm/kasan/report_generic.c+3−3
@@ -220,7 +220,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 219 unmodified lines
220220 const size_t tok_len = sep - *frame_descr;
221221
222222 if (tok_len + 1 > max_tok_len) {
223− pr_err("KASAN internal error: frame description too long: %s\n",
223+ pr_err("internal error: frame description too long: %s\n",
224224 *frame_descr);
225225 return false;
226226 }
@@ -233,7 +233,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 6 unmodified lines
233233 *frame_descr = sep + 1;
234234
235235 if (value != NULL && kstrtoul(token, 10, value)) {
236− pr_err("KASAN internal error: not a valid number: %s\n", token);
236+ pr_err("internal error: not a valid number: %s\n", token);
237237 return false;
238238 }
239239
@@ -323,7 +323,7 @@ static bool __must_check get_address_stack_frame_info(const void *addr, · 83 unmodified lines
323323
324324 frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
325325 if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
326− pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",
326+ pr_err("internal error: frame has invalid marker: %lu\n",
327327 frame[0]);
328328 return false;
329329 }
mm/kasan/shadow.c+1−1
@@ -324,7 +324,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 323 unmodified lines
324324 if (!page)
325325 return -ENOMEM;
326326
327− memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
327+ __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
328328 pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
329329
330330 spin_lock(&init_mm.page_table_lock);
mm/khugepaged.c+68−69
@@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, · 523 unmodified lines
524524 }
525525 }
526526
527−static bool is_refcount_suitable(struct page *page)
527+static bool is_refcount_suitable(struct folio *folio)
528528 {
529529 int expected_refcount;
530530
531− expected_refcount = total_mapcount(page);
532− if (PageSwapCache(page))
533− expected_refcount += compound_nr(page);
531+ expected_refcount = folio_mapcount(folio);
532+ if (folio_test_swapcache(folio))
533+ expected_refcount += folio_nr_pages(folio);
534534
535− return page_count(page) == expected_refcount;
535+ return folio_ref_count(folio) == expected_refcount;
536536 }
537537
538538 static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
@@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 3 unmodified lines
542542 struct list_head *compound_pagelist)
543543 {
544544 struct page *page = NULL;
545+ struct folio *folio = NULL;
545546 pte_t *_pte;
546547 int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
547548 bool writable = false;
@@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 28 unmodified lines
576577 goto out;
577578 }
578579
579− VM_BUG_ON_PAGE(!PageAnon(page), page);
580+ folio = page_folio(page);
581+ VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
580582
581583 if (page_mapcount(page) > 1) {
582584 ++shared;
@@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 5 unmodified lines
588590 }
589591 }
590592
591− if (PageCompound(page)) {
592− struct page *p;
593− page = compound_head(page);
593+ if (folio_test_large(folio)) {
594+ struct folio *f;
594595
595596 /*
596597 * Check if we have dealt with the compound page
597598 * already
598599 */
599− list_for_each_entry(p, compound_pagelist, lru) {
600− if (page == p)
600+ list_for_each_entry(f, compound_pagelist, lru) {
601+ if (folio == f)
601602 goto next;
602603 }
603604 }
@@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 4 unmodified lines
608609 * is needed to serialize against split_huge_page
609610 * when invoked from the VM.
610611 */
611− if (!trylock_page(page)) {
612+ if (!folio_trylock(folio)) {
612613 result = SCAN_PAGE_LOCK;
613614 goto out;
614615 }
@@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 9 unmodified lines
624625 * but not from this process. The other process cannot write to
625626 * the page, only trigger CoW.
626627 */
627− if (!is_refcount_suitable(page)) {
628− unlock_page(page);
628+ if (!is_refcount_suitable(folio)) {
629+ folio_unlock(folio);
629630 result = SCAN_PAGE_COUNT;
630631 goto out;
631632 }
@@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 2 unmodified lines
634635 * Isolate the page to avoid collapsing an hugepage
635636 * currently in use by the VM.
636637 */
637− if (!isolate_lru_page(page)) {
638− unlock_page(page);
638+ if (!folio_isolate_lru(folio)) {
639+ folio_unlock(folio);
639640 result = SCAN_DEL_PAGE_LRU;
640641 goto out;
641642 }
642− mod_node_page_state(page_pgdat(page),
643− NR_ISOLATED_ANON + page_is_file_lru(page),
644− compound_nr(page));
645− VM_BUG_ON_PAGE(!PageLocked(page), page);
646− VM_BUG_ON_PAGE(PageLRU(page), page);
643+ node_stat_mod_folio(folio,
644+ NR_ISOLATED_ANON + folio_is_file_lru(folio),
645+ folio_nr_pages(folio));
646+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
647+ VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
647648
648− if (PageCompound(page))
649− list_add_tail(&page->lru, compound_pagelist);
649+ if (folio_test_large(folio))
650+ list_add_tail(&folio->lru, compound_pagelist);
650651 next:
651652 /*
652653 * If collapse was initiated by khugepaged, check that there is
653654 * enough young pte to justify collapsing the page
654655 */
655656 if (cc->is_khugepaged &&
656− (pte_young(pteval) || page_is_young(page) ||
657− PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm,
657+ (pte_young(pteval) || folio_test_young(folio) ||
658+ folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm,
658659 address)))
659660 referenced++;
660661
@@ -668,13 +669,13 @@ next: · 7 unmodified lines
668669 result = SCAN_LACK_REFERENCED_PAGE;
669670 } else {
670671 result = SCAN_SUCCEED;
671− trace_mm_collapse_huge_page_isolate(page, none_or_zero,
672+ trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero,
672673 referenced, writable, result);
673674 return result;
674675 }
675676 out:
676677 release_pte_pages(pte, _pte, compound_pagelist);
677− trace_mm_collapse_huge_page_isolate(page, none_or_zero,
678+ trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero,
678679 referenced, writable, result);
679680 return result;
680681 }
@@ -887,16 +888,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) · 206 unmodified lines
887888 }
888889 #endif
889890
890−static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node,
891+static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node,
891892 nodemask_t *nmask)
892893 {
893− *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask);
894− if (unlikely(!*hpage)) {
894+ *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask);
895+
896+ if (unlikely(!*folio)) {
895897 count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
896898 return false;
897899 }
898900
899− folio_prep_large_rmappable((struct folio *)*hpage);
900901 count_vm_event(THP_COLLAPSE_ALLOC);
901902 return true;
902903 }
@@ -1063,17 +1064,20 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, · 160 unmodified lines
10631064 int node = hpage_collapse_find_target_node(cc);
10641065 struct folio *folio;
10651066
1066− if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask))
1067+ if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) {
1068+ *hpage = NULL;
10671069 return SCAN_ALLOC_HUGE_PAGE_FAIL;
1070+ }
10681071
1069− folio = page_folio(*hpage);
10701072 if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {
10711073 folio_put(folio);
10721074 *hpage = NULL;
10731075 return SCAN_CGROUP_CHARGE_FAIL;
10741076 }
1075− count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC);
10761077
1078+ count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1);
1079+
1080+ *hpage = folio_page(folio, 0);
10771081 return SCAN_SUCCEED;
10781082 }
10791083
@@ -1247,6 +1251,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 167 unmodified lines
12471251 int result = SCAN_FAIL, referenced = 0;
12481252 int none_or_zero = 0, shared = 0;
12491253 struct page *page = NULL;
1254+ struct folio *folio = NULL;
12501255 unsigned long _address;
12511256 spinlock_t *ptl;
12521257 int node = NUMA_NO_NODE, unmapped = 0;
@@ -1333,29 +1338,28 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 80 unmodified lines
13331338 }
13341339 }
13351340
1336− page = compound_head(page);
1337−
1341+ folio = page_folio(page);
13381342 /*
13391343 * Record which node the original page is from and save this
13401344 * information to cc->node_load[].
13411345 * Khugepaged will allocate hugepage from the node has the max
13421346 * hit record.
13431347 */
1344− node = page_to_nid(page);
1348+ node = folio_nid(folio);
13451349 if (hpage_collapse_scan_abort(node, cc)) {
13461350 result = SCAN_SCAN_ABORT;
13471351 goto out_unmap;
13481352 }
13491353 cc->node_load[node]++;
1350− if (!PageLRU(page)) {
1354+ if (!folio_test_lru(folio)) {
13511355 result = SCAN_PAGE_LRU;
13521356 goto out_unmap;
13531357 }
1354− if (PageLocked(page)) {
1358+ if (folio_test_locked(folio)) {
13551359 result = SCAN_PAGE_LOCK;
13561360 goto out_unmap;
13571361 }
1358− if (!PageAnon(page)) {
1362+ if (!folio_test_anon(folio)) {
13591363 result = SCAN_PAGE_ANON;
13601364 goto out_unmap;
13611365 }
@@ -1370,7 +1374,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 8 unmodified lines
13701374 * has excessive GUP pins (i.e. 512). Anyway the same check
13711375 * will be done again later the risk seems low.
13721376 */
1373− if (!is_refcount_suitable(page)) {
1377+ if (!is_refcount_suitable(folio)) {
13741378 result = SCAN_PAGE_COUNT;
13751379 goto out_unmap;
13761380 }
@@ -1380,8 +1384,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 3 unmodified lines
13801384 * enough young pte to justify collapsing the page
13811385 */
13821386 if (cc->is_khugepaged &&
1383− (pte_young(pteval) || page_is_young(page) ||
1384− PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm,
1387+ (pte_young(pteval) || folio_test_young(folio) ||
1388+ folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm,
13851389 address)))
13861390 referenced++;
13871391 }
@@ -1403,7 +1407,7 @@ out_unmap: · 15 unmodified lines
14031407 *mmap_locked = false;
14041408 }
14051409 out:
1406− trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
1410+ trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced,
14071411 none_or_zero, result, unmapped);
14081412 return result;
14091413 }
@@ -1473,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 63 unmodified lines
14731477 bool notified = false;
14741478 unsigned long haddr = addr & HPAGE_PMD_MASK;
14751479 struct vm_area_struct *vma = vma_lookup(mm, haddr);
1476− struct page *hpage;
1480+ struct folio *folio;
14771481 pte_t *start_pte, *pte;
14781482 pmd_t *pmd, pgt_pmd;
14791483 spinlock_t *pml = NULL, *ptl;
@@ -1506,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 26 unmodified lines
15061510 if (userfaultfd_wp(vma))
15071511 return SCAN_PTE_UFFD_WP;
15081512
1509− hpage = find_lock_page(vma->vm_file->f_mapping,
1513+ folio = filemap_lock_folio(vma->vm_file->f_mapping,
15101514 linear_page_index(vma, haddr));
1511− if (!hpage)
1515+ if (IS_ERR(folio))
15121516 return SCAN_PAGE_NULL;
15131517
1514− if (!PageHead(hpage)) {
1515− result = SCAN_FAIL;
1516− goto drop_hpage;
1517− }
1518−
1519− if (compound_order(hpage) != HPAGE_PMD_ORDER) {
1518+ if (folio_order(folio) != HPAGE_PMD_ORDER) {
15201519 result = SCAN_PAGE_COMPOUND;
1521− goto drop_hpage;
1520+ goto drop_folio;
15221521 }
15231522
15241523 result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
@@ -1532,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 7 unmodified lines
15321531 */
15331532 goto maybe_install_pmd;
15341533 default:
1535− goto drop_hpage;
1534+ goto drop_folio;
15361535 }
15371536
15381537 result = SCAN_FAIL;
15391538 start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);
15401539 if (!start_pte) /* mmap_lock + page lock should prevent this */
1541− goto drop_hpage;
1540+ goto drop_folio;
15421541
15431542 /* step 1: check all mapped PTEs are to the right huge page */
15441543 for (i = 0, addr = haddr, pte = start_pte;
@@ -1563,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 18 unmodified lines
15631562 * Note that uprobe, debugger, or MAP_PRIVATE may change the
15641563 * page table, but the new page will not be a subpage of hpage.
15651564 */
1566− if (hpage + i != page)
1565+ if (folio_page(folio, i) != page)
15671566 goto abort;
15681567 }
15691568
@@ -1578,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 8 unmodified lines
15781577 * page_table_lock) ptl nests inside pml. The less time we hold pml,
15791578 * the better; but userfaultfd's mfill_atomic_pte() on a private VMA
15801579 * inserts a valid as-if-COWed PTE without even looking up page cache.
1581− * So page lock of hpage does not protect from it, so we must not drop
1580+ * So page lock of folio does not protect from it, so we must not drop
15821581 * ptl before pgt_pmd is removed, so uffd private needs pml taken now.
15831582 */
15841583 if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED))
@@ -1602,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines
16021601 continue;
16031602 /*
16041603 * We dropped ptl after the first scan, to do the mmu_notifier:
1605− * page lock stops more PTEs of the hpage being faulted in, but
1604+ * page lock stops more PTEs of the folio being faulted in, but
16061605 * does not stop write faults COWing anon copies from existing
16071606 * PTEs; and does not stop those being swapped out or migrated.
16081607 */
@@ -1611,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines
16111610 goto abort;
16121611 }
16131612 page = vm_normal_page(vma, addr, ptent);
1614− if (hpage + i != page)
1613+ if (folio_page(folio, i) != page)
16151614 goto abort;
16161615
16171616 /*
@@ -1630,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 12 unmodified lines
16301629
16311630 /* step 3: set proper refcount and mm_counters. */
16321631 if (nr_ptes) {
1633− page_ref_sub(hpage, nr_ptes);
1634− add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes);
1632+ folio_ref_sub(folio, nr_ptes);
1633+ add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes);
16351634 }
16361635
16371636 /* step 4: remove empty page table */
@@ -1655,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines
16551654 maybe_install_pmd:
16561655 /* step 5: install pmd entry */
16571656 result = install_pmd
1658− ? set_huge_pmd(vma, haddr, pmd, hpage)
1657+ ? set_huge_pmd(vma, haddr, pmd, &folio->page)
16591658 : SCAN_SUCCEED;
1660− goto drop_hpage;
1659+ goto drop_folio;
16611660 abort:
16621661 if (nr_ptes) {
16631662 flush_tlb_mm(mm);
1664− page_ref_sub(hpage, nr_ptes);
1665− add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes);
1663+ folio_ref_sub(folio, nr_ptes);
1664+ add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes);
16661665 }
16671666 if (start_pte)
16681667 pte_unmap_unlock(start_pte, ptl);
@@ -1670,9 +1669,9 @@ abort: · 1 unmodified line
16701669 spin_unlock(pml);
16711670 if (notified)
16721671 mmu_notifier_invalidate_range_end(&range);
1673−drop_hpage:
1674− unlock_page(hpage);
1675− put_page(hpage);
1672+drop_folio:
1673+ folio_unlock(folio);
1674+ folio_put(folio);
16761675 return result;
16771676 }
16781677
mm/kmemleak.c+98−46
@@ -583,6 +583,19 @@ static void __remove_object(struct kmemleak_object *object) · 582 unmodified lines
583583 object->del_state |= DELSTATE_REMOVED;
584584 }
585585
586+static struct kmemleak_object *__find_and_remove_object(unsigned long ptr,
587+ int alias,
588+ bool is_phys)
589+{
590+ struct kmemleak_object *object;
591+
592+ object = __lookup_object(ptr, alias, is_phys);
593+ if (object)
594+ __remove_object(object);
595+
596+ return object;
597+}
598+
586599 /*
587600 * Look up an object in the object search tree and remove it from both
588601 * object_tree_root (or object_phys_tree_root) and object_list. The
@@ -596,9 +609,7 @@ static struct kmemleak_object *find_and_remove_object(unsigned long ptr, int ali · 7 unmodified lines
596609 struct kmemleak_object *object;
597610
598611 raw_spin_lock_irqsave(&kmemleak_lock, flags);
599− object = __lookup_object(ptr, alias, is_phys);
600− if (object)
601− __remove_object(object);
612+ object = __find_and_remove_object(ptr, alias, is_phys);
602613 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
603614
604615 return object;
@@ -623,27 +634,28 @@ static noinline depot_stack_handle_t set_track_prepare(void) · 18 unmodified lines
623634 return trace_handle;
624635 }
625636
626−/*
627− * Create the metadata (struct kmemleak_object) corresponding to an allocated
628− * memory block and add it to the object_list and object_tree_root (or
629− * object_phys_tree_root).
630− */
631−static void __create_object(unsigned long ptr, size_t size,
632− int min_count, gfp_t gfp, bool is_phys)
637+static struct kmemleak_object *__alloc_object(gfp_t gfp)
633638 {
634− unsigned long flags;
635− struct kmemleak_object *object, *parent;
636− struct rb_node **link, *rb_parent;
637− unsigned long untagged_ptr;
638− unsigned long untagged_objp;
639+ struct kmemleak_object *object;
639640
640641 object = mem_pool_alloc(gfp);
641642 if (!object) {
642643 pr_warn("Cannot allocate a kmemleak_object structure\n");
643644 kmemleak_disable();
644− return;
645645 }
646646
647+ return object;
648+}
649+
650+static int __link_object(struct kmemleak_object *object, unsigned long ptr,
651+ size_t size, int min_count, bool is_phys)
652+{
653+
654+ struct kmemleak_object *parent;
655+ struct rb_node **link, *rb_parent;
656+ unsigned long untagged_ptr;
657+ unsigned long untagged_objp;
658+
647659 INIT_LIST_HEAD(&object->object_list);
648660 INIT_LIST_HEAD(&object->gray_list);
649661 INIT_HLIST_HEAD(&object->area_list);
@@ -680,8 +692,6 @@ static void __create_object(unsigned long ptr, size_t size, · 30 unmodified lines
680692 /* kernel backtrace */
681693 object->trace_handle = set_track_prepare();
682694
683− raw_spin_lock_irqsave(&kmemleak_lock, flags);
684−
685695 untagged_ptr = (unsigned long)kasan_reset_tag((void *)ptr);
686696 /*
687697 * Only update min_addr and max_addr with object
@@ -710,16 +720,38 @@ static void __create_object(unsigned long ptr, size_t size, · 22 unmodified lines
710720 * be freed while the kmemleak_lock is held.
711721 */
712722 dump_object_info(parent);
713− kmem_cache_free(object_cache, object);
714− goto out;
723+ return -EEXIST;
715724 }
716725 }
717726 rb_link_node(&object->rb_node, rb_parent, link);
718727 rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root :
719728 &object_tree_root);
720729 list_add_tail_rcu(&object->object_list, &object_list);
721−out:
730+
731+ return 0;
732+}
733+
734+/*
735+ * Create the metadata (struct kmemleak_object) corresponding to an allocated
736+ * memory block and add it to the object_list and object_tree_root (or
737+ * object_phys_tree_root).
738+ */
739+static void __create_object(unsigned long ptr, size_t size,
740+ int min_count, gfp_t gfp, bool is_phys)
741+{
742+ struct kmemleak_object *object;
743+ unsigned long flags;
744+ int ret;
745+
746+ object = __alloc_object(gfp);
747+ if (!object)
748+ return;
749+
750+ raw_spin_lock_irqsave(&kmemleak_lock, flags);
751+ ret = __link_object(object, ptr, size, min_count, is_phys);
722752 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
753+ if (ret)
754+ mem_pool_free(object);
723755 }
724756
725757 /* Create kmemleak object which allocated with virtual address. */
@@ -782,16 +814,25 @@ static void delete_object_full(unsigned long ptr) · 56 unmodified lines
782814 */
783815 static void delete_object_part(unsigned long ptr, size_t size, bool is_phys)
784816 {
785− struct kmemleak_object *object;
786− unsigned long start, end;
817+ struct kmemleak_object *object, *object_l, *object_r;
818+ unsigned long start, end, flags;
819+
820+ object_l = __alloc_object(GFP_KERNEL);
821+ if (!object_l)
822+ return;
787823
788− object = find_and_remove_object(ptr, 1, is_phys);
824+ object_r = __alloc_object(GFP_KERNEL);
825+ if (!object_r)
826+ goto out;
827+
828+ raw_spin_lock_irqsave(&kmemleak_lock, flags);
829+ object = __find_and_remove_object(ptr, 1, is_phys);
789830 if (!object) {
790831 #ifdef DEBUG
791832 kmemleak_warn("Partially freeing unknown object at 0x%08lx (size %zu)\n",
792833 ptr, size);
793834 #endif
794− return;
835+ goto unlock;
795836 }
796837
797838 /*
@@ -801,14 +842,25 @@ static void delete_object_part(unsigned long ptr, size_t size, bool is_phys) · 3 unmodified lines
801842 */
802843 start = object->pointer;
803844 end = object->pointer + object->size;
804− if (ptr > start)
805− __create_object(start, ptr - start, object->min_count,
806− GFP_KERNEL, is_phys);
807− if (ptr + size < end)
808− __create_object(ptr + size, end - ptr - size, object->min_count,
809− GFP_KERNEL, is_phys);
845+ if ((ptr > start) &&
846+ !__link_object(object_l, start, ptr - start,
847+ object->min_count, is_phys))
848+ object_l = NULL;
849+ if ((ptr + size < end) &&
850+ !__link_object(object_r, ptr + size, end - ptr - size,
851+ object->min_count, is_phys))
852+ object_r = NULL;
853+
854+unlock:
855+ raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
856+ if (object)
857+ __delete_object(object);
810858
811− __delete_object(object);
859+out:
860+ if (object_l)
861+ mem_pool_free(object_l);
862+ if (object_r)
863+ mem_pool_free(object_r);
812864 }
813865
814866 static void __paint_it(struct kmemleak_object *object, int color)
@@ -975,7 +1027,7 @@ static void object_no_scan(unsigned long ptr) · 160 unmodified lines
9751027 void __ref kmemleak_alloc(const void *ptr, size_t size, int min_count,
9761028 gfp_t gfp)
9771029 {
978− pr_debug("%s(0x%p, %zu, %d)\n", __func__, ptr, size, min_count);
1030+ pr_debug("%s(0x%px, %zu, %d)\n", __func__, ptr, size, min_count);
9791031
9801032 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
9811033 create_object((unsigned long)ptr, size, min_count, gfp);
@@ -996,7 +1048,7 @@ void __ref kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, · 14 unmodified lines
9961048 {
9971049 unsigned int cpu;
9981050
999− pr_debug("%s(0x%p, %zu)\n", __func__, ptr, size);
1051+ pr_debug("%s(0x%px, %zu)\n", __func__, ptr, size);
10001052
10011053 /*
10021054 * Percpu allocations are only scanned and not reported as leaks
@@ -1020,7 +1072,7 @@ EXPORT_SYMBOL_GPL(kmemleak_alloc_percpu); · 17 unmodified lines
10201072 */
10211073 void __ref kmemleak_vmalloc(const struct vm_struct *area, size_t size, gfp_t gfp)
10221074 {
1023− pr_debug("%s(0x%p, %zu)\n", __func__, area, size);
1075+ pr_debug("%s(0x%px, %zu)\n", __func__, area, size);
10241076
10251077 /*
10261078 * A min_count = 2 is needed because vm_struct contains a reference to
@@ -1043,7 +1095,7 @@ EXPORT_SYMBOL_GPL(kmemleak_vmalloc); · 16 unmodified lines
10431095 */
10441096 void __ref kmemleak_free(const void *ptr)
10451097 {
1046− pr_debug("%s(0x%p)\n", __func__, ptr);
1098+ pr_debug("%s(0x%px)\n", __func__, ptr);
10471099
10481100 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))
10491101 delete_object_full((unsigned long)ptr);
@@ -1061,7 +1113,7 @@ EXPORT_SYMBOL_GPL(kmemleak_free); · 11 unmodified lines
10611113 */
10621114 void __ref kmemleak_free_part(const void *ptr, size_t size)
10631115 {
1064− pr_debug("%s(0x%p)\n", __func__, ptr);
1116+ pr_debug("%s(0x%px)\n", __func__, ptr);
10651117
10661118 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
10671119 delete_object_part((unsigned long)ptr, size, false);
@@ -1079,7 +1131,7 @@ void __ref kmemleak_free_percpu(const void __percpu *ptr) · 11 unmodified lines
10791131 {
10801132 unsigned int cpu;
10811133
1082− pr_debug("%s(0x%p)\n", __func__, ptr);
1134+ pr_debug("%s(0x%px)\n", __func__, ptr);
10831135
10841136 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))
10851137 for_each_possible_cpu(cpu)
@@ -1100,7 +1152,7 @@ void __ref kmemleak_update_trace(const void *ptr) · 14 unmodified lines
11001152 struct kmemleak_object *object;
11011153 unsigned long flags;
11021154
1103− pr_debug("%s(0x%p)\n", __func__, ptr);
1155+ pr_debug("%s(0x%px)\n", __func__, ptr);
11041156
11051157 if (!kmemleak_enabled || IS_ERR_OR_NULL(ptr))
11061158 return;
@@ -1131,7 +1183,7 @@ EXPORT_SYMBOL(kmemleak_update_trace); · 24 unmodified lines
11311183 */
11321184 void __ref kmemleak_not_leak(const void *ptr)
11331185 {
1134− pr_debug("%s(0x%p)\n", __func__, ptr);
1186+ pr_debug("%s(0x%px)\n", __func__, ptr);
11351187
11361188 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
11371189 make_gray_object((unsigned long)ptr);
@@ -1149,7 +1201,7 @@ EXPORT_SYMBOL(kmemleak_not_leak); · 11 unmodified lines
11491201 */
11501202 void __ref kmemleak_ignore(const void *ptr)
11511203 {
1152− pr_debug("%s(0x%p)\n", __func__, ptr);
1204+ pr_debug("%s(0x%px)\n", __func__, ptr);
11531205
11541206 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
11551207 make_black_object((unsigned long)ptr, false);
@@ -1169,7 +1221,7 @@ EXPORT_SYMBOL(kmemleak_ignore); · 13 unmodified lines
11691221 */
11701222 void __ref kmemleak_scan_area(const void *ptr, size_t size, gfp_t gfp)
11711223 {
1172− pr_debug("%s(0x%p)\n", __func__, ptr);
1224+ pr_debug("%s(0x%px)\n", __func__, ptr);
11731225
11741226 if (kmemleak_enabled && ptr && size && !IS_ERR(ptr))
11751227 add_scan_area((unsigned long)ptr, size, gfp);
@@ -1187,7 +1239,7 @@ EXPORT_SYMBOL(kmemleak_scan_area); · 11 unmodified lines
11871239 */
11881240 void __ref kmemleak_no_scan(const void *ptr)
11891241 {
1190− pr_debug("%s(0x%p)\n", __func__, ptr);
1242+ pr_debug("%s(0x%px)\n", __func__, ptr);
11911243
11921244 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
11931245 object_no_scan((unsigned long)ptr);
@@ -1203,7 +1255,7 @@ EXPORT_SYMBOL(kmemleak_no_scan); · 9 unmodified lines
12031255 */
12041256 void __ref kmemleak_alloc_phys(phys_addr_t phys, size_t size, gfp_t gfp)
12051257 {
1206− pr_debug("%s(0x%pa, %zu)\n", __func__, &phys, size);
1258+ pr_debug("%s(0x%px, %zu)\n", __func__, &phys, size);
12071259
12081260 if (kmemleak_enabled)
12091261 /*
@@ -1223,7 +1275,7 @@ EXPORT_SYMBOL(kmemleak_alloc_phys); · 13 unmodified lines
12231275 */
12241276 void __ref kmemleak_free_part_phys(phys_addr_t phys, size_t size)
12251277 {
1226− pr_debug("%s(0x%pa)\n", __func__, &phys);
1278+ pr_debug("%s(0x%px)\n", __func__, &phys);
12271279
12281280 if (kmemleak_enabled)
12291281 delete_object_part((unsigned long)phys, size, true);
@@ -1237,7 +1289,7 @@ EXPORT_SYMBOL(kmemleak_free_part_phys); · 7 unmodified lines
12371289 */
12381290 void __ref kmemleak_ignore_phys(phys_addr_t phys)
12391291 {
1240− pr_debug("%s(0x%pa)\n", __func__, &phys);
1292+ pr_debug("%s(0x%px)\n", __func__, &phys);
12411293
12421294 if (kmemleak_enabled)
12431295 make_black_object((unsigned long)phys, true);
mm/kmsan/core.c+31−96
@@ -83,131 +83,66 @@ depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, · 82 unmodified lines
8383 /* Copy the metadata following the memmove() behavior. */
8484 void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n)
8585 {
86+ depot_stack_handle_t prev_old_origin = 0, prev_new_origin = 0;
87+ int i, iter, step, src_off, dst_off, oiter_src, oiter_dst;
8688 depot_stack_handle_t old_origin = 0, new_origin = 0;
87− int src_slots, dst_slots, i, iter, step, skip_bits;
8889 depot_stack_handle_t *origin_src, *origin_dst;
89− void *shadow_src, *shadow_dst;
90− u32 *align_shadow_src, shadow;
90+ u8 *shadow_src, *shadow_dst;
91+ u32 *align_shadow_dst;
9192 bool backwards;
9293
9394 shadow_dst = kmsan_get_metadata(dst, KMSAN_META_SHADOW);
9495 if (!shadow_dst)
9596 return;
9697 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n));
98+ align_shadow_dst =
99+ (u32 *)ALIGN_DOWN((u64)shadow_dst, KMSAN_ORIGIN_SIZE);
97100
98101 shadow_src = kmsan_get_metadata(src, KMSAN_META_SHADOW);
99102 if (!shadow_src) {
100− /*
101− * @src is untracked: zero out destination shadow, ignore the
102− * origins, we're done.
103− */
104− __memset(shadow_dst, 0, n);
103+ /* @src is untracked: mark @dst as initialized. */
104+ kmsan_internal_unpoison_memory(dst, n, /*checked*/ false);
105105 return;
106106 }
107107 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n));
108108
109− __memmove(shadow_dst, shadow_src, n);
110−
111109 origin_dst = kmsan_get_metadata(dst, KMSAN_META_ORIGIN);
112110 origin_src = kmsan_get_metadata(src, KMSAN_META_ORIGIN);
113111 KMSAN_WARN_ON(!origin_dst || !origin_src);
114− src_slots = (ALIGN((u64)src + n, KMSAN_ORIGIN_SIZE) -
115− ALIGN_DOWN((u64)src, KMSAN_ORIGIN_SIZE)) /
116− KMSAN_ORIGIN_SIZE;
117− dst_slots = (ALIGN((u64)dst + n, KMSAN_ORIGIN_SIZE) -
118− ALIGN_DOWN((u64)dst, KMSAN_ORIGIN_SIZE)) /
119− KMSAN_ORIGIN_SIZE;
120− KMSAN_WARN_ON((src_slots < 1) || (dst_slots < 1));
121− KMSAN_WARN_ON((src_slots - dst_slots > 1) ||
122− (dst_slots - src_slots < -1));
123112
124113 backwards = dst > src;
125− i = backwards ? min(src_slots, dst_slots) - 1 : 0;
126− iter = backwards ? -1 : 1;
127−
128− align_shadow_src =
129− (u32 *)ALIGN_DOWN((u64)shadow_src, KMSAN_ORIGIN_SIZE);
130− for (step = 0; step < min(src_slots, dst_slots); step++, i += iter) {
131− KMSAN_WARN_ON(i < 0);
132− shadow = align_shadow_src[i];
133− if (i == 0) {
134− /*
135− * If @src isn't aligned on KMSAN_ORIGIN_SIZE, don't
136− * look at the first @src % KMSAN_ORIGIN_SIZE bytes
137− * of the first shadow slot.
138− */
139− skip_bits = ((u64)src % KMSAN_ORIGIN_SIZE) * 8;
140− shadow = (shadow >> skip_bits) << skip_bits;
114+ step = backwards ? -1 : 1;
115+ iter = backwards ? n - 1 : 0;
116+ src_off = (u64)src % KMSAN_ORIGIN_SIZE;
117+ dst_off = (u64)dst % KMSAN_ORIGIN_SIZE;
118+
119+ /* Copy shadow bytes one by one, updating the origins if necessary. */
120+ for (i = 0; i < n; i++, iter += step) {
121+ oiter_src = (iter + src_off) / KMSAN_ORIGIN_SIZE;
122+ oiter_dst = (iter + dst_off) / KMSAN_ORIGIN_SIZE;
123+ if (!shadow_src[iter]) {
124+ shadow_dst[iter] = 0;
125+ if (!align_shadow_dst[oiter_dst])
126+ origin_dst[oiter_dst] = 0;
127+ continue;
141128 }
142− if (i == src_slots - 1) {
143− /*
144− * If @src + n isn't aligned on
145− * KMSAN_ORIGIN_SIZE, don't look at the last
146− * (@src + n) % KMSAN_ORIGIN_SIZE bytes of the
147− * last shadow slot.
148− */
149− skip_bits = (((u64)src + n) % KMSAN_ORIGIN_SIZE) * 8;
150− shadow = (shadow << skip_bits) >> skip_bits;
151− }
152− /*
153− * Overwrite the origin only if the corresponding
154− * shadow is nonempty.
155− */
156− if (origin_src[i] && (origin_src[i] != old_origin) && shadow) {
157− old_origin = origin_src[i];
158− new_origin = kmsan_internal_chain_origin(old_origin);
129+ shadow_dst[iter] = shadow_src[iter];
130+ old_origin = origin_src[oiter_src];
131+ if (old_origin == prev_old_origin)
132+ new_origin = prev_new_origin;
133+ else {
159134 /*
160135 * kmsan_internal_chain_origin() may return
161136 * NULL, but we don't want to lose the previous
162137 * origin value.
163138 */
139+ new_origin = kmsan_internal_chain_origin(old_origin);
164140 if (!new_origin)
165141 new_origin = old_origin;
166142 }
167− if (shadow)
168− origin_dst[i] = new_origin;
169− else
170− origin_dst[i] = 0;
171− }
172− /*
173− * If dst_slots is greater than src_slots (i.e.
174− * dst_slots == src_slots + 1), there is an extra origin slot at the
175− * beginning or end of the destination buffer, for which we take the
176− * origin from the previous slot.
177− * This is only done if the part of the source shadow corresponding to
178− * slot is non-zero.
179− *
180− * E.g. if we copy 8 aligned bytes that are marked as uninitialized
181− * and have origins o111 and o222, to an unaligned buffer with offset 1,
182− * these two origins are copied to three origin slots, so one of then
183− * needs to be duplicated, depending on the copy direction (@backwards)
184− *
185− * src shadow: |uuuu|uuuu|....|
186− * src origin: |o111|o222|....|
187− *
188− * backwards = 0:
189− * dst shadow: |.uuu|uuuu|u...|
190− * dst origin: |....|o111|o222| - fill the empty slot with o111
191− * backwards = 1:
192− * dst shadow: |.uuu|uuuu|u...|
193− * dst origin: |o111|o222|....| - fill the empty slot with o222
194− */
195− if (src_slots < dst_slots) {
196− if (backwards) {
197− shadow = align_shadow_src[src_slots - 1];
198− skip_bits = (((u64)dst + n) % KMSAN_ORIGIN_SIZE) * 8;
199− shadow = (shadow << skip_bits) >> skip_bits;
200− if (shadow)
201− /* src_slots > 0, therefore dst_slots is at least 2 */
202− origin_dst[dst_slots - 1] =
203− origin_dst[dst_slots - 2];
204− } else {
205− shadow = align_shadow_src[0];
206− skip_bits = ((u64)dst % KMSAN_ORIGIN_SIZE) * 8;
207− shadow = (shadow >> skip_bits) << skip_bits;
208− if (shadow)
209− origin_dst[0] = origin_dst[1];
210− }
143+ origin_dst[oiter_dst] = new_origin;
144+ prev_new_origin = new_origin;
145+ prev_old_origin = old_origin;
211146 }
212147 }
213148
mm/kmsan/kmsan_test.c+71−38
@@ -67,6 +67,17 @@ static bool report_available(void) · 66 unmodified lines
6767 return READ_ONCE(observed.available);
6868 }
6969
70+/* Reset observed.available, so that the test can trigger another report. */
71+static void report_reset(void)
72+{
73+ unsigned long flags;
74+
75+ spin_lock_irqsave(&observed.lock, flags);
76+ WRITE_ONCE(observed.available, false);
77+ observed.ignore = false;
78+ spin_unlock_irqrestore(&observed.lock, flags);
79+}
80+
7081 /* Information we expect in a report. */
7182 struct expect_report {
7283 const char *error_type; /* Error type. */
@@ -407,33 +418,25 @@ static void test_printk(struct kunit *test) · 334 unmodified lines
407418 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
408419 }
409420
410−/*
411− * Prevent the compiler from optimizing @var away. Without this, Clang may
412− * notice that @var is uninitialized and drop memcpy() calls that use it.
413− *
414− * There is OPTIMIZER_HIDE_VAR() in linux/compier.h that we cannot use here,
415− * because it is implemented as inline assembly receiving @var as a parameter
416− * and will enforce a KMSAN check. Same is true for e.g. barrier_data(var).
417− */
418−#define DO_NOT_OPTIMIZE(var) barrier()
421+/* Prevent the compiler from inlining a memcpy() call. */
422+static noinline void *memcpy_noinline(volatile void *dst,
423+ const volatile void *src, size_t size)
424+{
425+ return memcpy((void *)dst, (const void *)src, size);
426+}
419427
420−/*
421− * Test case: ensure that memcpy() correctly copies initialized values.
422− * Also serves as a regression test to ensure DO_NOT_OPTIMIZE() does not cause
423− * extra checks.
424− */
428+/* Test case: ensure that memcpy() correctly copies initialized values. */
425429 static void test_init_memcpy(struct kunit *test)
426430 {
427431 EXPECTATION_NO_REPORT(expect);
428− volatile int src;
429− volatile int dst = 0;
432+ volatile long long src;
433+ volatile long long dst = 0;
430434
431− DO_NOT_OPTIMIZE(src);
432435 src = 1;
433436 kunit_info(
434437 test,
435438 "memcpy()ing aligned initialized src to aligned dst (no reports)\n");
436− memcpy((void *)&dst, (void *)&src, sizeof(src));
439+ memcpy_noinline((void *)&dst, (void *)&src, sizeof(src));
437440 kmsan_check_memory((void *)&dst, sizeof(dst));
438441 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
439442 }
@@ -451,8 +454,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 11 unmodified lines
451454 kunit_info(
452455 test,
453456 "memcpy()ing aligned uninit src to aligned dst (UMR report)\n");
454− DO_NOT_OPTIMIZE(uninit_src);
455− memcpy((void *)&dst, (void *)&uninit_src, sizeof(uninit_src));
457+ memcpy_noinline((void *)&dst, (void *)&uninit_src, sizeof(uninit_src));
456458 kmsan_check_memory((void *)&dst, sizeof(dst));
457459 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
458460 }
@@ -463,7 +465,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 4 unmodified lines
463465 *
464466 * Copying aligned 4-byte value to an unaligned one leads to touching two
465467 * aligned 4-byte values. This test case checks that KMSAN correctly reports an
466− * error on the first of the two values.
468+ * error on the mentioned two values.
467469 */
468470 static void test_memcpy_aligned_to_unaligned(struct kunit *test)
469471 {
@@ -474,33 +476,65 @@ static void test_memcpy_aligned_to_unaligned(struct kunit *test) · 4 unmodified lines
474476 kunit_info(
475477 test,
476478 "memcpy()ing aligned uninit src to unaligned dst (UMR report)\n");
477− DO_NOT_OPTIMIZE(uninit_src);
478− memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src));
479+ kmsan_check_memory((void *)&uninit_src, sizeof(uninit_src));
480+ memcpy_noinline((void *)&dst[1], (void *)&uninit_src,
481+ sizeof(uninit_src));
479482 kmsan_check_memory((void *)dst, 4);
480483 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
484+ report_reset();
485+ kmsan_check_memory((void *)&dst[4], sizeof(uninit_src));
486+ KUNIT_EXPECT_TRUE(test, report_matches(&expect));
481487 }
482488
483489 /*
484− * Test case: ensure that memcpy() correctly copies uninitialized values between
485− * aligned `src` and unaligned `dst`.
490+ * Test case: ensure that origin slots do not accidentally get overwritten with
491+ * zeroes during memcpy().
486492 *
487− * Copying aligned 4-byte value to an unaligned one leads to touching two
488− * aligned 4-byte values. This test case checks that KMSAN correctly reports an
489− * error on the second of the two values.
493+ * Previously, when copying memory from an aligned buffer to an unaligned one,
494+ * if there were zero origins corresponding to zero shadow values in the source
495+ * buffer, they could have ended up being copied to nonzero shadow values in the
496+ * destination buffer:
497+ *
498+ * memcpy(0xffff888080a00000, 0xffff888080900002, 8)
499+ *
500+ * src (0xffff888080900002): ..xx .... xx..
501+ * src origins: o111 0000 o222
502+ * dst (0xffff888080a00000): xx.. ..xx
503+ * dst origins: o111 0000
504+ * (or 0000 o222)
505+ *
506+ * (here . stands for an initialized byte, and x for an uninitialized one.
507+ *
508+ * Ensure that this does not happen anymore, and for both destination bytes
509+ * the origin is nonzero (i.e. KMSAN reports an error).
490510 */
491−static void test_memcpy_aligned_to_unaligned2(struct kunit *test)
511+static void test_memcpy_initialized_gap(struct kunit *test)
492512 {
493− EXPECTATION_UNINIT_VALUE_FN(expect,
494− "test_memcpy_aligned_to_unaligned2");
495− volatile int uninit_src;
513+ EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_initialized_gap");
514+ volatile char uninit_src[12];
496515 volatile char dst[8] = { 0 };
497516
498517 kunit_info(
499518 test,
500− "memcpy()ing aligned uninit src to unaligned dst - part 2 (UMR report)\n");
501− DO_NOT_OPTIMIZE(uninit_src);
502− memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src));
503− kmsan_check_memory((void *)&dst[4], sizeof(uninit_src));
519+ "unaligned 4-byte initialized value gets a nonzero origin after memcpy() - (2 UMR reports)\n");
520+
521+ uninit_src[0] = 42;
522+ uninit_src[1] = 42;
523+ uninit_src[4] = 42;
524+ uninit_src[5] = 42;
525+ uninit_src[6] = 42;
526+ uninit_src[7] = 42;
527+ uninit_src[10] = 42;
528+ uninit_src[11] = 42;
529+ memcpy_noinline((void *)&dst[0], (void *)&uninit_src[2], 8);
530+
531+ kmsan_check_memory((void *)&dst[0], 4);
532+ KUNIT_EXPECT_TRUE(test, report_matches(&expect));
533+ report_reset();
534+ kmsan_check_memory((void *)&dst[2], 4);
535+ KUNIT_EXPECT_FALSE(test, report_matches(&expect));
536+ report_reset();
537+ kmsan_check_memory((void *)&dst[4], 4);
504538 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
505539 }
506540
@@ -513,7 +547,6 @@ static void test_memcpy_aligned_to_unaligned2(struct kunit *test) · 6 unmodified lines
513547 \
514548 kunit_info(test, \
515549 "memset" #size "() should initialize memory\n"); \
516− DO_NOT_OPTIMIZE(uninit); \
517550 memset##size((uint##size##_t *)&uninit, 0, 1); \
518551 kmsan_check_memory((void *)&uninit, sizeof(uninit)); \
519552 KUNIT_EXPECT_TRUE(test, report_matches(&expect)); \
@@ -598,7 +631,7 @@ static struct kunit_case kmsan_test_cases[] = { · 78 unmodified lines
598631 KUNIT_CASE(test_init_memcpy),
599632 KUNIT_CASE(test_memcpy_aligned_to_aligned),
600633 KUNIT_CASE(test_memcpy_aligned_to_unaligned),
601− KUNIT_CASE(test_memcpy_aligned_to_unaligned2),
634+ KUNIT_CASE(test_memcpy_initialized_gap),
602635 KUNIT_CASE(test_memset16),
603636 KUNIT_CASE(test_memset32),
604637 KUNIT_CASE(test_memset64),
mm/kmsan/shadow.c+7−2
@@ -285,12 +285,17 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end) · 284 unmodified lines
285285 size = PAGE_ALIGN((u64)end - (u64)start);
286286 shadow = memblock_alloc(size, PAGE_SIZE);
287287 origin = memblock_alloc(size, PAGE_SIZE);
288+
289+ if (!shadow || !origin)
290+ panic("%s: Failed to allocate metadata memory for early boot range of size %llu",
291+ __func__, size);
292+
288293 for (u64 addr = 0; addr < size; addr += PAGE_SIZE) {
289294 page = virt_to_page_or_null((char *)start + addr);
290− shadow_p = virt_to_page_or_null((char *)shadow + addr);
295+ shadow_p = virt_to_page((char *)shadow + addr);
291296 set_no_shadow_origin_page(shadow_p);
292297 shadow_page_for(page) = shadow_p;
293− origin_p = virt_to_page_or_null((char *)origin + addr);
298+ origin_p = virt_to_page((char *)origin + addr);
294299 set_no_shadow_origin_page(origin_p);
295300 origin_page_for(page) = origin_p;
296301 }
mm/ksm.c+116−0
@@ -56,6 +56,8 @@ · 55 unmodified lines
5656 #define DO_NUMA(x) do { } while (0)
5757 #endif
5858
59+typedef u8 rmap_age_t;
60+
5961 /**
6062 * DOC: Overview
6163 *
@@ -193,6 +195,8 @@ struct ksm_stable_node { · 131 unmodified lines
193195 * @node: rb node of this rmap_item in the unstable tree
194196 * @head: pointer to stable_node heading this list in the stable tree
195197 * @hlist: link into hlist of rmap_items hanging off that stable_node
198+ * @age: number of scan iterations since creation
199+ * @remaining_skips: how many scans to skip
196200 */
197201 struct ksm_rmap_item {
198202 struct ksm_rmap_item *rmap_list;
@@ -205,6 +209,8 @@ struct ksm_rmap_item { · 6 unmodified lines
205209 struct mm_struct *mm;
206210 unsigned long address; /* + low bits used for flags below */
207211 unsigned int oldchecksum; /* when unstable */
212+ rmap_age_t age;
213+ rmap_age_t remaining_skips;
208214 union {
209215 struct rb_node node; /* when node of unstable tree */
210216 struct { /* when listed from stable tree */
@@ -281,9 +287,16 @@ static unsigned int zero_checksum __read_mostly; · 70 unmodified lines
281287 /* Whether to merge empty (zeroed) pages with actual zero pages */
282288 static bool ksm_use_zero_pages __read_mostly;
283289
290+/* Skip pages that couldn't be de-duplicated previously */
291+/* Default to true at least temporarily, for testing */
292+static bool ksm_smart_scan = true;
293+
284294 /* The number of zero pages which is placed by KSM */
285295 unsigned long ksm_zero_pages;
286296
297+/* The number of pages that have been skipped due to "smart scanning" */
298+static unsigned long ksm_pages_skipped;
299+
287300 #ifdef CONFIG_NUMA
288301 /* Zeroed when merging across nodes is not allowed */
289302 static unsigned int ksm_merge_across_nodes = 1;
@@ -2305,6 +2318,74 @@ static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, · 2015 unmodified lines
23052318 return rmap_item;
23062319 }
23072320
2321+/*
2322+ * Calculate skip age for the ksm page age. The age determines how often
2323+ * de-duplicating has already been tried unsuccessfully. If the age is
2324+ * smaller, the scanning of this page is skipped for less scans.
2325+ *
2326+ * @age: rmap_item age of page
2327+ */
2328+static unsigned int skip_age(rmap_age_t age)
2329+{
2330+ if (age <= 3)
2331+ return 1;
2332+ if (age <= 5)
2333+ return 2;
2334+ if (age <= 8)
2335+ return 4;
2336+
2337+ return 8;
2338+}
2339+
2340+/*
2341+ * Determines if a page should be skipped for the current scan.
2342+ *
2343+ * @page: page to check
2344+ * @rmap_item: associated rmap_item of page
2345+ */
2346+static bool should_skip_rmap_item(struct page *page,
2347+ struct ksm_rmap_item *rmap_item)
2348+{
2349+ rmap_age_t age;
2350+
2351+ if (!ksm_smart_scan)
2352+ return false;
2353+
2354+ /*
2355+ * Never skip pages that are already KSM; pages cmp_and_merge_page()
2356+ * will essentially ignore them, but we still have to process them
2357+ * properly.
2358+ */
2359+ if (PageKsm(page))
2360+ return false;
2361+
2362+ age = rmap_item->age;
2363+ if (age != U8_MAX)
2364+ rmap_item->age++;
2365+
2366+ /*
2367+ * Smaller ages are not skipped, they need to get a chance to go
2368+ * through the different phases of the KSM merging.
2369+ */
2370+ if (age < 3)
2371+ return false;
2372+
2373+ /*
2374+ * Are we still allowed to skip? If not, then don't skip it
2375+ * and determine how much more often we are allowed to skip next.
2376+ */
2377+ if (!rmap_item->remaining_skips) {
2378+ rmap_item->remaining_skips = skip_age(age);
2379+ return false;
2380+ }
2381+
2382+ /* Skip this page */
2383+ ksm_pages_skipped++;
2384+ rmap_item->remaining_skips--;
2385+ remove_rmap_item_from_tree(rmap_item);
2386+ return true;
2387+}
2388+
23082389 static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
23092390 {
23102391 struct mm_struct *mm;
@@ -2409,6 +2490,10 @@ next_mm: · 98 unmodified lines
24092490 if (rmap_item) {
24102491 ksm_scan.rmap_list =
24112492 &rmap_item->rmap_list;
2493+
2494+ if (should_skip_rmap_item(*page, rmap_item))
2495+ goto next_page;
2496+
24122497 ksm_scan.address += PAGE_SIZE;
24132498 } else
24142499 put_page(*page);
@@ -3383,6 +3468,13 @@ static ssize_t pages_volatile_show(struct kobject *kobj, · 968 unmodified lines
33833468 }
33843469 KSM_ATTR_RO(pages_volatile);
33853470
3471+static ssize_t pages_skipped_show(struct kobject *kobj,
3472+ struct kobj_attribute *attr, char *buf)
3473+{
3474+ return sysfs_emit(buf, "%lu\n", ksm_pages_skipped);
3475+}
3476+KSM_ATTR_RO(pages_skipped);
3477+
33863478 static ssize_t ksm_zero_pages_show(struct kobject *kobj,
33873479 struct kobj_attribute *attr, char *buf)
33883480 {
@@ -3449,6 +3541,28 @@ static ssize_t full_scans_show(struct kobject *kobj, · 60 unmodified lines
34493541 }
34503542 KSM_ATTR_RO(full_scans);
34513543
3544+static ssize_t smart_scan_show(struct kobject *kobj,
3545+ struct kobj_attribute *attr, char *buf)
3546+{
3547+ return sysfs_emit(buf, "%u\n", ksm_smart_scan);
3548+}
3549+
3550+static ssize_t smart_scan_store(struct kobject *kobj,
3551+ struct kobj_attribute *attr,
3552+ const char *buf, size_t count)
3553+{
3554+ int err;
3555+ bool value;
3556+
3557+ err = kstrtobool(buf, &value);
3558+ if (err)
3559+ return -EINVAL;
3560+
3561+ ksm_smart_scan = value;
3562+ return count;
3563+}
3564+KSM_ATTR(smart_scan);
3565+
34523566 static struct attribute *ksm_attrs[] = {
34533567 &sleep_millisecs_attr.attr,
34543568 &pages_to_scan_attr.attr,
@@ -3458,6 +3572,7 @@ static struct attribute *ksm_attrs[] = { · 3 unmodified lines
34583572 &pages_sharing_attr.attr,
34593573 &pages_unshared_attr.attr,
34603574 &pages_volatile_attr.attr,
3575+ &pages_skipped_attr.attr,
34613576 &ksm_zero_pages_attr.attr,
34623577 &full_scans_attr.attr,
34633578 #ifdef CONFIG_NUMA
@@ -3469,6 +3584,7 @@ static struct attribute *ksm_attrs[] = { · 5 unmodified lines
34693584 &stable_node_chains_prune_millisecs_attr.attr,
34703585 &use_zero_pages_attr.attr,
34713586 &general_profit_attr.attr,
3587+ &smart_scan_attr.attr,
34723588 NULL,
34733589 };
34743590
mm/madvise.c+6−27
@@ -141,7 +141,6 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 140 unmodified lines
141141 {
142142 struct mm_struct *mm = vma->vm_mm;
143143 int error;
144− pgoff_t pgoff;
145144 VMA_ITERATOR(vmi, mm, start);
146145
147146 if (new_flags == vma->vm_flags && anon_vma_name_eq(anon_vma_name(vma), anon_name)) {
@@ -149,30 +148,13 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 1 unmodified line
149148 return 0;
150149 }
151150
152− pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);
153− *prev = vma_merge(&vmi, mm, *prev, start, end, new_flags,
154− vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma),
155− vma->vm_userfaultfd_ctx, anon_name);
156− if (*prev) {
157− vma = *prev;
158− goto success;
159− }
151+ vma = vma_modify_flags_name(&vmi, *prev, vma, start, end, new_flags,
152+ anon_name);
153+ if (IS_ERR(vma))
154+ return PTR_ERR(vma);
160155
161156 *prev = vma;
162157
163− if (start != vma->vm_start) {
164− error = split_vma(&vmi, vma, start, 1);
165− if (error)
166− return error;
167− }
168−
169− if (end != vma->vm_end) {
170− error = split_vma(&vmi, vma, end, 0);
171− if (error)
172− return error;
173− }
174−
175−success:
176158 /* vm_flags is protected by the mmap_lock held in write mode. */
177159 vma_start_write(vma);
178160 vm_flags_reset(vma, new_flags);
@@ -746,11 +728,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 567 unmodified lines
746728 folio_mark_lazyfree(folio);
747729 }
748730
749− if (nr_swap) {
750− if (current->mm == mm)
751− sync_mm_rss(mm);
731+ if (nr_swap)
752732 add_mm_counter(mm, MM_SWAPENTS, nr_swap);
753− }
754733 if (start_pte) {
755734 arch_leave_lazy_mmu_mode();
756735 pte_unmap_unlock(start_pte, ptl);
@@ -991,7 +970,7 @@ static long madvise_remove(struct vm_area_struct *vma, · 234 unmodified lines
991970 return -EINVAL;
992971 }
993972
994− if ((vma->vm_flags & (VM_SHARED|VM_WRITE)) != (VM_SHARED|VM_WRITE))
973+ if (!vma_is_shared_maywrite(vma))
995974 return -EACCES;
996975
997976 offset = (loff_t)(start - vma->vm_start)
mm/memblock.c+36−13
@@ -892,6 +892,7 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 891 unmodified lines
892892
893893 /**
894894 * memblock_setclr_flag - set or clear flag for a memory region
895+ * @type: memblock type to set/clear flag for
895896 * @base: base address of the region
896897 * @size: size of the region
897898 * @set: set or clear the flag
@@ -901,10 +902,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 3 unmodified lines
901902 *
902903 * Return: 0 on success, -errno on failure.
903904 */
904−static int __init_memblock memblock_setclr_flag(phys_addr_t base,
905− phys_addr_t size, int set, int flag)
905+static int __init_memblock memblock_setclr_flag(struct memblock_type *type,
906+ phys_addr_t base, phys_addr_t size, int set, int flag)
906907 {
907− struct memblock_type *type = &memblock.memory;
908908 int i, ret, start_rgn, end_rgn;
909909
910910 ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn);
@@ -933,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, · 22 unmodified lines
933933 */
934934 int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size)
935935 {
936− return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG);
936+ return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG);
937937 }
938938
939939 /**
@@ -945,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) · 5 unmodified lines
945945 */
946946 int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
947947 {
948− return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG);
948+ return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG);
949949 }
950950
951951 /**
@@ -962,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 10 unmodified lines
962962
963963 system_has_some_mirror = true;
964964
965− return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR);
965+ return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR);
966966 }
967967
968968 /**
@@ -982,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 13 unmodified lines
982982 */
983983 int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size)
984984 {
985− return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP);
985+ return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP);
986986 }
987987
988988 /**
@@ -994,7 +994,25 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) · 5 unmodified lines
994994 */
995995 int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size)
996996 {
997− return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP);
997+ return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP);
998+}
999+
1000+/**
1001+ * memblock_reserved_mark_noinit - Mark a reserved memory region with flag
1002+ * MEMBLOCK_RSRV_NOINIT which results in the struct pages not being initialized
1003+ * for this region.
1004+ * @base: the base phys addr of the region
1005+ * @size: the size of the region
1006+ *
1007+ * struct pages will not be initialized for reserved memory regions marked with
1008+ * %MEMBLOCK_RSRV_NOINIT.
1009+ *
1010+ * Return: 0 on success, -errno on failure.
1011+ */
1012+int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size)
1013+{
1014+ return memblock_setclr_flag(&memblock.reserved, base, size, 1,
1015+ MEMBLOCK_RSRV_NOINIT);
9981016 }
9991017
10001018 static bool should_skip_region(struct memblock_type *type,
@@ -2113,13 +2131,18 @@ static void __init memmap_init_reserved_pages(void) · 1112 unmodified lines
21132131 memblock_set_node(start, end, &memblock.reserved, nid);
21142132 }
21152133
2116− /* initialize struct pages for the reserved regions */
2134+ /*
2135+ * initialize struct pages for reserved regions that don't have
2136+ * the MEMBLOCK_RSRV_NOINIT flag set
2137+ */
21172138 for_each_reserved_mem_region(region) {
2118− nid = memblock_get_region_node(region);
2119− start = region->base;
2120− end = start + region->size;
2139+ if (!memblock_is_reserved_noinit(region)) {
2140+ nid = memblock_get_region_node(region);
2141+ start = region->base;
2142+ end = start + region->size;
21212143
2122− reserve_bootmem_region(start, end, nid);
2144+ reserve_bootmem_region(start, end, nid);
2145+ }
21232146 }
21242147 }
21252148
mm/memcontrol.c+341−60
@@ -249,6 +249,9 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) · 248 unmodified lines
249249 return container_of(vmpr, struct mem_cgroup, vmpressure);
250250 }
251251
252+#define CURRENT_OBJCG_UPDATE_BIT 0
253+#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT)
254+
252255 #ifdef CONFIG_MEMCG_KMEM
253256 static DEFINE_SPINLOCK(objcg_lock);
254257
@@ -704,6 +707,8 @@ static const unsigned int memcg_vm_event_stat[] = { · 449 unmodified lines
704707 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
705708 THP_FAULT_ALLOC,
706709 THP_COLLAPSE_ALLOC,
710+ THP_SWPOUT,
711+ THP_SWPOUT_FALLBACK,
707712 #endif
708713 };
709714
@@ -761,6 +766,22 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) · 51 unmodified lines
761766 return x;
762767 }
763768
769+static int memcg_page_state_unit(int item);
770+
771+/*
772+ * Normalize the value passed into memcg_rstat_updated() to be in pages. Round
773+ * up non-zero sub-page updates to 1 page as zero page updates are ignored.
774+ */
775+static int memcg_state_val_in_pages(int idx, int val)
776+{
777+ int unit = memcg_page_state_unit(idx);
778+
779+ if (!val || unit == PAGE_SIZE)
780+ return val;
781+ else
782+ return max(val * unit / PAGE_SIZE, 1UL);
783+}
784+
764785 /**
765786 * __mod_memcg_state - update cgroup memory statistics
766787 * @memcg: the memory cgroup
@@ -773,7 +794,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) · 6 unmodified lines
773794 return;
774795
775796 __this_cpu_add(memcg->vmstats_percpu->state[idx], val);
776− memcg_rstat_updated(memcg, val);
797+ memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val));
777798 }
778799
779800 /* idx can be of type enum memcg_stat_item or node_stat_item. */
@@ -798,7 +819,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 18 unmodified lines
798819 memcg = pn->memcg;
799820
800821 /*
801− * The caller from rmap relay on disabled preemption becase they never
822+ * The caller from rmap relies on disabled preemption because they never
802823 * update their counter from in-interrupt context. For these two
803824 * counters we check that the update is never performed from an
804825 * interrupt context while other caller need to have disabled interrupt.
@@ -824,7 +845,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 19 unmodified lines
824845 /* Update lruvec */
825846 __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val);
826847
827− memcg_rstat_updated(memcg, val);
848+ memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val));
828849 memcg_stats_unlock();
829850 }
830851
@@ -1068,17 +1089,25 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 237 unmodified lines
10681089 }
10691090 EXPORT_SYMBOL(get_mem_cgroup_from_mm);
10701091
1071−static __always_inline bool memcg_kmem_bypass(void)
1092+/**
1093+ * get_mem_cgroup_from_current - Obtain a reference on current task's memcg.
1094+ */
1095+struct mem_cgroup *get_mem_cgroup_from_current(void)
10721096 {
1073− /* Allow remote memcg charging from any context. */
1074− if (unlikely(active_memcg()))
1075− return false;
1097+ struct mem_cgroup *memcg;
10761098
1077− /* Memcg to charge can't be determined. */
1078− if (!in_task() || !current->mm || (current->flags & PF_KTHREAD))
1079− return true;
1099+ if (mem_cgroup_disabled())
1100+ return NULL;
10801101
1081− return false;
1102+again:
1103+ rcu_read_lock();
1104+ memcg = mem_cgroup_from_task(current);
1105+ if (!css_tryget(&memcg->css)) {
1106+ rcu_read_unlock();
1107+ goto again;
1108+ }
1109+ rcu_read_unlock();
1110+ return memcg;
10821111 }
10831112
10841113 /**
@@ -1533,7 +1562,7 @@ static const struct memory_stat memory_stats[] = { · 448 unmodified lines
15331562 { "workingset_nodereclaim", WORKINGSET_NODERECLAIM },
15341563 };
15351564
1536−/* Translate stat items to the correct unit for memory.stat output */
1565+/* The actual unit of the state item, not the same as the output unit */
15371566 static int memcg_page_state_unit(int item)
15381567 {
15391568 switch (item) {
@@ -1541,6 +1570,22 @@ static int memcg_page_state_unit(int item) · 1 unmodified line
15411570 case MEMCG_ZSWAP_B:
15421571 case NR_SLAB_RECLAIMABLE_B:
15431572 case NR_SLAB_UNRECLAIMABLE_B:
1573+ return 1;
1574+ case NR_KERNEL_STACK_KB:
1575+ return SZ_1K;
1576+ default:
1577+ return PAGE_SIZE;
1578+ }
1579+}
1580+
1581+/* Translate stat items to the correct unit for memory.stat output */
1582+static int memcg_page_state_output_unit(int item)
1583+{
1584+ /*
1585+ * Workingset state is actually in pages, but we export it to userspace
1586+ * as a scalar count of events, so special case it here.
1587+ */
1588+ switch (item) {
15441589 case WORKINGSET_REFAULT_ANON:
15451590 case WORKINGSET_REFAULT_FILE:
15461591 case WORKINGSET_ACTIVATE_ANON:
@@ -1549,17 +1594,23 @@ static int memcg_page_state_unit(int item) · 2 unmodified lines
15491594 case WORKINGSET_RESTORE_FILE:
15501595 case WORKINGSET_NODERECLAIM:
15511596 return 1;
1552− case NR_KERNEL_STACK_KB:
1553− return SZ_1K;
15541597 default:
1555− return PAGE_SIZE;
1598+ return memcg_page_state_unit(item);
15561599 }
15571600 }
15581601
15591602 static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg,
15601603 int item)
15611604 {
1562− return memcg_page_state(memcg, item) * memcg_page_state_unit(item);
1605+ return memcg_page_state(memcg, item) *
1606+ memcg_page_state_output_unit(item);
1607+}
1608+
1609+static inline unsigned long memcg_page_state_local_output(
1610+ struct mem_cgroup *memcg, int item)
1611+{
1612+ return memcg_page_state_local(memcg, item) *
1613+ memcg_page_state_output_unit(item);
15631614 }
15641615
15651616 static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
@@ -2833,7 +2884,12 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · 1267 unmodified lines
28332884 return try_charge_memcg(memcg, gfp_mask, nr_pages);
28342885 }
28352886
2836−static inline void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
2887+/**
2888+ * mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call.
2889+ * @memcg: memcg previously charged.
2890+ * @nr_pages: number of pages previously charged.
2891+ */
2892+void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
28372893 {
28382894 if (mem_cgroup_is_root(memcg))
28392895 return;
@@ -2858,6 +2914,22 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 18 unmodified lines
28582914 folio->memcg_data = (unsigned long)memcg;
28592915 }
28602916
2917+/**
2918+ * mem_cgroup_commit_charge - commit a previously successful try_charge().
2919+ * @folio: folio to commit the charge to.
2920+ * @memcg: memcg previously charged.
2921+ */
2922+void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg)
2923+{
2924+ css_get(&memcg->css);
2925+ commit_charge(folio, memcg);
2926+
2927+ local_irq_disable();
2928+ mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio));
2929+ memcg_check_events(memcg, folio_nid(folio));
2930+ local_irq_enable();
2931+}
2932+
28612933 #ifdef CONFIG_MEMCG_KMEM
28622934 /*
28632935 * The allocated objcg pointers array is not accounted directly.
@@ -3007,28 +3079,105 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) · 143 unmodified lines
30073079
30083080 for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
30093081 objcg = rcu_dereference(memcg->objcg);
3010− if (objcg && obj_cgroup_tryget(objcg))
3082+ if (likely(objcg && obj_cgroup_tryget(objcg)))
30113083 break;
30123084 objcg = NULL;
30133085 }
30143086 return objcg;
30153087 }
30163088
3017−__always_inline struct obj_cgroup *get_obj_cgroup_from_current(void)
3089+static struct obj_cgroup *current_objcg_update(void)
30183090 {
3019− struct obj_cgroup *objcg = NULL;
30203091 struct mem_cgroup *memcg;
3092+ struct obj_cgroup *old, *objcg = NULL;
30213093
3022− if (memcg_kmem_bypass())
3023− return NULL;
3094+ do {
3095+ /* Atomically drop the update bit. */
3096+ old = xchg(&current->objcg, NULL);
3097+ if (old) {
3098+ old = (struct obj_cgroup *)
3099+ ((unsigned long)old & ~CURRENT_OBJCG_UPDATE_FLAG);
3100+ if (old)
3101+ obj_cgroup_put(old);
3102+
3103+ old = NULL;
3104+ }
30243105
3025− rcu_read_lock();
3026− if (unlikely(active_memcg()))
3027− memcg = active_memcg();
3028− else
3106+ /* If new objcg is NULL, no reason for the second atomic update. */
3107+ if (!current->mm || (current->flags & PF_KTHREAD))
3108+ return NULL;
3109+
3110+ /*
3111+ * Release the objcg pointer from the previous iteration,
3112+ * if try_cmpxcg() below fails.
3113+ */
3114+ if (unlikely(objcg)) {
3115+ obj_cgroup_put(objcg);
3116+ objcg = NULL;
3117+ }
3118+
3119+ /*
3120+ * Obtain the new objcg pointer. The current task can be
3121+ * asynchronously moved to another memcg and the previous
3122+ * memcg can be offlined. So let's get the memcg pointer
3123+ * and try get a reference to objcg under a rcu read lock.
3124+ */
3125+
3126+ rcu_read_lock();
30293127 memcg = mem_cgroup_from_task(current);
3030− objcg = __get_obj_cgroup_from_memcg(memcg);
3031− rcu_read_unlock();
3128+ objcg = __get_obj_cgroup_from_memcg(memcg);
3129+ rcu_read_unlock();
3130+
3131+ /*
3132+ * Try set up a new objcg pointer atomically. If it
3133+ * fails, it means the update flag was set concurrently, so
3134+ * the whole procedure should be repeated.
3135+ */
3136+ } while (!try_cmpxchg(&current->objcg, &old, objcg));
3137+
3138+ return objcg;
3139+}
3140+
3141+__always_inline struct obj_cgroup *current_obj_cgroup(void)
3142+{
3143+ struct mem_cgroup *memcg;
3144+ struct obj_cgroup *objcg;
3145+
3146+ if (in_task()) {
3147+ memcg = current->active_memcg;
3148+ if (unlikely(memcg))
3149+ goto from_memcg;
3150+
3151+ objcg = READ_ONCE(current->objcg);
3152+ if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG))
3153+ objcg = current_objcg_update();
3154+ /*
3155+ * Objcg reference is kept by the task, so it's safe
3156+ * to use the objcg by the current task.
3157+ */
3158+ return objcg;
3159+ }
3160+
3161+ memcg = this_cpu_read(int_active_memcg);
3162+ if (unlikely(memcg))
3163+ goto from_memcg;
3164+
3165+ return NULL;
3166+
3167+from_memcg:
3168+ for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
3169+ /*
3170+ * Memcg pointer is protected by scope (see set_active_memcg())
3171+ * and is pinning the corresponding objcg, so objcg can't go
3172+ * away and can be used within the scope without any additional
3173+ * protection.
3174+ */
3175+ objcg = rcu_dereference_check(memcg->objcg, 1);
3176+ if (likely(objcg))
3177+ break;
3178+ objcg = NULL;
3179+ }
3180+
30323181 return objcg;
30333182 }
30343183
@@ -3126,15 +3275,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) · 91 unmodified lines
31263275 struct obj_cgroup *objcg;
31273276 int ret = 0;
31283277
3129− objcg = get_obj_cgroup_from_current();
3278+ objcg = current_obj_cgroup();
31303279 if (objcg) {
31313280 ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order);
31323281 if (!ret) {
3282+ obj_cgroup_get(objcg);
31333283 page->memcg_data = (unsigned long)objcg |
31343284 MEMCG_DATA_KMEM;
31353285 return 0;
31363286 }
3137− obj_cgroup_put(objcg);
31383287 }
31393288 return ret;
31403289 }
@@ -3761,6 +3910,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) · 620 unmodified lines
37613910
37623911 objcg->memcg = memcg;
37633912 rcu_assign_pointer(memcg->objcg, objcg);
3913+ obj_cgroup_get(objcg);
3914+ memcg->orig_objcg = objcg;
37643915
37653916 static_branch_enable(&memcg_kmem_online_key);
37663917
@@ -4066,7 +4217,10 @@ static const unsigned int memcg1_stats[] = { · 299 unmodified lines
40664217 NR_WRITEBACK,
40674218 WORKINGSET_REFAULT_ANON,
40684219 WORKINGSET_REFAULT_FILE,
4220+#ifdef CONFIG_SWAP
40694221 MEMCG_SWAP,
4222+ NR_SWAPCACHE,
4223+#endif
40704224 };
40714225
40724226 static const char *const memcg1_stat_names[] = {
@@ -4081,7 +4235,10 @@ static const char *const memcg1_stat_names[] = { · 8 unmodified lines
40814235 "writeback",
40824236 "workingset_refault_anon",
40834237 "workingset_refault_file",
4238+#ifdef CONFIG_SWAP
40844239 "swap",
4240+ "swapcached",
4241+#endif
40854242 };
40864243
40874244 /* Universal VM events cgroup1 shows, original sort order */
@@ -4105,11 +4262,8 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 17 unmodified lines
41054262 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
41064263 unsigned long nr;
41074264
4108− if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account())
4109− continue;
4110− nr = memcg_page_state_local(memcg, memcg1_stats[i]);
4111− seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i],
4112− nr * memcg_page_state_unit(memcg1_stats[i]));
4265+ nr = memcg_page_state_local_output(memcg, memcg1_stats[i]);
4266+ seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr);
41134267 }
41144268
41154269 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
@@ -4129,18 +4283,15 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 13 unmodified lines
41294283 }
41304284 seq_buf_printf(s, "hierarchical_memory_limit %llu\n",
41314285 (u64)memory * PAGE_SIZE);
4132− if (do_memsw_account())
4133− seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",
4134− (u64)memsw * PAGE_SIZE);
4286+ seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",
4287+ (u64)memsw * PAGE_SIZE);
41354288
41364289 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
41374290 unsigned long nr;
41384291
4139− if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account())
4140− continue;
4141− nr = memcg_page_state(memcg, memcg1_stats[i]);
4292+ nr = memcg_page_state_output(memcg, memcg1_stats[i]);
41424293 seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i],
4143− (u64)nr * memcg_page_state_unit(memcg1_stats[i]));
4294+ (u64)nr);
41444295 }
41454296
41464297 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
@@ -5268,6 +5419,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) · 1121 unmodified lines
52685419 {
52695420 int node;
52705421
5422+ if (memcg->orig_objcg)
5423+ obj_cgroup_put(memcg->orig_objcg);
5424+
52715425 for_each_node(node)
52725426 free_mem_cgroup_per_node_info(memcg, node);
52735427 kfree(memcg->vmstats);
@@ -6076,7 +6230,7 @@ static void __mem_cgroup_clear_mc(void) · 802 unmodified lines
60766230
60776231 /* we must uncharge all the leftover precharges from mc.to */
60786232 if (mc.precharge) {
6079− cancel_charge(mc.to, mc.precharge);
6233+ mem_cgroup_cancel_charge(mc.to, mc.precharge);
60806234 mc.precharge = 0;
60816235 }
60826236 /*
@@ -6084,7 +6238,7 @@ static void __mem_cgroup_clear_mc(void) · 1 unmodified line
60846238 * we must uncharge here.
60856239 */
60866240 if (mc.moved_charge) {
6087− cancel_charge(mc.from, mc.moved_charge);
6241+ mem_cgroup_cancel_charge(mc.from, mc.moved_charge);
60886242 mc.moved_charge = 0;
60896243 }
60906244 /* we must fixup refcnts and charges */
@@ -6364,6 +6518,7 @@ static void mem_cgroup_move_task(void) · 273 unmodified lines
63646518 mem_cgroup_clear_mc();
63656519 }
63666520 }
6521+
63676522 #else /* !CONFIG_MMU */
63686523 static int mem_cgroup_can_attach(struct cgroup_taskset *tset)
63696524 {
@@ -6377,8 +6532,39 @@ static void mem_cgroup_move_task(void) · 7 unmodified lines
63776532 }
63786533 #endif
63796534
6535+#ifdef CONFIG_MEMCG_KMEM
6536+static void mem_cgroup_fork(struct task_struct *task)
6537+{
6538+ /*
6539+ * Set the update flag to cause task->objcg to be initialized lazily
6540+ * on the first allocation. It can be done without any synchronization
6541+ * because it's always performed on the current task, so does
6542+ * current_objcg_update().
6543+ */
6544+ task->objcg = (struct obj_cgroup *)CURRENT_OBJCG_UPDATE_FLAG;
6545+}
6546+
6547+static void mem_cgroup_exit(struct task_struct *task)
6548+{
6549+ struct obj_cgroup *objcg = task->objcg;
6550+
6551+ objcg = (struct obj_cgroup *)
6552+ ((unsigned long)objcg & ~CURRENT_OBJCG_UPDATE_FLAG);
6553+ if (objcg)
6554+ obj_cgroup_put(objcg);
6555+
6556+ /*
6557+ * Some kernel allocations can happen after this point,
6558+ * but let's ignore them. It can be done without any synchronization
6559+ * because it's always performed on the current task, so does
6560+ * current_objcg_update().
6561+ */
6562+ task->objcg = NULL;
6563+}
6564+#endif
6565+
63806566 #ifdef CONFIG_LRU_GEN
6381−static void mem_cgroup_attach(struct cgroup_taskset *tset)
6567+static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset)
63826568 {
63836569 struct task_struct *task;
63846570 struct cgroup_subsys_state *css;
@@ -6396,10 +6582,31 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) · 11 unmodified lines
63966582 task_unlock(task);
63976583 }
63986584 #else
6585+static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {}
6586+#endif /* CONFIG_LRU_GEN */
6587+
6588+#ifdef CONFIG_MEMCG_KMEM
6589+static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset)
6590+{
6591+ struct task_struct *task;
6592+ struct cgroup_subsys_state *css;
6593+
6594+ cgroup_taskset_for_each(task, css, tset) {
6595+ /* atomically set the update bit */
6596+ set_bit(CURRENT_OBJCG_UPDATE_BIT, (unsigned long *)&task->objcg);
6597+ }
6598+}
6599+#else
6600+static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {}
6601+#endif /* CONFIG_MEMCG_KMEM */
6602+
6603+#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM)
63996604 static void mem_cgroup_attach(struct cgroup_taskset *tset)
64006605 {
6606+ mem_cgroup_lru_gen_attach(tset);
6607+ mem_cgroup_kmem_attach(tset);
64016608 }
6402−#endif /* CONFIG_LRU_GEN */
6609+#endif
64036610
64046611 static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value)
64056612 {
@@ -6622,7 +6829,8 @@ static int memory_stat_show(struct seq_file *m, void *v) · 216 unmodified lines
66226829 static inline unsigned long lruvec_page_state_output(struct lruvec *lruvec,
66236830 int item)
66246831 {
6625− return lruvec_page_state(lruvec, item) * memcg_page_state_unit(item);
6832+ return lruvec_page_state(lruvec, item) *
6833+ memcg_page_state_output_unit(item);
66266834 }
66276835
66286836 static int memory_numa_stat_show(struct seq_file *m, void *v)
@@ -6808,9 +7016,15 @@ struct cgroup_subsys memory_cgrp_subsys = { · 179 unmodified lines
68087016 .css_reset = mem_cgroup_css_reset,
68097017 .css_rstat_flush = mem_cgroup_css_rstat_flush,
68107018 .can_attach = mem_cgroup_can_attach,
7019+#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM)
68117020 .attach = mem_cgroup_attach,
7021+#endif
68127022 .cancel_attach = mem_cgroup_cancel_attach,
68137023 .post_attach = mem_cgroup_move_task,
7024+#ifdef CONFIG_MEMCG_KMEM
7025+ .fork = mem_cgroup_fork,
7026+ .exit = mem_cgroup_exit,
7027+#endif
68147028 .dfl_cftypes = memory_files,
68157029 .legacy_cftypes = mem_cgroup_legacy_files,
68167030 .early_init = 0,
@@ -6990,20 +7204,13 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, · 173 unmodified lines
69907204 static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
69917205 gfp_t gfp)
69927206 {
6993− long nr_pages = folio_nr_pages(folio);
69947207 int ret;
69957208
6996− ret = try_charge(memcg, gfp, nr_pages);
7209+ ret = try_charge(memcg, gfp, folio_nr_pages(folio));
69977210 if (ret)
69987211 goto out;
69997212
7000− css_get(&memcg->css);
7001− commit_charge(folio, memcg);
7002−
7003− local_irq_disable();
7004− mem_cgroup_charge_statistics(memcg, nr_pages);
7005− memcg_check_events(memcg, folio_nid(folio));
7006− local_irq_enable();
7213+ mem_cgroup_commit_charge(folio, memcg);
70077214 out:
70087215 return ret;
70097216 }
@@ -7020,6 +7227,41 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) · 10 unmodified lines
70207227 return ret;
70217228 }
70227229
7230+/**
7231+ * mem_cgroup_hugetlb_try_charge - try to charge the memcg for a hugetlb folio
7232+ * @memcg: memcg to charge.
7233+ * @gfp: reclaim mode.
7234+ * @nr_pages: number of pages to charge.
7235+ *
7236+ * This function is called when allocating a huge page folio to determine if
7237+ * the memcg has the capacity for it. It does not commit the charge yet,
7238+ * as the hugetlb folio itself has not been obtained from the hugetlb pool.
7239+ *
7240+ * Once we have obtained the hugetlb folio, we can call
7241+ * mem_cgroup_commit_charge() to commit the charge. If we fail to obtain the
7242+ * folio, we should instead call mem_cgroup_cancel_charge() to undo the effect
7243+ * of try_charge().
7244+ *
7245+ * Returns 0 on success. Otherwise, an error code is returned.
7246+ */
7247+int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
7248+ long nr_pages)
7249+{
7250+ /*
7251+ * If hugetlb memcg charging is not enabled, do not fail hugetlb allocation,
7252+ * but do not attempt to commit charge later (or cancel on error) either.
7253+ */
7254+ if (mem_cgroup_disabled() || !memcg ||
7255+ !cgroup_subsys_on_dfl(memory_cgrp_subsys) ||
7256+ !(cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING))
7257+ return -EOPNOTSUPP;
7258+
7259+ if (try_charge(memcg, gfp, nr_pages))
7260+ return -ENOMEM;
7261+
7262+ return 0;
7263+}
7264+
70237265 /**
70247266 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
70257267 * @folio: folio to charge.
@@ -7216,16 +7458,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) · 190 unmodified lines
72167458 }
72177459
72187460 /**
7219− * mem_cgroup_migrate - Charge a folio's replacement.
7461+ * mem_cgroup_replace_folio - Charge a folio's replacement.
72207462 * @old: Currently circulating folio.
72217463 * @new: Replacement folio.
72227464 *
72237465 * Charge @new as a replacement folio for @old. @old will
7224− * be uncharged upon free.
7466+ * be uncharged upon free. This is only used by the page cache
7467+ * (in replace_page_cache_folio()).
72257468 *
72267469 * Both folios must be locked, @new->mapping must be set up.
72277470 */
7228−void mem_cgroup_migrate(struct folio *old, struct folio *new)
7471+void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
72297472 {
72307473 struct mem_cgroup *memcg;
72317474 long nr_pages = folio_nr_pages(new);
@@ -7264,6 +7507,44 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) · 32 unmodified lines
72647507 local_irq_restore(flags);
72657508 }
72667509
7510+/**
7511+ * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
7512+ * @old: Currently circulating folio.
7513+ * @new: Replacement folio.
7514+ *
7515+ * Transfer the memcg data from the old folio to the new folio for migration.
7516+ * The old folio's data info will be cleared. Note that the memory counters
7517+ * will remain unchanged throughout the process.
7518+ *
7519+ * Both folios must be locked, @new->mapping must be set up.
7520+ */
7521+void mem_cgroup_migrate(struct folio *old, struct folio *new)
7522+{
7523+ struct mem_cgroup *memcg;
7524+
7525+ VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
7526+ VM_BUG_ON_FOLIO(!folio_test_locked(new), new);
7527+ VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new);
7528+ VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new);
7529+
7530+ if (mem_cgroup_disabled())
7531+ return;
7532+
7533+ memcg = folio_memcg(old);
7534+ /*
7535+ * Note that it is normal to see !memcg for a hugetlb folio.
7536+ * For e.g, itt could have been allocated when memory_hugetlb_accounting
7537+ * was not selected.
7538+ */
7539+ VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old);
7540+ if (!memcg)
7541+ return;
7542+
7543+ /* Transfer the charge and the css ref */
7544+ commit_charge(new, memcg);
7545+ old->memcg_data = 0;
7546+}
7547+
72677548 DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key);
72687549 EXPORT_SYMBOL(memcg_sockets_enabled_key);
72697550
@@ -7763,7 +8044,7 @@ static struct cftype memsw_files[] = { · 493 unmodified lines
77638044 *
77648045 * This doesn't check for specific headroom, and it is not atomic
77658046 * either. But with zswap, the size of the allocation is only known
7766− * once compression has occured, and this optimistic pre-check avoids
8047+ * once compression has occurred, and this optimistic pre-check avoids
77678048 * spending cycles on compression when there is already no room left
77688049 * or zswap is disabled altogether somewhere in the hierarchy.
77698050 */
mm/memfd.c+0−6
@@ -315,12 +315,6 @@ SYSCALL_DEFINE2(memfd_create, · 314 unmodified lines
315315 if ((flags & MFD_EXEC) && (flags & MFD_NOEXEC_SEAL))
316316 return -EINVAL;
317317
318− if (!(flags & (MFD_EXEC | MFD_NOEXEC_SEAL))) {
319− pr_warn_once(
320− "%s[%d]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set\n",
321− current->comm, task_pid_nr(current));
322− }
323−
324318 error = check_sysctl_memfd_noexec(&flags);
325319 if (error < 0)
326320 return error;
mm/memory-failure.c+16−13
@@ -1713,20 +1713,23 @@ static void unmap_and_kill(struct list_head *to_kill, unsigned long pfn, · 1712 unmodified lines
17131713 kill_procs(to_kill, flags & MF_MUST_KILL, false, pfn, flags);
17141714 }
17151715
1716+/*
1717+ * Only dev_pagemap pages get here, such as fsdax when the filesystem
1718+ * either do not claim or fails to claim a hwpoison event, or devdax.
1719+ * The fsdax pages are initialized per base page, and the devdax pages
1720+ * could be initialized either as base pages, or as compound pages with
1721+ * vmemmap optimization enabled. Devdax is simplistic in its dealing with
1722+ * hwpoison, such that, if a subpage of a compound page is poisoned,
1723+ * simply mark the compound head page is by far sufficient.
1724+ */
17161725 static int mf_generic_kill_procs(unsigned long long pfn, int flags,
17171726 struct dev_pagemap *pgmap)
17181727 {
1719− struct page *page = pfn_to_page(pfn);
1728+ struct folio *folio = pfn_folio(pfn);
17201729 LIST_HEAD(to_kill);
17211730 dax_entry_t cookie;
17221731 int rc = 0;
17231732
1724− /*
1725− * Pages instantiated by device-dax (not filesystem-dax)
1726− * may be compound pages.
1727− */
1728− page = compound_head(page);
1729−
17301733 /*
17311734 * Prevent the inode from being freed while we are interrogating
17321735 * the address_space, typically this would be handled by
@@ -1734,11 +1737,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 1 unmodified line
17341737 * also prevents changes to the mapping of this pfn until
17351738 * poison signaling is complete.
17361739 */
1737− cookie = dax_lock_page(page);
1740+ cookie = dax_lock_folio(folio);
17381741 if (!cookie)
17391742 return -EBUSY;
17401743
1741− if (hwpoison_filter(page)) {
1744+ if (hwpoison_filter(&folio->page)) {
17421745 rc = -EOPNOTSUPP;
17431746 goto unlock;
17441747 }
@@ -1760,7 +1763,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 15 unmodified lines
17601763 * Use this flag as an indication that the dax page has been
17611764 * remapped UC to prevent speculative consumption of poison.
17621765 */
1763− SetPageHWPoison(page);
1766+ SetPageHWPoison(&folio->page);
17641767
17651768 /*
17661769 * Unlike System-RAM there is no possibility to swap in a
@@ -1769,11 +1772,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 2 unmodified lines
17691772 * SIGBUS (i.e. MF_MUST_KILL)
17701773 */
17711774 flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
1772− collect_procs(page, &to_kill, true);
1775+ collect_procs(&folio->page, &to_kill, true);
17731776
1774− unmap_and_kill(&to_kill, pfn, page->mapping, page->index, flags);
1777+ unmap_and_kill(&to_kill, pfn, folio->mapping, folio->index, flags);
17751778 unlock:
1776− dax_unlock_page(page, cookie);
1779+ dax_unlock_folio(folio, cookie);
17771780 return rc;
17781781 }
17791782
mm/memory-tiers.c+168−7
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 #include <linux/kobject.h>
66 #include <linux/memory.h>
77 #include <linux/memory-tiers.h>
8+#include <linux/notifier.h>
89
910 #include "internal.h"
1011
@@ -36,7 +37,7 @@ struct node_memory_type_map { · 25 unmodified lines
3637 static DEFINE_MUTEX(memory_tier_lock);
3738 static LIST_HEAD(memory_tiers);
3839 static struct node_memory_type_map node_memory_types[MAX_NUMNODES];
39−static struct memory_dev_type *default_dram_type;
40+struct memory_dev_type *default_dram_type;
4041
4142 static struct bus_type memory_tier_subsys = {
4243 .name = "memory_tiering",
@@ -105,6 +106,13 @@ static int top_tier_adistance; · 62 unmodified lines
105106 static struct demotion_nodes *node_demotion __read_mostly;
106107 #endif /* CONFIG_MIGRATION */
107108
109+static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms);
110+
111+static bool default_dram_perf_error;
112+static struct node_hmem_attrs default_dram_perf;
113+static int default_dram_perf_ref_nid = NUMA_NO_NODE;
114+static const char *default_dram_perf_ref_source;
115+
108116 static inline struct memory_tier *to_memory_tier(struct device *device)
109117 {
110118 return container_of(device, struct memory_tier, dev);
@@ -115,7 +123,7 @@ static __always_inline nodemask_t get_memtier_nodemask(struct memory_tier *memti · 4 unmodified lines
115123 nodemask_t nodes = NODE_MASK_NONE;
116124 struct memory_dev_type *memtype;
117125
118− list_for_each_entry(memtype, &memtier->memory_types, tier_sibiling)
126+ list_for_each_entry(memtype, &memtier->memory_types, tier_sibling)
119127 nodes_or(nodes, nodes, memtype->nodes);
120128
121129 return nodes;
@@ -174,7 +182,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 52 unmodified lines
174182 * If the memtype is already part of a memory tier,
175183 * just return that.
176184 */
177− if (!list_empty(&memtype->tier_sibiling)) {
185+ if (!list_empty(&memtype->tier_sibling)) {
178186 list_for_each_entry(memtier, &memory_tiers, list) {
179187 if (adistance == memtier->adistance_start)
180188 return memtier;
@@ -218,7 +226,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 37 unmodified lines
218226 memtier = new_memtier;
219227
220228 link_memtype:
221− list_add(&memtype->tier_sibiling, &memtier->memory_types);
229+ list_add(&memtype->tier_sibling, &memtier->memory_types);
222230 return memtier;
223231 }
224232
@@ -527,7 +535,7 @@ static bool clear_node_memory_tier(int node) · 302 unmodified lines
527535 memtype = node_memory_types[node].memtype;
528536 node_clear(node, memtype->nodes);
529537 if (nodes_empty(memtype->nodes)) {
530− list_del_init(&memtype->tier_sibiling);
538+ list_del_init(&memtype->tier_sibling);
531539 if (list_empty(&memtier->memory_types))
532540 destroy_memory_tier(memtier);
533541 }
@@ -553,7 +561,7 @@ struct memory_dev_type *alloc_memory_type(int adistance) · 19 unmodified lines
553561 return ERR_PTR(-ENOMEM);
554562
555563 memtype->adistance = adistance;
556− INIT_LIST_HEAD(&memtype->tier_sibiling);
564+ INIT_LIST_HEAD(&memtype->tier_sibling);
557565 memtype->nodes = NODE_MASK_NONE;
558566 kref_init(&memtype->kref);
559567 return memtype;
@@ -578,13 +586,14 @@ EXPORT_SYMBOL_GPL(init_node_memory_type); · 18 unmodified lines
578586 void clear_node_memory_type(int node, struct memory_dev_type *memtype)
579587 {
580588 mutex_lock(&memory_tier_lock);
581− if (node_memory_types[node].memtype == memtype)
589+ if (node_memory_types[node].memtype == memtype || !memtype)
582590 node_memory_types[node].map_count--;
583591 /*
584592 * If we umapped all the attached devices to this node,
585593 * clear the node memory type.
586594 */
587595 if (!node_memory_types[node].map_count) {
596+ memtype = node_memory_types[node].memtype;
588597 node_memory_types[node].memtype = NULL;
589598 put_memory_type(memtype);
590599 }
@@ -592,6 +601,158 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) · 1 unmodified line
592601 }
593602 EXPORT_SYMBOL_GPL(clear_node_memory_type);
594603
604+static void dump_hmem_attrs(struct node_hmem_attrs *attrs, const char *prefix)
605+{
606+ pr_info(
607+"%sread_latency: %u, write_latency: %u, read_bandwidth: %u, write_bandwidth: %u\n",
608+ prefix, attrs->read_latency, attrs->write_latency,
609+ attrs->read_bandwidth, attrs->write_bandwidth);
610+}
611+
612+int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
613+ const char *source)
614+{
615+ int rc = 0;
616+
617+ mutex_lock(&memory_tier_lock);
618+ if (default_dram_perf_error) {
619+ rc = -EIO;
620+ goto out;
621+ }
622+
623+ if (perf->read_latency + perf->write_latency == 0 ||
624+ perf->read_bandwidth + perf->write_bandwidth == 0) {
625+ rc = -EINVAL;
626+ goto out;
627+ }
628+
629+ if (default_dram_perf_ref_nid == NUMA_NO_NODE) {
630+ default_dram_perf = *perf;
631+ default_dram_perf_ref_nid = nid;
632+ default_dram_perf_ref_source = kstrdup(source, GFP_KERNEL);
633+ goto out;
634+ }
635+
636+ /*
637+ * The performance of all default DRAM nodes is expected to be
638+ * same (that is, the variation is less than 10%). And it
639+ * will be used as base to calculate the abstract distance of
640+ * other memory nodes.
641+ */
642+ if (abs(perf->read_latency - default_dram_perf.read_latency) * 10 >
643+ default_dram_perf.read_latency ||
644+ abs(perf->write_latency - default_dram_perf.write_latency) * 10 >
645+ default_dram_perf.write_latency ||
646+ abs(perf->read_bandwidth - default_dram_perf.read_bandwidth) * 10 >
647+ default_dram_perf.read_bandwidth ||
648+ abs(perf->write_bandwidth - default_dram_perf.write_bandwidth) * 10 >
649+ default_dram_perf.write_bandwidth) {
650+ pr_info(
651+"memory-tiers: the performance of DRAM node %d mismatches that of the reference\n"
652+"DRAM node %d.\n", nid, default_dram_perf_ref_nid);
653+ pr_info(" performance of reference DRAM node %d:\n",
654+ default_dram_perf_ref_nid);
655+ dump_hmem_attrs(&default_dram_perf, " ");
656+ pr_info(" performance of DRAM node %d:\n", nid);
657+ dump_hmem_attrs(perf, " ");
658+ pr_info(
659+" disable default DRAM node performance based abstract distance algorithm.\n");
660+ default_dram_perf_error = true;
661+ rc = -EINVAL;
662+ }
663+
664+out:
665+ mutex_unlock(&memory_tier_lock);
666+ return rc;
667+}
668+
669+int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist)
670+{
671+ if (default_dram_perf_error)
672+ return -EIO;
673+
674+ if (default_dram_perf_ref_nid == NUMA_NO_NODE)
675+ return -ENOENT;
676+
677+ if (perf->read_latency + perf->write_latency == 0 ||
678+ perf->read_bandwidth + perf->write_bandwidth == 0)
679+ return -EINVAL;
680+
681+ mutex_lock(&memory_tier_lock);
682+ /*
683+ * The abstract distance of a memory node is in direct proportion to
684+ * its memory latency (read + write) and inversely proportional to its
685+ * memory bandwidth (read + write). The abstract distance, memory
686+ * latency, and memory bandwidth of the default DRAM nodes are used as
687+ * the base.
688+ */
689+ *adist = MEMTIER_ADISTANCE_DRAM *
690+ (perf->read_latency + perf->write_latency) /
691+ (default_dram_perf.read_latency + default_dram_perf.write_latency) *
692+ (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) /
693+ (perf->read_bandwidth + perf->write_bandwidth);
694+ mutex_unlock(&memory_tier_lock);
695+
696+ return 0;
697+}
698+EXPORT_SYMBOL_GPL(mt_perf_to_adistance);
699+
700+/**
701+ * register_mt_adistance_algorithm() - Register memory tiering abstract distance algorithm
702+ * @nb: The notifier block which describe the algorithm
703+ *
704+ * Return: 0 on success, errno on error.
705+ *
706+ * Every memory tiering abstract distance algorithm provider needs to
707+ * register the algorithm with register_mt_adistance_algorithm(). To
708+ * calculate the abstract distance for a specified memory node, the
709+ * notifier function will be called unless some high priority
710+ * algorithm has provided result. The prototype of the notifier
711+ * function is as follows,
712+ *
713+ * int (*algorithm_notifier)(struct notifier_block *nb,
714+ * unsigned long nid, void *data);
715+ *
716+ * Where "nid" specifies the memory node, "data" is the pointer to the
717+ * returned abstract distance (that is, "int *adist"). If the
718+ * algorithm provides the result, NOTIFY_STOP should be returned.
719+ * Otherwise, return_value & %NOTIFY_STOP_MASK == 0 to allow the next
720+ * algorithm in the chain to provide the result.
721+ */
722+int register_mt_adistance_algorithm(struct notifier_block *nb)
723+{
724+ return blocking_notifier_chain_register(&mt_adistance_algorithms, nb);
725+}
726+EXPORT_SYMBOL_GPL(register_mt_adistance_algorithm);
727+
728+/**
729+ * unregister_mt_adistance_algorithm() - Unregister memory tiering abstract distance algorithm
730+ * @nb: the notifier block which describe the algorithm
731+ *
732+ * Return: 0 on success, errno on error.
733+ */
734+int unregister_mt_adistance_algorithm(struct notifier_block *nb)
735+{
736+ return blocking_notifier_chain_unregister(&mt_adistance_algorithms, nb);
737+}
738+EXPORT_SYMBOL_GPL(unregister_mt_adistance_algorithm);
739+
740+/**
741+ * mt_calc_adistance() - Calculate abstract distance with registered algorithms
742+ * @node: the node to calculate abstract distance for
743+ * @adist: the returned abstract distance
744+ *
745+ * Return: if return_value & %NOTIFY_STOP_MASK != 0, then some
746+ * abstract distance algorithm provides the result, and return it via
747+ * @adist. Otherwise, no algorithm can provide the result and @adist
748+ * will be kept as it is.
749+ */
750+int mt_calc_adistance(int node, int *adist)
751+{
752+ return blocking_notifier_call_chain(&mt_adistance_algorithms, node, adist);
753+}
754+EXPORT_SYMBOL_GPL(mt_calc_adistance);
755+
595756 static int __meminit memtier_hotplug_callback(struct notifier_block *self,
596757 unsigned long action, void *_arg)
597758 {
mm/memory.c+179−123
@@ -1,3 +1,4 @@
1+
12 // SPDX-License-Identifier: GPL-2.0-only
23 /*
34 * linux/mm/memory.c
@@ -471,8 +472,6 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) · 467 unmodified lines
471472 {
472473 int i;
473474
474− if (current->mm == mm)
475− sync_mm_rss(mm);
476475 for (i = 0; i < NR_MM_COUNTERS; i++)
477476 if (rss[i])
478477 add_mm_counter(mm, i, rss[i]);
@@ -691,6 +690,16 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, · 212 unmodified lines
691690 out:
692691 return pfn_to_page(pfn);
693692 }
693+
694+struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
695+ unsigned long addr, pmd_t pmd)
696+{
697+ struct page *page = vm_normal_page_pmd(vma, addr, pmd);
698+
699+ if (page)
700+ return page_folio(page);
701+ return NULL;
702+}
694703 #endif
695704
696705 static void restore_exclusive_pte(struct vm_area_struct *vma,
@@ -3006,23 +3015,24 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · 2309 unmodified lines
30063015 * case, all we need to do here is to mark the page as writable and update
30073016 * any related book-keeping.
30083017 */
3009−static inline void wp_page_reuse(struct vm_fault *vmf)
3018+static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio)
30103019 __releases(vmf->ptl)
30113020 {
30123021 struct vm_area_struct *vma = vmf->vma;
3013− struct page *page = vmf->page;
30143022 pte_t entry;
30153023
30163024 VM_BUG_ON(!(vmf->flags & FAULT_FLAG_WRITE));
3017− VM_BUG_ON(page && PageAnon(page) && !PageAnonExclusive(page));
30183025
3019− /*
3020− * Clear the pages cpupid information as the existing
3021− * information potentially belongs to a now completely
3022− * unrelated process.
3023− */
3024− if (page)
3025− page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
3026+ if (folio) {
3027+ VM_BUG_ON(folio_test_anon(folio) &&
3028+ !PageAnonExclusive(vmf->page));
3029+ /*
3030+ * Clear the folio's cpupid information as the existing
3031+ * information potentially belongs to a now completely
3032+ * unrelated process.
3033+ */
3034+ folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
3035+ }
30263036
30273037 flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
30283038 entry = pte_mkyoung(vmf->orig_pte);
@@ -3033,6 +3043,36 @@ static inline void wp_page_reuse(struct vm_fault *vmf) · 4 unmodified lines
30333043 count_vm_event(PGREUSE);
30343044 }
30353045
3046+/*
3047+ * We could add a bitflag somewhere, but for now, we know that all
3048+ * vm_ops that have a ->map_pages have been audited and don't need
3049+ * the mmap_lock to be held.
3050+ */
3051+static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf)
3052+{
3053+ struct vm_area_struct *vma = vmf->vma;
3054+
3055+ if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK))
3056+ return 0;
3057+ vma_end_read(vma);
3058+ return VM_FAULT_RETRY;
3059+}
3060+
3061+static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf)
3062+{
3063+ struct vm_area_struct *vma = vmf->vma;
3064+
3065+ if (likely(vma->anon_vma))
3066+ return 0;
3067+ if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
3068+ vma_end_read(vma);
3069+ return VM_FAULT_RETRY;
3070+ }
3071+ if (__anon_vma_prepare(vma))
3072+ return VM_FAULT_OOM;
3073+ return 0;
3074+}
3075+
30363076 /*
30373077 * Handle the case of a page which we actually need to copy to a new page,
30383078 * either due to COW or unsharing.
@@ -3060,27 +3100,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 21 unmodified lines
30603100 pte_t entry;
30613101 int page_copied = 0;
30623102 struct mmu_notifier_range range;
3063− int ret;
3103+ vm_fault_t ret;
30643104
30653105 delayacct_wpcopy_start();
30663106
30673107 if (vmf->page)
30683108 old_folio = page_folio(vmf->page);
3069− if (unlikely(anon_vma_prepare(vma)))
3070− goto oom;
3109+ ret = vmf_anon_prepare(vmf);
3110+ if (unlikely(ret))
3111+ goto out;
30713112
30723113 if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
30733114 new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
30743115 if (!new_folio)
30753116 goto oom;
30763117 } else {
3118+ int err;
30773119 new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma,
30783120 vmf->address, false);
30793121 if (!new_folio)
30803122 goto oom;
30813123
3082− ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf);
3083− if (ret) {
3124+ err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf);
3125+ if (err) {
30843126 /*
30853127 * COW failed, if the fault was solved by other,
30863128 * it's fine. If not, userspace would re-fault on
@@ -3093,7 +3135,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 6 unmodified lines
30933135 folio_put(old_folio);
30943136
30953137 delayacct_wpcopy_end();
3096− return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0;
3138+ return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0;
30973139 }
30983140 kmsan_copy_page_meta(&new_folio->page, vmf->page);
30993141 }
@@ -3203,11 +3245,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 103 unmodified lines
32033245 oom_free_new:
32043246 folio_put(new_folio);
32053247 oom:
3248+ ret = VM_FAULT_OOM;
3249+out:
32063250 if (old_folio)
32073251 folio_put(old_folio);
32083252
32093253 delayacct_wpcopy_end();
3210− return VM_FAULT_OOM;
3254+ return ret;
32113255 }
32123256
32133257 /**
@@ -3215,6 +3259,7 @@ oom: · 1 unmodified line
32153259 * writeable once the page is prepared
32163260 *
32173261 * @vmf: structure describing the fault
3262+ * @folio: the folio of vmf->page
32183263 *
32193264 * This function handles all that is needed to finish a write page fault in a
32203265 * shared mapping due to PTE being read-only once the mapped page is prepared.
@@ -3226,7 +3271,7 @@ oom: · 5 unmodified lines
32263271 * Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before
32273272 * we acquired PTE lock.
32283273 */
3229−vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf)
3274+static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, struct folio *folio)
32303275 {
32313276 WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
32323277 vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
@@ -3242,7 +3287,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) · 9 unmodified lines
32423287 pte_unmap_unlock(vmf->pte, vmf->ptl);
32433288 return VM_FAULT_NOPAGE;
32443289 }
3245− wp_page_reuse(vmf);
3290+ wp_page_reuse(vmf, folio);
32463291 return 0;
32473292 }
32483293
@@ -3258,18 +3303,17 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) · 9 unmodified lines
32583303 vm_fault_t ret;
32593304
32603305 pte_unmap_unlock(vmf->pte, vmf->ptl);
3261− if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
3262− vma_end_read(vmf->vma);
3263− return VM_FAULT_RETRY;
3264− }
3306+ ret = vmf_can_call_fault(vmf);
3307+ if (ret)
3308+ return ret;
32653309
32663310 vmf->flags |= FAULT_FLAG_MKWRITE;
32673311 ret = vma->vm_ops->pfn_mkwrite(vmf);
32683312 if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))
32693313 return ret;
3270− return finish_mkwrite_fault(vmf);
3314+ return finish_mkwrite_fault(vmf, NULL);
32713315 }
3272− wp_page_reuse(vmf);
3316+ wp_page_reuse(vmf, NULL);
32733317 return 0;
32743318 }
32753319
@@ -3285,10 +3329,10 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 9 unmodified lines
32853329 vm_fault_t tmp;
32863330
32873331 pte_unmap_unlock(vmf->pte, vmf->ptl);
3288− if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
3332+ tmp = vmf_can_call_fault(vmf);
3333+ if (tmp) {
32893334 folio_put(folio);
3290− vma_end_read(vmf->vma);
3291− return VM_FAULT_RETRY;
3335+ return tmp;
32923336 }
32933337
32943338 tmp = do_page_mkwrite(vmf, folio);
@@ -3297,14 +3341,14 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines
32973341 folio_put(folio);
32983342 return tmp;
32993343 }
3300− tmp = finish_mkwrite_fault(vmf);
3344+ tmp = finish_mkwrite_fault(vmf, folio);
33013345 if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {
33023346 folio_unlock(folio);
33033347 folio_put(folio);
33043348 return tmp;
33053349 }
33063350 } else {
3307− wp_page_reuse(vmf);
3351+ wp_page_reuse(vmf, folio);
33083352 folio_lock(folio);
33093353 }
33103354 ret |= fault_dirty_shared_page(vmf);
@@ -3313,6 +3357,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines
33133357 return ret;
33143358 }
33153359
3360+static bool wp_can_reuse_anon_folio(struct folio *folio,
3361+ struct vm_area_struct *vma)
3362+{
3363+ /*
3364+ * We have to verify under folio lock: these early checks are
3365+ * just an optimization to avoid locking the folio and freeing
3366+ * the swapcache if there is little hope that we can reuse.
3367+ *
3368+ * KSM doesn't necessarily raise the folio refcount.
3369+ */
3370+ if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
3371+ return false;
3372+ if (!folio_test_lru(folio))
3373+ /*
3374+ * We cannot easily detect+handle references from
3375+ * remote LRU caches or references to LRU folios.
3376+ */
3377+ lru_add_drain();
3378+ if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
3379+ return false;
3380+ if (!folio_trylock(folio))
3381+ return false;
3382+ if (folio_test_swapcache(folio))
3383+ folio_free_swap(folio);
3384+ if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) {
3385+ folio_unlock(folio);
3386+ return false;
3387+ }
3388+ /*
3389+ * Ok, we've got the only folio reference from our mapping
3390+ * and the folio is locked, it's dark out, and we're wearing
3391+ * sunglasses. Hit it.
3392+ */
3393+ folio_move_anon_rmap(folio, vma);
3394+ folio_unlock(folio);
3395+ return true;
3396+}
3397+
33163398 /*
33173399 * This routine handles present pages, when
33183400 * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -3341,11 +3423,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 22 unmodified lines
33413423 const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
33423424 struct vm_area_struct *vma = vmf->vma;
33433425 struct folio *folio = NULL;
3426+ pte_t pte;
33443427
33453428 if (likely(!unshare)) {
33463429 if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) {
3347− pte_unmap_unlock(vmf->pte, vmf->ptl);
3348− return handle_userfault(vmf, VM_UFFD_WP);
3430+ if (!userfaultfd_wp_async(vma)) {
3431+ pte_unmap_unlock(vmf->pte, vmf->ptl);
3432+ return handle_userfault(vmf, VM_UFFD_WP);
3433+ }
3434+
3435+ /*
3436+ * Nothing needed (cache flush, TLB invalidations,
3437+ * etc.) because we're only removing the uffd-wp bit,
3438+ * which is completely invisible to the user.
3439+ */
3440+ pte = pte_clear_uffd_wp(ptep_get(vmf->pte));
3441+
3442+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
3443+ /*
3444+ * Update this to be prepared for following up CoW
3445+ * handling
3446+ */
3447+ vmf->orig_pte = pte;
33493448 }
33503449
33513450 /*
@@ -3382,62 +3481,21 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 30 unmodified lines
33823481 /*
33833482 * Private mapping: create an exclusive anonymous page copy if reuse
33843483 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.
3484+ *
3485+ * If we encounter a page that is marked exclusive, we must reuse
3486+ * the page without further checks.
33853487 */
3386− if (folio && folio_test_anon(folio)) {
3387− /*
3388− * If the page is exclusive to this process we must reuse the
3389− * page without further checks.
3390− */
3391− if (PageAnonExclusive(vmf->page))
3392− goto reuse;
3393−
3394− /*
3395− * We have to verify under folio lock: these early checks are
3396− * just an optimization to avoid locking the folio and freeing
3397− * the swapcache if there is little hope that we can reuse.
3398− *
3399− * KSM doesn't necessarily raise the folio refcount.
3400− */
3401− if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
3402− goto copy;
3403− if (!folio_test_lru(folio))
3404− /*
3405− * We cannot easily detect+handle references from
3406− * remote LRU caches or references to LRU folios.
3407− */
3408− lru_add_drain();
3409− if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
3410− goto copy;
3411− if (!folio_trylock(folio))
3412− goto copy;
3413− if (folio_test_swapcache(folio))
3414− folio_free_swap(folio);
3415− if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) {
3416− folio_unlock(folio);
3417− goto copy;
3418− }
3419− /*
3420− * Ok, we've got the only folio reference from our mapping
3421− * and the folio is locked, it's dark out, and we're wearing
3422− * sunglasses. Hit it.
3423− */
3424− page_move_anon_rmap(vmf->page, vma);
3425− folio_unlock(folio);
3426−reuse:
3488+ if (folio && folio_test_anon(folio) &&
3489+ (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
3490+ if (!PageAnonExclusive(vmf->page))
3491+ SetPageAnonExclusive(vmf->page);
34273492 if (unlikely(unshare)) {
34283493 pte_unmap_unlock(vmf->pte, vmf->ptl);
34293494 return 0;
34303495 }
3431− wp_page_reuse(vmf);
3496+ wp_page_reuse(vmf, folio);
34323497 return 0;
34333498 }
3434−copy:
3435− if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) {
3436− pte_unmap_unlock(vmf->pte, vmf->ptl);
3437− vma_end_read(vmf->vma);
3438− return VM_FAULT_RETRY;
3439− }
3440−
34413499 /*
34423500 * Ok, we need to copy. Oh, well..
34433501 */
@@ -4563,10 +4621,9 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) · 1119 unmodified lines
45634621 return ret;
45644622 }
45654623
4566− if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
4567− vma_end_read(vmf->vma);
4568− return VM_FAULT_RETRY;
4569− }
4624+ ret = vmf_can_call_fault(vmf);
4625+ if (ret)
4626+ return ret;
45704627
45714628 ret = __do_fault(vmf);
45724629 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
@@ -4585,13 +4642,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) · 12 unmodified lines
45854642 struct vm_area_struct *vma = vmf->vma;
45864643 vm_fault_t ret;
45874644
4588− if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
4589− vma_end_read(vma);
4590− return VM_FAULT_RETRY;
4591− }
4592−
4593− if (unlikely(anon_vma_prepare(vma)))
4594− return VM_FAULT_OOM;
4645+ ret = vmf_can_call_fault(vmf);
4646+ if (!ret)
4647+ ret = vmf_anon_prepare(vmf);
4648+ if (ret)
4649+ return ret;
45954650
45964651 vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
45974652 if (!vmf->cow_page)
@@ -4630,10 +4685,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) · 32 unmodified lines
46304685 vm_fault_t ret, tmp;
46314686 struct folio *folio;
46324687
4633− if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
4634− vma_end_read(vma);
4635− return VM_FAULT_RETRY;
4636− }
4688+ ret = vmf_can_call_fault(vmf);
4689+ if (ret)
4690+ return ret;
46374691
46384692 ret = __do_fault(vmf);
46394693 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
@@ -4719,10 +4773,10 @@ static vm_fault_t do_fault(struct vm_fault *vmf) · 79 unmodified lines
47194773 return ret;
47204774 }
47214775
4722−int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
4776+int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma,
47234777 unsigned long addr, int page_nid, int *flags)
47244778 {
4725− get_page(page);
4779+ folio_get(folio);
47264780
47274781 /* Record the current PID acceesing VMA */
47284782 vma_set_access_pid_bit(vma);
@@ -4733,14 +4787,14 @@ int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, · 4 unmodified lines
47334787 *flags |= TNF_FAULT_LOCAL;
47344788 }
47354789
4736− return mpol_misplaced(page, vma, addr);
4790+ return mpol_misplaced(folio, vma, addr);
47374791 }
47384792
47394793 static vm_fault_t do_numa_page(struct vm_fault *vmf)
47404794 {
47414795 struct vm_area_struct *vma = vmf->vma;
4742− struct page *page = NULL;
4743− int page_nid = NUMA_NO_NODE;
4796+ struct folio *folio = NULL;
4797+ int nid = NUMA_NO_NODE;
47444798 bool writable = false;
47454799 int last_cpupid;
47464800 int target_nid;
@@ -4771,12 +4825,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 24 unmodified lines
47714825 can_change_pte_writable(vma, vmf->address, pte))
47724826 writable = true;
47734827
4774− page = vm_normal_page(vma, vmf->address, pte);
4775− if (!page || is_zone_device_page(page))
4828+ folio = vm_normal_folio(vma, vmf->address, pte);
4829+ if (!folio || folio_is_zone_device(folio))
47764830 goto out_map;
47774831
47784832 /* TODO: handle PTE-mapped THP */
4779− if (PageCompound(page))
4833+ if (folio_test_large(folio))
47804834 goto out_map;
47814835
47824836 /*
@@ -4791,34 +4845,33 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 8 unmodified lines
47914845 flags |= TNF_NO_GROUP;
47924846
47934847 /*
4794− * Flag if the page is shared between multiple address spaces. This
4848+ * Flag if the folio is shared between multiple address spaces. This
47954849 * is later used when determining whether to group tasks together
47964850 */
4797− if (page_mapcount(page) > 1 && (vma->vm_flags & VM_SHARED))
4851+ if (folio_estimated_sharers(folio) > 1 && (vma->vm_flags & VM_SHARED))
47984852 flags |= TNF_SHARED;
47994853
4800− page_nid = page_to_nid(page);
4854+ nid = folio_nid(folio);
48014855 /*
48024856 * For memory tiering mode, cpupid of slow memory page is used
48034857 * to record page access time. So use default value.
48044858 */
48054859 if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
4806− !node_is_toptier(page_nid))
4860+ !node_is_toptier(nid))
48074861 last_cpupid = (-1 & LAST_CPUPID_MASK);
48084862 else
4809− last_cpupid = page_cpupid_last(page);
4810− target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
4811− &flags);
4863+ last_cpupid = folio_last_cpupid(folio);
4864+ target_nid = numa_migrate_prep(folio, vma, vmf->address, nid, &flags);
48124865 if (target_nid == NUMA_NO_NODE) {
4813− put_page(page);
4866+ folio_put(folio);
48144867 goto out_map;
48154868 }
48164869 pte_unmap_unlock(vmf->pte, vmf->ptl);
48174870 writable = false;
48184871
48194872 /* Migrate to the requested node */
4820− if (migrate_misplaced_page(page, vma, target_nid)) {
4821− page_nid = target_nid;
4873+ if (migrate_misplaced_folio(folio, vma, target_nid)) {
4874+ nid = target_nid;
48224875 flags |= TNF_MIGRATED;
48234876 } else {
48244877 flags |= TNF_MIGRATE_FAIL;
@@ -4834,8 +4887,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 9 unmodified lines
48344887 }
48354888
48364889 out:
4837− if (page_nid != NUMA_NO_NODE)
4838− task_numa_fault(last_cpupid, page_nid, 1, flags);
4890+ if (nid != NUMA_NO_NODE)
4891+ task_numa_fault(last_cpupid, nid, 1, flags);
48394892 return 0;
48404893 out_map:
48414894 /*
@@ -4872,8 +4925,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 30 unmodified lines
48724925
48734926 if (vma_is_anonymous(vma)) {
48744927 if (likely(!unshare) &&
4875− userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd))
4928+ userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) {
4929+ if (userfaultfd_wp_async(vmf->vma))
4930+ goto split;
48764931 return handle_userfault(vmf, VM_UFFD_WP);
4932+ }
48774933 return do_huge_pmd_wp_page(vmf);
48784934 }
48794935
@@ -4885,6 +4941,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 5 unmodified lines
48854941 }
48864942 }
48874943
4944+split:
48884945 /* COW or write-notify handled on pte level: split pmd. */
48894946 __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL);
48904947
@@ -5736,8 +5793,8 @@ EXPORT_SYMBOL_GPL(generic_access_phys); · 845 unmodified lines
57365793 /*
57375794 * Access another process' address space as given in mm.
57385795 */
5739−int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf,
5740− int len, unsigned int gup_flags)
5796+static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
5797+ void *buf, int len, unsigned int gup_flags)
57415798 {
57425799 void *old_buf = buf;
57435800 int write = gup_flags & FOLL_WRITE;
@@ -5760,7 +5817,7 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 16 unmodified lines
57605817 struct page *page = get_user_page_vma_remote(mm, addr,
57615818 gup_flags, &vma);
57625819
5763− if (IS_ERR_OR_NULL(page)) {
5820+ if (IS_ERR(page)) {
57645821 /* We might need to expand the stack to access it */
57655822 vma = vma_lookup(mm, addr);
57665823 if (!vma) {
@@ -5774,7 +5831,6 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 7 unmodified lines
57745831 continue;
57755832 }
57765833
5777−
57785834 /*
57795835 * Check if this is a VM_IO | VM_PFNMAP VMA, which
57805836 * we can access using slightly different code.
mm/memory_hotplug.c+6−2
@@ -1689,7 +1689,7 @@ static int scan_movable_pages(unsigned long start, unsigned long end, · 1688 unmodified lines
16891689 */
16901690 if (HPageMigratable(head))
16911691 goto found;
1692− skip = compound_nr(head) - (page - head);
1692+ skip = compound_nr(head) - (pfn - page_to_pfn(head));
16931693 pfn += skip - 1;
16941694 }
16951695 return -ENOENT;
@@ -2012,12 +2012,16 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, · 316 unmodified lines
20122012 /* reinitialise watermarks and update pcp limits */
20132013 init_per_zone_wmark_min();
20142014
2015+ /*
2016+ * Make sure to mark the node as memory-less before rebuilding the zone
2017+ * list. Otherwise this node would still appear in the fallback lists.
2018+ */
2019+ node_states_clear_node(node, &arg);
20152020 if (!populated_zone(zone)) {
20162021 zone_pcp_reset(zone);
20172022 build_all_zonelists(NULL);
20182023 }
20192024
2020− node_states_clear_node(node, &arg);
20212025 if (arg.status_change_nid >= 0) {
20222026 kcompactd_stop(node);
20232027 kswapd_stop(node);
mm/mempolicy.c+455−571
@@ -25,7 +25,7 @@ · 24 unmodified lines
2525 * to the last. It would be better if bind would truly restrict
2626 * the allocation to memory nodes instead
2727 *
28− * preferred Try a specific node first before normal fallback.
28+ * preferred Try a specific node first before normal fallback.
2929 * As a special case NUMA_NO_NODE here means do the allocation
3030 * on the local CPU. This is normally identical to default,
3131 * but useful to set in a VMA when you have a non default
@@ -52,7 +52,7 @@ · 20 unmodified lines
5252 * on systems with highmem kernel lowmem allocation don't get policied.
5353 * Same with GFP_DMA allocations.
5454 *
55− * For shmfs/tmpfs/hugetlbfs shared memory the policy is shared between
55+ * For shmem/tmpfs shared memory the policy is shared between
5656 * all users and remembered even when nobody has memory mapped.
5757 */
5858
@@ -111,7 +111,8 @@ · 52 unmodified lines
111111
112112 /* Internal flags */
113113 #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0) /* Skip checks for continuous vmas */
114−#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */
114+#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */
115+#define MPOL_MF_WRLOCK (MPOL_MF_INTERNAL << 2) /* Write-lock walked vmas */
115116
116117 static struct kmem_cache *policy_cache;
117118 static struct kmem_cache *sn_cache;
@@ -267,9 +268,6 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 149 unmodified lines
267268 {
268269 struct mempolicy *policy;
269270
270− pr_debug("setting mode %d flags %d nodes[0] %lx\n",
271− mode, flags, nodes ? nodes_addr(*nodes)[0] : NUMA_NO_NODE);
272−
273271 if (mode == MPOL_DEFAULT) {
274272 if (nodes && !nodes_empty(*nodes))
275273 return ERR_PTR(-EINVAL);
@@ -297,6 +295,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 21 unmodified lines
297295 return ERR_PTR(-EINVAL);
298296 } else if (nodes_empty(*nodes))
299297 return ERR_PTR(-EINVAL);
298+
300299 policy = kmem_cache_alloc(policy_cache, GFP_KERNEL);
301300 if (!policy)
302301 return ERR_PTR(-ENOMEM);
@@ -309,11 +308,11 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 6 unmodified lines
309308 }
310309
311310 /* Slow path of a mpol destructor. */
312−void __mpol_put(struct mempolicy *p)
311+void __mpol_put(struct mempolicy *pol)
313312 {
314− if (!atomic_dec_and_test(&p->refcnt))
313+ if (!atomic_dec_and_test(&pol->refcnt))
315314 return;
316− kmem_cache_free(policy_cache, p);
315+ kmem_cache_free(policy_cache, pol);
317316 }
318317
319318 static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
@@ -370,7 +369,6 @@ static void mpol_rebind_policy(struct mempolicy *pol, const nodemask_t *newmask) · 50 unmodified lines
370369 *
371370 * Called with task's alloc_lock held.
372371 */
373−
374372 void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new)
375373 {
376374 mpol_rebind_policy(tsk->mempolicy, new);
@@ -381,7 +379,6 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) · 4 unmodified lines
381379 *
382380 * Call holding a reference to mm. Takes mm->mmap_lock during call.
383381 */
384−
385382 void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)
386383 {
387384 struct vm_area_struct *vma;
@@ -420,8 +417,25 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { · 32 unmodified lines
420417 },
421418 };
422419
423−static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,
420+static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
424421 unsigned long flags);
422+static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
423+ pgoff_t ilx, int *nid);
424+
425+static bool strictly_unmovable(unsigned long flags)
426+{
427+ /*
428+ * STRICT without MOVE flags lets do_mbind() fail immediately with -EIO
429+ * if any misplaced page is found.
430+ */
431+ return (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ==
432+ MPOL_MF_STRICT;
433+}
434+
435+struct migration_mpol { /* for alloc_migration_target_by_mpol() */
436+ struct mempolicy *pol;
437+ pgoff_t ilx;
438+};
425439
426440 struct queue_pages {
427441 struct list_head *pagelist;
@@ -430,7 +444,8 @@ struct queue_pages { · 2 unmodified lines
430444 unsigned long start;
431445 unsigned long end;
432446 struct vm_area_struct *first;
433− bool has_unmovable;
447+ struct folio *large; /* note last large folio encountered */
448+ long nr_failed; /* could not be isolated at this time */
434449 };
435450
436451 /*
@@ -448,61 +463,37 @@ static inline bool queue_folio_required(struct folio *folio, · 11 unmodified lines
448463 return node_isset(nid, *qp->nmask) == !(flags & MPOL_MF_INVERT);
449464 }
450465
451−/*
452− * queue_folios_pmd() has three possible return values:
453− * 0 - folios are placed on the right node or queued successfully, or
454− * special page is met, i.e. zero page, or unmovable page is found
455− * but continue walking (indicated by queue_pages.has_unmovable).
456− * -EIO - is migration entry or only MPOL_MF_STRICT was specified and an
457− * existing folio was already on a node that does not follow the
458− * policy.
459− */
460−static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,
461− unsigned long end, struct mm_walk *walk)
462− __releases(ptl)
466+static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
463467 {
464− int ret = 0;
465468 struct folio *folio;
466469 struct queue_pages *qp = walk->private;
467− unsigned long flags;
468470
469471 if (unlikely(is_pmd_migration_entry(*pmd))) {
470− ret = -EIO;
471− goto unlock;
472+ qp->nr_failed++;
473+ return;
472474 }
473475 folio = pfn_folio(pmd_pfn(*pmd));
474476 if (is_huge_zero_page(&folio->page)) {
475477 walk->action = ACTION_CONTINUE;
476− goto unlock;
478+ return;
477479 }
478480 if (!queue_folio_required(folio, qp))
479− goto unlock;
480−
481− flags = qp->flags;
482− /* go to folio migration */
483− if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
484− if (!vma_migratable(walk->vma) ||
485− migrate_folio_add(folio, qp->pagelist, flags)) {
486− qp->has_unmovable = true;
487− goto unlock;
488− }
489− } else
490− ret = -EIO;
491−unlock:
492− spin_unlock(ptl);
493− return ret;
481+ return;
482+ if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
483+ !vma_migratable(walk->vma) ||
484+ !migrate_folio_add(folio, qp->pagelist, qp->flags))
485+ qp->nr_failed++;
494486 }
495487
496488 /*
497− * Scan through pages checking if pages follow certain conditions,
498− * and move them to the pagelist if they do.
489+ * Scan through folios, checking if they satisfy the required conditions,
490+ * moving them from LRU to local pagelist for migration if they do (or not).
499491 *
500− * queue_folios_pte_range() has three possible return values:
501− * 0 - folios are placed on the right node or queued successfully, or
502− * special page is met, i.e. zero page, or unmovable page is found
503− * but continue walking (indicated by queue_pages.has_unmovable).
504− * -EIO - only MPOL_MF_STRICT was specified and an existing folio was already
505− * on a node that does not follow the policy.
492+ * queue_folios_pte_range() has two possible return values:
493+ * 0 - continue walking to scan for more, even if an existing folio on the
494+ * wrong node could not be isolated and queued for migration.
495+ * -EIO - only MPOL_MF_STRICT was specified, without MPOL_MF_MOVE or ..._ALL,
496+ * and an existing folio was on a node that does not follow the policy.
506497 */
507498 static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
508499 unsigned long end, struct mm_walk *walk)
@@ -516,8 +507,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 7 unmodified lines
516507 spinlock_t *ptl;
517508
518509 ptl = pmd_trans_huge_lock(pmd, vma);
519− if (ptl)
520− return queue_folios_pmd(pmd, ptl, addr, end, walk);
510+ if (ptl) {
511+ queue_folios_pmd(pmd, walk);
512+ spin_unlock(ptl);
513+ goto out;
514+ }
521515
522516 mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
523517 if (!pte) {
@@ -526,8 +520,13 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 2 unmodified lines
526520 }
527521 for (; addr != end; pte++, addr += PAGE_SIZE) {
528522 ptent = ptep_get(pte);
529− if (!pte_present(ptent))
523+ if (pte_none(ptent))
524+ continue;
525+ if (!pte_present(ptent)) {
526+ if (is_migration_entry(pte_to_swp_entry(ptent)))
527+ qp->nr_failed++;
530528 continue;
529+ }
531530 folio = vm_normal_folio(vma, addr, ptent);
532531 if (!folio || folio_is_zone_device(folio))
533532 continue;
@@ -539,95 +538,87 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 5 unmodified lines
539538 continue;
540539 if (!queue_folio_required(folio, qp))
541540 continue;
542− if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
543− /*
544− * MPOL_MF_STRICT must be specified if we get here.
545− * Continue walking vmas due to MPOL_MF_MOVE* flags.
546− */
547− if (!vma_migratable(vma))
548− qp->has_unmovable = true;
549−
541+ if (folio_test_large(folio)) {
550542 /*
551− * Do not abort immediately since there may be
552− * temporary off LRU pages in the range. Still
553− * need migrate other LRU pages.
543+ * A large folio can only be isolated from LRU once,
544+ * but may be mapped by many PTEs (and Copy-On-Write may
545+ * intersperse PTEs of other, order 0, folios). This is
546+ * a common case, so don't mistake it for failure (but
547+ * there can be other cases of multi-mapped pages which
548+ * this quick check does not help to filter out - and a
549+ * search of the pagelist might grow to be prohibitive).
550+ *
551+ * migrate_pages(&pagelist) returns nr_failed folios, so
552+ * check "large" now so that queue_pages_range() returns
553+ * a comparable nr_failed folios. This does imply that
554+ * if folio could not be isolated for some racy reason
555+ * at its first PTE, later PTEs will not give it another
556+ * chance of isolation; but keeps the accounting simple.
554557 */
555− if (migrate_folio_add(folio, qp->pagelist, flags))
556− qp->has_unmovable = true;
557− } else
558− break;
558+ if (folio == qp->large)
559+ continue;
560+ qp->large = folio;
561+ }
562+ if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
563+ !vma_migratable(vma) ||
564+ !migrate_folio_add(folio, qp->pagelist, flags)) {
565+ qp->nr_failed++;
566+ if (strictly_unmovable(flags))
567+ break;
568+ }
559569 }
560570 pte_unmap_unlock(mapped_pte, ptl);
561571 cond_resched();
562−
563− return addr != end ? -EIO : 0;
572+out:
573+ if (qp->nr_failed && strictly_unmovable(flags))
574+ return -EIO;
575+ return 0;
564576 }
565577
566578 static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
567579 unsigned long addr, unsigned long end,
568580 struct mm_walk *walk)
569581 {
570− int ret = 0;
571582 #ifdef CONFIG_HUGETLB_PAGE
572583 struct queue_pages *qp = walk->private;
573− unsigned long flags = (qp->flags & MPOL_MF_VALID);
584+ unsigned long flags = qp->flags;
574585 struct folio *folio;
575586 spinlock_t *ptl;
576587 pte_t entry;
577588
578589 ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte);
579590 entry = huge_ptep_get(pte);
580− if (!pte_present(entry))
591+ if (!pte_present(entry)) {
592+ if (unlikely(is_hugetlb_entry_migration(entry)))
593+ qp->nr_failed++;
581594 goto unlock;
595+ }
582596 folio = pfn_folio(pte_pfn(entry));
583597 if (!queue_folio_required(folio, qp))
584598 goto unlock;
585−
586− if (flags == MPOL_MF_STRICT) {
587− /*
588− * STRICT alone means only detecting misplaced folio and no
589− * need to further check other vma.
590− */
591− ret = -EIO;
599+ if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
600+ !vma_migratable(walk->vma)) {
601+ qp->nr_failed++;
592602 goto unlock;
593603 }
594−
595− if (!vma_migratable(walk->vma)) {
596− /*
597− * Must be STRICT with MOVE*, otherwise .test_walk() have
598− * stopped walking current vma.
599− * Detecting misplaced folio but allow migrating folios which
600− * have been queued.
601− */
602− qp->has_unmovable = true;
603− goto unlock;
604− }
605−
606604 /*
607− * With MPOL_MF_MOVE, we try to migrate only unshared folios. If it
608− * is shared it is likely not worth migrating.
605+ * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio.
606+ * Choosing not to migrate a shared folio is not counted as a failure.
609607 *
610608 * To check if the folio is shared, ideally we want to make sure
611609 * every page is mapped to the same process. Doing that is very
612− * expensive, so check the estimated mapcount of the folio instead.
610+ * expensive, so check the estimated sharers of the folio instead.
613611 */
614− if (flags & (MPOL_MF_MOVE_ALL) ||
615− (flags & MPOL_MF_MOVE && folio_estimated_sharers(folio) == 1 &&
616− !hugetlb_pmd_shared(pte))) {
617− if (!isolate_hugetlb(folio, qp->pagelist) &&
618− (flags & MPOL_MF_STRICT))
619− /*
620− * Failed to isolate folio but allow migrating pages
621− * which have been queued.
622− */
623− qp->has_unmovable = true;
624− }
612+ if ((flags & MPOL_MF_MOVE_ALL) ||
613+ (folio_estimated_sharers(folio) == 1 && !hugetlb_pmd_shared(pte)))
614+ if (!isolate_hugetlb(folio, qp->pagelist))
615+ qp->nr_failed++;
625616 unlock:
626617 spin_unlock(ptl);
627−#else
628− BUG();
618+ if (qp->nr_failed && strictly_unmovable(flags))
619+ return -EIO;
629620 #endif
630− return ret;
621+ return 0;
631622 }
632623
633624 #ifdef CONFIG_NUMA_BALANCING
@@ -656,12 +647,6 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, · 22 unmodified lines
656647
657648 return nr_updated;
658649 }
659−#else
660−static unsigned long change_prot_numa(struct vm_area_struct *vma,
661− unsigned long addr, unsigned long end)
662−{
663− return 0;
664−}
665650 #endif /* CONFIG_NUMA_BALANCING */
666651
667652 static int queue_pages_test_walk(unsigned long start, unsigned long end,
@@ -700,16 +685,11 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, · 32 unmodified lines
700685 if (endvma > end)
701686 endvma = end;
702687
703− if (flags & MPOL_MF_LAZY) {
704− /* Similar to task_numa_work, skip inaccessible VMAs */
705− if (!is_vm_hugetlb_page(vma) && vma_is_accessible(vma) &&
706− !(vma->vm_flags & VM_MIXEDMAP))
707− change_prot_numa(vma, start, endvma);
708− return 1;
709− }
710−
711− /* queue pages from current vma */
712− if (flags & MPOL_MF_VALID)
688+ /*
689+ * Check page nodes, and queue pages to move, in the current vma.
690+ * But if no moving, and no strict checking, the scan can be skipped.
691+ */
692+ if (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
713693 return 0;
714694 return 1;
715695 }
@@ -731,22 +711,21 @@ static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { · 15 unmodified lines
731711 /*
732712 * Walk through page tables and collect pages to be migrated.
733713 *
734− * If pages found in a given range are on a set of nodes (determined by
735− * @nodes and @flags,) it's isolated and queued to the pagelist which is
736− * passed via @private.
714+ * If pages found in a given range are not on the required set of @nodes,
715+ * and migration is allowed, they are isolated and queued to @pagelist.
737716 *
738− * queue_pages_range() has three possible return values:
739− * 1 - there is unmovable page, but MPOL_MF_MOVE* & MPOL_MF_STRICT were
740− * specified.
741− * 0 - queue pages successfully or no misplaced page.
742− * errno - i.e. misplaced pages with MPOL_MF_STRICT specified (-EIO) or
743− * memory range specified by nodemask and maxnode points outside
744− * your accessible address space (-EFAULT)
717+ * queue_pages_range() may return:
718+ * 0 - all pages already on the right node, or successfully queued for moving
719+ * (or neither strict checking nor moving requested: only range checking).
720+ * >0 - this number of misplaced folios could not be queued for moving
721+ * (a hugetlbfs page or a transparent huge page being counted as 1).
722+ * -EIO - a misplaced page found, when MPOL_MF_STRICT specified without MOVEs.
723+ * -EFAULT - a hole in the memory range, when MPOL_MF_DISCONTIG_OK unspecified.
745724 */
746−static int
725+static long
747726 queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end,
748727 nodemask_t *nodes, unsigned long flags,
749− struct list_head *pagelist, bool lock_vma)
728+ struct list_head *pagelist)
750729 {
751730 int err;
752731 struct queue_pages qp = {
@@ -756,20 +735,17 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 3 unmodified lines
756735 .start = start,
757736 .end = end,
758737 .first = NULL,
759− .has_unmovable = false,
760738 };
761− const struct mm_walk_ops *ops = lock_vma ?
739+ const struct mm_walk_ops *ops = (flags & MPOL_MF_WRLOCK) ?
762740 &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops;
763741
764742 err = walk_page_range(mm, start, end, ops, &qp);
765743
766− if (qp.has_unmovable)
767− err = 1;
768744 if (!qp.first)
769745 /* whole range in hole */
770746 err = -EFAULT;
771747
772− return err;
748+ return err ? : qp.nr_failed;
773749 }
774750
775751 /*
@@ -777,7 +753,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 1 unmodified line
777753 * This must be called with the mmap_lock held for writing.
778754 */
779755 static int vma_replace_policy(struct vm_area_struct *vma,
780− struct mempolicy *pol)
756+ struct mempolicy *pol)
781757 {
782758 int err;
783759 struct mempolicy *old;
@@ -785,11 +761,6 @@ static int vma_replace_policy(struct vm_area_struct *vma, · 1 unmodified line
785761
786762 vma_assert_write_locked(vma);
787763
788− pr_debug("vma %lx-%lx/%lx vm_ops %p vm_file %p set_policy %p\n",
789− vma->vm_start, vma->vm_end, vma->vm_pgoff,
790− vma->vm_ops, vma->vm_file,
791− vma->vm_ops ? vma->vm_ops->set_policy : NULL);
792−
793764 new = mpol_dup(pol);
794765 if (IS_ERR(new))
795766 return PTR_ERR(new);
@@ -815,10 +786,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 19 unmodified lines
815786 struct vm_area_struct **prev, unsigned long start,
816787 unsigned long end, struct mempolicy *new_pol)
817788 {
818− struct vm_area_struct *merged;
819789 unsigned long vmstart, vmend;
820− pgoff_t pgoff;
821− int err;
822790
823791 vmend = min(end, vma->vm_end);
824792 if (start > vma->vm_start) {
@@ -828,31 +796,14 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 3 unmodified lines
828796 vmstart = vma->vm_start;
829797 }
830798
831− if (mpol_equal(vma_policy(vma), new_pol)) {
799+ if (mpol_equal(vma->vm_policy, new_pol)) {
832800 *prev = vma;
833801 return 0;
834802 }
835803
836− pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT);
837− merged = vma_merge(vmi, vma->vm_mm, *prev, vmstart, vmend, vma->vm_flags,
838− vma->anon_vma, vma->vm_file, pgoff, new_pol,
839− vma->vm_userfaultfd_ctx, anon_vma_name(vma));
840− if (merged) {
841− *prev = merged;
842− return vma_replace_policy(merged, new_pol);
843− }
844−
845− if (vma->vm_start != vmstart) {
846− err = split_vma(vmi, vma, vmstart, 1);
847− if (err)
848− return err;
849− }
850−
851− if (vma->vm_end != vmend) {
852− err = split_vma(vmi, vma, vmend, 0);
853− if (err)
854− return err;
855− }
804+ vma = vma_modify_policy(vmi, *prev, vma, vmstart, vmend, new_pol);
805+ if (IS_ERR(vma))
806+ return PTR_ERR(vma);
856807
857808 *prev = vma;
858809 return vma_replace_policy(vma, new_pol);
@@ -900,18 +851,18 @@ out: · 41 unmodified lines
900851 *
901852 * Called with task's alloc_lock held
902853 */
903−static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
854+static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
904855 {
905856 nodes_clear(*nodes);
906− if (p == &default_policy)
857+ if (pol == &default_policy)
907858 return;
908859
909− switch (p->mode) {
860+ switch (pol->mode) {
910861 case MPOL_BIND:
911862 case MPOL_INTERLEAVE:
912863 case MPOL_PREFERRED:
913864 case MPOL_PREFERRED_MANY:
914− *nodes = p->nodes;
865+ *nodes = pol->nodes;
915866 break;
916867 case MPOL_LOCAL:
917868 /* return empty node mask for local allocation */
@@ -958,6 +909,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 40 unmodified lines
958909 }
959910
960911 if (flags & MPOL_F_ADDR) {
912+ pgoff_t ilx; /* ignored here */
961913 /*
962914 * Do NOT fall back to task policy if the
963915 * vma/shared policy at addr is NULL. We
@@ -969,10 +921,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 5 unmodified lines
969921 mmap_read_unlock(mm);
970922 return -EFAULT;
971923 }
972− if (vma->vm_ops && vma->vm_ops->get_policy)
973− pol = vma->vm_ops->get_policy(vma, addr);
974− else
975− pol = vma->vm_policy;
924+ pol = __get_vma_policy(vma, addr, &ilx);
976925 } else if (addr)
977926 return -EINVAL;
978927
@@ -1032,16 +981,16 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 53 unmodified lines
1032981 }
1033982
1034983 #ifdef CONFIG_MIGRATION
1035−static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,
984+static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
1036985 unsigned long flags)
1037986 {
1038987 /*
1039− * We try to migrate only unshared folios. If it is shared it
1040− * is likely not worth migrating.
988+ * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio.
989+ * Choosing not to migrate a shared folio is not counted as a failure.
1041990 *
1042991 * To check if the folio is shared, ideally we want to make sure
1043992 * every page is mapped to the same process. Doing that is very
1044− * expensive, so check the estimated mapcount of the folio instead.
993+ * expensive, so check the estimated sharers of the folio instead.
1045994 */
1046995 if ((flags & MPOL_MF_MOVE_ALL) || folio_estimated_sharers(folio) == 1) {
1047996 if (folio_isolate_lru(folio)) {
@@ -1049,32 +998,31 @@ static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, · 1 unmodified line
1049998 node_stat_mod_folio(folio,
1050999 NR_ISOLATED_ANON + folio_is_file_lru(folio),
10511000 folio_nr_pages(folio));
1052− } else if (flags & MPOL_MF_STRICT) {
1001+ } else {
10531002 /*
10541003 * Non-movable folio may reach here. And, there may be
10551004 * temporary off LRU folios or non-LRU movable folios.
10561005 * Treat them as unmovable folios since they can't be
1057− * isolated, so they can't be moved at the moment. It
1058− * should return -EIO for this case too.
1006+ * isolated, so they can't be moved at the moment.
10591007 */
1060− return -EIO;
1008+ return false;
10611009 }
10621010 }
1063−
1064− return 0;
1011+ return true;
10651012 }
10661013
10671014 /*
10681015 * Migrate pages from one node to a target node.
10691016 * Returns error or the number of pages not migrated.
10701017 */
1071−static int migrate_to_node(struct mm_struct *mm, int source, int dest,
1072− int flags)
1018+static long migrate_to_node(struct mm_struct *mm, int source, int dest,
1019+ int flags)
10731020 {
10741021 nodemask_t nmask;
10751022 struct vm_area_struct *vma;
10761023 LIST_HEAD(pagelist);
1077− int err = 0;
1024+ long nr_failed;
1025+ long err = 0;
10781026 struct migration_target_control mtc = {
10791027 .nid = dest,
10801028 .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
@@ -1083,23 +1031,30 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 2 unmodified lines
10831031 nodes_clear(nmask);
10841032 node_set(source, nmask);
10851033
1034+ VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)));
1035+
1036+ mmap_read_lock(mm);
1037+ vma = find_vma(mm, 0);
1038+
10861039 /*
1087− * This does not "check" the range but isolates all pages that
1040+ * This does not migrate the range, but isolates all pages that
10881041 * need migration. Between passing in the full user address
1089− * space range and MPOL_MF_DISCONTIG_OK, this call can not fail.
1042+ * space range and MPOL_MF_DISCONTIG_OK, this call cannot fail,
1043+ * but passes back the count of pages which could not be isolated.
10901044 */
1091− vma = find_vma(mm, 0);
1092− VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)));
1093− queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask,
1094− flags | MPOL_MF_DISCONTIG_OK, &pagelist, false);
1045+ nr_failed = queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask,
1046+ flags | MPOL_MF_DISCONTIG_OK, &pagelist);
1047+ mmap_read_unlock(mm);
10951048
10961049 if (!list_empty(&pagelist)) {
10971050 err = migrate_pages(&pagelist, alloc_migration_target, NULL,
1098− (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);
1051+ (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);
10991052 if (err)
11001053 putback_movable_pages(&pagelist);
11011054 }
11021055
1056+ if (err >= 0)
1057+ err += nr_failed;
11031058 return err;
11041059 }
11051060
@@ -1112,14 +1067,12 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 6 unmodified lines
11121067 int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,
11131068 const nodemask_t *to, int flags)
11141069 {
1115− int busy = 0;
1116− int err = 0;
1070+ long nr_failed = 0;
1071+ long err = 0;
11171072 nodemask_t tmp;
11181073
11191074 lru_cache_disable();
11201075
1121− mmap_read_lock(mm);
1122−
11231076 /*
11241077 * Find a 'source' bit set in 'tmp' whose corresponding 'dest'
11251078 * bit in 'to' is not also set in 'tmp'. Clear the found 'source'
@@ -1195,59 +1148,58 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 69 unmodified lines
11951148 node_clear(source, tmp);
11961149 err = migrate_to_node(mm, source, dest, flags);
11971150 if (err > 0)
1198− busy += err;
1151+ nr_failed += err;
11991152 if (err < 0)
12001153 break;
12011154 }
1202− mmap_read_unlock(mm);
12031155
12041156 lru_cache_enable();
12051157 if (err < 0)
12061158 return err;
1207− return busy;
1208−
1159+ return (nr_failed < INT_MAX) ? nr_failed : INT_MAX;
12091160 }
12101161
12111162 /*
1212− * Allocate a new page for page migration based on vma policy.
1213− * Start by assuming the page is mapped by the same vma as contains @start.
1214− * Search forward from there, if not. N.B., this assumes that the
1215− * list of pages handed to migrate_pages()--which is how we get here--
1216− * is in virtual address order.
1163+ * Allocate a new folio for page migration, according to NUMA mempolicy.
12171164 */
1218−static struct folio *new_folio(struct folio *src, unsigned long start)
1165+static struct folio *alloc_migration_target_by_mpol(struct folio *src,
1166+ unsigned long private)
12191167 {
1220− struct vm_area_struct *vma;
1221− unsigned long address;
1222− VMA_ITERATOR(vmi, current->mm, start);
1223− gfp_t gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL;
1168+ struct migration_mpol *mmpol = (struct migration_mpol *)private;
1169+ struct mempolicy *pol = mmpol->pol;
1170+ pgoff_t ilx = mmpol->ilx;
1171+ struct page *page;
1172+ unsigned int order;
1173+ int nid = numa_node_id();
1174+ gfp_t gfp;
12241175
1225− for_each_vma(vmi, vma) {
1226− address = page_address_in_vma(&src->page, vma);
1227− if (address != -EFAULT)
1228− break;
1229− }
1176+ order = folio_order(src);
1177+ ilx += src->index >> order;
12301178
12311179 if (folio_test_hugetlb(src)) {
1232− return alloc_hugetlb_folio_vma(folio_hstate(src),
1233− vma, address);
1180+ nodemask_t *nodemask;
1181+ struct hstate *h;
1182+
1183+ h = folio_hstate(src);
1184+ gfp = htlb_alloc_mask(h);
1185+ nodemask = policy_nodemask(gfp, pol, ilx, &nid);
1186+ return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp);
12341187 }
12351188
12361189 if (folio_test_large(src))
12371190 gfp = GFP_TRANSHUGE;
1191+ else
1192+ gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP;
12381193
1239− /*
1240− * if !vma, vma_alloc_folio() will use task or system default policy
1241− */
1242− return vma_alloc_folio(gfp, folio_order(src), vma, address,
1243− folio_test_large(src));
1194+ page = alloc_pages_mpol(gfp, order, pol, ilx, nid);
1195+ return page_rmappable_folio(page);
12441196 }
12451197 #else
12461198
1247−static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,
1199+static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
12481200 unsigned long flags)
12491201 {
1250− return -EIO;
1202+ return false;
12511203 }
12521204
12531205 int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,
@@ -1256,7 +1208,8 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 2 unmodified lines
12561208 return -ENOSYS;
12571209 }
12581210
1259−static struct folio *new_folio(struct folio *src, unsigned long start)
1211+static struct folio *alloc_migration_target_by_mpol(struct folio *src,
1212+ unsigned long private)
12601213 {
12611214 return NULL;
12621215 }
@@ -1269,10 +1222,11 @@ static long do_mbind(unsigned long start, unsigned long len, · 6 unmodified lines
12691222 struct mm_struct *mm = current->mm;
12701223 struct vm_area_struct *vma, *prev;
12711224 struct vma_iterator vmi;
1225+ struct migration_mpol mmpol;
12721226 struct mempolicy *new;
12731227 unsigned long end;
1274− int err;
1275− int ret;
1228+ long err;
1229+ long nr_failed;
12761230 LIST_HEAD(pagelist);
12771231
12781232 if (flags & ~(unsigned long)MPOL_MF_VALID)
@@ -1298,9 +1252,6 @@ static long do_mbind(unsigned long start, unsigned long len, · 19 unmodified lines
12981252 if (IS_ERR(new))
12991253 return PTR_ERR(new);
13001254
1301− if (flags & MPOL_MF_LAZY)
1302− new->flags |= MPOL_F_MOF;
1303−
13041255 /*
13051256 * If we are using the default policy then operation
13061257 * on discontinuous address spaces is okay after all
@@ -1308,14 +1259,8 @@ static long do_mbind(unsigned long start, unsigned long len, · 1 unmodified line
13081259 if (!new)
13091260 flags |= MPOL_MF_DISCONTIG_OK;
13101261
1311− pr_debug("mbind %lx-%lx mode:%d flags:%d nodes:%lx\n",
1312− start, start + len, mode, mode_flags,
1313− nmask ? nodes_addr(*nmask)[0] : NUMA_NO_NODE);
1314−
1315− if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
1316−
1262+ if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
13171263 lru_cache_disable();
1318− }
13191264 {
13201265 NODEMASK_SCRATCH(scratch);
13211266 if (scratch) {
@@ -1331,45 +1276,81 @@ static long do_mbind(unsigned long start, unsigned long len, · 9 unmodified lines
13311276 goto mpol_out;
13321277
13331278 /*
1334− * Lock the VMAs before scanning for pages to migrate, to ensure we don't
1335− * miss a concurrently inserted page.
1279+ * Lock the VMAs before scanning for pages to migrate,
1280+ * to ensure we don't miss a concurrently inserted page.
13361281 */
1337− ret = queue_pages_range(mm, start, end, nmask,
1338− flags | MPOL_MF_INVERT, &pagelist, true);
1282+ nr_failed = queue_pages_range(mm, start, end, nmask,
1283+ flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist);
13391284
1340− if (ret < 0) {
1341− err = ret;
1342− goto up_out;
1343− }
1344−
1345− vma_iter_init(&vmi, mm, start);
1346− prev = vma_prev(&vmi);
1347− for_each_vma_range(vmi, vma, end) {
1348− err = mbind_range(&vmi, vma, &prev, start, end, new);
1349− if (err)
1350− break;
1285+ if (nr_failed < 0) {
1286+ err = nr_failed;
1287+ nr_failed = 0;
1288+ } else {
1289+ vma_iter_init(&vmi, mm, start);
1290+ prev = vma_prev(&vmi);
1291+ for_each_vma_range(vmi, vma, end) {
1292+ err = mbind_range(&vmi, vma, &prev, start, end, new);
1293+ if (err)
1294+ break;
1295+ }
13511296 }
13521297
1353− if (!err) {
1354− int nr_failed = 0;
1355−
1356− if (!list_empty(&pagelist)) {
1357− WARN_ON_ONCE(flags & MPOL_MF_LAZY);
1358− nr_failed = migrate_pages(&pagelist, new_folio, NULL,
1359− start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL);
1360− if (nr_failed)
1361− putback_movable_pages(&pagelist);
1298+ if (!err && !list_empty(&pagelist)) {
1299+ /* Convert MPOL_DEFAULT's NULL to task or default policy */
1300+ if (!new) {
1301+ new = get_task_policy(current);
1302+ mpol_get(new);
13621303 }
1304+ mmpol.pol = new;
1305+ mmpol.ilx = 0;
13631306
1364− if (((ret > 0) || nr_failed) && (flags & MPOL_MF_STRICT))
1365− err = -EIO;
1366− } else {
1367−up_out:
1368− if (!list_empty(&pagelist))
1369− putback_movable_pages(&pagelist);
1307+ /*
1308+ * In the interleaved case, attempt to allocate on exactly the
1309+ * targeted nodes, for the first VMA to be migrated; for later
1310+ * VMAs, the nodes will still be interleaved from the targeted
1311+ * nodemask, but one by one may be selected differently.
1312+ */
1313+ if (new->mode == MPOL_INTERLEAVE) {
1314+ struct page *page;
1315+ unsigned int order;
1316+ unsigned long addr = -EFAULT;
1317+
1318+ list_for_each_entry(page, &pagelist, lru) {
1319+ if (!PageKsm(page))
1320+ break;
1321+ }
1322+ if (!list_entry_is_head(page, &pagelist, lru)) {
1323+ vma_iter_init(&vmi, mm, start);
1324+ for_each_vma_range(vmi, vma, end) {
1325+ addr = page_address_in_vma(page, vma);
1326+ if (addr != -EFAULT)
1327+ break;
1328+ }
1329+ }
1330+ if (addr != -EFAULT) {
1331+ order = compound_order(page);
1332+ /* We already know the pol, but not the ilx */
1333+ mpol_cond_put(get_vma_policy(vma, addr, order,
1334+ &mmpol.ilx));
1335+ /* Set base from which to increment by index */
1336+ mmpol.ilx -= page->index >> order;
1337+ }
1338+ }
13701339 }
13711340
13721341 mmap_write_unlock(mm);
1342+
1343+ if (!err && !list_empty(&pagelist)) {
1344+ nr_failed |= migrate_pages(&pagelist,
1345+ alloc_migration_target_by_mpol, NULL,
1346+ (unsigned long)&mmpol, MIGRATE_SYNC,
1347+ MR_MEMPOLICY_MBIND, NULL);
1348+ }
1349+
1350+ if (nr_failed && (flags & MPOL_MF_STRICT))
1351+ err = -EIO;
1352+ if (!list_empty(&pagelist))
1353+ putback_movable_pages(&pagelist);
13731354 mpol_out:
13741355 mpol_put(new);
13751356 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
@@ -1690,7 +1671,6 @@ out: · 314 unmodified lines
16901671 out_put:
16911672 put_task_struct(task);
16921673 goto out;
1693−
16941674 }
16951675
16961676 SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode,
@@ -1700,7 +1680,6 @@ SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, · 3 unmodified lines
17001680 return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes);
17011681 }
17021682
1703−
17041683 /* Retrieve NUMA policy */
17051684 static int kernel_get_mempolicy(int __user *policy,
17061685 unsigned long __user *nmask,
@@ -1767,34 +1746,19 @@ bool vma_migratable(struct vm_area_struct *vma) · 60 unmodified lines
17671746 }
17681747
17691748 struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
1770− unsigned long addr)
1749+ unsigned long addr, pgoff_t *ilx)
17711750 {
1772− struct mempolicy *pol = NULL;
1773−
1774− if (vma) {
1775− if (vma->vm_ops && vma->vm_ops->get_policy) {
1776− pol = vma->vm_ops->get_policy(vma, addr);
1777− } else if (vma->vm_policy) {
1778− pol = vma->vm_policy;
1779−
1780− /*
1781− * shmem_alloc_page() passes MPOL_F_SHARED policy with
1782− * a pseudo vma whose vma->vm_ops=NULL. Take a reference
1783− * count on these policies which will be dropped by
1784− * mpol_cond_put() later
1785− */
1786− if (mpol_needs_cond_ref(pol))
1787− mpol_get(pol);
1788− }
1789− }
1790−
1791− return pol;
1751+ *ilx = 0;
1752+ return (vma->vm_ops && vma->vm_ops->get_policy) ?
1753+ vma->vm_ops->get_policy(vma, addr, ilx) : vma->vm_policy;
17921754 }
17931755
17941756 /*
1795− * get_vma_policy(@vma, @addr)
1757+ * get_vma_policy(@vma, @addr, @order, @ilx)
17961758 * @vma: virtual memory area whose policy is sought
17971759 * @addr: address in @vma for shared policy lookup
1760+ * @order: 0, or appropriate huge_page_order for interleaving
1761+ * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE
17981762 *
17991763 * Returns effective policy for a VMA at specified address.
18001764 * Falls back to current->mempolicy or system default policy, as necessary.
@@ -1803,14 +1767,18 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, · 2 unmodified lines
18031767 * freeing by another task. It is the caller's responsibility to free the
18041768 * extra reference for shared policies.
18051769 */
1806−static struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
1807− unsigned long addr)
1770+struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
1771+ unsigned long addr, int order, pgoff_t *ilx)
18081772 {
1809− struct mempolicy *pol = __get_vma_policy(vma, addr);
1773+ struct mempolicy *pol;
18101774
1775+ pol = __get_vma_policy(vma, addr, ilx);
18111776 if (!pol)
18121777 pol = get_task_policy(current);
1813−
1778+ if (pol->mode == MPOL_INTERLEAVE) {
1779+ *ilx += vma->vm_pgoff >> order;
1780+ *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
1781+ }
18141782 return pol;
18151783 }
18161784
@@ -1820,8 +1788,9 @@ bool vma_policy_mof(struct vm_area_struct *vma) · 3 unmodified lines
18201788
18211789 if (vma->vm_ops && vma->vm_ops->get_policy) {
18221790 bool ret = false;
1791+ pgoff_t ilx; /* ignored here */
18231792
1824− pol = vma->vm_ops->get_policy(vma, vma->vm_start);
1793+ pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx);
18251794 if (pol && (pol->flags & MPOL_F_MOF))
18261795 ret = true;
18271796 mpol_cond_put(pol);
@@ -1856,64 +1825,15 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) · 28 unmodified lines
18561825 return zone >= dynamic_policy_zone;
18571826 }
18581827
1859−/*
1860− * Return a nodemask representing a mempolicy for filtering nodes for
1861− * page allocation
1862− */
1863−nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy)
1864−{
1865− int mode = policy->mode;
1866−
1867− /* Lower zones don't get a nodemask applied for MPOL_BIND */
1868− if (unlikely(mode == MPOL_BIND) &&
1869− apply_policy_zone(policy, gfp_zone(gfp)) &&
1870− cpuset_nodemask_valid_mems_allowed(&policy->nodes))
1871− return &policy->nodes;
1872−
1873− if (mode == MPOL_PREFERRED_MANY)
1874− return &policy->nodes;
1875−
1876− return NULL;
1877−}
1878−
1879−/*
1880− * Return the preferred node id for 'prefer' mempolicy, and return
1881− * the given id for all other policies.
1882− *
1883− * policy_node() is always coupled with policy_nodemask(), which
1884− * secures the nodemask limit for 'bind' and 'prefer-many' policy.
1885− */
1886−static int policy_node(gfp_t gfp, struct mempolicy *policy, int nd)
1887−{
1888− if (policy->mode == MPOL_PREFERRED) {
1889− nd = first_node(policy->nodes);
1890− } else {
1891− /*
1892− * __GFP_THISNODE shouldn't even be used with the bind policy
1893− * because we might easily break the expectation to stay on the
1894− * requested node and not break the policy.
1895− */
1896− WARN_ON_ONCE(policy->mode == MPOL_BIND && (gfp & __GFP_THISNODE));
1897− }
1898−
1899− if ((policy->mode == MPOL_BIND ||
1900− policy->mode == MPOL_PREFERRED_MANY) &&
1901− policy->home_node != NUMA_NO_NODE)
1902− return policy->home_node;
1903−
1904− return nd;
1905−}
1906−
19071828 /* Do dynamic interleaving for a process */
1908−static unsigned interleave_nodes(struct mempolicy *policy)
1829+static unsigned int interleave_nodes(struct mempolicy *policy)
19091830 {
1910− unsigned next;
1911− struct task_struct *me = current;
1831+ unsigned int nid;
19121832
1913− next = next_node_in(me->il_prev, policy->nodes);
1914− if (next < MAX_NUMNODES)
1915− me->il_prev = next;
1916− return next;
1833+ nid = next_node_in(current->il_prev, policy->nodes);
1834+ if (nid < MAX_NUMNODES)
1835+ current->il_prev = nid;
1836+ return nid;
19171837 }
19181838
19191839 /*
@@ -1964,11 +1884,11 @@ unsigned int mempolicy_slab_node(void) · 44 unmodified lines
19641884 }
19651885
19661886 /*
1967− * Do static interleaving for a VMA with known offset @n. Returns the n'th
1968− * node in pol->nodes (starting from n=0), wrapping around if n exceeds the
1969− * number of present nodes.
1887+ * Do static interleaving for interleave index @ilx. Returns the ilx'th
1888+ * node in pol->nodes (starting from ilx=0), wrapping around if ilx
1889+ * exceeds the number of present nodes.
19701890 */
1971−static unsigned offset_il_node(struct mempolicy *pol, unsigned long n)
1891+static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx)
19721892 {
19731893 nodemask_t nodemask = pol->nodes;
19741894 unsigned int target, nnodes;
@@ -1986,33 +1906,54 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) · 11 unmodified lines
19861906 nnodes = nodes_weight(nodemask);
19871907 if (!nnodes)
19881908 return numa_node_id();
1989− target = (unsigned int)n % nnodes;
1909+ target = ilx % nnodes;
19901910 nid = first_node(nodemask);
19911911 for (i = 0; i < target; i++)
19921912 nid = next_node(nid, nodemask);
19931913 return nid;
19941914 }
19951915
1996−/* Determine a node number for interleave */
1997−static inline unsigned interleave_nid(struct mempolicy *pol,
1998− struct vm_area_struct *vma, unsigned long addr, int shift)
1916+/*
1917+ * Return a nodemask representing a mempolicy for filtering nodes for
1918+ * page allocation, together with preferred node id (or the input node id).
1919+ */
1920+static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
1921+ pgoff_t ilx, int *nid)
19991922 {
2000− if (vma) {
2001− unsigned long off;
1923+ nodemask_t *nodemask = NULL;
20021924
1925+ switch (pol->mode) {
1926+ case MPOL_PREFERRED:
1927+ /* Override input node id */
1928+ *nid = first_node(pol->nodes);
1929+ break;
1930+ case MPOL_PREFERRED_MANY:
1931+ nodemask = &pol->nodes;
1932+ if (pol->home_node != NUMA_NO_NODE)
1933+ *nid = pol->home_node;
1934+ break;
1935+ case MPOL_BIND:
1936+ /* Restrict to nodemask (but not on lower zones) */
1937+ if (apply_policy_zone(pol, gfp_zone(gfp)) &&
1938+ cpuset_nodemask_valid_mems_allowed(&pol->nodes))
1939+ nodemask = &pol->nodes;
1940+ if (pol->home_node != NUMA_NO_NODE)
1941+ *nid = pol->home_node;
20031942 /*
2004− * for small pages, there is no difference between
2005− * shift and PAGE_SHIFT, so the bit-shift is safe.
2006− * for huge pages, since vm_pgoff is in units of small
2007− * pages, we need to shift off the always 0 bits to get
2008− * a useful offset.
1943+ * __GFP_THISNODE shouldn't even be used with the bind policy
1944+ * because we might easily break the expectation to stay on the
1945+ * requested node and not break the policy.
20091946 */
2010− BUG_ON(shift < PAGE_SHIFT);
2011− off = vma->vm_pgoff >> (shift - PAGE_SHIFT);
2012− off += (addr - vma->vm_start) >> shift;
2013− return offset_il_node(pol, off);
2014− } else
2015− return interleave_nodes(pol);
1947+ WARN_ON_ONCE(gfp & __GFP_THISNODE);
1948+ break;
1949+ case MPOL_INTERLEAVE:
1950+ /* Override input node id */
1951+ *nid = (ilx == NO_INTERLEAVE_INDEX) ?
1952+ interleave_nodes(pol) : interleave_nid(pol, ilx);
1953+ break;
1954+ }
1955+
1956+ return nodemask;
20161957 }
20171958
20181959 #ifdef CONFIG_HUGETLBFS
@@ -2028,27 +1969,16 @@ static inline unsigned interleave_nid(struct mempolicy *pol, · 9 unmodified lines
20281969 * to the struct mempolicy for conditional unref after allocation.
20291970 * If the effective policy is 'bind' or 'prefer-many', returns a pointer
20301971 * to the mempolicy's @nodemask for filtering the zonelist.
2031− *
2032− * Must be protected by read_mems_allowed_begin()
20331972 */
20341973 int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags,
2035− struct mempolicy **mpol, nodemask_t **nodemask)
1974+ struct mempolicy **mpol, nodemask_t **nodemask)
20361975 {
1976+ pgoff_t ilx;
20371977 int nid;
2038− int mode;
2039−
2040− *mpol = get_vma_policy(vma, addr);
2041− *nodemask = NULL;
2042− mode = (*mpol)->mode;
20431978
2044− if (unlikely(mode == MPOL_INTERLEAVE)) {
2045− nid = interleave_nid(*mpol, vma, addr,
2046− huge_page_shift(hstate_vma(vma)));
2047− } else {
2048− nid = policy_node(gfp_flags, *mpol, numa_node_id());
2049− if (mode == MPOL_BIND || mode == MPOL_PREFERRED_MANY)
2050− *nodemask = &(*mpol)->nodes;
2051− }
1979+ nid = numa_node_id();
1980+ *mpol = get_vma_policy(vma, addr, hstate_vma(vma)->order, &ilx);
1981+ *nodemask = policy_nodemask(gfp_flags, *mpol, ilx, &nid);
20521982 return nid;
20531983 }
20541984
@@ -2126,27 +2056,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk, · 71 unmodified lines
21262056 return ret;
21272057 }
21282058
2129−/* Allocate a page in interleaved policy.
2130− Own path because it needs to do special accounting. */
2131−static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
2132− unsigned nid)
2133−{
2134− struct page *page;
2135−
2136− page = __alloc_pages(gfp, order, nid, NULL);
2137− /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
2138− if (!static_branch_likely(&vm_numa_stat_key))
2139− return page;
2140− if (page && page_to_nid(page) == nid) {
2141− preempt_disable();
2142− __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT);
2143− preempt_enable();
2144− }
2145− return page;
2146−}
2147−
21482059 static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
2149− int nid, struct mempolicy *pol)
2060+ int nid, nodemask_t *nodemask)
21502061 {
21512062 struct page *page;
21522063 gfp_t preferred_gfp;
@@ -2159,7 +2070,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 6 unmodified lines
21592070 */
21602071 preferred_gfp = gfp | __GFP_NOWARN;
21612072 preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
2162− page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes);
2073+ page = __alloc_pages(preferred_gfp, order, nid, nodemask);
21632074 if (!page)
21642075 page = __alloc_pages(gfp, order, nid, NULL);
21652076
@@ -2167,61 +2078,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 1 unmodified line
21672078 }
21682079
21692080 /**
2170− * vma_alloc_folio - Allocate a folio for a VMA.
2081+ * alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
21712082 * @gfp: GFP flags.
2172− * @order: Order of the folio.
2173− * @vma: Pointer to VMA or NULL if not available.
2174− * @addr: Virtual address of the allocation. Must be inside @vma.
2175− * @hugepage: For hugepages try only the preferred node if possible.
2083+ * @order: Order of the page allocation.
2084+ * @pol: Pointer to the NUMA mempolicy.
2085+ * @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()).
2086+ * @nid: Preferred node (usually numa_node_id() but @mpol may override it).
21762087 *
2177− * Allocate a folio for a specific address in @vma, using the appropriate
2178− * NUMA policy. When @vma is not NULL the caller must hold the mmap_lock
2179− * of the mm_struct of the VMA to prevent it from going away. Should be
2180− * used for all allocations for folios that will be mapped into user space.
2181− *
2182− * Return: The folio on success or NULL if allocation fails.
2088+ * Return: The page on success or NULL if allocation fails.
21832089 */
2184−struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,
2185− unsigned long addr, bool hugepage)
2090+struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
2091+ struct mempolicy *pol, pgoff_t ilx, int nid)
21862092 {
2187− struct mempolicy *pol;
2188− int node = numa_node_id();
2189− struct folio *folio;
2190− int preferred_nid;
2191− nodemask_t *nmask;
2192−
2193− pol = get_vma_policy(vma, addr);
2194−
2195− if (pol->mode == MPOL_INTERLEAVE) {
2196− struct page *page;
2197− unsigned nid;
2198−
2199− nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order);
2200− mpol_cond_put(pol);
2201− gfp |= __GFP_COMP;
2202− page = alloc_page_interleave(gfp, order, nid);
2203− folio = (struct folio *)page;
2204− if (folio && order > 1)
2205− folio_prep_large_rmappable(folio);
2206− goto out;
2207− }
2208−
2209− if (pol->mode == MPOL_PREFERRED_MANY) {
2210− struct page *page;
2093+ nodemask_t *nodemask;
2094+ struct page *page;
22112095
2212− node = policy_node(gfp, pol, node);
2213− gfp |= __GFP_COMP;
2214− page = alloc_pages_preferred_many(gfp, order, node, pol);
2215− mpol_cond_put(pol);
2216− folio = (struct folio *)page;
2217− if (folio && order > 1)
2218− folio_prep_large_rmappable(folio);
2219− goto out;
2220− }
2096+ nodemask = policy_nodemask(gfp, pol, ilx, &nid);
22212097
2222− if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) {
2223− int hpage_node = node;
2098+ if (pol->mode == MPOL_PREFERRED_MANY)
2099+ return alloc_pages_preferred_many(gfp, order, nid, nodemask);
22242100
2101+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
2102+ /* filter "hugepage" allocation, unless from alloc_pages() */
2103+ order == HPAGE_PMD_ORDER && ilx != NO_INTERLEAVE_INDEX) {
22252104 /*
22262105 * For hugepage allocation and non-interleave policy which
22272106 * allows the current node (or other explicitly preferred
@@ -2232,39 +2111,68 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, · 4 unmodified lines
22322111 * If the policy is interleave or does not allow the current
22332112 * node in its nodemask, we allocate the standard way.
22342113 */
2235− if (pol->mode == MPOL_PREFERRED)
2236− hpage_node = first_node(pol->nodes);
2237−
2238− nmask = policy_nodemask(gfp, pol);
2239− if (!nmask || node_isset(hpage_node, *nmask)) {
2240− mpol_cond_put(pol);
2114+ if (pol->mode != MPOL_INTERLEAVE &&
2115+ (!nodemask || node_isset(nid, *nodemask))) {
22412116 /*
22422117 * First, try to allocate THP only on local node, but
22432118 * don't reclaim unnecessarily, just compact.
22442119 */
2245− folio = __folio_alloc_node(gfp | __GFP_THISNODE |
2246− __GFP_NORETRY, order, hpage_node);
2247−
2120+ page = __alloc_pages_node(nid,
2121+ gfp | __GFP_THISNODE | __GFP_NORETRY, order);
2122+ if (page || !(gfp & __GFP_DIRECT_RECLAIM))
2123+ return page;
22482124 /*
22492125 * If hugepage allocations are configured to always
22502126 * synchronous compact or the vma has been madvised
22512127 * to prefer hugepage backing, retry allowing remote
22522128 * memory with both reclaim and compact as well.
22532129 */
2254− if (!folio && (gfp & __GFP_DIRECT_RECLAIM))
2255− folio = __folio_alloc(gfp, order, hpage_node,
2256− nmask);
2130+ }
2131+ }
22572132
2258− goto out;
2133+ page = __alloc_pages(gfp, order, nid, nodemask);
2134+
2135+ if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) {
2136+ /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */
2137+ if (static_branch_likely(&vm_numa_stat_key) &&
2138+ page_to_nid(page) == nid) {
2139+ preempt_disable();
2140+ __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT);
2141+ preempt_enable();
22592142 }
22602143 }
22612144
2262− nmask = policy_nodemask(gfp, pol);
2263− preferred_nid = policy_node(gfp, pol, node);
2264− folio = __folio_alloc(gfp, order, preferred_nid, nmask);
2145+ return page;
2146+}
2147+
2148+/**
2149+ * vma_alloc_folio - Allocate a folio for a VMA.
2150+ * @gfp: GFP flags.
2151+ * @order: Order of the folio.
2152+ * @vma: Pointer to VMA.
2153+ * @addr: Virtual address of the allocation. Must be inside @vma.
2154+ * @hugepage: Unused (was: For hugepages try only preferred node if possible).
2155+ *
2156+ * Allocate a folio for a specific address in @vma, using the appropriate
2157+ * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the
2158+ * VMA to prevent it from going away. Should be used for all allocations
2159+ * for folios that will be mapped into user space, excepting hugetlbfs, and
2160+ * excepting where direct use of alloc_pages_mpol() is more appropriate.
2161+ *
2162+ * Return: The folio on success or NULL if allocation fails.
2163+ */
2164+struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,
2165+ unsigned long addr, bool hugepage)
2166+{
2167+ struct mempolicy *pol;
2168+ pgoff_t ilx;
2169+ struct page *page;
2170+
2171+ pol = get_vma_policy(vma, addr, order, &ilx);
2172+ page = alloc_pages_mpol(gfp | __GFP_COMP, order,
2173+ pol, ilx, numa_node_id());
22652174 mpol_cond_put(pol);
2266−out:
2267− return folio;
2175+ return page_rmappable_folio(page);
22682176 }
22692177 EXPORT_SYMBOL(vma_alloc_folio);
22702178
@@ -2282,40 +2190,25 @@ EXPORT_SYMBOL(vma_alloc_folio); · 11 unmodified lines
22822190 * flags are used.
22832191 * Return: The page on success or NULL if allocation fails.
22842192 */
2285−struct page *alloc_pages(gfp_t gfp, unsigned order)
2193+struct page *alloc_pages(gfp_t gfp, unsigned int order)
22862194 {
22872195 struct mempolicy *pol = &default_policy;
2288− struct page *page;
2289−
2290− if (!in_interrupt() && !(gfp & __GFP_THISNODE))
2291− pol = get_task_policy(current);
22922196
22932197 /*
22942198 * No reference counting needed for current->mempolicy
22952199 * nor system default_policy
22962200 */
2297− if (pol->mode == MPOL_INTERLEAVE)
2298− page = alloc_page_interleave(gfp, order, interleave_nodes(pol));
2299− else if (pol->mode == MPOL_PREFERRED_MANY)
2300− page = alloc_pages_preferred_many(gfp, order,
2301− policy_node(gfp, pol, numa_node_id()), pol);
2302− else
2303− page = __alloc_pages(gfp, order,
2304− policy_node(gfp, pol, numa_node_id()),
2305− policy_nodemask(gfp, pol));
2201+ if (!in_interrupt() && !(gfp & __GFP_THISNODE))
2202+ pol = get_task_policy(current);
23062203
2307− return page;
2204+ return alloc_pages_mpol(gfp, order,
2205+ pol, NO_INTERLEAVE_INDEX, numa_node_id());
23082206 }
23092207 EXPORT_SYMBOL(alloc_pages);
23102208
2311−struct folio *folio_alloc(gfp_t gfp, unsigned order)
2209+struct folio *folio_alloc(gfp_t gfp, unsigned int order)
23122210 {
2313− struct page *page = alloc_pages(gfp | __GFP_COMP, order);
2314− struct folio *folio = (struct folio *)page;
2315−
2316− if (folio && order > 1)
2317− folio_prep_large_rmappable(folio);
2318− return folio;
2211+ return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order));
23192212 }
23202213 EXPORT_SYMBOL(folio_alloc);
23212214
@@ -2384,6 +2277,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 62 unmodified lines
23842277 unsigned long nr_pages, struct page **page_array)
23852278 {
23862279 struct mempolicy *pol = &default_policy;
2280+ nodemask_t *nodemask;
2281+ int nid;
23872282
23882283 if (!in_interrupt() && !(gfp & __GFP_THISNODE))
23892284 pol = get_task_policy(current);
@@ -2396,14 +2291,15 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 6 unmodified lines
23962291 return alloc_pages_bulk_array_preferred_many(gfp,
23972292 numa_node_id(), pol, nr_pages, page_array);
23982293
2399− return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()),
2400− policy_nodemask(gfp, pol), nr_pages, NULL,
2401− page_array);
2294+ nid = numa_node_id();
2295+ nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid);
2296+ return __alloc_pages_bulk(gfp, nid, nodemask,
2297+ nr_pages, NULL, page_array);
24022298 }
24032299
24042300 int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
24052301 {
2406− struct mempolicy *pol = mpol_dup(vma_policy(src));
2302+ struct mempolicy *pol = mpol_dup(src->vm_policy);
24072303
24082304 if (IS_ERR(pol))
24092305 return PTR_ERR(pol);
@@ -2488,8 +2384,8 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 78 unmodified lines
24882384 * lookup first element intersecting start-end. Caller holds sp->lock for
24892385 * reading or for writing
24902386 */
2491−static struct sp_node *
2492−sp_lookup(struct shared_policy *sp, unsigned long start, unsigned long end)
2387+static struct sp_node *sp_lookup(struct shared_policy *sp,
2388+ pgoff_t start, pgoff_t end)
24932389 {
24942390 struct rb_node *n = sp->root.rb_node;
24952391
@@ -2540,13 +2436,11 @@ static void sp_insert(struct shared_policy *sp, struct sp_node *new) · 44 unmodified lines
25402436 }
25412437 rb_link_node(&new->nd, parent, p);
25422438 rb_insert_color(&new->nd, &sp->root);
2543− pr_debug("inserting %lx-%lx: %d\n", new->start, new->end,
2544− new->policy ? new->policy->mode : 0);
25452439 }
25462440
25472441 /* Find shared policy intersecting idx */
2548−struct mempolicy *
2549−mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx)
2442+struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
2443+ pgoff_t idx)
25502444 {
25512445 struct mempolicy *pol = NULL;
25522446 struct sp_node *sn;
@@ -2570,39 +2464,38 @@ static void sp_free(struct sp_node *n) · 17 unmodified lines
25702464 }
25712465
25722466 /**
2573− * mpol_misplaced - check whether current page node is valid in policy
2467+ * mpol_misplaced - check whether current folio node is valid in policy
25742468 *
2575− * @page: page to be checked
2576− * @vma: vm area where page mapped
2577− * @addr: virtual address where page mapped
2469+ * @folio: folio to be checked
2470+ * @vma: vm area where folio mapped
2471+ * @addr: virtual address in @vma for shared policy lookup and interleave policy
25782472 *
2579− * Lookup current policy node id for vma,addr and "compare to" page's
2473+ * Lookup current policy node id for vma,addr and "compare to" folio's
25802474 * node id. Policy determination "mimics" alloc_page_vma().
25812475 * Called from fault path where we know the vma and faulting address.
25822476 *
25832477 * Return: NUMA_NO_NODE if the page is in a node that is valid for this
2584− * policy, or a suitable node ID to allocate a replacement page from.
2478+ * policy, or a suitable node ID to allocate a replacement folio from.
25852479 */
2586−int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long addr)
2480+int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
2481+ unsigned long addr)
25872482 {
25882483 struct mempolicy *pol;
2484+ pgoff_t ilx;
25892485 struct zoneref *z;
2590− int curnid = page_to_nid(page);
2591− unsigned long pgoff;
2486+ int curnid = folio_nid(folio);
25922487 int thiscpu = raw_smp_processor_id();
25932488 int thisnid = cpu_to_node(thiscpu);
25942489 int polnid = NUMA_NO_NODE;
25952490 int ret = NUMA_NO_NODE;
25962491
2597− pol = get_vma_policy(vma, addr);
2492+ pol = get_vma_policy(vma, addr, folio_order(folio), &ilx);
25982493 if (!(pol->flags & MPOL_F_MOF))
25992494 goto out;
26002495
26012496 switch (pol->mode) {
26022497 case MPOL_INTERLEAVE:
2603− pgoff = vma->vm_pgoff;
2604− pgoff += (addr - vma->vm_start) >> PAGE_SHIFT;
2605− polnid = offset_il_node(pol, pgoff);
2498+ polnid = interleave_nid(pol, ilx);
26062499 break;
26072500
26082501 case MPOL_PREFERRED:
@@ -2643,11 +2536,12 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long · 34 unmodified lines
26432536 BUG();
26442537 }
26452538
2646− /* Migrate the page towards the node whose CPU is referencing it */
2539+ /* Migrate the folio towards the node whose CPU is referencing it */
26472540 if (pol->flags & MPOL_F_MORON) {
26482541 polnid = thisnid;
26492542
2650− if (!should_numa_migrate_memory(current, page, curnid, thiscpu))
2543+ if (!should_numa_migrate_memory(current, folio, curnid,
2544+ thiscpu))
26512545 goto out;
26522546 }
26532547
@@ -2678,7 +2572,6 @@ void mpol_put_task_policy(struct task_struct *task) · 24 unmodified lines
26782572
26792573 static void sp_delete(struct shared_policy *sp, struct sp_node *n)
26802574 {
2681− pr_debug("deleting %lx-l%lx\n", n->start, n->end);
26822575 rb_erase(&n->nd, &sp->root);
26832576 sp_free(n);
26842577 }
@@ -2713,8 +2606,8 @@ static struct sp_node *sp_alloc(unsigned long start, unsigned long end, · 28 unmodified lines
27132606 }
27142607
27152608 /* Replace a policy range. */
2716−static int shared_policy_replace(struct shared_policy *sp, unsigned long start,
2717− unsigned long end, struct sp_node *new)
2609+static int shared_policy_replace(struct shared_policy *sp, pgoff_t start,
2610+ pgoff_t end, struct sp_node *new)
27182611 {
27192612 struct sp_node *n;
27202613 struct sp_node *n_new = NULL;
@@ -2797,30 +2690,30 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) · 76 unmodified lines
27972690 rwlock_init(&sp->lock);
27982691
27992692 if (mpol) {
2800− struct vm_area_struct pvma;
2801− struct mempolicy *new;
2693+ struct sp_node *sn;
2694+ struct mempolicy *npol;
28022695 NODEMASK_SCRATCH(scratch);
28032696
28042697 if (!scratch)
28052698 goto put_mpol;
2806− /* contextualize the tmpfs mount point mempolicy */
2807− new = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);
2808− if (IS_ERR(new))
2699+
2700+ /* contextualize the tmpfs mount point mempolicy to this file */
2701+ npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);
2702+ if (IS_ERR(npol))
28092703 goto free_scratch; /* no valid nodemask intersection */
28102704
28112705 task_lock(current);
2812− ret = mpol_set_nodemask(new, &mpol->w.user_nodemask, scratch);
2706+ ret = mpol_set_nodemask(npol, &mpol->w.user_nodemask, scratch);
28132707 task_unlock(current);
28142708 if (ret)
2815− goto put_new;
2816−
2817− /* Create pseudo-vma that contains just the policy */
2818− vma_init(&pvma, NULL);
2819− pvma.vm_end = TASK_SIZE; /* policy covers entire file */
2820− mpol_set_shared_policy(sp, &pvma, new); /* adds ref */
2821−
2822−put_new:
2823− mpol_put(new); /* drop initial ref */
2709+ goto put_npol;
2710+
2711+ /* alloc node covering entire file; adds ref to file's npol */
2712+ sn = sp_alloc(0, MAX_LFS_FILESIZE >> PAGE_SHIFT, npol);
2713+ if (sn)
2714+ sp_insert(sp, sn);
2715+put_npol:
2716+ mpol_put(npol); /* drop initial ref on file's npol */
28242717 free_scratch:
28252718 NODEMASK_SCRATCH_FREE(scratch);
28262719 put_mpol:
@@ -2828,46 +2721,40 @@ put_mpol: · 1 unmodified line
28282721 }
28292722 }
28302723
2831−int mpol_set_shared_policy(struct shared_policy *info,
2832− struct vm_area_struct *vma, struct mempolicy *npol)
2724+int mpol_set_shared_policy(struct shared_policy *sp,
2725+ struct vm_area_struct *vma, struct mempolicy *pol)
28332726 {
28342727 int err;
28352728 struct sp_node *new = NULL;
28362729 unsigned long sz = vma_pages(vma);
28372730
2838− pr_debug("set_shared_policy %lx sz %lu %d %d %lx\n",
2839− vma->vm_pgoff,
2840− sz, npol ? npol->mode : -1,
2841− npol ? npol->flags : -1,
2842− npol ? nodes_addr(npol->nodes)[0] : NUMA_NO_NODE);
2843−
2844− if (npol) {
2845− new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, npol);
2731+ if (pol) {
2732+ new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, pol);
28462733 if (!new)
28472734 return -ENOMEM;
28482735 }
2849− err = shared_policy_replace(info, vma->vm_pgoff, vma->vm_pgoff+sz, new);
2736+ err = shared_policy_replace(sp, vma->vm_pgoff, vma->vm_pgoff + sz, new);
28502737 if (err && new)
28512738 sp_free(new);
28522739 return err;
28532740 }
28542741
28552742 /* Free a backing policy store on inode delete. */
2856−void mpol_free_shared_policy(struct shared_policy *p)
2743+void mpol_free_shared_policy(struct shared_policy *sp)
28572744 {
28582745 struct sp_node *n;
28592746 struct rb_node *next;
28602747
2861− if (!p->root.rb_node)
2748+ if (!sp->root.rb_node)
28622749 return;
2863− write_lock(&p->lock);
2864− next = rb_first(&p->root);
2750+ write_lock(&sp->lock);
2751+ next = rb_first(&sp->root);
28652752 while (next) {
28662753 n = rb_entry(next, struct sp_node, nd);
28672754 next = rb_next(&n->nd);
2868− sp_delete(p, n);
2755+ sp_delete(sp, n);
28692756 }
2870− write_unlock(&p->lock);
2757+ write_unlock(&sp->lock);
28712758 }
28722759
28732760 #ifdef CONFIG_NUMA_BALANCING
@@ -2917,7 +2804,6 @@ static inline void __init check_numabalancing_enable(void) · 43 unmodified lines
29172804 }
29182805 #endif /* CONFIG_NUMA_BALANCING */
29192806
2920−/* assumes fs == KERNEL_DS */
29212807 void __init numa_policy_init(void)
29222808 {
29232809 nodemask_t interleave_nodes;
@@ -2980,7 +2866,6 @@ void numa_default_policy(void) · 56 unmodified lines
29802866 /*
29812867 * Parse and format mempolicy from/to strings
29822868 */
2983−
29842869 static const char * const policy_modes[] =
29852870 {
29862871 [MPOL_DEFAULT] = "default",
@@ -2991,7 +2876,6 @@ static const char * const policy_modes[] = · 4 unmodified lines
29912876 [MPOL_PREFERRED_MANY] = "prefer (many)",
29922877 };
29932878
2994−
29952879 #ifdef CONFIG_TMPFS
29962880 /**
29972881 * mpol_parse_str - parse string to mempolicy, for tmpfs mpol mount option.
mm/migrate.c+117−108
@@ -157,8 +157,8 @@ void putback_movable_pages(struct list_head *l) · 156 unmodified lines
157157 list_del(&folio->lru);
158158 /*
159159 * We isolated non-lru movable folio so here we can use
160− * __PageMovable because LRU folio's mapping cannot have
161− * PAGE_MAPPING_MOVABLE.
160+ * __folio_test_movable because LRU folio's mapping cannot
161+ * have PAGE_MAPPING_MOVABLE.
162162 */
163163 if (unlikely(__folio_test_movable(folio))) {
164164 VM_BUG_ON_FOLIO(!folio_test_isolated(folio), folio);
@@ -249,7 +249,7 @@ static bool remove_migration_pte(struct folio *folio, · 84 unmodified lines
249249
250250 pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
251251 if (folio_test_anon(folio))
252− hugepage_add_anon_rmap(new, vma, pvmw.address,
252+ hugepage_add_anon_rmap(folio, vma, pvmw.address,
253253 rmap_flags);
254254 else
255255 page_dup_file_rmap(new, true);
@@ -524,7 +524,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 268 unmodified lines
524524 int expected_count;
525525
526526 xas_lock_irq(&xas);
527− expected_count = 2 + folio_has_private(src);
527+ expected_count = folio_expected_refs(mapping, src);
528528 if (!folio_ref_freeze(src, expected_count)) {
529529 xas_unlock_irq(&xas);
530530 return -EAGAIN;
@@ -533,11 +533,11 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 2 unmodified lines
533533 dst->index = src->index;
534534 dst->mapping = src->mapping;
535535
536− folio_get(dst);
536+ folio_ref_add(dst, folio_nr_pages(dst));
537537
538538 xas_store(&xas, dst);
539539
540− folio_ref_unfreeze(src, expected_count - 1);
540+ folio_ref_unfreeze(src, expected_count - folio_nr_pages(src));
541541
542542 xas_unlock_irq(&xas);
543543
@@ -588,20 +588,20 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 44 unmodified lines
588588 * Copy NUMA information to the new page, to prevent over-eager
589589 * future migrations of this same page.
590590 */
591− cpupid = page_cpupid_xchg_last(&folio->page, -1);
591+ cpupid = folio_xchg_last_cpupid(folio, -1);
592592 /*
593593 * For memory tiering mode, when migrate between slow and fast
594594 * memory node, reset cpupid, because that is used to record
595595 * page access time in slow memory node.
596596 */
597597 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) {
598− bool f_toptier = node_is_toptier(page_to_nid(&folio->page));
599− bool t_toptier = node_is_toptier(page_to_nid(&newfolio->page));
598+ bool f_toptier = node_is_toptier(folio_nid(folio));
599+ bool t_toptier = node_is_toptier(folio_nid(newfolio));
600600
601601 if (f_toptier != t_toptier)
602602 cpupid = -1;
603603 }
604− page_cpupid_xchg_last(&newfolio->page, cpupid);
604+ folio_xchg_last_cpupid(newfolio, cpupid);
605605
606606 folio_migrate_ksm(newfolio, folio);
607607 /*
@@ -633,8 +633,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 25 unmodified lines
633633
634634 folio_copy_owner(newfolio, folio);
635635
636− if (!folio_test_hugetlb(folio))
637− mem_cgroup_migrate(folio, newfolio);
636+ mem_cgroup_migrate(folio, newfolio);
638637 }
639638 EXPORT_SYMBOL(folio_migrate_flags);
640639
@@ -946,7 +945,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 305 unmodified lines
946945 enum migrate_mode mode)
947946 {
948947 int rc = -EAGAIN;
949− bool is_lru = !__PageMovable(&src->page);
948+ bool is_lru = !__folio_test_movable(src);
950949
951950 VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
952951 VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
@@ -993,7 +992,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 40 unmodified lines
993992 * src is freed; but stats require that PageAnon be left as PageAnon.
994993 */
995994 if (rc == MIGRATEPAGE_SUCCESS) {
996− if (__PageMovable(&src->page)) {
995+ if (__folio_test_movable(src)) {
997996 VM_BUG_ON_FOLIO(!folio_test_isolated(src), src);
998997
999998 /*
@@ -1028,22 +1027,28 @@ union migration_ptr { · 28 unmodified lines
10281027 struct anon_vma *anon_vma;
10291028 struct address_space *mapping;
10301029 };
1030+
1031+enum {
1032+ PAGE_WAS_MAPPED = BIT(0),
1033+ PAGE_WAS_MLOCKED = BIT(1),
1034+};
1035+
10311036 static void __migrate_folio_record(struct folio *dst,
1032− unsigned long page_was_mapped,
1037+ unsigned long old_page_state,
10331038 struct anon_vma *anon_vma)
10341039 {
10351040 union migration_ptr ptr = { .anon_vma = anon_vma };
10361041 dst->mapping = ptr.mapping;
1037− dst->private = (void *)page_was_mapped;
1042+ dst->private = (void *)old_page_state;
10381043 }
10391044
10401045 static void __migrate_folio_extract(struct folio *dst,
1041− int *page_was_mappedp,
1046+ int *old_page_state,
10421047 struct anon_vma **anon_vmap)
10431048 {
10441049 union migration_ptr ptr = { .mapping = dst->mapping };
10451050 *anon_vmap = ptr.anon_vma;
1046− *page_was_mappedp = (unsigned long)dst->private;
1051+ *old_page_state = (unsigned long)dst->private;
10471052 dst->mapping = NULL;
10481053 dst->private = NULL;
10491054 }
@@ -1085,7 +1090,7 @@ static void migrate_folio_done(struct folio *src, · 35 unmodified lines
10851090 /*
10861091 * Compaction can migrate also non-LRU pages which are
10871092 * not accounted to NR_ISOLATED_*. They can be recognized
1088− * as __PageMovable
1093+ * as __folio_test_movable
10891094 */
10901095 if (likely(!__folio_test_movable(src)))
10911096 mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
@@ -1104,9 +1109,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 12 unmodified lines
11041109 {
11051110 struct folio *dst;
11061111 int rc = -EAGAIN;
1107− int page_was_mapped = 0;
1112+ int old_page_state = 0;
11081113 struct anon_vma *anon_vma = NULL;
1109− bool is_lru = !__PageMovable(&src->page);
1114+ bool is_lru = !__folio_test_movable(src);
11101115 bool locked = false;
11111116 bool dst_locked = false;
11121117
@@ -1158,6 +1163,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 45 unmodified lines
11581163 folio_lock(src);
11591164 }
11601165 locked = true;
1166+ if (folio_test_mlocked(src))
1167+ old_page_state |= PAGE_WAS_MLOCKED;
11611168
11621169 if (folio_test_writeback(src)) {
11631170 /*
@@ -1207,7 +1214,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 43 unmodified lines
12071214 dst_locked = true;
12081215
12091216 if (unlikely(!is_lru)) {
1210− __migrate_folio_record(dst, page_was_mapped, anon_vma);
1217+ __migrate_folio_record(dst, old_page_state, anon_vma);
12111218 return MIGRATEPAGE_UNMAP;
12121219 }
12131220
@@ -1233,11 +1240,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 19 unmodified lines
12331240 VM_BUG_ON_FOLIO(folio_test_anon(src) &&
12341241 !folio_test_ksm(src) && !anon_vma, src);
12351242 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
1236− page_was_mapped = 1;
1243+ old_page_state |= PAGE_WAS_MAPPED;
12371244 }
12381245
12391246 if (!folio_mapped(src)) {
1240− __migrate_folio_record(dst, page_was_mapped, anon_vma);
1247+ __migrate_folio_record(dst, old_page_state, anon_vma);
12411248 return MIGRATEPAGE_UNMAP;
12421249 }
12431250
@@ -1249,7 +1256,8 @@ out: · 5 unmodified lines
12491256 if (rc == -EAGAIN)
12501257 ret = NULL;
12511258
1252− migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret);
1259+ migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1260+ anon_vma, locked, ret);
12531261 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
12541262
12551263 return rc;
@@ -1262,12 +1270,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 6 unmodified lines
12621270 struct list_head *ret)
12631271 {
12641272 int rc;
1265− int page_was_mapped = 0;
1273+ int old_page_state = 0;
12661274 struct anon_vma *anon_vma = NULL;
1267− bool is_lru = !__PageMovable(&src->page);
1275+ bool is_lru = !__folio_test_movable(src);
12681276 struct list_head *prev;
12691277
1270− __migrate_folio_extract(dst, &page_was_mapped, &anon_vma);
1278+ __migrate_folio_extract(dst, &old_page_state, &anon_vma);
12711279 prev = dst->lru.prev;
12721280 list_del(&dst->lru);
12731281
@@ -1288,10 +1296,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 14 unmodified lines
12881296 * isolated from the unevictable LRU: but this case is the easiest.
12891297 */
12901298 folio_add_lru(dst);
1291− if (page_was_mapped)
1299+ if (old_page_state & PAGE_WAS_MLOCKED)
12921300 lru_add_drain();
12931301
1294− if (page_was_mapped)
1302+ if (old_page_state & PAGE_WAS_MAPPED)
12951303 remove_migration_ptes(src, dst, false);
12961304
12971305 out_unlock_both:
@@ -1323,11 +1331,12 @@ out: · 25 unmodified lines
13231331 */
13241332 if (rc == -EAGAIN) {
13251333 list_add(&dst->lru, prev);
1326− __migrate_folio_record(dst, page_was_mapped, anon_vma);
1334+ __migrate_folio_record(dst, old_page_state, anon_vma);
13271335 return rc;
13281336 }
13291337
1330− migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret);
1338+ migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1339+ anon_vma, true, ret);
13311340 migrate_folio_undo_dst(dst, true, put_new_folio, private);
13321341
13331342 return rc;
@@ -1496,6 +1505,7 @@ struct migrate_pages_stats { · 162 unmodified lines
14961505 int nr_thp_succeeded; /* THP migrated successfully */
14971506 int nr_thp_failed; /* THP failed to be migrated */
14981507 int nr_thp_split; /* THP split before migrating */
1508+ int nr_split; /* Large folio (include THP) split before migrating */
14991509 };
15001510
15011511 /*
@@ -1615,6 +1625,7 @@ static int migrate_pages_batch(struct list_head *from, · 113 unmodified lines
16151625 int nr_retry_pages = 0;
16161626 int pass = 0;
16171627 bool is_thp = false;
1628+ bool is_large = false;
16181629 struct folio *folio, *folio2, *dst = NULL, *dst2;
16191630 int rc, rc_saved = 0, nr_pages;
16201631 LIST_HEAD(unmap_folios);
@@ -1630,7 +1641,8 @@ static int migrate_pages_batch(struct list_head *from, · 9 unmodified lines
16301641 nr_retry_pages = 0;
16311642
16321643 list_for_each_entry_safe(folio, folio2, from, lru) {
1633− is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
1644+ is_large = folio_test_large(folio);
1645+ is_thp = is_large && folio_test_pmd_mappable(folio);
16341646 nr_pages = folio_nr_pages(folio);
16351647
16361648 cond_resched();
@@ -1650,6 +1662,7 @@ static int migrate_pages_batch(struct list_head *from, · 13 unmodified lines
16501662 stats->nr_thp_failed++;
16511663 if (!try_split_folio(folio, split_folios)) {
16521664 stats->nr_thp_split++;
1665+ stats->nr_split++;
16531666 continue;
16541667 }
16551668 stats->nr_failed_pages += nr_pages;
@@ -1678,11 +1691,12 @@ static int migrate_pages_batch(struct list_head *from, · 22 unmodified lines
16781691 nr_failed++;
16791692 stats->nr_thp_failed += is_thp;
16801693 /* Large folio NUMA faulting doesn't split to retry. */
1681− if (folio_test_large(folio) && !nosplit) {
1694+ if (is_large && !nosplit) {
16821695 int ret = try_split_folio(folio, split_folios);
16831696
16841697 if (!ret) {
16851698 stats->nr_thp_split += is_thp;
1699+ stats->nr_split++;
16861700 break;
16871701 } else if (reason == MR_LONGTERM_PIN &&
16881702 ret == -EAGAIN) {
@@ -1795,12 +1809,12 @@ out: · 106 unmodified lines
17951809 dst = list_first_entry(&dst_folios, struct folio, lru);
17961810 dst2 = list_next_entry(dst, lru);
17971811 list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) {
1798− int page_was_mapped = 0;
1812+ int old_page_state = 0;
17991813 struct anon_vma *anon_vma = NULL;
18001814
1801− __migrate_folio_extract(dst, &page_was_mapped, &anon_vma);
1802− migrate_folio_undo_src(folio, page_was_mapped, anon_vma,
1803− true, ret_folios);
1815+ __migrate_folio_extract(dst, &old_page_state, &anon_vma);
1816+ migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
1817+ anon_vma, true, ret_folios);
18041818 list_del(&dst->lru);
18051819 migrate_folio_undo_dst(dst, true, put_new_folio, private);
18061820 dst = dst2;
@@ -1828,6 +1842,7 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 21 unmodified lines
18281842 stats->nr_succeeded += astats.nr_succeeded;
18291843 stats->nr_thp_succeeded += astats.nr_thp_succeeded;
18301844 stats->nr_thp_split += astats.nr_thp_split;
1845+ stats->nr_split += astats.nr_split;
18311846 if (rc < 0) {
18321847 stats->nr_failed_pages += astats.nr_failed_pages;
18331848 stats->nr_thp_failed += astats.nr_thp_failed;
@@ -1835,7 +1850,11 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 1 unmodified line
18351850 return rc;
18361851 }
18371852 stats->nr_thp_failed += astats.nr_thp_split;
1838− nr_failed += astats.nr_thp_split;
1853+ /*
1854+ * Do not count rc, as pages will be retried below.
1855+ * Count nr_split only, since it includes nr_thp_split.
1856+ */
1857+ nr_failed += astats.nr_split;
18391858 /*
18401859 * Fall back to migrate all failed folios one by one synchronously. All
18411860 * failed folios except split THPs will be retried, so their failure
@@ -1970,7 +1989,8 @@ out: · 128 unmodified lines
19701989 count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);
19711990 trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,
19721991 stats.nr_thp_succeeded, stats.nr_thp_failed,
1973− stats.nr_thp_split, mode, reason);
1992+ stats.nr_thp_split, stats.nr_split, mode,
1993+ reason);
19741994
19751995 if (ret_succeeded)
19761996 *ret_succeeded = stats.nr_succeeded;
@@ -2029,8 +2049,7 @@ static int store_status(int __user *status, int start, int value, int nr) · 52 unmodified lines
20292049 return 0;
20302050 }
20312051
2032−static int do_move_pages_to_node(struct mm_struct *mm,
2033− struct list_head *pagelist, int node)
2052+static int do_move_pages_to_node(struct list_head *pagelist, int node)
20342053 {
20352054 int err;
20362055 struct migration_target_control mtc = {
@@ -2060,8 +2079,8 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 23 unmodified lines
20602079 struct vm_area_struct *vma;
20612080 unsigned long addr;
20622081 struct page *page;
2082+ struct folio *folio;
20632083 int err;
2064− bool isolated;
20652084
20662085 mmap_read_lock(mm);
20672086 addr = (unsigned long)untagged_addr_remote(mm, p);
@@ -2082,51 +2101,44 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 14 unmodified lines
20822101 if (!page)
20832102 goto out;
20842103
2085− if (is_zone_device_page(page))
2086− goto out_putpage;
2104+ folio = page_folio(page);
2105+ if (folio_is_zone_device(folio))
2106+ goto out_putfolio;
20872107
20882108 err = 0;
2089− if (page_to_nid(page) == node)
2090− goto out_putpage;
2109+ if (folio_nid(folio) == node)
2110+ goto out_putfolio;
20912111
20922112 err = -EACCES;
20932113 if (page_mapcount(page) > 1 && !migrate_all)
2094− goto out_putpage;
2114+ goto out_putfolio;
20952115
2096− if (PageHuge(page)) {
2097− if (PageHead(page)) {
2098− isolated = isolate_hugetlb(page_folio(page), pagelist);
2099− err = isolated ? 1 : -EBUSY;
2100− }
2116+ err = -EBUSY;
2117+ if (folio_test_hugetlb(folio)) {
2118+ if (isolate_hugetlb(folio, pagelist))
2119+ err = 1;
21012120 } else {
2102− struct page *head;
2103−
2104− head = compound_head(page);
2105− isolated = isolate_lru_page(head);
2106− if (!isolated) {
2107− err = -EBUSY;
2108− goto out_putpage;
2109− }
2121+ if (!folio_isolate_lru(folio))
2122+ goto out_putfolio;
21102123
21112124 err = 1;
2112− list_add_tail(&head->lru, pagelist);
2113− mod_node_page_state(page_pgdat(head),
2114− NR_ISOLATED_ANON + page_is_file_lru(head),
2115− thp_nr_pages(head));
2125+ list_add_tail(&folio->lru, pagelist);
2126+ node_stat_mod_folio(folio,
2127+ NR_ISOLATED_ANON + folio_is_file_lru(folio),
2128+ folio_nr_pages(folio));
21162129 }
2117−out_putpage:
2130+out_putfolio:
21182131 /*
2119− * Either remove the duplicate refcount from
2120− * isolate_lru_page() or drop the page ref if it was
2121− * not isolated.
2132+ * Either remove the duplicate refcount from folio_isolate_lru()
2133+ * or drop the folio ref if it was not isolated.
21222134 */
2123− put_page(page);
2135+ folio_put(folio);
21242136 out:
21252137 mmap_read_unlock(mm);
21262138 return err;
21272139 }
21282140
2129−static int move_pages_and_store_status(struct mm_struct *mm, int node,
2141+static int move_pages_and_store_status(int node,
21302142 struct list_head *pagelist, int __user *status,
21312143 int start, int i, unsigned long nr_pages)
21322144 {
@@ -2135,7 +2147,7 @@ static int move_pages_and_store_status(struct mm_struct *mm, int node, · 2 unmodified lines
21352147 if (list_empty(pagelist))
21362148 return 0;
21372149
2138− err = do_move_pages_to_node(mm, pagelist, node);
2150+ err = do_move_pages_to_node(pagelist, node);
21392151 if (err) {
21402152 /*
21412153 * Positive err means the number of failed
@@ -2203,7 +2215,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 61 unmodified lines
22032215 current_node = node;
22042216 start = i;
22052217 } else if (node != current_node) {
2206− err = move_pages_and_store_status(mm, current_node,
2218+ err = move_pages_and_store_status(current_node,
22072219 &pagelist, status, start, i, nr_pages);
22082220 if (err)
22092221 goto out;
@@ -2238,7 +2250,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 28 unmodified lines
22382250 if (err)
22392251 goto out_flush;
22402252
2241− err = move_pages_and_store_status(mm, current_node, &pagelist,
2253+ err = move_pages_and_store_status(current_node, &pagelist,
22422254 status, start, i, nr_pages);
22432255 if (err) {
22442256 /* We have accounted for page i */
@@ -2250,7 +2262,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 5 unmodified lines
22502262 }
22512263 out_flush:
22522264 /* Make sure we do not overwrite the existing error */
2253− err1 = move_pages_and_store_status(mm, current_node, &pagelist,
2265+ err1 = move_pages_and_store_status(current_node, &pagelist,
22542266 status, start, i, nr_pages);
22552267 if (err >= 0)
22562268 err = err1;
@@ -2491,16 +2503,9 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src, · 234 unmodified lines
24912503 return __folio_alloc_node(gfp, order, nid);
24922504 }
24932505
2494−static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)
2506+static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
24952507 {
2496− int nr_pages = thp_nr_pages(page);
2497− int order = compound_order(page);
2498−
2499− VM_BUG_ON_PAGE(order && !PageTransHuge(page), page);
2500−
2501− /* Do not migrate THP mapped by multiple processes */
2502− if (PageTransHuge(page) && total_mapcount(page) > 1)
2503− return 0;
2508+ int nr_pages = folio_nr_pages(folio);
25042509
25052510 /* Avoid migrating to a node that is nearly full */
25062511 if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
@@ -2512,75 +2517,79 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · 5 unmodified lines
25122517 if (managed_zone(pgdat->node_zones + z))
25132518 break;
25142519 }
2515− wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE);
2520+ wakeup_kswapd(pgdat->node_zones + z, 0,
2521+ folio_order(folio), ZONE_MOVABLE);
25162522 return 0;
25172523 }
25182524
2519− if (!isolate_lru_page(page))
2525+ if (!folio_isolate_lru(folio))
25202526 return 0;
25212527
2522− mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_is_file_lru(page),
2528+ node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
25232529 nr_pages);
25242530
25252531 /*
2526− * Isolating the page has taken another reference, so the
2527− * caller's reference can be safely dropped without the page
2532+ * Isolating the folio has taken another reference, so the
2533+ * caller's reference can be safely dropped without the folio
25282534 * disappearing underneath us during migration.
25292535 */
2530− put_page(page);
2536+ folio_put(folio);
25312537 return 1;
25322538 }
25332539
25342540 /*
2535− * Attempt to migrate a misplaced page to the specified destination
2541+ * Attempt to migrate a misplaced folio to the specified destination
25362542 * node. Caller is expected to have an elevated reference count on
2537− * the page that will be dropped by this function before returning.
2543+ * the folio that will be dropped by this function before returning.
25382544 */
2539−int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
2540− int node)
2545+int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
2546+ int node)
25412547 {
25422548 pg_data_t *pgdat = NODE_DATA(node);
25432549 int isolated;
25442550 int nr_remaining;
25452551 unsigned int nr_succeeded;
25462552 LIST_HEAD(migratepages);
2547− int nr_pages = thp_nr_pages(page);
2553+ int nr_pages = folio_nr_pages(folio);
25482554
25492555 /*
2550− * Don't migrate file pages that are mapped in multiple processes
2556+ * Don't migrate file folios that are mapped in multiple processes
25512557 * with execute permissions as they are probably shared libraries.
2558+ * To check if the folio is shared, ideally we want to make sure
2559+ * every page is mapped to the same process. Doing that is very
2560+ * expensive, so check the estimated mapcount of the folio instead.
25522561 */
2553− if (page_mapcount(page) != 1 && page_is_file_lru(page) &&
2562+ if (folio_estimated_sharers(folio) != 1 && folio_is_file_lru(folio) &&
25542563 (vma->vm_flags & VM_EXEC))
25552564 goto out;
25562565
25572566 /*
2558− * Also do not migrate dirty pages as not all filesystems can move
2559− * dirty pages in MIGRATE_ASYNC mode which is a waste of cycles.
2567+ * Also do not migrate dirty folios as not all filesystems can move
2568+ * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles.
25602569 */
2561− if (page_is_file_lru(page) && PageDirty(page))
2570+ if (folio_is_file_lru(folio) && folio_test_dirty(folio))
25622571 goto out;
25632572
2564− isolated = numamigrate_isolate_page(pgdat, page);
2573+ isolated = numamigrate_isolate_folio(pgdat, folio);
25652574 if (!isolated)
25662575 goto out;
25672576
2568− list_add(&page->lru, &migratepages);
2577+ list_add(&folio->lru, &migratepages);
25692578 nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
25702579 NULL, node, MIGRATE_ASYNC,
25712580 MR_NUMA_MISPLACED, &nr_succeeded);
25722581 if (nr_remaining) {
25732582 if (!list_empty(&migratepages)) {
2574− list_del(&page->lru);
2575− mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
2576− page_is_file_lru(page), -nr_pages);
2577− putback_lru_page(page);
2583+ list_del(&folio->lru);
2584+ node_stat_mod_folio(folio, NR_ISOLATED_ANON +
2585+ folio_is_file_lru(folio), -nr_pages);
2586+ folio_putback_lru(folio);
25782587 }
25792588 isolated = 0;
25802589 }
25812590 if (nr_succeeded) {
25822591 count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
2583− if (!node_is_toptier(page_to_nid(page)) && node_is_toptier(node))
2592+ if (!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
25842593 mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,
25852594 nr_succeeded);
25862595 }
@@ -2588,7 +2597,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, · 1 unmodified line
25882597 return isolated;
25892598
25902599 out:
2591− put_page(page);
2600+ folio_put(folio);
25922601 return 0;
25932602 }
25942603 #endif /* CONFIG_NUMA_BALANCING */
mm/mlock.c+72−23
@@ -305,6 +305,62 @@ void munlock_folio(struct folio *folio) · 304 unmodified lines
305305 local_unlock(&mlock_fbatch.lock);
306306 }
307307
308+static inline unsigned int folio_mlock_step(struct folio *folio,
309+ pte_t *pte, unsigned long addr, unsigned long end)
310+{
311+ unsigned int count, i, nr = folio_nr_pages(folio);
312+ unsigned long pfn = folio_pfn(folio);
313+ pte_t ptent = ptep_get(pte);
314+
315+ if (!folio_test_large(folio))
316+ return 1;
317+
318+ count = pfn + nr - pte_pfn(ptent);
319+ count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT);
320+
321+ for (i = 0; i < count; i++, pte++) {
322+ pte_t entry = ptep_get(pte);
323+
324+ if (!pte_present(entry))
325+ break;
326+ if (pte_pfn(entry) - pfn >= nr)
327+ break;
328+ }
329+
330+ return i;
331+}
332+
333+static inline bool allow_mlock_munlock(struct folio *folio,
334+ struct vm_area_struct *vma, unsigned long start,
335+ unsigned long end, unsigned int step)
336+{
337+ /*
338+ * For unlock, allow munlock large folio which is partially
339+ * mapped to VMA. As it's possible that large folio is
340+ * mlocked and VMA is split later.
341+ *
342+ * During memory pressure, such kind of large folio can
343+ * be split. And the pages are not in VM_LOCKed VMA
344+ * can be reclaimed.
345+ */

101 more lines aren’t shown here.

mm/mm_init.c+1−3

This diff is large: its 18 lines aren’t shown here.

mm/mmap.c+116−33

This diff is large: its 273 lines aren’t shown here.

mm/mmzone.c+3−3

This diff is large: its 23 lines aren’t shown here.

mm/mprotect.c+24−34

This diff is large: its 130 lines aren’t shown here.

mm/mremap.c+86−21

This diff is large: its 164 lines aren’t shown here.

mm/nommu.c+4−4

This diff is large: its 22 lines aren’t shown here.

mm/oom_kill.c+8−8

This diff is large: its 58 lines aren’t shown here.

mm/page-writeback.c+17−20

This diff is large: its 60 lines aren’t shown here.

mm/page_alloc.c+267−76

This diff is large: its 593 lines aren’t shown here.

mm/page_io.c+4−4

This diff is large: its 30 lines aren’t shown here.

mm/page_owner.c+2−2

This diff is large: its 14 lines aren’t shown here.

mm/percpu.c+3−5

This diff is large: its 33 lines aren’t shown here.

mm/rmap.c+101−63

This diff is large: its 290 lines aren’t shown here.

mm/shmem.c+294−294

This diff is large: its 1055 lines aren’t shown here.

mm/show_mem.c+4−7

This diff is large: its 34 lines aren’t shown here.

mm/shrinker.c+809−0

This diff is large: its 810 lines aren’t shown here.

mm/shrinker_debug.c+10−25

This diff is large: its 124 lines aren’t shown here.

mm/slab.h+8−7

This diff is large: its 42 lines aren’t shown here.

mm/swap.h+5−4

This diff is large: its 32 lines aren’t shown here.

mm/swap_state.c+57−35

This diff is large: its 222 lines aren’t shown here.

mm/util.c+1−0

This diff is large: its 8 lines aren’t shown here.

mm/vmalloc.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/vmscan.c+22−712

This diff is large: its 868 lines aren’t shown here.

mm/vmstat.c+17−13

This diff is large: its 112 lines aren’t shown here.

mm/workingset.c+16−13

This diff is large: its 58 lines aren’t shown here.

mm/zsmalloc.c+16−13

This diff is large: its 69 lines aren’t shown here.

mm/zswap.c+12−3

This diff is large: its 58 lines aren’t shown here.

net/sunrpc/auth.c+12−8

This diff is large: its 40 lines aren’t shown here.

tools/include/uapi/linux/fs.h+59−0

This diff is large: its 64 lines aren’t shown here.

tools/include/uapi/linux/prctl.h+2−1

This diff is large: its 10 lines aren’t shown here.

tools/mm/page_owner_sort.c+103−114

This diff is large: its 398 lines aren’t shown here.

tools/testing/radix-tree/linux.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/cgroup/.gitignore+1−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/cgroup/Makefile+2−0

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/cgroup/test_hugetlb_memcg.c+234−0

This diff is large: its 235 lines aren’t shown here.

tools/testing/selftests/cgroup/test_zswap.c+48−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/clone3/clone3.c+13−0

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.sh+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/.gitignore+2−0

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+3−1

This diff is large: its 25 lines aren’t shown here.

tools/testing/selftests/mm/config+1−0

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/mm/gup_longterm.c+2−1

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb-madvise.c+0−19

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb_fault_after_madv.c+73−0

This diff is large: its 74 lines aren’t shown here.

tools/testing/selftests/mm/ksm_functional_tests.c+65−1

This diff is large: its 96 lines aren’t shown here.

tools/testing/selftests/mm/mdwe_test.c+122−15

This diff is large: its 219 lines aren’t shown here.

tools/testing/selftests/mm/mremap_test.c+257−44

This diff is large: its 437 lines aren’t shown here.

tools/testing/selftests/mm/pagemap_ioctl.c+1660−0

This diff is large: its 1661 lines aren’t shown here.

tools/testing/selftests/mm/run_vmtests.sh+8−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/mm/vm_util.c+19−0

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/mm/vm_util.h+1−0

This diff is large: its 8 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t