History

Merge tag 'mm-stable-2023-11-01-14-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton: "Many singleton patches against the MM code. The patch series which are included in this merge do the following: - Kemeng Shi has contributed some compation maintenance work in the series 'Fixes and cleanups to compaction' - Joel Fernandes has a patchset ('Optimize mremap during mutual alignment within PMD') which fixes an obscure issue with mremap()'s pagetable handling during a subsequent exec(), based upon an implementation which Linus suggested - More DAMON/DAMOS maintenance and feature work from SeongJae Park i the following patch series: mm/damon: misc fixups for documents, comments and its tracepoint mm/damon: add a tracepoint for damos apply target regions mm/damon: provide pseudo-moving sum based access rate mm/damon: implement DAMOS apply intervals mm/damon/core-test: Fix memory leaks in core-test mm/damon/sysfs-schemes: Do DAMOS tried regions update for only one apply interval - In the series 'Do not try to access unaccepted memory' Adrian Hunter provides some fixups for the recently-added 'unaccepted memory' feature. To increase the feature's checking coverage. 'Plug a few gaps where RAM is exposed without checking if it is unaccepted memory' - In the series 'cleanups for lockless slab shrink' Qi Zheng has done some maintenance work which is preparation for the lockless slab shrinking code - Qi Zheng has redone the earlier (and reverted) attempt to make slab shrinking lockless in the series 'use refcount+RCU method to implement lockless slab shrink' - David Hildenbrand contributes some maintenance work for the rmap code in the series 'Anon rmap cleanups' - Kefeng Wang does more folio conversions and some maintenance work in the migration code. Series 'mm: migrate: more folio conversion and unification' - Matthew Wilcox has fixed an issue in the buffer_head code which was causing long stalls under some heavy memory/IO loads. Some cleanups were added on the way. Series 'Add and use bdev_getblk()' - In the series 'Use nth_page() in place of direct struct page manipulation' Zi Yan has fixed a potential issue with the direct manipulation of hugetlb page frames - In the series 'mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO' has improved our handling of gigantic pages in the hugetlb vmmemmep optimizaton code. This provides significant boot time improvements when significant amounts of gigantic pages are in use - Matthew Wilcox has sent the series 'Small hugetlb cleanups' - code rationalization and folio conversions in the hugetlb code - Yin Fengwei has improved mlock()'s handling of large folios in the series 'support large folio for mlock' - In the series 'Expose swapcache stat for memcg v1' Liu Shixin has added statistics for memcg v1 users which are available (and useful) under memcg v2 - Florent Revest has enhanced the MDWE (Memory-Deny-Write-Executable) prctl so that userspace may direct the kernel to not automatically propagate the denial to child processes. The series is named 'MDWE without inheritance' - Kefeng Wang has provided the series 'mm: convert numa balancing functions to use a folio' which does what it says - In the series 'mm/ksm: add fork-exec support for prctl' Stefan Roesch makes is possible for a process to propagate KSM treatment across exec() - Huang Ying has enhanced memory tiering's calculation of memory distances. This is used to permit the dax/kmem driver to use 'high bandwidth memory' in addition to Optane Data Center Persistent Memory Modules (DCPMM). The series is named 'memory tiering: calculate abstract distance based on ACPI HMAT' - In the series 'Smart scanning mode for KSM' Stefan Roesch has optimized KSM by teaching it to retain and use some historical information from previous scans - Yosry Ahmed has fixed some inconsistencies in memcg statistics in the series 'mm: memcg: fix tracking of pending stats updates values' - In the series 'Implement IOCTL to get and optionally clear info about PTEs' Peter Xu has added an ioctl to /proc/<pid>/pagemap which permits us to atomically read-then-clear page softdirty state. This is mainly used by CRIU - Hugh Dickins contributed the series 'shmem,tmpfs: general maintenance', a bunch of relatively minor maintenance tweaks to this code - Matthew Wilcox has increased the use of the VMA lock over file-backed page faults in the series 'Handle more faults under the VMA lock'. Some rationalizations of the fault path became possible as a result - In the series 'mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()' David Hildenbrand has implemented some cleanups and folio conversions - In the series 'various improvements to the GUP interface' Lorenzo Stoakes has simplified and improved the GUP interface with an eye to providing groundwork for future improvements - Andrey Konovalov has sent along the series 'kasan: assorted fixes and improvements' which does those things - Some page allocator maintenance work from Kemeng Shi in the series 'Two minor cleanups to break_down_buddy_pages' - In thes series 'New selftest for mm' Breno Leitao has developed another MM self test which tickles a race we had between madvise() and page faults - In the series 'Add folio_end_read' Matthew Wilcox provides cleanups and an optimization to the core pagecache code - Nhat Pham has added memcg accounting for hugetlb memory in the series 'hugetlb memcg accounting' - Cleanups and rationalizations to the pagemap code from Lorenzo Stoakes, in the series 'Abstract vma_merge() and split_vma()' - Audra Mitchell has fixed issues in the procfs page_owner code's new timestamping feature which was causing some misbehaviours. In the series 'Fix page_owner's use of free timestamps' - Lorenzo Stoakes has fixed the handling of new mappings of sealed files in the series 'permit write-sealed memfd read-only shared mappings' - Mike Kravetz has optimized the hugetlb vmemmap optimization in the series 'Batch hugetlb vmemmap modification operations' - Some buffer_head folio conversions and cleanups from Matthew Wilcox in the series 'Finish the create_empty_buffers() transition' - As a page allocator performance optimization Huang Ying has added automatic tuning to the allocator's per-cpu-pages feature, in the series 'mm: PCP high auto-tuning' - Roman Gushchin has contributed the patchset 'mm: improve performance of accounted kernel memory allocations' which improves their performance by ~30% as measured by a micro-benchmark - folio conversions from Kefeng Wang in the series 'mm: convert page cpupid functions to folios' - Some kmemleak fixups in Liu Shixin's series 'Some bugfix about kmemleak' - Qi Zheng has improved our handling of memoryless nodes by keeping them off the allocation fallback list. This is done in the series 'handle memoryless nodes more appropriately' - khugepaged conversions from Vishal Moola in the series 'Some khugepaged folio conversions'" [ bcachefs conflicts with the dynamically allocated shrinkers have been resolved as per Stephen Rothwell in https://lore.kernel.org/all/20230913093553.4290421e@canb.auug.org.au/ with help from Qi Zheng. The clone3 test filtering conflict was half-arsed by yours truly ] * tag 'mm-stable-2023-11-01-14-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (406 commits) mm/damon/sysfs: update monitoring target regions for online input commit mm/damon/sysfs: remove requested targets when online-commit inputs selftests: add a sanity check for zswap Documentation: maple_tree: fix word spelling error mm/vmalloc: fix the unchecked dereference warning in vread_iter() zswap: export compression failure stats Documentation: ubsan: drop "the" from article title mempolicy: migration attempt to match interleave nodes mempolicy: mmap_lock is not needed while migrating folios mempolicy: alloc_pages_mpol() for NUMA policy without vma mm: add page_rmappable_folio() wrapper mempolicy: remove confusing MPOL_MF_LAZY dead code mempolicy: mpol_shared_policy_init() without pseudo-vma mempolicy trivia: use pgoff_t in shared mempolicy tree mempolicy trivia: slightly more consistent naming mempolicy trivia: delete those ancient pr_debug()s mempolicy: fix migrate_pages(2) syscall return nr_failed kernfs: drop shared NUMA mempolicy hooks hugetlbfs: drop shared NUMA mempolicy pretence mm/damon/sysfs-test: add a unit test for damon_sysfs_set_targets() ...

Linus Torvalds committed 3 years ago

Commit ecae0bd2 parents bc3012f 9732336281 files · +11683 −5277Browse files

Unified · Split

281 files
  1. Documentation/ABI/testing/sysfs-kernel-mm-damon +7 −0
  2. Documentation/admin-guide/cgroup-v1/memory.rst +1 −0
  3. Documentation/admin-guide/cgroup-v2.rst +38 −0
  4. Documentation/admin-guide/mm/damon/usage.rst +81 −43
  5. Documentation/admin-guide/mm/ksm.rst +11 −0
  6. Documentation/admin-guide/mm/pagemap.rst +89 −0
  7. Documentation/admin-guide/mm/userfaultfd.rst +35 −0
  8. Documentation/core-api/maple_tree.rst +1 −1
  9. Documentation/dev-tools/kasan.rst +5 −2
  10. Documentation/dev-tools/kcsan.rst +2 −2
  11. Documentation/dev-tools/kmsan.rst +3 −3
  12. Documentation/dev-tools/ubsan.rst +4 −2
  13. Documentation/mm/damon/design.rst +22 −4
  14. Documentation/trace/postprocess/trace-vmscan-postprocess.pl +18 −24
  15. MAINTAINERS +2 −0
  16. arch/alpha/include/asm/bitops.h +20 −0
  17. arch/arm64/include/asm/tlb.h +4 −1
  18. arch/arm64/kernel/mte.c +2 −2
  19. arch/arm64/mm/kasan_init.c +5 −1
  20. arch/loongarch/include/asm/pgalloc.h +1 −0
  21. arch/m68k/include/asm/bitops.h +21 −0
  22. arch/mips/include/asm/bitops.h +24 −1
  23. arch/mips/include/asm/pgalloc.h +1 −0
  24. arch/mips/lib/bitops.c +14 −0
  25. arch/mips/mm/cache.c +1 −1
  26. arch/powerpc/include/asm/bitops.h +5 −16
  27. arch/riscv/include/asm/bitops.h +12 −0
  28. arch/s390/include/asm/bitops.h +10 −0
  29. arch/x86/include/asm/bitops.h +5 −6
  30. arch/x86/kvm/mmu/mmu.c +10 −8
  31. arch/x86/mm/pgtable.c +3 −0
  32. drivers/acpi/acpi_pad.c +1 −1
  33. drivers/acpi/numa/hmat.c +93 −53
  34. drivers/android/binder_alloc.c +18 −12
  35. drivers/base/cacheinfo.c +50 −1
  36. drivers/dax/kmem.c +49 −13
  37. drivers/firmware/efi/unaccepted_memory.c +20 −0
  38. drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +16 −13
  39. drivers/gpu/drm/i915/i915_drv.h +1 −1
  40. drivers/gpu/drm/msm/msm_drv.c +3 −1
  41. drivers/gpu/drm/msm/msm_drv.h +2 −2
  42. drivers/gpu/drm/msm/msm_gem_shrinker.c +20 −13
  43. drivers/gpu/drm/panfrost/panfrost_device.h +1 −1
  44. drivers/gpu/drm/panfrost/panfrost_drv.c +5 −1
  45. drivers/gpu/drm/panfrost/panfrost_gem.h +1 −1
  46. drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +17 −12
  47. drivers/gpu/drm/ttm/ttm_pool.c +31 −9
  48. drivers/md/bcache/bcache.h +1 −1
  49. drivers/md/bcache/btree.c +16 −11
  50. drivers/md/bcache/sysfs.c +2 −1
  51. drivers/md/dm-bufio.c +17 −11
  52. drivers/md/dm-cache-metadata.c +1 −1
  53. drivers/md/dm-zoned-metadata.c +16 −12
  54. drivers/md/raid5.c +15 −11
  55. drivers/md/raid5.h +1 −1
  56. drivers/misc/vmw_balloon.c +12 −26
  57. drivers/virtio/virtio_balloon.c +14 −10
  58. drivers/xen/xenbus/xenbus_probe_backend.c +10 −7
  59. fs/bcachefs/btree_cache.c +12 −10
  60. fs/bcachefs/btree_key_cache.c +12 −9
  61. fs/bcachefs/btree_types.h +2 −2
  62. fs/bcachefs/fs.c +1 −1
  63. fs/bcachefs/sysfs.c +1 −1
  64. fs/btrfs/super.c +1 −1
  65. fs/buffer.c +60 −65
  66. fs/dax.c +12 −12
  67. fs/erofs/utils.c +12 −7
  68. fs/exec.c +1 −3
  69. fs/ext4/ext4.h +1 −1
  70. fs/ext4/extents_status.c +13 −10
  71. fs/ext4/inode.c +5 −9
  72. fs/ext4/move_extent.c +5 −6
  73. fs/ext4/readpage.c +3 −11
  74. fs/ext4/super.c +10 −3
  75. fs/f2fs/super.c +23 −8
  76. fs/gfs2/aops.c +1 −1
  77. fs/gfs2/bmap.c +23 −25
  78. fs/gfs2/glock.c +10 −9
  79. fs/gfs2/main.c +3 −3
  80. fs/gfs2/meta_io.c +25 −36
  81. fs/gfs2/quota.c +38 −24
  82. fs/gfs2/quota.h +2 −1
  83. fs/hugetlbfs/inode.c +23 −61
  84. fs/iomap/buffered-io.c +35 −22
  85. fs/jbd2/journal.c +18 −11
  86. fs/kernfs/file.c +0 −49
  87. fs/kernfs/mount.c +1 −1
  88. fs/mbcache.c +12 −10
  89. fs/mpage.c +1 −2
  90. fs/nfs/nfs42xattr.c +44 −43
  91. fs/nfs/super.c +13 −8
  92. fs/nfsd/filecache.c +13 −10
  93. fs/nfsd/netns.h +2 −2
  94. fs/nfsd/nfs4state.c +11 −8
  95. fs/nfsd/nfscache.c +16 −15
  96. fs/nilfs2/mdt.c +34 −32
  97. fs/nilfs2/page.c +39 −37
  98. fs/nilfs2/page.h +0 −11
  99. fs/nilfs2/segment.c +3 −4
  100. fs/ntfs/aops.c +119 −136
  101. fs/ntfs/file.c +41 −48
  102. fs/ntfs3/file.c +16 −15
  103. fs/ocfs2/aops.c +10 −9
  104. fs/proc/kcore.c +2 −1
  105. fs/proc/root.c +1 −1
  106. fs/proc/task_mmu.c +731 −2
  107. fs/quota/dquot.c +9 −8
  108. fs/reiserfs/inode.c +40 −40
  109. fs/super.c +19 −16
  110. fs/ubifs/super.c +11 −10
  111. fs/ufs/balloc.c +9 −11
  112. fs/ufs/inode.c +12 −13
  113. fs/ufs/util.c +16 −18
  114. fs/ufs/util.h +4 −6
  115. fs/userfaultfd.c +40 −56
  116. fs/xfs/xfs_buf.c +13 −11
  117. fs/xfs/xfs_buf.h +1 −1
  118. fs/xfs/xfs_icache.c +15 −11
  119. fs/xfs/xfs_mount.c +2 −2
  120. fs/xfs/xfs_mount.h +1 −1
  121. fs/xfs/xfs_qm.c +14 −13
  122. fs/xfs/xfs_qm.h +1 −1
  123. include/asm-generic/bitops/instrumented-lock.h +13 −15
  124. include/asm-generic/bitops/lock.h +4 −16
  125. include/asm-generic/pgalloc.h +6 −1
  126. include/linux/bootmem_info.h +1 −1
  127. include/linux/buffer_head.h +54 −27
  128. include/linux/cacheinfo.h +1 −0
  129. include/linux/cgroup-defs.h +5 −0
  130. include/linux/damon.h +60 −14
  131. include/linux/dax.h +5 −5
  132. include/linux/fs.h +3 −3
  133. include/linux/gfp.h +11 −1
  134. include/linux/hugetlb.h +14 −11
  135. include/linux/hugetlb_cgroup.h +0 −11
  136. include/linux/jbd2.h +1 −1
  137. include/linux/memblock.h +9 −0
  138. include/linux/memcontrol.h +65 −28
  139. include/linux/memory-tiers.h +40 −1
  140. include/linux/mempolicy.h +31 −27
  141. include/linux/migrate.h +2 −2
  142. include/linux/mm.h +130 −46
  143. include/linux/mm_inline.h +1 −19
  144. include/linux/mm_types.h +45 −4
  145. include/linux/mmu_notifier.h +8 −1
  146. include/linux/mmzone.h +24 −5
  147. include/linux/page-flags.h +19 −0
  148. include/linux/pagemap.h +3 −30
  149. include/linux/percpu_counter.h +30 −0
  150. include/linux/rmap.h +2 −2
  151. include/linux/sched.h +4 −0
  152. include/linux/sched/coredump.h +16 −3
  153. include/linux/sched/mm.h +4 −0
  154. include/linux/sched/numa_balancing.h +3 −3
  155. include/linux/shmem_fs.h +10 −6
  156. include/linux/shrinker.h +55 −32
  157. include/linux/userfaultfd_k.h +27 −1
  158. include/linux/wait.h +3 −6
  159. include/trace/events/damon.h +42 −3
  160. include/trace/events/migrate.h +14 −10
  161. include/trace/events/vmscan.h +2 −6
  162. include/uapi/linux/fs.h +59 −0
  163. include/uapi/linux/mempolicy.h +1 −1
  164. include/uapi/linux/prctl.h +2 −1
  165. include/uapi/linux/userfaultfd.h +8 −1
  166. ipc/shm.c +8 −13
  167. kernel/cgroup/cgroup.c +14 −1
  168. kernel/events/uprobes.c +2 −2
  169. kernel/exit.c +0 −4
  170. kernel/fork.c +2 −6
  171. kernel/kcsan/kcsan_test.c +3 −6
  172. kernel/kcsan/selftest.c +3 −6
  173. kernel/kthread.c +0 −1
  174. kernel/ptrace.c +1 −1
  175. kernel/rcu/tree.c +12 −9
  176. kernel/rcu/tree_nocb.h +10 −9
  177. kernel/sched/fair.c +6 −6
  178. kernel/sched/wait.c +10 −50
  179. kernel/sys.c +26 −6
  180. lib/percpu_counter.c +79 −0
  181. mm/Kconfig +12 −0
  182. mm/Makefile +2 −2
  183. mm/bootmem_info.c +1 −1
  184. mm/cma.c +1 −1
  185. mm/compaction.c +52 −39
  186. mm/damon/Kconfig +12 −0
  187. mm/damon/core-test.h +28 −1
  188. mm/damon/core.c +226 −69
  189. mm/damon/dbgfs.c +2 −1
  190. mm/damon/lru_sort.c +3 −3
  191. mm/damon/ops-common.c +2 −3
  192. mm/damon/paddr.c +5 −6
  193. mm/damon/reclaim.c +2 −0
  194. mm/damon/sysfs-common.h +2 −0
  195. mm/damon/sysfs-schemes.c +128 −5
  196. mm/damon/sysfs-test.h +86 −0
  197. mm/damon/sysfs.c +83 −40
  198. mm/damon/vaddr.c +13 −9
  199. mm/debug_vm_pgtable.c +2 −2
  200. mm/filemap.c +82 −133
  201. mm/gup.c +13 −3
  202. mm/huge_memory.c +79 −55
  203. mm/hugetlb.c +458 −167
  204. mm/hugetlb_cgroup.c +2 −18
  205. mm/hugetlb_vmemmap.c +252 −49
  206. mm/hugetlb_vmemmap.h +24 −7
  207. mm/internal.h +136 −11
  208. mm/kasan/kasan_test.c +9 −10
  209. mm/kasan/kasan_test_module.c +1 −1
  210. mm/kasan/quarantine.c +3 −1
  211. mm/kasan/report.c +2 −2
  212. mm/kasan/report_generic.c +3 −3
  213. mm/kasan/shadow.c +1 −1
  214. mm/khugepaged.c +68 −69
  215. mm/kmemleak.c +98 −46
  216. mm/kmsan/core.c +31 −96
  217. mm/kmsan/kmsan_test.c +71 −38
  218. mm/kmsan/shadow.c +7 −2
  219. mm/ksm.c +116 −0
  220. mm/madvise.c +6 −27
  221. mm/memblock.c +36 −13
  222. mm/memcontrol.c +341 −60
  223. mm/memfd.c +0 −6
  224. mm/memory-failure.c +16 −13
  225. mm/memory-tiers.c +168 −7
  226. mm/memory.c +179 −123
  227. mm/memory_hotplug.c +6 −2
  228. mm/mempolicy.c +455 −571
  229. mm/migrate.c +117 −108
  230. mm/mlock.c +72 −23
  231. mm/mm_init.c +1 −3
  232. mm/mmap.c +116 −33
  233. mm/mmzone.c +3 −3
  234. mm/mprotect.c +24 −34
  235. mm/mremap.c +86 −21
  236. mm/nommu.c +4 −4
  237. mm/oom_kill.c +8 −8
  238. mm/page-writeback.c +17 −20
  239. mm/page_alloc.c +267 −76
  240. mm/page_io.c +4 −4
  241. mm/page_owner.c +2 −2
  242. mm/percpu.c +3 −5
  243. mm/rmap.c +101 −63
  244. mm/shmem.c +294 −294
  245. mm/show_mem.c +4 −7
  246. mm/shrinker.c +809 −0
  247. mm/shrinker_debug.c +10 −25
  248. mm/slab.h +8 −7
  249. mm/swap.h +5 −4
  250. mm/swap_state.c +57 −35
  251. mm/util.c +1 −0
  252. mm/vmalloc.c +1 −1
  253. mm/vmscan.c +22 −712
  254. mm/vmstat.c +17 −13
  255. mm/workingset.c +16 −13
  256. mm/zsmalloc.c +16 −13
  257. mm/zswap.c +12 −3
  258. net/sunrpc/auth.c +12 −8
  259. tools/include/uapi/linux/fs.h +59 −0
  260. tools/include/uapi/linux/prctl.h +2 −1
  261. tools/mm/page_owner_sort.c +103 −114
  262. tools/testing/radix-tree/linux.c +2 −2
  263. tools/testing/selftests/cgroup/.gitignore +1 −0
  264. tools/testing/selftests/cgroup/Makefile +2 −0
  265. tools/testing/selftests/cgroup/test_hugetlb_memcg.c +234 −0
  266. tools/testing/selftests/cgroup/test_zswap.c +48 −0
  267. tools/testing/selftests/clone3/clone3.c +13 −0
  268. tools/testing/selftests/damon/sysfs.sh +1 −0
  269. tools/testing/selftests/mm/.gitignore +2 −0
  270. tools/testing/selftests/mm/Makefile +3 −1
  271. tools/testing/selftests/mm/config +1 −0
  272. tools/testing/selftests/mm/gup_longterm.c +2 −1
  273. tools/testing/selftests/mm/hugetlb-madvise.c +0 −19
  274. tools/testing/selftests/mm/hugetlb_fault_after_madv.c +73 −0
  275. tools/testing/selftests/mm/ksm_functional_tests.c +65 −1
  276. tools/testing/selftests/mm/mdwe_test.c +122 −15
  277. tools/testing/selftests/mm/mremap_test.c +257 −44
  278. tools/testing/selftests/mm/pagemap_ioctl.c +1660 −0
  279. tools/testing/selftests/mm/run_vmtests.sh +8 −0
  280. tools/testing/selftests/mm/vm_util.c +19 −0
  281. tools/testing/selftests/mm/vm_util.h +1 −0
Documentation/ABI/testing/sysfs-kernel-mm-damon+7−0
@@ -151,6 +151,13 @@ Contact: SeongJae Park <sj@kernel.org> · 150 unmodified lines
151Description: Writing to and reading from this file sets and gets the action151Description: Writing to and reading from this file sets and gets the action
152 of the scheme.152 of the scheme.
153 153
154What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us
155Date: Sep 2023
156Contact: SeongJae Park <sj@kernel.org>
157Description: Writing a value to this file sets the action apply interval of
158 the scheme in microseconds. Reading this file returns the
159 value.
160
154What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/min161What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/min
155Date: Mar 2022162Date: Mar 2022
156Contact: SeongJae Park <sj@kernel.org>163Contact: SeongJae Park <sj@kernel.org>
Documentation/admin-guide/cgroup-v1/memory.rst+1−0
@@ -551,6 +551,7 @@ memory.stat file includes following statistics: · 550 unmodified lines
551 event happens each time a page is unaccounted from the551 event happens each time a page is unaccounted from the
552 cgroup.552 cgroup.
553 swap # of bytes of swap usage553 swap # of bytes of swap usage
554 swapcached # of bytes of swap cached in memory
554 dirty # of bytes that are waiting to get written back to the disk.555 dirty # of bytes that are waiting to get written back to the disk.
555 writeback # of bytes of file/anon cache that are queued for syncing to556 writeback # of bytes of file/anon cache that are queued for syncing to
556 disk.557 disk.
Documentation/admin-guide/cgroup-v2.rst+38−0
@@ -210,6 +210,35 @@ cgroup v2 currently supports the following mount options. · 209 unmodified lines
210 relying on the original semantics (e.g. specifying bogusly210 relying on the original semantics (e.g. specifying bogusly
211 high 'bypass' protection values at higher tree levels).211 high 'bypass' protection values at higher tree levels).
212 212
213 memory_hugetlb_accounting
214 Count HugeTLB memory usage towards the cgroup's overall
215 memory usage for the memory controller (for the purpose of
216 statistics reporting and memory protetion). This is a new
217 behavior that could regress existing setups, so it must be
218 explicitly opted in with this mount option.
219
220 A few caveats to keep in mind:
221
222 * There is no HugeTLB pool management involved in the memory
223 controller. The pre-allocated pool does not belong to anyone.
224 Specifically, when a new HugeTLB folio is allocated to
225 the pool, it is not accounted for from the perspective of the
226 memory controller. It is only charged to a cgroup when it is
227 actually used (for e.g at page fault time). Host memory
228 overcommit management has to consider this when configuring
229 hard limits. In general, HugeTLB pool management should be
230 done via other mechanisms (such as the HugeTLB controller).
231 * Failure to charge a HugeTLB folio to the memory controller
232 results in SIGBUS. This could happen even if the HugeTLB pool
233 still has pages available (but the cgroup limit is hit and
234 reclaim attempt fails).
235 * Charging HugeTLB memory towards the memory controller affects
236 memory protection and reclaim dynamics. Any userspace tuning
237 (of low, min limits for e.g) needs to take this into account.
238 * HugeTLB pages utilized while this option is not selected
239 will not be tracked by the memory controller (even if cgroup
240 v2 is remounted later on).
241
213 242
214Organizing Processes and Threads243Organizing Processes and Threads
215--------------------------------244--------------------------------
@@ -1539,6 +1568,15 @@ PAGE_SIZE multiple when read back. · 1323 unmodified lines
1539 collapsing an existing range of pages. This counter is not1568 collapsing an existing range of pages. This counter is not
1540 present when CONFIG_TRANSPARENT_HUGEPAGE is not set.1569 present when CONFIG_TRANSPARENT_HUGEPAGE is not set.
1541 1570
1571 thp_swpout (npn)
1572 Number of transparent hugepages which are swapout in one piece
1573 without splitting.
1574
1575 thp_swpout_fallback (npn)
1576 Number of transparent hugepages which were split before swapout.
1577 Usually because failed to allocate some continuous swap space
1578 for the huge page.
1579
1542 memory.numa_stat1580 memory.numa_stat
1543 A read-only nested-keyed file which exists on non-root cgroups.1581 A read-only nested-keyed file which exists on non-root cgroups.
1544 1582
Documentation/admin-guide/mm/damon/usage.rst+81−43
@@ -20,18 +20,18 @@ DAMON provides below interfaces for different users. · 19 unmodified lines
20 you can write and use your personalized DAMON sysfs wrapper programs that20 you can write and use your personalized DAMON sysfs wrapper programs that
21 reads/writes the sysfs files instead of you. The `DAMON user space tool21 reads/writes the sysfs files instead of you. The `DAMON user space tool
22 <https://github.com/awslabs/damo>`_ is one example of such programs.22 <https://github.com/awslabs/damo>`_ is one example of such programs.
23- *debugfs interface. (DEPRECATED!)*
24 :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
25 <sysfs_interface>`. This is deprecated, so users should move to the
26 :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
27 move, please report your usecase to damon@lists.linux.dev and
28 linux-mm@kvack.org.
29- *Kernel Space Programming Interface.*23- *Kernel Space Programming Interface.*
30 :doc:`This </mm/damon/api>` is for kernel space programmers. Using this,24 :doc:`This </mm/damon/api>` is for kernel space programmers. Using this,
31 users can utilize every feature of DAMON most flexibly and efficiently by25 users can utilize every feature of DAMON most flexibly and efficiently by
32 writing kernel space DAMON application programs for you. You can even extend26 writing kernel space DAMON application programs for you. You can even extend
33 DAMON for various address spaces. For detail, please refer to the interface27 DAMON for various address spaces. For detail, please refer to the interface
34 :doc:`document </mm/damon/api>`.28 :doc:`document </mm/damon/api>`.
29- *debugfs interface. (DEPRECATED!)*
30 :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
31 <sysfs_interface>`. This is deprecated, so users should move to the
32 :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
33 move, please report your usecase to damon@lists.linux.dev and
34 linux-mm@kvack.org.
35 35
36.. _sysfs_interface:36.. _sysfs_interface:
37 37
@@ -76,7 +76,7 @@ comma (","). :: · 38 unmodified lines
76 │ │ │ │ │ │ │ │ ...76 │ │ │ │ │ │ │ │ ...
77 │ │ │ │ │ │ ...77 │ │ │ │ │ │ ...
78 │ │ │ │ │ schemes/nr_schemes78 │ │ │ │ │ schemes/nr_schemes
79 │ │ │ │ │ │ 0/action79 │ │ │ │ │ │ 0/action,apply_interval_us
80 │ │ │ │ │ │ │ access_pattern/80 │ │ │ │ │ │ │ access_pattern/
81 │ │ │ │ │ │ │ │ sz/min,max81 │ │ │ │ │ │ │ │ sz/min,max
82 │ │ │ │ │ │ │ │ nr_accesses/min,max82 │ │ │ │ │ │ │ │ nr_accesses/min,max
@@ -105,14 +105,12 @@ having the root permission could use this directory. · 22 unmodified lines
105kdamonds/105kdamonds/
106---------106---------
107 107
108The monitoring-related information including request specifications and results
109are called DAMON context. DAMON executes each context with a kernel thread
110called kdamond, and multiple kdamonds could run in parallel.
111
112Under the ``admin`` directory, one directory, ``kdamonds``, which has files for108Under the ``admin`` directory, one directory, ``kdamonds``, which has files for
113controlling the kdamonds exist. In the beginning, this directory has only one109controlling the kdamonds (refer to
114file, ``nr_kdamonds``. Writing a number (``N``) to the file creates the number110:ref:`design <damon_design_execution_model_and_data_structures>` for more
115of child directories named ``0`` to ``N-1``. Each directory represents each111details) exists. In the beginning, this directory has only one file,
112``nr_kdamonds``. Writing a number (``N``) to the file creates the number of
113child directories named ``0`` to ``N-1``. Each directory represents each
116kdamond.114kdamond.
117 115
118kdamonds/<N>/116kdamonds/<N>/
@@ -150,9 +148,10 @@ kdamonds/<N>/contexts/ · 31 unmodified lines
150 148
151In the beginning, this directory has only one file, ``nr_contexts``. Writing a149In the beginning, this directory has only one file, ``nr_contexts``. Writing a
152number (``N``) to the file creates the number of child directories named as150number (``N``) to the file creates the number of child directories named as
153``0`` to ``N-1``. Each directory represents each monitoring context. At the151``0`` to ``N-1``. Each directory represents each monitoring context (refer to
154moment, only one context per kdamond is supported, so only ``0`` or ``1`` can152:ref:`design <damon_design_execution_model_and_data_structures>` for more
155be written to the file.153details). At the moment, only one context per kdamond is supported, so only
154``0`` or ``1`` can be written to the file.
156 155
157.. _sysfs_contexts:156.. _sysfs_contexts:
158 157
@@ -270,8 +269,8 @@ schemes/<N>/ · 111 unmodified lines
270------------269------------
271 270
272In each scheme directory, five directories (``access_pattern``, ``quotas``,271In each scheme directory, five directories (``access_pattern``, ``quotas``,
273``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and one file272``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files
274(``action``) exist.273(``action`` and ``apply_interval``) exist.
275 274
276The ``action`` file is for setting and getting the scheme's :ref:`action275The ``action`` file is for setting and getting the scheme's :ref:`action
277<damon_design_damos_action>`. The keywords that can be written to and read276<damon_design_damos_action>`. The keywords that can be written to and read
@@ -297,6 +296,9 @@ Note that support of each action depends on the running DAMON operations set · 19 unmodified lines
297 - ``stat``: Do nothing but count the statistics.296 - ``stat``: Do nothing but count the statistics.
298 Supported by all operations sets.297 Supported by all operations sets.
299 298
299The ``apply_interval_us`` file is for setting and getting the scheme's
300:ref:`apply_interval <damon_design_damos>` in microseconds.
301
300schemes/<N>/access_pattern/302schemes/<N>/access_pattern/
301---------------------------303---------------------------
302 304
@@ -392,7 +394,7 @@ pages of all memory cgroups except ``/having_care_already``.:: · 89 unmodified lines
392 echo N > 1/matching394 echo N > 1/matching
393 395
394Note that ``anon`` and ``memcg`` filters are currently supported only when396Note that ``anon`` and ``memcg`` filters are currently supported only when
395``paddr`` `implementation <sysfs_contexts>` is being used.397``paddr`` :ref:`implementation <sysfs_contexts>` is being used.
396 398
397Also, memory regions that are filtered out by ``addr`` or ``target`` filters399Also, memory regions that are filtered out by ``addr`` or ``target`` filters
398are not counted as the scheme has tried to those, while regions that filtered400are not counted as the scheme has tried to those, while regions that filtered
@@ -430,9 +432,9 @@ that reading it returns the total size of the scheme tried regions, and creates · 31 unmodified lines
430directories named integer starting from ``0`` under this directory. Each432directories named integer starting from ``0`` under this directory. Each
431directory contains files exposing detailed information about each of the memory433directory contains files exposing detailed information about each of the memory
432region that the corresponding scheme's ``action`` has tried to be applied under434region that the corresponding scheme's ``action`` has tried to be applied under
433this directory, during next :ref:`aggregation interval435this directory, during next :ref:`apply interval <damon_design_damos>` of the
434<sysfs_monitoring_attrs>`. The information includes address range,436corresponding scheme. The information includes address range, ``nr_accesses``,
435``nr_accesses``, and ``age`` of the region.437and ``age`` of the region.
436 438
437Writing ``update_schemes_tried_bytes`` to the relevant ``kdamonds/<N>/state``439Writing ``update_schemes_tried_bytes`` to the relevant ``kdamonds/<N>/state``
438file will only update the ``total_bytes`` file, and will not create the440file will only update the ``total_bytes`` file, and will not create the
@@ -495,6 +497,62 @@ Please note that it's highly recommended to use user space tools like `damo · 56 unmodified lines
495<https://github.com/awslabs/damo>`_ rather than manually reading and writing497<https://github.com/awslabs/damo>`_ rather than manually reading and writing
496the files as above. Above is only for an example.498the files as above. Above is only for an example.
497 499
500.. _tracepoint:
501
502Tracepoints for Monitoring Results
503==================================
504
505Users can get the monitoring results via the :ref:`tried_regions
506<sysfs_schemes_tried_regions>`. The interface is useful for getting a
507snapshot, but it could be inefficient for fully recording all the monitoring
508results. For the purpose, two trace points, namely ``damon:damon_aggregated``
509and ``damon:damos_before_apply``, are provided. ``damon:damon_aggregated``
510provides the whole monitoring results, while ``damon:damos_before_apply``
511provides the monitoring results for regions that each DAMON-based Operation
512Scheme (:ref:`DAMOS <damon_design_damos>`) is gonna be applied. Hence,
513``damon:damos_before_apply`` is more useful for recording internal behavior of
514DAMOS, or DAMOS target access
515:ref:`pattern <damon_design_damos_access_pattern>` based query-like efficient
516monitoring results recording.
517
518While the monitoring is turned on, you could record the tracepoint events and
519show results using tracepoint supporting tools like ``perf``. For example::
520
521 # echo on > monitor_on
522 # perf record -e damon:damon_aggregated &
523 # sleep 5
524 # kill 9 $(pidof perf)
525 # echo off > monitor_on
526 # perf script
527 kdamond.0 46568 [027] 79357.842179: damon:damon_aggregated: target_id=0 nr_regions=11 122509119488-135708762112: 0 864
528 [...]
529
530Each line of the perf script output represents each monitoring region. The
531first five fields are as usual other tracepoint outputs. The sixth field
532(``target_id=X``) shows the ide of the monitoring target of the region. The
533seventh field (``nr_regions=X``) shows the total number of monitoring regions
534for the target. The eighth field (``X-Y:``) shows the start (``X``) and end
535(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the
536``nr_accesses`` of the region (refer to
537:ref:`design <damon_design_region_based_sampling>` for more details of the
538counter). Finally the tenth field (``X``) shows the ``age`` of the region
539(refer to :ref:`design <damon_design_age_tracking>` for more details of the
540counter).
541
542If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would
543be somewhat like below::
544
545 kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136
546 [...]
547
548Each line of the output represents each monitoring region that each DAMON-based
549Operation Scheme was about to be applied at the traced time. The first five
550fields are as usual. It shows the index of the DAMON context (``ctx_idx=X``)
551of the scheme in the list of the contexts of the context's kdamond, the index
552of the scheme (``scheme_idx=X``) in the list of the schemes of the context, in
553addition to the output of ``damon_aggregated`` tracepoint.
554
555
498.. _debugfs_interface:556.. _debugfs_interface:
499 557
500debugfs Interface (DEPRECATED!)558debugfs Interface (DEPRECATED!)
@@ -790,23 +848,3 @@ directory by putting the name of the context to the ``rm_contexts`` file. :: · 289 unmodified lines
790 848
791Note that ``mk_contexts``, ``rm_contexts``, and ``monitor_on`` files are in the849Note that ``mk_contexts``, ``rm_contexts``, and ``monitor_on`` files are in the
792root directory only.850root directory only.
793
794
795.. _tracepoint:
796
797Tracepoint for Monitoring Results
798=================================
799
800Users can get the monitoring results via the :ref:`tried_regions
801<sysfs_schemes_tried_regions>` or a tracepoint, ``damon:damon_aggregated``.
802While the tried regions directory is useful for getting a snapshot, the
803tracepoint is useful for getting a full record of the results. While the
804monitoring is turned on, you could record the tracepoint events and show
805results using tracepoint supporting tools like ``perf``. For example::
806
807 # echo on > monitor_on
808 # perf record -e damon:damon_aggregated &
809 # sleep 5
810 # kill 9 $(pidof perf)
811 # echo off > monitor_on
812 # perf script
Documentation/admin-guide/mm/ksm.rst+11−0
@@ -155,6 +155,15 @@ stable_node_chains_prune_millisecs · 154 unmodified lines
155 scan. It's a noop if not a single KSM page hit the155 scan. It's a noop if not a single KSM page hit the
156 ``max_page_sharing`` yet.156 ``max_page_sharing`` yet.
157 157
158smart_scan
159 Historically KSM checked every candidate page for each scan. It did
160 not take into account historic information. When smart scan is
161 enabled, pages that have previously not been de-duplicated get
162 skipped. How often these pages are skipped depends on how often
163 de-duplication has already been tried and failed. By default this
164 optimization is enabled. The ``pages_skipped`` metric shows how
165 effective the setting is.
166
158The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``:167The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``:
159 168
160general_profit169general_profit
@@ -169,6 +178,8 @@ pages_unshared · 8 unmodified lines
169 how many pages unique but repeatedly checked for merging178 how many pages unique but repeatedly checked for merging
170pages_volatile179pages_volatile
171 how many pages changing too fast to be placed in a tree180 how many pages changing too fast to be placed in a tree
181pages_skipped
182 how many pages did the "smart" page scanning algorithm skip
172full_scans183full_scans
173 how many times all mergeable areas have been scanned184 how many times all mergeable areas have been scanned
174stable_node_chains185stable_node_chains
Documentation/admin-guide/mm/pagemap.rst+89−0
@@ -227,3 +227,92 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is · 226 unmodified lines
227always 12 at most architectures). Since Linux 3.11 their meaning changes227always 12 at most architectures). Since Linux 3.11 their meaning changes
228after first clear of soft-dirty bits. Since Linux 4.2 they are used for228after first clear of soft-dirty bits. Since Linux 4.2 they are used for
229flags unconditionally.229flags unconditionally.
230
231Pagemap Scan IOCTL
232==================
233
234The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally
235clear the info about page table entries. The following operations are supported
236in this IOCTL:
237
238- Scan the address range and get the memory ranges matching the provided criteria.
239 This is performed when the output buffer is specified.
240- Write-protect the pages. The ``PM_SCAN_WP_MATCHING`` is used to write-protect
241 the pages of interest. The ``PM_SCAN_CHECK_WPASYNC`` aborts the operation if
242 non-Async Write Protected pages are found. The ``PM_SCAN_WP_MATCHING`` can be
243 used with or without ``PM_SCAN_CHECK_WPASYNC``.
244- Both of those operations can be combined into one atomic operation where we can
245 get and write protect the pages as well.
246
247Following flags about pages are currently supported:
248
249- ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled
250- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected
251- ``PAGE_IS_FILE`` - Page is file backed
252- ``PAGE_IS_PRESENT`` - Page is present in the memory
253- ``PAGE_IS_SWAPPED`` - Page is in swapped
254- ``PAGE_IS_PFNZERO`` - Page has zero PFN
255- ``PAGE_IS_HUGE`` - Page is THP or Hugetlb backed
256
257The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
258
259 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size``
260 field. This field will be helpful in recognizing the structure if extensions
261 are done later.
262 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING``
263 and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get
264 operation is optionally performed depending upon if the output buffer is
265 provided or not.
266 3. The range is specified through ``start`` and ``end``.
267 4. The walk can abort before visiting the complete range such as the user buffer
268 can get full etc. The walk ending address is specified in``end_walk``.
269 5. The output buffer of ``struct page_region`` array and size is specified in
270 ``vec`` and ``vec_len``.
271 6. The optional maximum requested pages are specified in the ``max_pages``.
272 7. The masks are specified in ``category_mask``, ``category_anyof_mask``,
273 ``category_inverted`` and ``return_mask``.
274
275Find pages which have been written and WP them as well::
276
277 struct pm_scan_arg arg = {
278 .size = sizeof(arg),
279 .flags = PM_SCAN_CHECK_WPASYNC | PM_SCAN_CHECK_WPASYNC,
280 ..
281 .category_mask = PAGE_IS_WRITTEN,
282 .return_mask = PAGE_IS_WRITTEN,
283 };
284
285Find pages which have been written, are file backed, not swapped and either
286present or huge::
287
288 struct pm_scan_arg arg = {
289 .size = sizeof(arg),
290 .flags = 0,
291 ..
292 .category_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED,
293 .category_inverted = PAGE_IS_SWAPPED,
294 .category_anyof_mask = PAGE_IS_PRESENT | PAGE_IS_HUGE,
295 .return_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED |
296 PAGE_IS_PRESENT | PAGE_IS_HUGE,
297 };
298
299The ``PAGE_IS_WRITTEN`` flag can be considered as a better-performing alternative
300of soft-dirty flag. It doesn't get affected by VMA merging of the kernel and hence
301the user can find the true soft-dirty pages in case of normal pages. (There may
302still be extra dirty pages reported for THP or Hugetlb pages.)
303
304"PAGE_IS_WRITTEN" category is used with uffd write protect-enabled ranges to
305implement memory dirty tracking in userspace:
306
307 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall.
308 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features
309 are set by ``UFFDIO_API`` IOCTL.
310 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode
311 through ``UFFDIO_REGISTER`` IOCTL.
312 4. Then any part of the registered memory or the whole memory region must
313 be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_WP_MATCHING``
314 or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the
315 same operation. The former is better in terms of performance.
316 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which
317 have been written to since they were last marked and/or optionally write protect
318 the pages as well.
Documentation/admin-guide/mm/userfaultfd.rst+35−0
@@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls · 243 unmodified lines
244support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``244support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP``
245respectively) to configure the mapping this way.245respectively) to configure the mapping this way.
246 246
247If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set,
248any vma registered with write-protection will work in async mode rather
249than the default sync mode.
250
251In async mode, there will be no message generated when a write operation
252happens, meanwhile the write-protection will be resolved automatically by
253the kernel. It can be seen as a more accurate version of soft-dirty
254tracking and it can be different in a few ways:
255
256 - The dirty result will not be affected by vma changes (e.g. vma
257 merging) because the dirty is only tracked by the pte.
258
259 - It supports range operations by default, so one can enable tracking on
260 any range of memory as long as page aligned.
261
262 - Dirty information will not get lost if the pte was zapped due to
263 various reasons (e.g. during split of a shmem transparent huge page).
264
265 - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
266 set; dirty when uffd-wp bit cleared), it has different semantics on
267 some of the memory operations. For example: ``MADV_DONTNEED`` on
268 anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
269 dirtying of memory by dropping uffd-wp bit during the procedure.
270
271The user app can collect the "written/dirty" status by looking up the
272uffd-wp bit for the pages being interested in /proc/pagemap.
273
274The page will not be under track of uffd-wp async mode until the page is
275explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
276flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault
277that was tracked by async mode userfaultfd-wp is invalid.
278
279When userfaultfd-wp async mode is used alone, it can be applied to all
280kinds of memory.
281
247Memory Poisioning Emulation282Memory Poisioning Emulation
248---------------------------283---------------------------
249 284
Documentation/core-api/maple_tree.rst+1−1
@@ -175,7 +175,7 @@ will return the previous entry which occurs before the entry at index. · 174 unmodified lines
175mas_find() will find the first entry which exists at or above index on175mas_find() will find the first entry which exists at or above index on
176the first call, and the next entry from every subsequent calls.176the first call, and the next entry from every subsequent calls.
177 177
178mas_find_rev() will find the fist entry which exists at or below the last on178mas_find_rev() will find the first entry which exists at or below the last on
179the first call, and the previous entry from every subsequent calls.179the first call, and the previous entry from every subsequent calls.
180 180
181If the user needs to yield the lock during an operation, then the maple state181If the user needs to yield the lock during an operation, then the maple state
Documentation/dev-tools/kasan.rst+5−2
@@ -1,5 +1,8 @@
1The Kernel Address Sanitizer (KASAN)1.. SPDX-License-Identifier: GPL-2.0
2====================================2.. Copyright (C) 2023, Google LLC.
3
4Kernel Address Sanitizer (KASAN)
5================================
3 6
4Overview7Overview
5--------8--------
Documentation/dev-tools/kcsan.rst+2−2
@@ -1,8 +1,8 @@
1.. SPDX-License-Identifier: GPL-2.01.. SPDX-License-Identifier: GPL-2.0
2.. Copyright (C) 2019, Google LLC.2.. Copyright (C) 2019, Google LLC.
3 3
4The Kernel Concurrency Sanitizer (KCSAN)4Kernel Concurrency Sanitizer (KCSAN)
5========================================5====================================
6 6
7The Kernel Concurrency Sanitizer (KCSAN) is a dynamic race detector, which7The Kernel Concurrency Sanitizer (KCSAN) is a dynamic race detector, which
8relies on compile-time instrumentation, and uses a watchpoint-based sampling8relies on compile-time instrumentation, and uses a watchpoint-based sampling
Documentation/dev-tools/kmsan.rst+3−3
@@ -1,9 +1,9 @@
1.. SPDX-License-Identifier: GPL-2.01.. SPDX-License-Identifier: GPL-2.0
2.. Copyright (C) 2022, Google LLC.2.. Copyright (C) 2022, Google LLC.
3 3
4===================================4===============================
5The Kernel Memory Sanitizer (KMSAN)5Kernel Memory Sanitizer (KMSAN)
6===================================6===============================
7 7
8KMSAN is a dynamic error detector aimed at finding uses of uninitialized8KMSAN is a dynamic error detector aimed at finding uses of uninitialized
9values. It is based on compiler instrumentation, and is quite similar to the9values. It is based on compiler instrumentation, and is quite similar to the
Documentation/dev-tools/ubsan.rst+4−2
@@ -1,5 +1,7 @@
1The Undefined Behavior Sanitizer - UBSAN1.. SPDX-License-Identifier: GPL-2.0
2========================================2
3Undefined Behavior Sanitizer - UBSAN
4====================================
3 5
4UBSAN is a runtime undefined behaviour checker.6UBSAN is a runtime undefined behaviour checker.
5 7
Documentation/mm/damon/design.rst+22−4
@@ -154,6 +154,8 @@ The monitoring overhead of this mechanism will arbitrarily increase as the · 153 unmodified lines
154size of the target workload grows.154size of the target workload grows.
155 155
156 156
157.. _damon_design_region_based_sampling:
158
157Region Based Sampling159Region Based Sampling
158~~~~~~~~~~~~~~~~~~~~~160~~~~~~~~~~~~~~~~~~~~~
159 161
@@ -163,9 +165,10 @@ assumption (pages in a region have the same access frequencies) is kept, only · 3 unmodified lines
163one page in the region is required to be checked. Thus, for each ``sampling165one page in the region is required to be checked. Thus, for each ``sampling
164interval``, DAMON randomly picks one page in each region, waits for one166interval``, DAMON randomly picks one page in each region, waits for one
165``sampling interval``, checks whether the page is accessed meanwhile, and167``sampling interval``, checks whether the page is accessed meanwhile, and
166increases the access frequency of the region if so. Therefore, the monitoring168increases the access frequency counter of the region if so. The counter is
167overhead is controllable by setting the number of regions. DAMON allows users169called ``nr_regions`` of the region. Therefore, the monitoring overhead is
168to set the minimum and the maximum number of regions for the trade-off.170controllable by setting the number of regions. DAMON allows users to set the
171minimum and the maximum number of regions for the trade-off.
169 172
170This scheme, however, cannot preserve the quality of the output if the173This scheme, however, cannot preserve the quality of the output if the
171assumption is not guaranteed.174assumption is not guaranteed.
@@ -190,6 +193,8 @@ In this way, DAMON provides its best-effort quality and minimal overhead while · 18 unmodified lines
190keeping the bounds users set for their trade-off.193keeping the bounds users set for their trade-off.
191 194
192 195
196.. _damon_design_age_tracking:
197
193Age Tracking198Age Tracking
194~~~~~~~~~~~~199~~~~~~~~~~~~
195 200
@@ -254,7 +259,8 @@ works, DAMON provides a feature called Data Access Monitoring-based Operation · 58 unmodified lines
254Schemes (DAMOS). It lets users specify their desired schemes at a high259Schemes (DAMOS). It lets users specify their desired schemes at a high
255level. For such specifications, DAMON starts monitoring, finds regions having260level. For such specifications, DAMON starts monitoring, finds regions having
256the access pattern of interest, and applies the user-desired operation actions261the access pattern of interest, and applies the user-desired operation actions
257to the regions as soon as found.262to the regions, for every user-specified time interval called
263``apply_interval``.
258 264
259 265
260.. _damon_design_damos_action:266.. _damon_design_damos_action:
@@ -471,3 +477,15 @@ modules for proactive reclamation and LRU lists manipulation are provided. For · 210 unmodified lines
471more detail, please read the usage documents for those477more detail, please read the usage documents for those
472(:doc:`/admin-guide/mm/damon/reclaim` and478(:doc:`/admin-guide/mm/damon/reclaim` and
473:doc:`/admin-guide/mm/damon/lru_sort`).479:doc:`/admin-guide/mm/damon/lru_sort`).
480
481
482.. _damon_design_execution_model_and_data_structures:
483
484Execution Model and Data Structures
485===================================
486
487The monitoring-related information including the monitoring request
488specification and DAMON-based operation schemes are stored in a data structure
489called DAMON ``context``. DAMON executes each context with a kernel thread
490called ``kdamond``. Multiple kdamonds could run in parallel, for different
491types of monitoring.
Documentation/trace/postprocess/trace-vmscan-postprocess.pl+18−24
@@ -107,14 +107,14 @@ GetOptions( · 106 unmodified lines
107);107);
108 108
109# Defaults for dynamically discovered regex's109# Defaults for dynamically discovered regex's
110my $regex_direct_begin_default = 'order=([0-9]*) may_writepage=([0-9]*) gfp_flags=([A-Z_|]*)';110my $regex_direct_begin_default = 'order=([0-9]*) gfp_flags=([A-Z_|]*)';
111my $regex_direct_end_default = 'nr_reclaimed=([0-9]*)';111my $regex_direct_end_default = 'nr_reclaimed=([0-9]*)';
112my $regex_kswapd_wake_default = 'nid=([0-9]*) order=([0-9]*)';112my $regex_kswapd_wake_default = 'nid=([0-9]*) order=([0-9]*)';
113my $regex_kswapd_sleep_default = 'nid=([0-9]*)';113my $regex_kswapd_sleep_default = 'nid=([0-9]*)';
114my $regex_wakeup_kswapd_default = 'nid=([0-9]*) zid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)';114my $regex_wakeup_kswapd_default = 'nid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)';
115my $regex_lru_isolate_default = 'isolate_mode=([0-9]*) classzone_idx=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)';115my $regex_lru_isolate_default = 'classzone=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)';
116my $regex_lru_shrink_inactive_default = 'nid=([0-9]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) nr_dirty=([0-9]*) nr_writeback=([0-9]*) nr_congested=([0-9]*) nr_immediate=([0-9]*) nr_activate_anon=([0-9]*) nr_activate_file=([0-9]*) nr_ref_keep=([0-9]*) nr_unmap_fail=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)';116my $regex_lru_shrink_inactive_default = 'nid=([0-9]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) nr_dirty=([0-9]*) nr_writeback=([0-9]*) nr_congested=([0-9]*) nr_immediate=([0-9]*) nr_activate_anon=([0-9]*) nr_activate_file=([0-9]*) nr_ref_keep=([0-9]*) nr_unmap_fail=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)';
117my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_scanned=([0-9]*) nr_rotated=([0-9]*) priority=([0-9]*)';117my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_taken=([0-9]*) nr_active=([0-9]*) nr_deactivated=([0-9]*) nr_referenced=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)' ;
118my $regex_writepage_default = 'page=([0-9a-f]*) pfn=([0-9]*) flags=([A-Z_|]*)';118my $regex_writepage_default = 'page=([0-9a-f]*) pfn=([0-9]*) flags=([A-Z_|]*)';
119 119
120# Dyanically discovered regex120# Dyanically discovered regex
@@ -184,8 +184,7 @@ sub generate_traceevent_regex { · 63 unmodified lines
184$regex_direct_begin = generate_traceevent_regex(184$regex_direct_begin = generate_traceevent_regex(
185 "vmscan/mm_vmscan_direct_reclaim_begin",185 "vmscan/mm_vmscan_direct_reclaim_begin",
186 $regex_direct_begin_default,186 $regex_direct_begin_default,
187 "order", "may_writepage",187 "order", "gfp_flags");
188 "gfp_flags");
189$regex_direct_end = generate_traceevent_regex(188$regex_direct_end = generate_traceevent_regex(
190 "vmscan/mm_vmscan_direct_reclaim_end",189 "vmscan/mm_vmscan_direct_reclaim_end",
191 $regex_direct_end_default,190 $regex_direct_end_default,
@@ -201,11 +200,11 @@ $regex_kswapd_sleep = generate_traceevent_regex( · 9 unmodified lines
201$regex_wakeup_kswapd = generate_traceevent_regex(200$regex_wakeup_kswapd = generate_traceevent_regex(
202 "vmscan/mm_vmscan_wakeup_kswapd",201 "vmscan/mm_vmscan_wakeup_kswapd",
203 $regex_wakeup_kswapd_default,202 $regex_wakeup_kswapd_default,
204 "nid", "zid", "order", "gfp_flags");203 "nid", "order", "gfp_flags");
205$regex_lru_isolate = generate_traceevent_regex(204$regex_lru_isolate = generate_traceevent_regex(
206 "vmscan/mm_vmscan_lru_isolate",205 "vmscan/mm_vmscan_lru_isolate",
207 $regex_lru_isolate_default,206 $regex_lru_isolate_default,
208 "isolate_mode", "classzone_idx", "order",207 "classzone", "order",
209 "nr_requested", "nr_scanned", "nr_skipped", "nr_taken",208 "nr_requested", "nr_scanned", "nr_skipped", "nr_taken",
210 "lru");209 "lru");
211$regex_lru_shrink_inactive = generate_traceevent_regex(210$regex_lru_shrink_inactive = generate_traceevent_regex(
@@ -218,11 +217,10 @@ $regex_lru_shrink_inactive = generate_traceevent_regex( · 6 unmodified lines
218$regex_lru_shrink_active = generate_traceevent_regex(217$regex_lru_shrink_active = generate_traceevent_regex(
219 "vmscan/mm_vmscan_lru_shrink_active",218 "vmscan/mm_vmscan_lru_shrink_active",
220 $regex_lru_shrink_active_default,219 $regex_lru_shrink_active_default,
221 "nid", "zid",220 "nid", "nr_taken", "nr_active", "nr_deactivated", "nr_referenced",
222 "lru",221 "priority", "flags");
223 "nr_scanned", "nr_rotated", "priority");
224$regex_writepage = generate_traceevent_regex(222$regex_writepage = generate_traceevent_regex(
225 "vmscan/mm_vmscan_writepage",223 "vmscan/mm_vmscan_write_folio",
226 $regex_writepage_default,224 $regex_writepage_default,
227 "page", "pfn", "flags");225 "page", "pfn", "flags");
228 226
@@ -371,7 +369,7 @@ EVENT_PROCESS: · 142 unmodified lines
371 print " $regex_wakeup_kswapd\n";369 print " $regex_wakeup_kswapd\n";
372 next;370 next;
373 }371 }
374 my $order = $3;372 my $order = $2;
375 $perprocesspid{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD_PERORDER}[$order]++;373 $perprocesspid{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD_PERORDER}[$order]++;
376 } elsif ($tracepoint eq "mm_vmscan_lru_isolate") {374 } elsif ($tracepoint eq "mm_vmscan_lru_isolate") {
377 $details = $6;375 $details = $6;
@@ -381,18 +379,14 @@ EVENT_PROCESS: · 3 unmodified lines
381 print " $regex_lru_isolate/o\n";379 print " $regex_lru_isolate/o\n";
382 next;380 next;
383 }381 }
384 my $isolate_mode = $1;382 my $nr_scanned = $4;
385 my $nr_scanned = $5;383 my $lru = $7;
386 my $file = $8;384
387 385 # To closer match vmstat scanning statistics, only count
388 # To closer match vmstat scanning statistics, only count isolate_both386 # inactive lru as scanning
389 # and isolate_inactive as scanning. isolate_active is rotation387 if ($lru =~ /inactive_/) {
390 # isolate_inactive == 1
391 # isolate_active == 2
392 # isolate_both == 3
393 if ($isolate_mode != 2) {
394 $perprocesspid{$process_pid}->{HIGH_NR_SCANNED} += $nr_scanned;388 $perprocesspid{$process_pid}->{HIGH_NR_SCANNED} += $nr_scanned;
395 if ($file =~ /_file/) {389 if ($lru =~ /_file/) {
396 $perprocesspid{$process_pid}->{HIGH_NR_FILE_SCANNED} += $nr_scanned;390 $perprocesspid{$process_pid}->{HIGH_NR_FILE_SCANNED} += $nr_scanned;
397 } else {391 } else {
398 $perprocesspid{$process_pid}->{HIGH_NR_ANON_SCANNED} += $nr_scanned;392 $perprocesspid{$process_pid}->{HIGH_NR_ANON_SCANNED} += $nr_scanned;
MAINTAINERS+2−0
@@ -5332,6 +5332,7 @@ S: Maintained · 5331 unmodified lines
5332F: mm/memcontrol.c5332F: mm/memcontrol.c
5333F: mm/swap_cgroup.c5333F: mm/swap_cgroup.c
5334F: tools/testing/selftests/cgroup/memcg_protection.m5334F: tools/testing/selftests/cgroup/memcg_protection.m
5335F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
5335F: tools/testing/selftests/cgroup/test_kmem.c5336F: tools/testing/selftests/cgroup/test_kmem.c
5336F: tools/testing/selftests/cgroup/test_memcontrol.c5337F: tools/testing/selftests/cgroup/test_memcontrol.c
5337 5338
@@ -9754,6 +9755,7 @@ F: include/linux/hugetlb.h · 4416 unmodified lines
9754F: mm/hugetlb.c9755F: mm/hugetlb.c
9755F: mm/hugetlb_vmemmap.c9756F: mm/hugetlb_vmemmap.c
9756F: mm/hugetlb_vmemmap.h9757F: mm/hugetlb_vmemmap.h
9758F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
9757 9759
9758HVA ST MEDIA DRIVER9760HVA ST MEDIA DRIVER
9759M: Jean-Christophe Trotin <jean-christophe.trotin@foss.st.com>9761M: Jean-Christophe Trotin <jean-christophe.trotin@foss.st.com>
arch/alpha/include/asm/bitops.h+20−0
@@ -286,6 +286,26 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 285 unmodified lines
286#define arch_test_bit generic_test_bit286#define arch_test_bit generic_test_bit
287#define arch_test_bit_acquire generic_test_bit_acquire287#define arch_test_bit_acquire generic_test_bit_acquire
288 288
289static inline bool xor_unlock_is_negative_byte(unsigned long mask,
290 volatile unsigned long *p)
291{
292 unsigned long temp, old;
293
294 __asm__ __volatile__(
295 "1: ldl_l %0,%4\n"
296 " mov %0,%2\n"
297 " xor %0,%3,%0\n"
298 " stl_c %0,%1\n"
299 " beq %0,2f\n"
300 ".subsection 2\n"
301 "2: br 1b\n"
302 ".previous"
303 :"=&r" (temp), "=m" (*p), "=&r" (old)
304 :"Ir" (mask), "m" (*p));
305
306 return (old & BIT(7)) != 0;
307}
308
289/*309/*
290 * ffz = Find First Zero in word. Undefined if no zero exists,310 * ffz = Find First Zero in word. Undefined if no zero exists,
291 * so code should check against ~0UL first..311 * so code should check against ~0UL first..
arch/arm64/include/asm/tlb.h+4−1
@@ -96,7 +96,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, · 95 unmodified lines
96static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,96static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
97 unsigned long addr)97 unsigned long addr)
98{98{
99 tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp));99 struct ptdesc *ptdesc = virt_to_ptdesc(pudp);
100
101 pagetable_pud_dtor(ptdesc);
102 tlb_remove_ptdesc(tlb, ptdesc);
100}103}
101#endif104#endif
102 105
arch/arm64/kernel/mte.c+2−2
@@ -411,8 +411,8 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, · 410 unmodified lines
411 struct page *page = get_user_page_vma_remote(mm, addr,411 struct page *page = get_user_page_vma_remote(mm, addr,
412 gup_flags, &vma);412 gup_flags, &vma);
413 413
414 if (IS_ERR_OR_NULL(page)) {414 if (IS_ERR(page)) {
415 err = page == NULL ? -EIO : PTR_ERR(page);415 err = PTR_ERR(page);
416 break;416 break;
417 }417 }
418 418
arch/arm64/mm/kasan_init.c+5−1
@@ -300,7 +300,11 @@ void __init kasan_init(void) · 299 unmodified lines
300 kasan_init_shadow();300 kasan_init_shadow();
301 kasan_init_depth();301 kasan_init_depth();
302#if defined(CONFIG_KASAN_GENERIC)302#if defined(CONFIG_KASAN_GENERIC)
303 /* CONFIG_KASAN_SW_TAGS also requires kasan_init_sw_tags(). */303 /*
304 * Generic KASAN is now fully initialized.
305 * Software and Hardware Tag-Based modes still require
306 * kasan_init_sw_tags() and kasan_init_hw_tags() correspondingly.
307 */
304 pr_info("KernelAddressSanitizer initialized (generic)\n");308 pr_info("KernelAddressSanitizer initialized (generic)\n");
305#endif309#endif
306}310}
arch/loongarch/include/asm/pgalloc.h+1−0
@@ -84,6 +84,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 83 unmodified lines
84 84
85 if (!ptdesc)85 if (!ptdesc)
86 return NULL;86 return NULL;
87 pagetable_pud_ctor(ptdesc);
87 pud = ptdesc_address(ptdesc);88 pud = ptdesc_address(ptdesc);
88 89
89 pud_init(pud);90 pud_init(pud);
arch/m68k/include/asm/bitops.h+21−0
@@ -319,6 +319,27 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 318 unmodified lines
319 return test_and_change_bit(nr, addr);319 return test_and_change_bit(nr, addr);
320}320}
321 321
322static inline bool xor_unlock_is_negative_byte(unsigned long mask,
323 volatile unsigned long *p)
324{
325#ifdef CONFIG_COLDFIRE
326 __asm__ __volatile__ ("eorl %1, %0"
327 : "+m" (*p)
328 : "d" (mask)
329 : "memory");
330 return *p & (1 << 7);
331#else
332 char result;
333 char *cp = (char *)p + 3; /* m68k is big-endian */
334
335 __asm__ __volatile__ ("eor.b %1, %2; smi %0"
336 : "=d" (result)
337 : "di" (mask), "o" (*cp)
338 : "memory");
339 return result;
340#endif
341}
342
322/*343/*
323 * The true 68020 and more advanced processors support the "bfffo"344 * The true 68020 and more advanced processors support the "bfffo"
324 * instruction for finding bits. ColdFire and simple 68000 parts345 * instruction for finding bits. ColdFire and simple 68000 parts
arch/mips/include/asm/bitops.h+24−1
@@ -73,7 +73,8 @@ int __mips_test_and_clear_bit(unsigned long nr, · 72 unmodified lines
73 volatile unsigned long *addr);73 volatile unsigned long *addr);
74int __mips_test_and_change_bit(unsigned long nr,74int __mips_test_and_change_bit(unsigned long nr,
75 volatile unsigned long *addr);75 volatile unsigned long *addr);
76 76bool __mips_xor_is_negative_byte(unsigned long mask,
77 volatile unsigned long *addr);
77 78
78/*79/*
79 * set_bit - Atomically set a bit in memory80 * set_bit - Atomically set a bit in memory
@@ -279,6 +280,28 @@ static inline int test_and_change_bit(unsigned long nr, · 199 unmodified lines
279 return res;280 return res;
280}281}
281 282
283static inline bool xor_unlock_is_negative_byte(unsigned long mask,
284 volatile unsigned long *p)
285{
286 unsigned long orig;
287 bool res;
288
289 smp_mb__before_atomic();
290
291 if (!kernel_uses_llsc) {
292 res = __mips_xor_is_negative_byte(mask, p);
293 } else {
294 orig = __test_bit_op(*p, "%0",
295 "xor\t%1, %0, %3",
296 "ir"(mask));
297 res = (orig & BIT(7)) != 0;
298 }
299
300 smp_llsc_mb();
301
302 return res;
303}
304
282#undef __bit_op305#undef __bit_op
283#undef __test_bit_op306#undef __test_bit_op
284 307
arch/mips/include/asm/pgalloc.h+1−0
@@ -95,6 +95,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 94 unmodified lines
95 95
96 if (!ptdesc)96 if (!ptdesc)
97 return NULL;97 return NULL;
98 pagetable_pud_ctor(ptdesc);
98 pud = ptdesc_address(ptdesc);99 pud = ptdesc_address(ptdesc);
99 100
100 pud_init(pud);101 pud_init(pud);
arch/mips/lib/bitops.c+14−0
@@ -146,3 +146,17 @@ int __mips_test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 145 unmodified lines
146 return res;146 return res;
147}147}
148EXPORT_SYMBOL(__mips_test_and_change_bit);148EXPORT_SYMBOL(__mips_test_and_change_bit);
149
150bool __mips_xor_is_negative_byte(unsigned long mask,
151 volatile unsigned long *addr)
152{
153 unsigned long flags;
154 unsigned long data;
155
156 raw_local_irq_save(flags);
157 data = *addr;
158 *addr = data ^ mask;
159 raw_local_irq_restore(flags);
160
161 return (data & BIT(7)) != 0;
162}
arch/mips/mm/cache.c+1−1
@@ -117,7 +117,7 @@ void __flush_dcache_pages(struct page *page, unsigned int nr) · 116 unmodified lines
117 * get faulted into the tlb (and thus flushed) anyways.117 * get faulted into the tlb (and thus flushed) anyways.
118 */118 */
119 for (i = 0; i < nr; i++) {119 for (i = 0; i < nr; i++) {
120 addr = (unsigned long)kmap_local_page(page + i);120 addr = (unsigned long)kmap_local_page(nth_page(page, i));
121 flush_data_cache_page(addr);121 flush_data_cache_page(addr);
122 kunmap_local((void *)addr);122 kunmap_local((void *)addr);
123 }123 }
arch/powerpc/include/asm/bitops.h+5−16
@@ -233,35 +233,24 @@ static inline int arch_test_and_change_bit(unsigned long nr, · 232 unmodified lines
233 return test_and_change_bits(BIT_MASK(nr), addr + BIT_WORD(nr)) != 0;233 return test_and_change_bits(BIT_MASK(nr), addr + BIT_WORD(nr)) != 0;
234}234}
235 235
236#ifdef CONFIG_PPC64236static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
237static inline unsigned long237 volatile unsigned long *p)
238clear_bit_unlock_return_word(int nr, volatile unsigned long *addr)
239{238{
240 unsigned long old, t;239 unsigned long old, t;
241 unsigned long *p = (unsigned long *)addr + BIT_WORD(nr);
242 unsigned long mask = BIT_MASK(nr);
243 240
244 __asm__ __volatile__ (241 __asm__ __volatile__ (
245 PPC_RELEASE_BARRIER242 PPC_RELEASE_BARRIER
246"1:" PPC_LLARX "%0,0,%3,0\n"243"1:" PPC_LLARX "%0,0,%3,0\n"
247 "andc %1,%0,%2\n"244 "xor %1,%0,%2\n"
248 PPC_STLCX "%1,0,%3\n"245 PPC_STLCX "%1,0,%3\n"
249 "bne- 1b\n"246 "bne- 1b\n"
250 : "=&r" (old), "=&r" (t)247 : "=&r" (old), "=&r" (t)
251 : "r" (mask), "r" (p)248 : "r" (mask), "r" (p)
252 : "cc", "memory");249 : "cc", "memory");
253 250
254 return old;251 return (old & BIT_MASK(7)) != 0;
255}252}
256 253#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
257/*
258 * This is a special function for mm/filemap.c
259 * Bit 7 corresponds to PG_waiters.
260 */
261#define arch_clear_bit_unlock_is_negative_byte(nr, addr) \
262 (clear_bit_unlock_return_word(nr, addr) & BIT_MASK(7))
263
264#endif /* CONFIG_PPC64 */
265 254
266#include <asm-generic/bitops/non-atomic.h>255#include <asm-generic/bitops/non-atomic.h>
267 256
arch/riscv/include/asm/bitops.h+12−0
@@ -191,6 +191,18 @@ static inline void __clear_bit_unlock( · 190 unmodified lines
191 clear_bit_unlock(nr, addr);191 clear_bit_unlock(nr, addr);
192}192}
193 193
194static inline bool xor_unlock_is_negative_byte(unsigned long mask,
195 volatile unsigned long *addr)
196{
197 unsigned long res;
198 __asm__ __volatile__ (
199 __AMO(xor) ".rl %0, %2, %1"
200 : "=r" (res), "+A" (*addr)
201 : "r" (__NOP(mask))
202 : "memory");
203 return (res & BIT(7)) != 0;
204}
205
194#undef __test_and_op_bit206#undef __test_and_op_bit
195#undef __op_bit207#undef __op_bit
196#undef __NOP208#undef __NOP
arch/s390/include/asm/bitops.h+10−0
@@ -201,6 +201,16 @@ static inline void arch___clear_bit_unlock(unsigned long nr, · 200 unmodified lines
201 arch___clear_bit(nr, ptr);201 arch___clear_bit(nr, ptr);
202}202}
203 203
204static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
205 volatile unsigned long *ptr)
206{
207 unsigned long old;
208
209 old = __atomic64_xor_barrier(mask, (long *)ptr);
210 return old & BIT(7);
211}
212#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
213
204#include <asm-generic/bitops/instrumented-atomic.h>214#include <asm-generic/bitops/instrumented-atomic.h>
205#include <asm-generic/bitops/instrumented-non-atomic.h>215#include <asm-generic/bitops/instrumented-non-atomic.h>
206#include <asm-generic/bitops/instrumented-lock.h>216#include <asm-generic/bitops/instrumented-lock.h>
arch/x86/include/asm/bitops.h+5−6
@@ -94,18 +94,17 @@ arch___clear_bit(unsigned long nr, volatile unsigned long *addr) · 93 unmodified lines
94 asm volatile(__ASM_SIZE(btr) " %1,%0" : : ADDR, "Ir" (nr) : "memory");94 asm volatile(__ASM_SIZE(btr) " %1,%0" : : ADDR, "Ir" (nr) : "memory");
95}95}
96 96
97static __always_inline bool97static __always_inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
98arch_clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr)98 volatile unsigned long *addr)
99{99{
100 bool negative;100 bool negative;
101 asm volatile(LOCK_PREFIX "andb %2,%1"101 asm volatile(LOCK_PREFIX "xorb %2,%1"
102 CC_SET(s)102 CC_SET(s)
103 : CC_OUT(s) (negative), WBYTE_ADDR(addr)103 : CC_OUT(s) (negative), WBYTE_ADDR(addr)
104 : "ir" ((char) ~(1 << nr)) : "memory");104 : "iq" ((char)mask) : "memory");
105 return negative;105 return negative;
106}106}
107#define arch_clear_bit_unlock_is_negative_byte \107#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte
108 arch_clear_bit_unlock_is_negative_byte
109 108
110static __always_inline void109static __always_inline void
111arch___clear_bit_unlock(long nr, volatile unsigned long *addr)110arch___clear_bit_unlock(long nr, volatile unsigned long *addr)
arch/x86/kvm/mmu/mmu.c+10−8
@@ -6800,11 +6800,7 @@ static unsigned long mmu_shrink_count(struct shrinker *shrink, · 6799 unmodified lines
6800 return percpu_counter_read_positive(&kvm_total_used_mmu_pages);6800 return percpu_counter_read_positive(&kvm_total_used_mmu_pages);
6801}6801}
6802 6802
6803static struct shrinker mmu_shrinker = {6803static struct shrinker *mmu_shrinker;
6804 .count_objects = mmu_shrink_count,
6805 .scan_objects = mmu_shrink_scan,
6806 .seeks = DEFAULT_SEEKS * 10,
6807};
6808 6804
6809static void mmu_destroy_caches(void)6805static void mmu_destroy_caches(void)
6810{6806{
@@ -6937,10 +6933,16 @@ int kvm_mmu_vendor_module_init(void) · 126 unmodified lines
6937 if (percpu_counter_init(&kvm_total_used_mmu_pages, 0, GFP_KERNEL))6933 if (percpu_counter_init(&kvm_total_used_mmu_pages, 0, GFP_KERNEL))
6938 goto out;6934 goto out;
6939 6935
6940 ret = register_shrinker(&mmu_shrinker, "x86-mmu");6936 mmu_shrinker = shrinker_alloc(0, "x86-mmu");
6941 if (ret)6937 if (!mmu_shrinker)
6942 goto out_shrinker;6938 goto out_shrinker;
6943 6939
6940 mmu_shrinker->count_objects = mmu_shrink_count;
6941 mmu_shrinker->scan_objects = mmu_shrink_scan;
6942 mmu_shrinker->seeks = DEFAULT_SEEKS * 10;
6943
6944 shrinker_register(mmu_shrinker);
6945
6944 return 0;6946 return 0;
6945 6947
6946out_shrinker:6948out_shrinker:
@@ -6962,7 +6964,7 @@ void kvm_mmu_vendor_module_exit(void) · 15 unmodified lines
6962{6964{
6963 mmu_destroy_caches();6965 mmu_destroy_caches();
6964 percpu_counter_destroy(&kvm_total_used_mmu_pages);6966 percpu_counter_destroy(&kvm_total_used_mmu_pages);
6965 unregister_shrinker(&mmu_shrinker);6967 shrinker_free(mmu_shrinker);
6966}6968}
6967 6969
6968/*6970/*
arch/x86/mm/pgtable.c+3−0
@@ -76,6 +76,9 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) · 75 unmodified lines
76#if CONFIG_PGTABLE_LEVELS > 376#if CONFIG_PGTABLE_LEVELS > 3
77void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)77void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)
78{78{
79 struct ptdesc *ptdesc = virt_to_ptdesc(pud);
80
81 pagetable_pud_dtor(ptdesc);
79 paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);82 paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
80 paravirt_tlb_remove_table(tlb, virt_to_page(pud));83 paravirt_tlb_remove_table(tlb, virt_to_page(pud));
81}84}
drivers/acpi/acpi_pad.c+1−1
@@ -101,7 +101,7 @@ static void round_robin_cpu(unsigned int tsk_index) · 100 unmodified lines
101 for_each_cpu(cpu, pad_busy_cpus)101 for_each_cpu(cpu, pad_busy_cpus)
102 cpumask_or(tmp, tmp, topology_sibling_cpumask(cpu));102 cpumask_or(tmp, tmp, topology_sibling_cpumask(cpu));
103 cpumask_andnot(tmp, cpu_online_mask, tmp);103 cpumask_andnot(tmp, cpu_online_mask, tmp);
104 /* avoid HT sibilings if possible */104 /* avoid HT siblings if possible */
105 if (cpumask_empty(tmp))105 if (cpumask_empty(tmp))
106 cpumask_andnot(tmp, cpu_online_mask, pad_busy_cpus);106 cpumask_andnot(tmp, cpu_online_mask, pad_busy_cpus);
107 if (cpumask_empty(tmp)) {107 if (cpumask_empty(tmp)) {
drivers/acpi/numa/hmat.c+93−53
@@ -24,6 +24,7 @@ · 23 unmodified lines
24#include <linux/node.h>24#include <linux/node.h>
25#include <linux/sysfs.h>25#include <linux/sysfs.h>
26#include <linux/dax.h>26#include <linux/dax.h>
27#include <linux/memory-tiers.h>
27 28
28static u8 hmat_revision;29static u8 hmat_revision;
29static int hmat_disable __initdata;30static int hmat_disable __initdata;
@@ -582,28 +583,25 @@ static int initiators_to_nodemask(unsigned long *p_nodes) · 552 unmodified lines
582 return 0;583 return 0;
583}584}
584 585
585static void hmat_register_target_initiators(struct memory_target *target)586static void hmat_update_target_attrs(struct memory_target *target,
587 unsigned long *p_nodes, int access)
586{588{
587 static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
588 struct memory_initiator *initiator;589 struct memory_initiator *initiator;
589 unsigned int mem_nid, cpu_nid;590 unsigned int cpu_nid;
590 struct memory_locality *loc = NULL;591 struct memory_locality *loc = NULL;
591 u32 best = 0;592 u32 best = 0;
592 bool access0done = false;
593 int i;593 int i;
594 594
595 mem_nid = pxm_to_node(target->memory_pxm);595 bitmap_zero(p_nodes, MAX_NUMNODES);
596 /*596 /*
597 * If the Address Range Structure provides a local processor pxm, link597 * If the Address Range Structure provides a local processor pxm, set
598 * only that one. Otherwise, find the best performance attributes and598 * only that one. Otherwise, find the best performance attributes and
599 * register all initiators that match.599 * collect all initiators that match.
600 */600 */
601 if (target->processor_pxm != PXM_INVAL) {601 if (target->processor_pxm != PXM_INVAL) {
602 cpu_nid = pxm_to_node(target->processor_pxm);602 cpu_nid = pxm_to_node(target->processor_pxm);
603 register_memory_node_under_compute_node(mem_nid, cpu_nid, 0);603 if (access == 0 || node_state(cpu_nid, N_CPU)) {
604 access0done = true;604 set_bit(target->processor_pxm, p_nodes);
605 if (node_state(cpu_nid, N_CPU)) {
606 register_memory_node_under_compute_node(mem_nid, cpu_nid, 1);
607 return;605 return;
608 }606 }
609 }607 }
@@ -617,47 +615,10 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines
617 * We'll also use the sorting to prime the candidate nodes with known615 * We'll also use the sorting to prime the candidate nodes with known
618 * initiators.616 * initiators.
619 */617 */
620 bitmap_zero(p_nodes, MAX_NUMNODES);
621 list_sort(NULL, &initiators, initiator_cmp);618 list_sort(NULL, &initiators, initiator_cmp);
622 if (initiators_to_nodemask(p_nodes) < 0)619 if (initiators_to_nodemask(p_nodes) < 0)
623 return;620 return;
624 621
625 if (!access0done) {
626 for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) {
627 loc = localities_types[i];
628 if (!loc)
629 continue;
630
631 best = 0;
632 list_for_each_entry(initiator, &initiators, node) {
633 u32 value;
634
635 if (!test_bit(initiator->processor_pxm, p_nodes))
636 continue;
637
638 value = hmat_initiator_perf(target, initiator,
639 loc->hmat_loc);
640 if (hmat_update_best(loc->hmat_loc->data_type, value, &best))
641 bitmap_clear(p_nodes, 0, initiator->processor_pxm);
642 if (value != best)
643 clear_bit(initiator->processor_pxm, p_nodes);
644 }
645 if (best)
646 hmat_update_target_access(target, loc->hmat_loc->data_type,
647 best, 0);
648 }
649
650 for_each_set_bit(i, p_nodes, MAX_NUMNODES) {
651 cpu_nid = pxm_to_node(i);
652 register_memory_node_under_compute_node(mem_nid, cpu_nid, 0);
653 }
654 }
655
656 /* Access 1 ignores Generic Initiators */
657 bitmap_zero(p_nodes, MAX_NUMNODES);
658 if (initiators_to_nodemask(p_nodes) < 0)
659 return;
660
661 for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) {622 for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) {
662 loc = localities_types[i];623 loc = localities_types[i];
663 if (!loc)624 if (!loc)
@@ -667,7 +628,7 @@ static void hmat_register_target_initiators(struct memory_target *target) · 3 unmodified lines
667 list_for_each_entry(initiator, &initiators, node) {628 list_for_each_entry(initiator, &initiators, node) {
668 u32 value;629 u32 value;
669 630
670 if (!initiator->has_cpu) {631 if (access == 1 && !initiator->has_cpu) {
671 clear_bit(initiator->processor_pxm, p_nodes);632 clear_bit(initiator->processor_pxm, p_nodes);
672 continue;633 continue;
673 }634 }
@@ -681,14 +642,33 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines
681 clear_bit(initiator->processor_pxm, p_nodes);642 clear_bit(initiator->processor_pxm, p_nodes);
682 }643 }
683 if (best)644 if (best)
684 hmat_update_target_access(target, loc->hmat_loc->data_type, best, 1);645 hmat_update_target_access(target, loc->hmat_loc->data_type, best, access);
685 }646 }
647}
648
649static void __hmat_register_target_initiators(struct memory_target *target,
650 unsigned long *p_nodes,
651 int access)
652{
653 unsigned int mem_nid, cpu_nid;
654 int i;
655
656 mem_nid = pxm_to_node(target->memory_pxm);
657 hmat_update_target_attrs(target, p_nodes, access);
686 for_each_set_bit(i, p_nodes, MAX_NUMNODES) {658 for_each_set_bit(i, p_nodes, MAX_NUMNODES) {
687 cpu_nid = pxm_to_node(i);659 cpu_nid = pxm_to_node(i);
688 register_memory_node_under_compute_node(mem_nid, cpu_nid, 1);660 register_memory_node_under_compute_node(mem_nid, cpu_nid, access);
689 }661 }
690}662}
691 663
664static void hmat_register_target_initiators(struct memory_target *target)
665{
666 static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
667
668 __hmat_register_target_initiators(target, p_nodes, 0);
669 __hmat_register_target_initiators(target, p_nodes, 1);
670}
671
692static void hmat_register_target_cache(struct memory_target *target)672static void hmat_register_target_cache(struct memory_target *target)
693{673{
694 unsigned mem_nid = pxm_to_node(target->memory_pxm);674 unsigned mem_nid = pxm_to_node(target->memory_pxm);
@@ -780,6 +760,61 @@ static int hmat_callback(struct notifier_block *self, · 85 unmodified lines
780 return NOTIFY_OK;760 return NOTIFY_OK;
781}761}
782 762
763static int hmat_set_default_dram_perf(void)
764{
765 int rc;
766 int nid, pxm;
767 struct memory_target *target;
768 struct node_hmem_attrs *attrs;
769
770 if (!default_dram_type)
771 return -EIO;
772
773 for_each_node_mask(nid, default_dram_type->nodes) {
774 pxm = node_to_pxm(nid);
775 target = find_mem_target(pxm);
776 if (!target)
777 continue;
778 attrs = &target->hmem_attrs[1];
779 rc = mt_set_default_dram_perf(nid, attrs, "ACPI HMAT");
780 if (rc)
781 return rc;
782 }
783
784 return 0;
785}
786
787static int hmat_calculate_adistance(struct notifier_block *self,
788 unsigned long nid, void *data)
789{
790 static DECLARE_BITMAP(p_nodes, MAX_NUMNODES);
791 struct memory_target *target;
792 struct node_hmem_attrs *perf;
793 int *adist = data;
794 int pxm;
795
796 pxm = node_to_pxm(nid);
797 target = find_mem_target(pxm);
798 if (!target)
799 return NOTIFY_OK;
800
801 mutex_lock(&target_lock);
802 hmat_update_target_attrs(target, p_nodes, 1);
803 mutex_unlock(&target_lock);
804
805 perf = &target->hmem_attrs[1];
806
807 if (mt_perf_to_adistance(perf, adist))
808 return NOTIFY_OK;
809
810 return NOTIFY_STOP;
811}
812
813static struct notifier_block hmat_adist_nb __meminitdata = {
814 .notifier_call = hmat_calculate_adistance,
815 .priority = 100,
816};
817
783static __init void hmat_free_structures(void)818static __init void hmat_free_structures(void)
784{819{
785 struct memory_target *target, *tnext;820 struct memory_target *target, *tnext;
@@ -862,8 +897,13 @@ static __init int hmat_init(void) · 76 unmodified lines
862 hmat_register_targets();897 hmat_register_targets();
863 898
864 /* Keep the table and structures if the notifier may use them */899 /* Keep the table and structures if the notifier may use them */
865 if (!hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI))900 if (hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI))
866 return 0;901 goto out_put;
902
903 if (!hmat_set_default_dram_perf())
904 register_mt_adistance_algorithm(&hmat_adist_nb);
905
906 return 0;
867out_put:907out_put:
868 hmat_free_structures();908 hmat_free_structures();
869 acpi_put_table(tbl);909 acpi_put_table(tbl);
drivers/android/binder_alloc.c+18−12
@@ -1053,11 +1053,7 @@ binder_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1052 unmodified lines
1053 NULL, sc->nr_to_scan);1053 NULL, sc->nr_to_scan);
1054}1054}
1055 1055
1056static struct shrinker binder_shrinker = {1056static struct shrinker *binder_shrinker;
1057 .count_objects = binder_shrink_count,
1058 .scan_objects = binder_shrink_scan,
1059 .seeks = DEFAULT_SEEKS,
1060};
1061 1057
1062/**1058/**
1063 * binder_alloc_init() - called by binder_open() for per-proc initialization1059 * binder_alloc_init() - called by binder_open() for per-proc initialization
@@ -1077,19 +1073,29 @@ void binder_alloc_init(struct binder_alloc *alloc) · 13 unmodified lines
1077 1073
1078int binder_alloc_shrinker_init(void)1074int binder_alloc_shrinker_init(void)
1079{1075{
1080 int ret = list_lru_init(&binder_alloc_lru);1076 int ret;
1081 1077
1082 if (ret == 0) {1078 ret = list_lru_init(&binder_alloc_lru);
1083 ret = register_shrinker(&binder_shrinker, "android-binder");1079 if (ret)
1084 if (ret)1080 return ret;
1085 list_lru_destroy(&binder_alloc_lru);1081
1082 binder_shrinker = shrinker_alloc(0, "android-binder");
1083 if (!binder_shrinker) {
1084 list_lru_destroy(&binder_alloc_lru);
1085 return -ENOMEM;
1086 }1086 }
1087 return ret;1087
1088 binder_shrinker->count_objects = binder_shrink_count;
1089 binder_shrinker->scan_objects = binder_shrink_scan;
1090
1091 shrinker_register(binder_shrinker);
1092
1093 return 0;
1088}1094}
1089 1095
1090void binder_alloc_shrinker_exit(void)1096void binder_alloc_shrinker_exit(void)
1091{1097{
1092 unregister_shrinker(&binder_shrinker);1098 shrinker_free(binder_shrinker);
1093 list_lru_destroy(&binder_alloc_lru);1099 list_lru_destroy(&binder_alloc_lru);
1094}1100}
1095 1101
drivers/base/cacheinfo.c+50−1
@@ -898,6 +898,48 @@ err: · 897 unmodified lines
898 return rc;898 return rc;
899}899}
900 900
901/*
902 * Calculate the size of the per-CPU data cache slice. This can be
903 * used to estimate the size of the data cache slice that can be used
904 * by one CPU under ideal circumstances. UNIFIED caches are counted
905 * in addition to DATA caches. So, please consider code cache usage
906 * when use the result.
907 *
908 * Because the cache inclusive/non-inclusive information isn't
909 * available, we just use the size of the per-CPU slice of LLC to make
910 * the result more predictable across architectures.
911 */
912static void update_per_cpu_data_slice_size_cpu(unsigned int cpu)
913{
914 struct cpu_cacheinfo *ci;
915 struct cacheinfo *llc;
916 unsigned int nr_shared;
917
918 if (!last_level_cache_is_valid(cpu))
919 return;
920
921 ci = ci_cacheinfo(cpu);
922 llc = per_cpu_cacheinfo_idx(cpu, cache_leaves(cpu) - 1);
923
924 if (llc->type != CACHE_TYPE_DATA && llc->type != CACHE_TYPE_UNIFIED)
925 return;
926
927 nr_shared = cpumask_weight(&llc->shared_cpu_map);
928 if (nr_shared)
929 ci->per_cpu_data_slice_size = llc->size / nr_shared;
930}
931
932static void update_per_cpu_data_slice_size(bool cpu_online, unsigned int cpu)
933{
934 unsigned int icpu;
935
936 for_each_online_cpu(icpu) {
937 if (!cpu_online && icpu == cpu)
938 continue;
939 update_per_cpu_data_slice_size_cpu(icpu);
940 }
941}
942
901static int cacheinfo_cpu_online(unsigned int cpu)943static int cacheinfo_cpu_online(unsigned int cpu)
902{944{
903 int rc = detect_cache_attributes(cpu);945 int rc = detect_cache_attributes(cpu);
@@ -906,7 +948,12 @@ static int cacheinfo_cpu_online(unsigned int cpu) · 2 unmodified lines
906 return rc;948 return rc;
907 rc = cache_add_dev(cpu);949 rc = cache_add_dev(cpu);
908 if (rc)950 if (rc)
909 free_cache_attributes(cpu);951 goto err;
952 update_per_cpu_data_slice_size(true, cpu);
953 setup_pcp_cacheinfo();
954 return 0;
955err:
956 free_cache_attributes(cpu);
910 return rc;957 return rc;
911}958}
912 959
@@ -916,6 +963,8 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu) · 3 unmodified lines
916 cpu_cache_sysfs_exit(cpu);963 cpu_cache_sysfs_exit(cpu);
917 964
918 free_cache_attributes(cpu);965 free_cache_attributes(cpu);
966 update_per_cpu_data_slice_size(false, cpu);
967 setup_pcp_cacheinfo();
919 return 0;968 return 0;
920}969}
921 970
drivers/dax/kmem.c+49−13
@@ -49,14 +49,52 @@ struct dax_kmem_data { · 48 unmodified lines
49 struct resource *res[];49 struct resource *res[];
50};50};
51 51
52static struct memory_dev_type *dax_slowmem_type;52static DEFINE_MUTEX(kmem_memory_type_lock);
53static LIST_HEAD(kmem_memory_types);
54
55static struct memory_dev_type *kmem_find_alloc_memory_type(int adist)
56{
57 bool found = false;
58 struct memory_dev_type *mtype;
59
60 mutex_lock(&kmem_memory_type_lock);
61 list_for_each_entry(mtype, &kmem_memory_types, list) {
62 if (mtype->adistance == adist) {
63 found = true;
64 break;
65 }
66 }
67 if (!found) {
68 mtype = alloc_memory_type(adist);
69 if (!IS_ERR(mtype))
70 list_add(&mtype->list, &kmem_memory_types);
71 }
72 mutex_unlock(&kmem_memory_type_lock);
73
74 return mtype;
75}
76
77static void kmem_put_memory_types(void)
78{
79 struct memory_dev_type *mtype, *mtn;
80
81 mutex_lock(&kmem_memory_type_lock);
82 list_for_each_entry_safe(mtype, mtn, &kmem_memory_types, list) {
83 list_del(&mtype->list);
84 put_memory_type(mtype);
85 }
86 mutex_unlock(&kmem_memory_type_lock);
87}
88
53static int dev_dax_kmem_probe(struct dev_dax *dev_dax)89static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
54{90{
55 struct device *dev = &dev_dax->dev;91 struct device *dev = &dev_dax->dev;
56 unsigned long total_len = 0;92 unsigned long total_len = 0;
57 struct dax_kmem_data *data;93 struct dax_kmem_data *data;
94 struct memory_dev_type *mtype;
58 int i, rc, mapped = 0;95 int i, rc, mapped = 0;
59 int numa_node;96 int numa_node;
97 int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
60 98
61 /*99 /*
62 * Ensure good NUMA information for the persistent memory.100 * Ensure good NUMA information for the persistent memory.
@@ -71,6 +109,11 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 8 unmodified lines
71 return -EINVAL;109 return -EINVAL;
72 }110 }
73 111
112 mt_calc_adistance(numa_node, &adist);
113 mtype = kmem_find_alloc_memory_type(adist);
114 if (IS_ERR(mtype))
115 return PTR_ERR(mtype);
116
74 for (i = 0; i < dev_dax->nr_range; i++) {117 for (i = 0; i < dev_dax->nr_range; i++) {
75 struct range range;118 struct range range;
76 119
@@ -88,7 +131,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 11 unmodified lines
88 return -EINVAL;131 return -EINVAL;
89 }132 }
90 133
91 init_node_memory_type(numa_node, dax_slowmem_type);134 init_node_memory_type(numa_node, mtype);
92 135
93 rc = -ENOMEM;136 rc = -ENOMEM;
94 data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL);137 data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL);
@@ -167,7 +210,7 @@ err_reg_mgid: · 72 unmodified lines
167err_res_name:210err_res_name:
168 kfree(data);211 kfree(data);
169err_dax_kmem_data:212err_dax_kmem_data:
170 clear_node_memory_type(numa_node, dax_slowmem_type);213 clear_node_memory_type(numa_node, mtype);
171 return rc;214 return rc;
172}215}
173 216
@@ -219,7 +262,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 45 unmodified lines
219 * for that. This implies this reference will be around262 * for that. This implies this reference will be around
220 * till next reboot.263 * till next reboot.
221 */264 */
222 clear_node_memory_type(node, dax_slowmem_type);265 clear_node_memory_type(node, NULL);
223 }266 }
224}267}
225#else268#else
@@ -251,12 +294,6 @@ static int __init dax_kmem_init(void) · 25 unmodified lines
251 if (!kmem_name)294 if (!kmem_name)
252 return -ENOMEM;295 return -ENOMEM;
253 296
254 dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE);
255 if (IS_ERR(dax_slowmem_type)) {
256 rc = PTR_ERR(dax_slowmem_type);
257 goto err_dax_slowmem_type;
258 }
259
260 rc = dax_driver_register(&device_dax_kmem_driver);297 rc = dax_driver_register(&device_dax_kmem_driver);
261 if (rc)298 if (rc)
262 goto error_dax_driver;299 goto error_dax_driver;
@@ -264,8 +301,7 @@ static int __init dax_kmem_init(void) · 1 unmodified line
264 return rc;301 return rc;
265 302
266error_dax_driver:303error_dax_driver:
267 put_memory_type(dax_slowmem_type);304 kmem_put_memory_types();
268err_dax_slowmem_type:
269 kfree_const(kmem_name);305 kfree_const(kmem_name);
270 return rc;306 return rc;
271}307}
@@ -275,7 +311,7 @@ static void __exit dax_kmem_exit(void) · 3 unmodified lines
275 dax_driver_unregister(&device_dax_kmem_driver);311 dax_driver_unregister(&device_dax_kmem_driver);
276 if (!any_hotremove_failed)312 if (!any_hotremove_failed)
277 kfree_const(kmem_name);313 kfree_const(kmem_name);
278 put_memory_type(dax_slowmem_type);314 kmem_put_memory_types();
279}315}
280 316
281MODULE_AUTHOR("Intel Corporation");317MODULE_AUTHOR("Intel Corporation");
drivers/firmware/efi/unaccepted_memory.c+20−0
@@ -3,6 +3,7 @@ · 2 unmodified lines
3#include <linux/efi.h>3#include <linux/efi.h>
4#include <linux/memblock.h>4#include <linux/memblock.h>
5#include <linux/spinlock.h>5#include <linux/spinlock.h>
6#include <linux/crash_dump.h>
6#include <asm/unaccepted_memory.h>7#include <asm/unaccepted_memory.h>
7 8
8/* Protects unaccepted memory bitmap and accepting_list */9/* Protects unaccepted memory bitmap and accepting_list */
@@ -201,3 +202,22 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · 192 unmodified lines
201 202
202 return ret;203 return ret;
203}204}
205
206#ifdef CONFIG_PROC_VMCORE
207static bool unaccepted_memory_vmcore_pfn_is_ram(struct vmcore_cb *cb,
208 unsigned long pfn)
209{
210 return !pfn_is_unaccepted_memory(pfn);
211}
212
213static struct vmcore_cb vmcore_cb = {
214 .pfn_is_ram = unaccepted_memory_vmcore_pfn_is_ram,
215};
216
217static int __init unaccepted_memory_init_kdump(void)
218{
219 register_vmcore_cb(&vmcore_cb);
220 return 0;
221}
222core_initcall(unaccepted_memory_init_kdump);
223#endif /* CONFIG_PROC_VMCORE */
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c+16−13
@@ -288,8 +288,7 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915) · 287 unmodified lines
288static unsigned long288static unsigned long
289i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)289i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
290{290{
291 struct drm_i915_private *i915 =291 struct drm_i915_private *i915 = shrinker->private_data;
292 container_of(shrinker, struct drm_i915_private, mm.shrinker);
293 unsigned long num_objects;292 unsigned long num_objects;
294 unsigned long count;293 unsigned long count;
295 294
@@ -306,8 +305,8 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 10 unmodified lines
306 if (num_objects) {305 if (num_objects) {
307 unsigned long avg = 2 * count / num_objects;306 unsigned long avg = 2 * count / num_objects;
308 307
309 i915->mm.shrinker.batch =308 i915->mm.shrinker->batch =
310 max((i915->mm.shrinker.batch + avg) >> 1,309 max((i915->mm.shrinker->batch + avg) >> 1,
311 128ul /* default SHRINK_BATCH */);310 128ul /* default SHRINK_BATCH */);
312 }311 }
313 312
@@ -317,8 +316,7 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 3 unmodified lines
317static unsigned long316static unsigned long
318i915_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)317i915_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
319{318{
320 struct drm_i915_private *i915 =319 struct drm_i915_private *i915 = shrinker->private_data;
321 container_of(shrinker, struct drm_i915_private, mm.shrinker);
322 unsigned long freed;320 unsigned long freed;
323 321
324 sc->nr_scanned = 0;322 sc->nr_scanned = 0;
@@ -430,12 +428,17 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr · 105 unmodified lines
430 428
431void i915_gem_driver_register__shrinker(struct drm_i915_private *i915)429void i915_gem_driver_register__shrinker(struct drm_i915_private *i915)
432{430{
433 i915->mm.shrinker.scan_objects = i915_gem_shrinker_scan;431 i915->mm.shrinker = shrinker_alloc(0, "drm-i915_gem");
434 i915->mm.shrinker.count_objects = i915_gem_shrinker_count;432 if (!i915->mm.shrinker) {
435 i915->mm.shrinker.seeks = DEFAULT_SEEKS;433 drm_WARN_ON(&i915->drm, 1);
436 i915->mm.shrinker.batch = 4096;434 } else {
437 drm_WARN_ON(&i915->drm, register_shrinker(&i915->mm.shrinker,435 i915->mm.shrinker->scan_objects = i915_gem_shrinker_scan;
438 "drm-i915_gem"));436 i915->mm.shrinker->count_objects = i915_gem_shrinker_count;
437 i915->mm.shrinker->batch = 4096;
438 i915->mm.shrinker->private_data = i915;
439
440 shrinker_register(i915->mm.shrinker);
441 }
439 442
440 i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom;443 i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom;
441 drm_WARN_ON(&i915->drm, register_oom_notifier(&i915->mm.oom_notifier));444 drm_WARN_ON(&i915->drm, register_oom_notifier(&i915->mm.oom_notifier));
@@ -451,7 +454,7 @@ void i915_gem_driver_unregister__shrinker(struct drm_i915_private *i915) · 9 unmodified lines
451 unregister_vmap_purge_notifier(&i915->mm.vmap_notifier));454 unregister_vmap_purge_notifier(&i915->mm.vmap_notifier));
452 drm_WARN_ON(&i915->drm,455 drm_WARN_ON(&i915->drm,
453 unregister_oom_notifier(&i915->mm.oom_notifier));456 unregister_oom_notifier(&i915->mm.oom_notifier));
454 unregister_shrinker(&i915->mm.shrinker);457 shrinker_free(i915->mm.shrinker);
455}458}
456 459
457void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915,460void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915,
drivers/gpu/drm/i915/i915_drv.h+1−1
@@ -163,7 +163,7 @@ struct i915_gem_mm { · 162 unmodified lines
163 163
164 struct notifier_block oom_notifier;164 struct notifier_block oom_notifier;
165 struct notifier_block vmap_notifier;165 struct notifier_block vmap_notifier;
166 struct shrinker shrinker;166 struct shrinker *shrinker;
167 167
168#ifdef CONFIG_MMU_NOTIFIER168#ifdef CONFIG_MMU_NOTIFIER
169 /**169 /**
drivers/gpu/drm/msm/msm_drv.c+3−1
@@ -265,7 +265,9 @@ static int msm_drm_init(struct device *dev, const struct drm_driver *drv) · 264 unmodified lines
265 if (ret)265 if (ret)
266 goto err_deinit_vram;266 goto err_deinit_vram;
267 267
268 msm_gem_shrinker_init(ddev);268 ret = msm_gem_shrinker_init(ddev);
269 if (ret)
270 goto err_msm_uninit;
269 271
270 if (priv->kms_init) {272 if (priv->kms_init) {
271 ret = msm_drm_kms_init(dev, drv);273 ret = msm_drm_kms_init(dev, drv);
drivers/gpu/drm/msm/msm_drv.h+2−2
@@ -218,7 +218,7 @@ struct msm_drm_private { · 217 unmodified lines
218 } vram;218 } vram;
219 219
220 struct notifier_block vmap_notifier;220 struct notifier_block vmap_notifier;
221 struct shrinker shrinker;221 struct shrinker *shrinker;
222 222
223 struct drm_atomic_state *pm_state;223 struct drm_atomic_state *pm_state;
224 224
@@ -280,7 +280,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data, · 55 unmodified lines
280unsigned long msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan);280unsigned long msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan);
281#endif281#endif
282 282
283void msm_gem_shrinker_init(struct drm_device *dev);283int msm_gem_shrinker_init(struct drm_device *dev);
284void msm_gem_shrinker_cleanup(struct drm_device *dev);284void msm_gem_shrinker_cleanup(struct drm_device *dev);
285 285
286struct sg_table *msm_gem_prime_get_sg_table(struct drm_gem_object *obj);286struct sg_table *msm_gem_prime_get_sg_table(struct drm_gem_object *obj);
drivers/gpu/drm/msm/msm_gem_shrinker.c+20−13
@@ -34,8 +34,7 @@ static bool can_block(struct shrink_control *sc) · 33 unmodified lines
34static unsigned long34static unsigned long
35msm_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)35msm_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
36{36{
37 struct msm_drm_private *priv =37 struct msm_drm_private *priv = shrinker->private_data;
38 container_of(shrinker, struct msm_drm_private, shrinker);
39 unsigned count = priv->lru.dontneed.count;38 unsigned count = priv->lru.dontneed.count;
40 39
41 if (can_swap())40 if (can_swap())
@@ -100,8 +99,7 @@ active_evict(struct drm_gem_object *obj) · 58 unmodified lines
100static unsigned long99static unsigned long
101msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)100msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
102{101{
103 struct msm_drm_private *priv =102 struct msm_drm_private *priv = shrinker->private_data;
104 container_of(shrinker, struct msm_drm_private, shrinker);
105 struct {103 struct {
106 struct drm_gem_lru *lru;104 struct drm_gem_lru *lru;
107 bool (*shrink)(struct drm_gem_object *obj);105 bool (*shrink)(struct drm_gem_object *obj);
@@ -148,10 +146,11 @@ msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan) · 40 unmodified lines
148 struct shrink_control sc = {146 struct shrink_control sc = {
149 .nr_to_scan = nr_to_scan,147 .nr_to_scan = nr_to_scan,
150 };148 };
151 int ret;149 unsigned long ret = SHRINK_STOP;
152 150
153 fs_reclaim_acquire(GFP_KERNEL);151 fs_reclaim_acquire(GFP_KERNEL);
154 ret = msm_gem_shrinker_scan(&priv->shrinker, &sc);152 if (priv->shrinker)
153 ret = msm_gem_shrinker_scan(priv->shrinker, &sc);
155 fs_reclaim_release(GFP_KERNEL);154 fs_reclaim_release(GFP_KERNEL);
156 155
157 return ret;156 return ret;
@@ -210,16 +209,24 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr) · 52 unmodified lines
210 *209 *
211 * This function registers and sets up the msm shrinker.210 * This function registers and sets up the msm shrinker.
212 */211 */
213void msm_gem_shrinker_init(struct drm_device *dev)212int msm_gem_shrinker_init(struct drm_device *dev)
214{213{
215 struct msm_drm_private *priv = dev->dev_private;214 struct msm_drm_private *priv = dev->dev_private;
216 priv->shrinker.count_objects = msm_gem_shrinker_count;215
217 priv->shrinker.scan_objects = msm_gem_shrinker_scan;216 priv->shrinker = shrinker_alloc(0, "drm-msm_gem");
218 priv->shrinker.seeks = DEFAULT_SEEKS;217 if (!priv->shrinker)
219 WARN_ON(register_shrinker(&priv->shrinker, "drm-msm_gem"));218 return -ENOMEM;
219
220 priv->shrinker->count_objects = msm_gem_shrinker_count;
221 priv->shrinker->scan_objects = msm_gem_shrinker_scan;
222 priv->shrinker->private_data = priv;
223
224 shrinker_register(priv->shrinker);
220 225
221 priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap;226 priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap;
222 WARN_ON(register_vmap_purge_notifier(&priv->vmap_notifier));227 WARN_ON(register_vmap_purge_notifier(&priv->vmap_notifier));
228
229 return 0;
223}230}
224 231
225/**232/**
@@ -232,8 +239,8 @@ void msm_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines
232{239{
233 struct msm_drm_private *priv = dev->dev_private;240 struct msm_drm_private *priv = dev->dev_private;
234 241
235 if (priv->shrinker.nr_deferred) {242 if (priv->shrinker) {
236 WARN_ON(unregister_vmap_purge_notifier(&priv->vmap_notifier));243 WARN_ON(unregister_vmap_purge_notifier(&priv->vmap_notifier));
237 unregister_shrinker(&priv->shrinker);244 shrinker_free(priv->shrinker);
238 }245 }
239}246}
drivers/gpu/drm/panfrost/panfrost_device.h+1−1
@@ -119,7 +119,7 @@ struct panfrost_device { · 118 unmodified lines
119 119
120 struct mutex shrinker_lock;120 struct mutex shrinker_lock;
121 struct list_head shrinker_list;121 struct list_head shrinker_list;
122 struct shrinker shrinker;122 struct shrinker *shrinker;
123 123
124 struct panfrost_devfreq pfdevfreq;124 struct panfrost_devfreq pfdevfreq;
125 125
drivers/gpu/drm/panfrost/panfrost_drv.c+5−1
@@ -659,10 +659,14 @@ static int panfrost_probe(struct platform_device *pdev) · 658 unmodified lines
659 if (err < 0)659 if (err < 0)
660 goto err_out1;660 goto err_out1;
661 661
662 panfrost_gem_shrinker_init(ddev);662 err = panfrost_gem_shrinker_init(ddev);
663 if (err)
664 goto err_out2;
663 665
664 return 0;666 return 0;
665 667
668err_out2:
669 drm_dev_unregister(ddev);
666err_out1:670err_out1:
667 pm_runtime_disable(pfdev->dev);671 pm_runtime_disable(pfdev->dev);
668 panfrost_device_fini(pfdev);672 panfrost_device_fini(pfdev);
drivers/gpu/drm/panfrost/panfrost_gem.h+1−1
@@ -86,7 +86,7 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo, · 85 unmodified lines
86void panfrost_gem_mapping_put(struct panfrost_gem_mapping *mapping);86void panfrost_gem_mapping_put(struct panfrost_gem_mapping *mapping);
87void panfrost_gem_teardown_mappings_locked(struct panfrost_gem_object *bo);87void panfrost_gem_teardown_mappings_locked(struct panfrost_gem_object *bo);
88 88
89void panfrost_gem_shrinker_init(struct drm_device *dev);89int panfrost_gem_shrinker_init(struct drm_device *dev);
90void panfrost_gem_shrinker_cleanup(struct drm_device *dev);90void panfrost_gem_shrinker_cleanup(struct drm_device *dev);
91 91
92#endif /* __PANFROST_GEM_H__ */92#endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c+17−12
@@ -18,8 +18,7 @@ · 17 unmodified lines
18static unsigned long18static unsigned long
19panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)19panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)
20{20{
21 struct panfrost_device *pfdev =21 struct panfrost_device *pfdev = shrinker->private_data;
22 container_of(shrinker, struct panfrost_device, shrinker);
23 struct drm_gem_shmem_object *shmem;22 struct drm_gem_shmem_object *shmem;
24 unsigned long count = 0;23 unsigned long count = 0;
25 24
@@ -65,8 +64,7 @@ unlock_mappings: · 39 unmodified lines
65static unsigned long64static unsigned long
66panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)65panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)
67{66{
68 struct panfrost_device *pfdev =67 struct panfrost_device *pfdev = shrinker->private_data;
69 container_of(shrinker, struct panfrost_device, shrinker);
70 struct drm_gem_shmem_object *shmem, *tmp;68 struct drm_gem_shmem_object *shmem, *tmp;
71 unsigned long freed = 0;69 unsigned long freed = 0;
72 70
@@ -97,13 +95,21 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) · 24 unmodified lines
97 *95 *
98 * This function registers and sets up the panfrost shrinker.96 * This function registers and sets up the panfrost shrinker.
99 */97 */
100void panfrost_gem_shrinker_init(struct drm_device *dev)98int panfrost_gem_shrinker_init(struct drm_device *dev)
101{99{
102 struct panfrost_device *pfdev = dev->dev_private;100 struct panfrost_device *pfdev = dev->dev_private;
103 pfdev->shrinker.count_objects = panfrost_gem_shrinker_count;101
104 pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan;102 pfdev->shrinker = shrinker_alloc(0, "drm-panfrost");
105 pfdev->shrinker.seeks = DEFAULT_SEEKS;103 if (!pfdev->shrinker)
106 WARN_ON(register_shrinker(&pfdev->shrinker, "drm-panfrost"));104 return -ENOMEM;
105
106 pfdev->shrinker->count_objects = panfrost_gem_shrinker_count;
107 pfdev->shrinker->scan_objects = panfrost_gem_shrinker_scan;
108 pfdev->shrinker->private_data = pfdev;
109
110 shrinker_register(pfdev->shrinker);
111
112 return 0;
107}113}
108 114
109/**115/**
@@ -116,7 +122,6 @@ void panfrost_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines
116{122{
117 struct panfrost_device *pfdev = dev->dev_private;123 struct panfrost_device *pfdev = dev->dev_private;
118 124
119 if (pfdev->shrinker.nr_deferred) {125 if (pfdev->shrinker)
120 unregister_shrinker(&pfdev->shrinker);126 shrinker_free(pfdev->shrinker);
121 }
122}127}
drivers/gpu/drm/ttm/ttm_pool.c+31−9
@@ -73,7 +73,8 @@ static struct ttm_pool_type global_dma32_uncached[MAX_ORDER + 1]; · 72 unmodified lines
73 73
74static spinlock_t shrinker_lock;74static spinlock_t shrinker_lock;
75static struct list_head shrinker_list;75static struct list_head shrinker_list;
76static struct shrinker mm_shrinker;76static struct shrinker *mm_shrinker;
77static DECLARE_RWSEM(pool_shrink_rwsem);
77 78
78/* Allocate pages of size 1 << order with the given gfp_flags */79/* Allocate pages of size 1 << order with the given gfp_flags */
79static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,80static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
@@ -317,6 +318,7 @@ static unsigned int ttm_pool_shrink(void) · 237 unmodified lines
317 unsigned int num_pages;318 unsigned int num_pages;
318 struct page *p;319 struct page *p;
319 320
321 down_read(&pool_shrink_rwsem);
320 spin_lock(&shrinker_lock);322 spin_lock(&shrinker_lock);
321 pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list);323 pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list);
322 list_move_tail(&pt->shrinker_list, &shrinker_list);324 list_move_tail(&pt->shrinker_list, &shrinker_list);
@@ -329,6 +331,7 @@ static unsigned int ttm_pool_shrink(void) · 6 unmodified lines
329 } else {331 } else {
330 num_pages = 0;332 num_pages = 0;
331 }333 }
334 up_read(&pool_shrink_rwsem);
332 335
333 return num_pages;336 return num_pages;
334}337}
@@ -572,6 +575,18 @@ void ttm_pool_init(struct ttm_pool *pool, struct device *dev, · 237 unmodified lines
572}575}
573EXPORT_SYMBOL(ttm_pool_init);576EXPORT_SYMBOL(ttm_pool_init);
574 577
578/**
579 * ttm_pool_synchronize_shrinkers - Wait for all running shrinkers to complete.
580 *
581 * This is useful to guarantee that all shrinker invocations have seen an
582 * update, before freeing memory, similar to rcu.
583 */
584static void ttm_pool_synchronize_shrinkers(void)
585{
586 down_write(&pool_shrink_rwsem);
587 up_write(&pool_shrink_rwsem);
588}
589
575/**590/**
576 * ttm_pool_fini - Cleanup a pool591 * ttm_pool_fini - Cleanup a pool
577 *592 *
@@ -593,7 +608,7 @@ void ttm_pool_fini(struct ttm_pool *pool) · 15 unmodified lines
593 /* We removed the pool types from the LRU, but we need to also make sure608 /* We removed the pool types from the LRU, but we need to also make sure
594 * that no shrinker is concurrently freeing pages from the pool.609 * that no shrinker is concurrently freeing pages from the pool.
595 */610 */
596 synchronize_shrinkers();611 ttm_pool_synchronize_shrinkers();
597}612}
598EXPORT_SYMBOL(ttm_pool_fini);613EXPORT_SYMBOL(ttm_pool_fini);
599 614
@@ -734,8 +749,8 @@ static int ttm_pool_debugfs_shrink_show(struct seq_file *m, void *data) · 134 unmodified lines
734 struct shrink_control sc = { .gfp_mask = GFP_NOFS };749 struct shrink_control sc = { .gfp_mask = GFP_NOFS };
735 750
736 fs_reclaim_acquire(GFP_KERNEL);751 fs_reclaim_acquire(GFP_KERNEL);
737 seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(&mm_shrinker, &sc),752 seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(mm_shrinker, &sc),
738 ttm_pool_shrinker_scan(&mm_shrinker, &sc));753 ttm_pool_shrinker_scan(mm_shrinker, &sc));
739 fs_reclaim_release(GFP_KERNEL);754 fs_reclaim_release(GFP_KERNEL);
740 755
741 return 0;756 return 0;
@@ -779,10 +794,17 @@ int ttm_pool_mgr_init(unsigned long num_pages) · 37 unmodified lines
779 &ttm_pool_debugfs_shrink_fops);794 &ttm_pool_debugfs_shrink_fops);
780#endif795#endif
781 796
782 mm_shrinker.count_objects = ttm_pool_shrinker_count;797 mm_shrinker = shrinker_alloc(0, "drm-ttm_pool");
783 mm_shrinker.scan_objects = ttm_pool_shrinker_scan;798 if (!mm_shrinker)
784 mm_shrinker.seeks = 1;799 return -ENOMEM;
785 return register_shrinker(&mm_shrinker, "drm-ttm_pool");800
801 mm_shrinker->count_objects = ttm_pool_shrinker_count;
802 mm_shrinker->scan_objects = ttm_pool_shrinker_scan;
803 mm_shrinker->seeks = 1;
804
805 shrinker_register(mm_shrinker);
806
807 return 0;
786}808}
787 809
788/**810/**
@@ -802,6 +824,6 @@ void ttm_pool_mgr_fini(void) · 13 unmodified lines
802 ttm_pool_type_fini(&global_dma32_uncached[i]);824 ttm_pool_type_fini(&global_dma32_uncached[i]);
803 }825 }
804 826
805 unregister_shrinker(&mm_shrinker);827 shrinker_free(mm_shrinker);
806 WARN_ON(!list_empty(&shrinker_list));828 WARN_ON(!list_empty(&shrinker_list));
807}829}
drivers/md/bcache/bcache.h+1−1
@@ -543,7 +543,7 @@ struct cache_set { · 542 unmodified lines
543 struct bio_set bio_split;543 struct bio_set bio_split;
544 544
545 /* For the btree cache */545 /* For the btree cache */
546 struct shrinker shrink;546 struct shrinker *shrink;
547 547
548 /* For the btree cache and anything allocation related */548 /* For the btree cache and anything allocation related */
549 struct mutex bucket_lock;549 struct mutex bucket_lock;
drivers/md/bcache/btree.c+16−11
@@ -667,7 +667,7 @@ out_unlock: · 666 unmodified lines
667static unsigned long bch_mca_scan(struct shrinker *shrink,667static unsigned long bch_mca_scan(struct shrinker *shrink,
668 struct shrink_control *sc)668 struct shrink_control *sc)
669{669{
670 struct cache_set *c = container_of(shrink, struct cache_set, shrink);670 struct cache_set *c = shrink->private_data;
671 struct btree *b, *t;671 struct btree *b, *t;
672 unsigned long i, nr = sc->nr_to_scan;672 unsigned long i, nr = sc->nr_to_scan;
673 unsigned long freed = 0;673 unsigned long freed = 0;
@@ -734,7 +734,7 @@ out: · 60 unmodified lines
734static unsigned long bch_mca_count(struct shrinker *shrink,734static unsigned long bch_mca_count(struct shrinker *shrink,
735 struct shrink_control *sc)735 struct shrink_control *sc)
736{736{
737 struct cache_set *c = container_of(shrink, struct cache_set, shrink);737 struct cache_set *c = shrink->private_data;
738 738
739 if (c->shrinker_disabled)739 if (c->shrinker_disabled)
740 return 0;740 return 0;
@@ -752,8 +752,8 @@ void bch_btree_cache_free(struct cache_set *c) · 11 unmodified lines
752 752
753 closure_init_stack(&cl);753 closure_init_stack(&cl);
754 754
755 if (c->shrink.list.next)755 if (c->shrink)
756 unregister_shrinker(&c->shrink);756 shrinker_free(c->shrink);
757 757
758 mutex_lock(&c->bucket_lock);758 mutex_lock(&c->bucket_lock);
759 759
@@ -828,14 +828,19 @@ int bch_btree_cache_alloc(struct cache_set *c) · 68 unmodified lines
828 c->verify_data = NULL;828 c->verify_data = NULL;
829#endif829#endif
830 830
831 c->shrink.count_objects = bch_mca_count;831 c->shrink = shrinker_alloc(0, "md-bcache:%pU", c->set_uuid);
832 c->shrink.scan_objects = bch_mca_scan;832 if (!c->shrink) {
833 c->shrink.seeks = 4;833 pr_warn("bcache: %s: could not allocate shrinker\n", __func__);
834 c->shrink.batch = c->btree_pages * 2;834 return 0;
835 }
836
837 c->shrink->count_objects = bch_mca_count;
838 c->shrink->scan_objects = bch_mca_scan;
839 c->shrink->seeks = 4;
840 c->shrink->batch = c->btree_pages * 2;
841 c->shrink->private_data = c;
835 842
836 if (register_shrinker(&c->shrink, "md-bcache:%pU", c->set_uuid))843 shrinker_register(c->shrink);
837 pr_warn("bcache: %s: could not register shrinker\n",
838 __func__);
839 844
840 return 0;845 return 0;
841}846}
drivers/md/bcache/sysfs.c+2−1
@@ -866,7 +866,8 @@ STORE(__bch_cache_set) · 865 unmodified lines
866 866
867 sc.gfp_mask = GFP_KERNEL;867 sc.gfp_mask = GFP_KERNEL;
868 sc.nr_to_scan = strtoul_or_return(buf);868 sc.nr_to_scan = strtoul_or_return(buf);
869 c->shrink.scan_objects(&c->shrink, &sc);869 if (c->shrink)
870 c->shrink->scan_objects(c->shrink, &sc);
870 }871 }
871 872
872 sysfs_strtoul_clamp(congested_read_threshold_us,873 sysfs_strtoul_clamp(congested_read_threshold_us,
drivers/md/dm-bufio.c+17−11
@@ -963,7 +963,7 @@ struct dm_bufio_client { · 962 unmodified lines
963 963
964 sector_t start;964 sector_t start;
965 965
966 struct shrinker shrinker;966 struct shrinker *shrinker;
967 struct work_struct shrink_work;967 struct work_struct shrink_work;
968 atomic_long_t need_shrink;968 atomic_long_t need_shrink;
969 969
@@ -2368,7 +2368,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 1398 unmodified lines
2368{2368{
2369 struct dm_bufio_client *c;2369 struct dm_bufio_client *c;
2370 2370
2371 c = container_of(shrink, struct dm_bufio_client, shrinker);2371 c = shrink->private_data;
2372 atomic_long_add(sc->nr_to_scan, &c->need_shrink);2372 atomic_long_add(sc->nr_to_scan, &c->need_shrink);
2373 queue_work(dm_bufio_wq, &c->shrink_work);2373 queue_work(dm_bufio_wq, &c->shrink_work);
2374 2374
@@ -2377,7 +2377,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 2 unmodified lines
2377 2377
2378static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc)2378static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
2379{2379{
2380 struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker);2380 struct dm_bufio_client *c = shrink->private_data;
2381 unsigned long count = cache_total(&c->cache);2381 unsigned long count = cache_total(&c->cache);
2382 unsigned long retain_target = get_retain_buffers(c);2382 unsigned long retain_target = get_retain_buffers(c);
2383 unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink);2383 unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink);
@@ -2490,14 +2490,20 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign · 106 unmodified lines
2490 INIT_WORK(&c->shrink_work, shrink_work);2490 INIT_WORK(&c->shrink_work, shrink_work);
2491 atomic_long_set(&c->need_shrink, 0);2491 atomic_long_set(&c->need_shrink, 0);
2492 2492
2493 c->shrinker.count_objects = dm_bufio_shrink_count;2493 c->shrinker = shrinker_alloc(0, "dm-bufio:(%u:%u)",
2494 c->shrinker.scan_objects = dm_bufio_shrink_scan;2494 MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));
2495 c->shrinker.seeks = 1;2495 if (!c->shrinker) {
2496 c->shrinker.batch = 0;2496 r = -ENOMEM;
2497 r = register_shrinker(&c->shrinker, "dm-bufio:(%u:%u)",
2498 MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));
2499 if (r)
2500 goto bad;2497 goto bad;
2498 }
2499
2500 c->shrinker->count_objects = dm_bufio_shrink_count;
2501 c->shrinker->scan_objects = dm_bufio_shrink_scan;
2502 c->shrinker->seeks = 1;
2503 c->shrinker->batch = 0;
2504 c->shrinker->private_data = c;
2505
2506 shrinker_register(c->shrinker);
2501 2507
2502 mutex_lock(&dm_bufio_clients_lock);2508 mutex_lock(&dm_bufio_clients_lock);
2503 dm_bufio_client_count++;2509 dm_bufio_client_count++;
@@ -2537,7 +2543,7 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c) · 33 unmodified lines
2537 2543
2538 drop_buffers(c);2544 drop_buffers(c);
2539 2545
2540 unregister_shrinker(&c->shrinker);2546 shrinker_free(c->shrinker);
2541 flush_work(&c->shrink_work);2547 flush_work(&c->shrink_work);
2542 2548
2543 mutex_lock(&dm_bufio_clients_lock);2549 mutex_lock(&dm_bufio_clients_lock);
drivers/md/dm-cache-metadata.c+1−1
@@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) · 1827 unmodified lines
1828 * Replacement block manager (new_bm) is created and old_bm destroyed outside of1828 * Replacement block manager (new_bm) is created and old_bm destroyed outside of
1829 * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of1829 * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of
1830 * shrinker associated with the block manager's bufio client vs cmd root_lock).1830 * shrinker associated with the block manager's bufio client vs cmd root_lock).
1831 * - must take shrinker_rwsem without holding cmd->root_lock1831 * - must take shrinker_mutex without holding cmd->root_lock
1832 */1832 */
1833 new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT,1833 new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT,
1834 CACHE_MAX_CONCURRENT_LOCKS);1834 CACHE_MAX_CONCURRENT_LOCKS);
drivers/md/dm-zoned-metadata.c+16−12
@@ -187,7 +187,7 @@ struct dmz_metadata { · 186 unmodified lines
187 struct rb_root mblk_rbtree;187 struct rb_root mblk_rbtree;
188 struct list_head mblk_lru_list;188 struct list_head mblk_lru_list;
189 struct list_head mblk_dirty_list;189 struct list_head mblk_dirty_list;
190 struct shrinker mblk_shrinker;190 struct shrinker *mblk_shrinker;
191 191
192 /* Zone allocation management */192 /* Zone allocation management */
193 struct mutex map_lock;193 struct mutex map_lock;
@@ -615,7 +615,7 @@ static unsigned long dmz_shrink_mblock_cache(struct dmz_metadata *zmd, · 421 unmodified lines
615static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink,615static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink,
616 struct shrink_control *sc)616 struct shrink_control *sc)
617{617{
618 struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker);618 struct dmz_metadata *zmd = shrink->private_data;
619 619
620 return atomic_read(&zmd->nr_mblks);620 return atomic_read(&zmd->nr_mblks);
621}621}
@@ -626,7 +626,7 @@ static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, · 4 unmodified lines
626static unsigned long dmz_mblock_shrinker_scan(struct shrinker *shrink,626static unsigned long dmz_mblock_shrinker_scan(struct shrinker *shrink,
627 struct shrink_control *sc)627 struct shrink_control *sc)
628{628{
629 struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker);629 struct dmz_metadata *zmd = shrink->private_data;
630 unsigned long count;630 unsigned long count;
631 631
632 spin_lock(&zmd->mblk_lock);632 spin_lock(&zmd->mblk_lock);
@@ -2936,19 +2936,23 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, · 2303 unmodified lines
2936 */2936 */
2937 zmd->min_nr_mblks = 2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks * 16;2937 zmd->min_nr_mblks = 2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks * 16;
2938 zmd->max_nr_mblks = zmd->min_nr_mblks + 512;2938 zmd->max_nr_mblks = zmd->min_nr_mblks + 512;
2939 zmd->mblk_shrinker.count_objects = dmz_mblock_shrinker_count;
2940 zmd->mblk_shrinker.scan_objects = dmz_mblock_shrinker_scan;
2941 zmd->mblk_shrinker.seeks = DEFAULT_SEEKS;
2942 2939
2943 /* Metadata cache shrinker */2940 /* Metadata cache shrinker */
2944 ret = register_shrinker(&zmd->mblk_shrinker, "dm-zoned-meta:(%u:%u)",2941 zmd->mblk_shrinker = shrinker_alloc(0, "dm-zoned-meta:(%u:%u)",
2945 MAJOR(dev->bdev->bd_dev),2942 MAJOR(dev->bdev->bd_dev),
2946 MINOR(dev->bdev->bd_dev));2943 MINOR(dev->bdev->bd_dev));
2947 if (ret) {2944 if (!zmd->mblk_shrinker) {
2948 dmz_zmd_err(zmd, "Register metadata cache shrinker failed");2945 ret = -ENOMEM;
2946 dmz_zmd_err(zmd, "Allocate metadata cache shrinker failed");
2949 goto err;2947 goto err;
2950 }2948 }
2951 2949
2950 zmd->mblk_shrinker->count_objects = dmz_mblock_shrinker_count;
2951 zmd->mblk_shrinker->scan_objects = dmz_mblock_shrinker_scan;
2952 zmd->mblk_shrinker->private_data = zmd;
2953
2954 shrinker_register(zmd->mblk_shrinker);
2955
2952 dmz_zmd_info(zmd, "DM-Zoned metadata version %d", zmd->sb_version);2956 dmz_zmd_info(zmd, "DM-Zoned metadata version %d", zmd->sb_version);
2953 for (i = 0; i < zmd->nr_devs; i++)2957 for (i = 0; i < zmd->nr_devs; i++)
2954 dmz_print_dev(zmd, i);2958 dmz_print_dev(zmd, i);
@@ -2995,7 +2999,7 @@ err: · 40 unmodified lines
2995 */2999 */
2996void dmz_dtr_metadata(struct dmz_metadata *zmd)3000void dmz_dtr_metadata(struct dmz_metadata *zmd)
2997{3001{
2998 unregister_shrinker(&zmd->mblk_shrinker);3002 shrinker_free(zmd->mblk_shrinker);
2999 dmz_cleanup_metadata(zmd);3003 dmz_cleanup_metadata(zmd);
3000 kfree(zmd);3004 kfree(zmd);
3001}3005}
drivers/md/raid5.c+15−11
@@ -7378,7 +7378,7 @@ static void free_conf(struct r5conf *conf) · 7377 unmodified lines
7378 7378
7379 log_exit(conf);7379 log_exit(conf);
7380 7380
7381 unregister_shrinker(&conf->shrinker);7381 shrinker_free(conf->shrinker);
7382 free_thread_groups(conf);7382 free_thread_groups(conf);
7383 shrink_stripes(conf);7383 shrink_stripes(conf);
7384 raid5_free_percpu(conf);7384 raid5_free_percpu(conf);
@@ -7426,7 +7426,7 @@ static int raid5_alloc_percpu(struct r5conf *conf) · 41 unmodified lines
7426static unsigned long raid5_cache_scan(struct shrinker *shrink,7426static unsigned long raid5_cache_scan(struct shrinker *shrink,
7427 struct shrink_control *sc)7427 struct shrink_control *sc)
7428{7428{
7429 struct r5conf *conf = container_of(shrink, struct r5conf, shrinker);7429 struct r5conf *conf = shrink->private_data;
7430 unsigned long ret = SHRINK_STOP;7430 unsigned long ret = SHRINK_STOP;
7431 7431
7432 if (mutex_trylock(&conf->cache_size_mutex)) {7432 if (mutex_trylock(&conf->cache_size_mutex)) {
@@ -7447,7 +7447,7 @@ static unsigned long raid5_cache_scan(struct shrinker *shrink, · 14 unmodified lines
7447static unsigned long raid5_cache_count(struct shrinker *shrink,7447static unsigned long raid5_cache_count(struct shrinker *shrink,
7448 struct shrink_control *sc)7448 struct shrink_control *sc)
7449{7449{
7450 struct r5conf *conf = container_of(shrink, struct r5conf, shrinker);7450 struct r5conf *conf = shrink->private_data;
7451 7451
7452 if (conf->max_nr_stripes < conf->min_nr_stripes)7452 if (conf->max_nr_stripes < conf->min_nr_stripes)
7453 /* unlikely, but not impossible */7453 /* unlikely, but not impossible */
@@ -7682,18 +7682,22 @@ static struct r5conf *setup_conf(struct mddev *mddev) · 228 unmodified lines
7682 * it reduces the queue depth and so can hurt throughput.7682 * it reduces the queue depth and so can hurt throughput.
7683 * So set it rather large, scaled by number of devices.7683 * So set it rather large, scaled by number of devices.
7684 */7684 */
7685 conf->shrinker.seeks = DEFAULT_SEEKS * conf->raid_disks * 4;7685 conf->shrinker = shrinker_alloc(0, "md-raid5:%s", mdname(mddev));
7686 conf->shrinker.scan_objects = raid5_cache_scan;7686 if (!conf->shrinker) {
7687 conf->shrinker.count_objects = raid5_cache_count;7687 ret = -ENOMEM;
7688 conf->shrinker.batch = 128;7688 pr_warn("md/raid:%s: couldn't allocate shrinker.\n",
7689 conf->shrinker.flags = 0;
7690 ret = register_shrinker(&conf->shrinker, "md-raid5:%s", mdname(mddev));
7691 if (ret) {
7692 pr_warn("md/raid:%s: couldn't register shrinker.\n",
7693 mdname(mddev));7689 mdname(mddev));
7694 goto abort;7690 goto abort;
7695 }7691 }
7696 7692
7693 conf->shrinker->seeks = DEFAULT_SEEKS * conf->raid_disks * 4;
7694 conf->shrinker->scan_objects = raid5_cache_scan;
7695 conf->shrinker->count_objects = raid5_cache_count;
7696 conf->shrinker->batch = 128;
7697 conf->shrinker->private_data = conf;
7698
7699 shrinker_register(conf->shrinker);
7700
7697 sprintf(pers_name, "raid%d", mddev->new_level);7701 sprintf(pers_name, "raid%d", mddev->new_level);
7698 rcu_assign_pointer(conf->thread,7702 rcu_assign_pointer(conf->thread,
7699 md_register_thread(raid5d, mddev, pers_name));7703 md_register_thread(raid5d, mddev, pers_name));
drivers/md/raid5.h+1−1
@@ -670,7 +670,7 @@ struct r5conf { · 669 unmodified lines
670 wait_queue_head_t wait_for_stripe;670 wait_queue_head_t wait_for_stripe;
671 wait_queue_head_t wait_for_overlap;671 wait_queue_head_t wait_for_overlap;
672 unsigned long cache_state;672 unsigned long cache_state;
673 struct shrinker shrinker;673 struct shrinker *shrinker;
674 int pool_size; /* number of disks in stripeheads in pool */674 int pool_size; /* number of disks in stripeheads in pool */
675 spinlock_t device_lock;675 spinlock_t device_lock;
676 struct disk_info *disks;676 struct disk_info *disks;
drivers/misc/vmw_balloon.c+12−26
@@ -380,16 +380,7 @@ struct vmballoon { · 379 unmodified lines
380 /**380 /**
381 * @shrinker: shrinker interface that is used to avoid over-inflation.381 * @shrinker: shrinker interface that is used to avoid over-inflation.
382 */382 */
383 struct shrinker shrinker;383 struct shrinker *shrinker;
384
385 /**
386 * @shrinker_registered: whether the shrinker was registered.
387 *
388 * The shrinker interface does not handle gracefully the removal of
389 * shrinker that was not registered before. This indication allows to
390 * simplify the unregistration process.
391 */
392 bool shrinker_registered;
393};384};
394 385
395static struct vmballoon balloon;386static struct vmballoon balloon;
@@ -1568,29 +1559,27 @@ static unsigned long vmballoon_shrinker_count(struct shrinker *shrinker, · 1172 unmodified lines
1568 1559
1569static void vmballoon_unregister_shrinker(struct vmballoon *b)1560static void vmballoon_unregister_shrinker(struct vmballoon *b)
1570{1561{
1571 if (b->shrinker_registered)1562 shrinker_free(b->shrinker);
1572 unregister_shrinker(&b->shrinker);1563 b->shrinker = NULL;
1573 b->shrinker_registered = false;
1574}1564}
1575 1565
1576static int vmballoon_register_shrinker(struct vmballoon *b)1566static int vmballoon_register_shrinker(struct vmballoon *b)
1577{1567{
1578 int r;
1579
1580 /* Do nothing if the shrinker is not enabled */1568 /* Do nothing if the shrinker is not enabled */
1581 if (!vmwballoon_shrinker_enable)1569 if (!vmwballoon_shrinker_enable)
1582 return 0;1570 return 0;
1583 1571
1584 b->shrinker.scan_objects = vmballoon_shrinker_scan;1572 b->shrinker = shrinker_alloc(0, "vmw-balloon");
1585 b->shrinker.count_objects = vmballoon_shrinker_count;1573 if (!b->shrinker)
1586 b->shrinker.seeks = DEFAULT_SEEKS;1574 return -ENOMEM;
1587 1575
1588 r = register_shrinker(&b->shrinker, "vmw-balloon");1576 b->shrinker->scan_objects = vmballoon_shrinker_scan;
1577 b->shrinker->count_objects = vmballoon_shrinker_count;
1578 b->shrinker->private_data = b;
1589 1579
1590 if (r == 0)1580 shrinker_register(b->shrinker);
1591 b->shrinker_registered = true;
1592 1581
1593 return r;1582 return 0;
1594}1583}
1595 1584
1596/*1585/*
@@ -1883,7 +1872,7 @@ static int __init vmballoon_init(void) · 286 unmodified lines
1883 1872
1884 error = vmballoon_register_shrinker(&balloon);1873 error = vmballoon_register_shrinker(&balloon);
1885 if (error)1874 if (error)
1886 goto fail;1875 return error;
1887 1876
1888 /*1877 /*
1889 * Initialization of compaction must be done after the call to1878 * Initialization of compaction must be done after the call to
@@ -1905,9 +1894,6 @@ static int __init vmballoon_init(void) · 15 unmodified lines
1905 vmballoon_debugfs_init(&balloon);1894 vmballoon_debugfs_init(&balloon);
1906 1895
1907 return 0;1896 return 0;
1908fail:
1909 vmballoon_unregister_shrinker(&balloon);
1910 return error;
1911}1897}
1912 1898
1913/*1899/*
drivers/virtio/virtio_balloon.c+14−10
@@ -111,7 +111,7 @@ struct virtio_balloon { · 110 unmodified lines
111 struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];111 struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
112 112
113 /* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */113 /* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */
114 struct shrinker shrinker;114 struct shrinker *shrinker;
115 115
116 /* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */116 /* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */
117 struct notifier_block oom_nb;117 struct notifier_block oom_nb;
@@ -820,8 +820,7 @@ static unsigned long shrink_free_pages(struct virtio_balloon *vb, · 702 unmodified lines
820static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker,820static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker,
821 struct shrink_control *sc)821 struct shrink_control *sc)
822{822{
823 struct virtio_balloon *vb = container_of(shrinker,823 struct virtio_balloon *vb = shrinker->private_data;
824 struct virtio_balloon, shrinker);
825 824
826 return shrink_free_pages(vb, sc->nr_to_scan);825 return shrink_free_pages(vb, sc->nr_to_scan);
827}826}
@@ -829,8 +828,7 @@ static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, · 1 unmodified line
829static unsigned long virtio_balloon_shrinker_count(struct shrinker *shrinker,828static unsigned long virtio_balloon_shrinker_count(struct shrinker *shrinker,
830 struct shrink_control *sc)829 struct shrink_control *sc)
831{830{
832 struct virtio_balloon *vb = container_of(shrinker,831 struct virtio_balloon *vb = shrinker->private_data;
833 struct virtio_balloon, shrinker);
834 832
835 return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES;833 return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES;
836}834}
@@ -851,16 +849,22 @@ static int virtio_balloon_oom_notify(struct notifier_block *nb, · 14 unmodified lines
851 849
852static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb)850static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb)
853{851{
854 unregister_shrinker(&vb->shrinker);852 shrinker_free(vb->shrinker);
855}853}
856 854
857static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)855static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)
858{856{
859 vb->shrinker.scan_objects = virtio_balloon_shrinker_scan;857 vb->shrinker = shrinker_alloc(0, "virtio-balloon");
860 vb->shrinker.count_objects = virtio_balloon_shrinker_count;858 if (!vb->shrinker)
861 vb->shrinker.seeks = DEFAULT_SEEKS;859 return -ENOMEM;
862 860
863 return register_shrinker(&vb->shrinker, "virtio-balloon");861 vb->shrinker->scan_objects = virtio_balloon_shrinker_scan;
862 vb->shrinker->count_objects = virtio_balloon_shrinker_count;
863 vb->shrinker->private_data = vb;
864
865 shrinker_register(vb->shrinker);
866
867 return 0;
864}868}
865 869
866static int virtballoon_probe(struct virtio_device *vdev)870static int virtballoon_probe(struct virtio_device *vdev)
drivers/xen/xenbus/xenbus_probe_backend.c+10−7
@@ -284,13 +284,9 @@ static unsigned long backend_shrink_memory_count(struct shrinker *shrinker, · 283 unmodified lines
284 return 0;284 return 0;
285}285}
286 286
287static struct shrinker backend_memory_shrinker = {
288 .count_objects = backend_shrink_memory_count,
289 .seeks = DEFAULT_SEEKS,
290};
291
292static int __init xenbus_probe_backend_init(void)287static int __init xenbus_probe_backend_init(void)
293{288{
289 struct shrinker *backend_memory_shrinker;
294 static struct notifier_block xenstore_notifier = {290 static struct notifier_block xenstore_notifier = {
295 .notifier_call = backend_probe_and_watch291 .notifier_call = backend_probe_and_watch
296 };292 };
@@ -305,8 +301,15 @@ static int __init xenbus_probe_backend_init(void) · 8 unmodified lines
305 301
306 register_xenstore_notifier(&xenstore_notifier);302 register_xenstore_notifier(&xenstore_notifier);
307 303
308 if (register_shrinker(&backend_memory_shrinker, "xen-backend"))304 backend_memory_shrinker = shrinker_alloc(0, "xen-backend");
309 pr_warn("shrinker registration failed\n");305 if (!backend_memory_shrinker) {
306 pr_warn("shrinker allocation failed\n");
307 return 0;
308 }
309
310 backend_memory_shrinker->count_objects = backend_shrink_memory_count;
311
312 shrinker_register(backend_memory_shrinker);
310 313
311 return 0;314 return 0;
312}315}
fs/bcachefs/btree_cache.c+12−10
@@ -285,8 +285,7 @@ static int btree_node_write_and_reclaim(struct bch_fs *c, struct btree *b) · 284 unmodified lines
285static unsigned long bch2_btree_cache_scan(struct shrinker *shrink,285static unsigned long bch2_btree_cache_scan(struct shrinker *shrink,
286 struct shrink_control *sc)286 struct shrink_control *sc)
287{287{
288 struct bch_fs *c = container_of(shrink, struct bch_fs,288 struct bch_fs *c = shrink->private_data;
289 btree_cache.shrink);
290 struct btree_cache *bc = &c->btree_cache;289 struct btree_cache *bc = &c->btree_cache;
291 struct btree *b, *t;290 struct btree *b, *t;
292 unsigned long nr = sc->nr_to_scan;291 unsigned long nr = sc->nr_to_scan;
@@ -384,8 +383,7 @@ out_nounlock: · 91 unmodified lines
384static unsigned long bch2_btree_cache_count(struct shrinker *shrink,383static unsigned long bch2_btree_cache_count(struct shrinker *shrink,
385 struct shrink_control *sc)384 struct shrink_control *sc)
386{385{
387 struct bch_fs *c = container_of(shrink, struct bch_fs,386 struct bch_fs *c = shrink->private_data;
388 btree_cache.shrink);
389 struct btree_cache *bc = &c->btree_cache;387 struct btree_cache *bc = &c->btree_cache;
390 388
391 if (bch2_btree_shrinker_disabled)389 if (bch2_btree_shrinker_disabled)
@@ -400,7 +398,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 8 unmodified lines
400 struct btree *b;398 struct btree *b;
401 unsigned i, flags;399 unsigned i, flags;
402 400
403 unregister_shrinker(&bc->shrink);401 shrinker_free(bc->shrink);
404 402
405 /* vfree() can allocate memory: */403 /* vfree() can allocate memory: */
406 flags = memalloc_nofs_save();404 flags = memalloc_nofs_save();
@@ -454,6 +452,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 47 unmodified lines
454int bch2_fs_btree_cache_init(struct bch_fs *c)452int bch2_fs_btree_cache_init(struct bch_fs *c)
455{453{
456 struct btree_cache *bc = &c->btree_cache;454 struct btree_cache *bc = &c->btree_cache;
455 struct shrinker *shrink;
457 unsigned i;456 unsigned i;
458 int ret = 0;457 int ret = 0;
459 458
@@ -473,12 +472,15 @@ int bch2_fs_btree_cache_init(struct bch_fs *c) · 13 unmodified lines
473 472
474 mutex_init(&c->verify_lock);473 mutex_init(&c->verify_lock);
475 474
476 bc->shrink.count_objects = bch2_btree_cache_count;475 shrink = shrinker_alloc(0, "%s/btree_cache", c->name);
477 bc->shrink.scan_objects = bch2_btree_cache_scan;476 if (!shrink)
478 bc->shrink.seeks = 4;
479 ret = register_shrinker(&bc->shrink, "%s/btree_cache", c->name);
480 if (ret)
481 goto err;477 goto err;
478 bc->shrink = shrink;
479 shrink->count_objects = bch2_btree_cache_count;
480 shrink->scan_objects = bch2_btree_cache_scan;
481 shrink->seeks = 4;
482 shrink->private_data = c;
483 shrinker_register(shrink);
482 484
483 return 0;485 return 0;
484err:486err:
fs/bcachefs/btree_key_cache.c+12−9
@@ -834,8 +834,7 @@ void bch2_btree_key_cache_drop(struct btree_trans *trans, · 833 unmodified lines
834static unsigned long bch2_btree_key_cache_scan(struct shrinker *shrink,834static unsigned long bch2_btree_key_cache_scan(struct shrinker *shrink,
835 struct shrink_control *sc)835 struct shrink_control *sc)
836{836{
837 struct bch_fs *c = container_of(shrink, struct bch_fs,837 struct bch_fs *c = shrink->private_data;
838 btree_key_cache.shrink);
839 struct btree_key_cache *bc = &c->btree_key_cache;838 struct btree_key_cache *bc = &c->btree_key_cache;
840 struct bucket_table *tbl;839 struct bucket_table *tbl;
841 struct bkey_cached *ck, *t;840 struct bkey_cached *ck, *t;
@@ -932,8 +931,7 @@ out: · 90 unmodified lines
932static unsigned long bch2_btree_key_cache_count(struct shrinker *shrink,931static unsigned long bch2_btree_key_cache_count(struct shrinker *shrink,
933 struct shrink_control *sc)932 struct shrink_control *sc)
934{933{
935 struct bch_fs *c = container_of(shrink, struct bch_fs,934 struct bch_fs *c = shrink->private_data;
936 btree_key_cache.shrink);
937 struct btree_key_cache *bc = &c->btree_key_cache;935 struct btree_key_cache *bc = &c->btree_key_cache;
938 long nr = atomic_long_read(&bc->nr_keys) -936 long nr = atomic_long_read(&bc->nr_keys) -
939 atomic_long_read(&bc->nr_dirty);937 atomic_long_read(&bc->nr_dirty);
@@ -953,7 +951,7 @@ void bch2_fs_btree_key_cache_exit(struct btree_key_cache *bc) · 13 unmodified lines
953 int cpu;951 int cpu;
954#endif952#endif
955 953
956 unregister_shrinker(&bc->shrink);954 shrinker_free(bc->shrink);
957 955
958 mutex_lock(&bc->lock);956 mutex_lock(&bc->lock);
959 957
@@ -1027,6 +1025,7 @@ void bch2_fs_btree_key_cache_init_early(struct btree_key_cache *c) · 67 unmodified lines
1027int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc)1025int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc)
1028{1026{
1029 struct bch_fs *c = container_of(bc, struct bch_fs, btree_key_cache);1027 struct bch_fs *c = container_of(bc, struct bch_fs, btree_key_cache);
1028 struct shrinker *shrink;
1030 1029
1031#ifdef __KERNEL__1030#ifdef __KERNEL__
1032 bc->pcpu_freed = alloc_percpu(struct btree_key_cache_freelist);1031 bc->pcpu_freed = alloc_percpu(struct btree_key_cache_freelist);
@@ -1039,11 +1038,15 @@ int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc) · 6 unmodified lines
1039 1038
1040 bc->table_init_done = true;1039 bc->table_init_done = true;
1041 1040
1042 bc->shrink.seeks = 0;1041 shrink = shrinker_alloc(0, "%s/btree_key_cache", c->name);
1043 bc->shrink.count_objects = bch2_btree_key_cache_count;1042 if (!shrink)
1044 bc->shrink.scan_objects = bch2_btree_key_cache_scan;
1045 if (register_shrinker(&bc->shrink, "%s/btree_key_cache", c->name))
1046 return -BCH_ERR_ENOMEM_fs_btree_cache_init;1043 return -BCH_ERR_ENOMEM_fs_btree_cache_init;
1044 bc->shrink = shrink;
1045 shrink->seeks = 0;
1046 shrink->count_objects = bch2_btree_key_cache_count;
1047 shrink->scan_objects = bch2_btree_key_cache_scan;
1048 shrink->private_data = c;
1049 shrinker_register(shrink);
1047 return 0;1050 return 0;
1048}1051}
1049 1052
fs/bcachefs/btree_types.h+2−2
@@ -163,7 +163,7 @@ struct btree_cache { · 162 unmodified lines
163 unsigned used;163 unsigned used;
164 unsigned reserve;164 unsigned reserve;
165 atomic_t dirty;165 atomic_t dirty;
166 struct shrinker shrink;166 struct shrinker *shrink;
167 167
168 /*168 /*
169 * If we need to allocate memory for a new btree node and that169 * If we need to allocate memory for a new btree node and that
@@ -321,7 +321,7 @@ struct btree_key_cache { · 151 unmodified lines
321 bool table_init_done;321 bool table_init_done;
322 struct list_head freed_pcpu;322 struct list_head freed_pcpu;
323 struct list_head freed_nonpcpu;323 struct list_head freed_nonpcpu;
324 struct shrinker shrink;324 struct shrinker *shrink;
325 unsigned shrink_iter;325 unsigned shrink_iter;
326 struct btree_key_cache_freelist __percpu *pcpu_freed;326 struct btree_key_cache_freelist __percpu *pcpu_freed;
327 327
fs/bcachefs/fs.c+1−1
@@ -1904,7 +1904,7 @@ got_sb: · 1903 unmodified lines
1904 sb->s_flags |= SB_POSIXACL;1904 sb->s_flags |= SB_POSIXACL;
1905#endif1905#endif
1906 1906
1907 sb->s_shrink.seeks = 0;1907 sb->s_shrink->seeks = 0;
1908 1908
1909 vinode = bch2_vfs_inode_get(c, BCACHEFS_ROOT_SUBVOL_INUM);1909 vinode = bch2_vfs_inode_get(c, BCACHEFS_ROOT_SUBVOL_INUM);
1910 ret = PTR_ERR_OR_ZERO(vinode);1910 ret = PTR_ERR_OR_ZERO(vinode);
fs/bcachefs/sysfs.c+1−1
@@ -494,7 +494,7 @@ STORE(bch2_fs) · 493 unmodified lines
494 494
495 sc.gfp_mask = GFP_KERNEL;495 sc.gfp_mask = GFP_KERNEL;
496 sc.nr_to_scan = strtoul_or_return(buf);496 sc.nr_to_scan = strtoul_or_return(buf);
497 c->btree_cache.shrink.scan_objects(&c->btree_cache.shrink, &sc);497 c->btree_cache.shrink->scan_objects(c->btree_cache.shrink, &sc);
498 }498 }
499 499
500 if (attr == &sysfs_btree_wakeup)500 if (attr == &sysfs_btree_wakeup)
fs/btrfs/super.c+1−1
@@ -1472,7 +1472,7 @@ static struct dentry *btrfs_mount_root(struct file_system_type *fs_type, · 1471 unmodified lines
1472 error = -EBUSY;1472 error = -EBUSY;
1473 } else {1473 } else {
1474 snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev);1474 snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev);
1475 shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", fs_type->name,1475 shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fs_type->name,
1476 s->s_id);1476 s->s_id);
1477 btrfs_sb(s)->bdev_holder = fs_type;1477 btrfs_sb(s)->bdev_holder = fs_type;
1478 error = btrfs_fill_super(s, fs_devices, data);1478 error = btrfs_fill_super(s, fs_devices, data);
fs/buffer.c+60−65
@@ -282,13 +282,7 @@ static void end_buffer_async_read(struct buffer_head *bh, int uptodate) · 281 unmodified lines
282 } while (tmp != bh);282 } while (tmp != bh);
283 spin_unlock_irqrestore(&first->b_uptodate_lock, flags);283 spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
284 284
285 /*285 folio_end_read(folio, folio_uptodate);
286 * If all of the buffers are uptodate then we can set the page
287 * uptodate.
288 */
289 if (folio_uptodate)
290 folio_mark_uptodate(folio);
291 folio_unlock(folio);
292 return;286 return;
293 287
294still_busy:288still_busy:
@@ -915,16 +909,12 @@ int remove_inode_buffers(struct inode *inode) · 620 unmodified lines
915 * which may not fail from ordinary buffer allocations.909 * which may not fail from ordinary buffer allocations.
916 */910 */
917struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,911struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
918 bool retry)912 gfp_t gfp)
919{913{
920 struct buffer_head *bh, *head;914 struct buffer_head *bh, *head;
921 gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
922 long offset;915 long offset;
923 struct mem_cgroup *memcg, *old_memcg;916 struct mem_cgroup *memcg, *old_memcg;
924 917
925 if (retry)
926 gfp |= __GFP_NOFAIL;
927
928 /* The folio lock pins the memcg */918 /* The folio lock pins the memcg */
929 memcg = folio_memcg(folio);919 memcg = folio_memcg(folio);
930 old_memcg = set_active_memcg(memcg);920 old_memcg = set_active_memcg(memcg);
@@ -967,7 +957,11 @@ EXPORT_SYMBOL_GPL(folio_alloc_buffers); · 36 unmodified lines
967struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,957struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
968 bool retry)958 bool retry)
969{959{
970 return folio_alloc_buffers(page_folio(page), size, retry);960 gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
961 if (retry)
962 gfp |= __GFP_NOFAIL;
963
964 return folio_alloc_buffers(page_folio(page), size, gfp);
971}965}
972EXPORT_SYMBOL_GPL(alloc_page_buffers);966EXPORT_SYMBOL_GPL(alloc_page_buffers);
973 967
@@ -1043,20 +1037,11 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 69 unmodified lines
1043 struct buffer_head *bh;1037 struct buffer_head *bh;
1044 sector_t end_block;1038 sector_t end_block;
1045 int ret = 0;1039 int ret = 0;
1046 gfp_t gfp_mask;
1047
1048 gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp;
1049
1050 /*
1051 * XXX: __getblk_slow() can not really deal with failure and
1052 * will endlessly loop on improvised global reclaim. Prefer
1053 * looping in the allocator rather than here, at least that
1054 * code knows what it's doing.
1055 */
1056 gfp_mask |= __GFP_NOFAIL;
1057 1040
1058 folio = __filemap_get_folio(inode->i_mapping, index,1041 folio = __filemap_get_folio(inode->i_mapping, index,
1059 FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp_mask);1042 FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
1043 if (IS_ERR(folio))
1044 return PTR_ERR(folio);
1060 1045
1061 bh = folio_buffers(folio);1046 bh = folio_buffers(folio);
1062 if (bh) {1047 if (bh) {
@@ -1069,7 +1054,10 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 6 unmodified lines
1069 goto failed;1054 goto failed;
1070 }1055 }
1071 1056
1072 bh = folio_alloc_buffers(folio, size, true);1057 ret = -ENOMEM;
1058 bh = folio_alloc_buffers(folio, size, gfp | __GFP_ACCOUNT);
1059 if (!bh)
1060 goto failed;
1073 1061
1074 /*1062 /*
1075 * Link the folio to the buffers and initialise them. Take the1063 * Link the folio to the buffers and initialise them. Take the
@@ -1420,33 +1408,36 @@ __find_get_block(struct block_device *bdev, sector_t block, unsigned size) · 344 unmodified lines
1420}1408}
1421EXPORT_SYMBOL(__find_get_block);1409EXPORT_SYMBOL(__find_get_block);
1422 1410
1423/*1411/**
1424 * __getblk_gfp() will locate (and, if necessary, create) the buffer_head1412 * bdev_getblk - Get a buffer_head in a block device's buffer cache.
1425 * which corresponds to the passed block_device, block and size. The1413 * @bdev: The block device.
1426 * returned buffer has its reference count incremented.1414 * @block: The block number.
1415 * @size: The size of buffer_heads for this @bdev.
1416 * @gfp: The memory allocation flags to use.
1427 *1417 *
1428 * __getblk_gfp() will lock up the machine if grow_dev_page's1418 * Return: The buffer head, or NULL if memory could not be allocated.
1429 * try_to_free_buffers() attempt is failing. FIXME, perhaps?
1430 */1419 */
1431struct buffer_head *1420struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
1432__getblk_gfp(struct block_device *bdev, sector_t block,1421 unsigned size, gfp_t gfp)
1433 unsigned size, gfp_t gfp)
1434{1422{
1435 struct buffer_head *bh = __find_get_block(bdev, block, size);1423 struct buffer_head *bh = __find_get_block(bdev, block, size);
1436 1424
1437 might_sleep();1425 might_alloc(gfp);
1438 if (bh == NULL)1426 if (bh)
1439 bh = __getblk_slow(bdev, block, size, gfp);1427 return bh;
1440 return bh;1428
1429 return __getblk_slow(bdev, block, size, gfp);
1441}1430}
1442EXPORT_SYMBOL(__getblk_gfp);1431EXPORT_SYMBOL(bdev_getblk);
1443 1432
1444/*1433/*
1445 * Do async read-ahead on a buffer..1434 * Do async read-ahead on a buffer..
1446 */1435 */
1447void __breadahead(struct block_device *bdev, sector_t block, unsigned size)1436void __breadahead(struct block_device *bdev, sector_t block, unsigned size)
1448{1437{
1449 struct buffer_head *bh = __getblk(bdev, block, size);1438 struct buffer_head *bh = bdev_getblk(bdev, block, size,
1439 GFP_NOWAIT | __GFP_MOVABLE);
1440
1450 if (likely(bh)) {1441 if (likely(bh)) {
1451 bh_readahead(bh, REQ_RAHEAD);1442 bh_readahead(bh, REQ_RAHEAD);
1452 brelse(bh);1443 brelse(bh);
@@ -1470,7 +1461,17 @@ struct buffer_head * · 17 unmodified lines
1470__bread_gfp(struct block_device *bdev, sector_t block,1461__bread_gfp(struct block_device *bdev, sector_t block,
1471 unsigned size, gfp_t gfp)1462 unsigned size, gfp_t gfp)
1472{1463{
1473 struct buffer_head *bh = __getblk_gfp(bdev, block, size, gfp);1464 struct buffer_head *bh;
1465
1466 gfp |= mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
1467
1468 /*
1469 * Prefer looping in the allocator rather than here, at least that
1470 * code knows what it's doing.
1471 */
1472 gfp |= __GFP_NOFAIL;
1473
1474 bh = bdev_getblk(bdev, block, size, gfp);
1474 1475
1475 if (likely(bh) && !buffer_uptodate(bh))1476 if (likely(bh) && !buffer_uptodate(bh))
1476 bh = __bread_slow(bh);1477 bh = __bread_slow(bh);
@@ -1640,12 +1641,13 @@ EXPORT_SYMBOL(block_invalidate_folio); · 163 unmodified lines
1640 * block_dirty_folio() via private_lock. try_to_free_buffers1641 * block_dirty_folio() via private_lock. try_to_free_buffers
1641 * is already excluded via the folio lock.1642 * is already excluded via the folio lock.
1642 */1643 */
1643void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize,1644struct buffer_head *create_empty_buffers(struct folio *folio,
1644 unsigned long b_state)1645 unsigned long blocksize, unsigned long b_state)
1645{1646{
1646 struct buffer_head *bh, *head, *tail;1647 struct buffer_head *bh, *head, *tail;
1648 gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL;
1647 1649
1648 head = folio_alloc_buffers(folio, blocksize, true);1650 head = folio_alloc_buffers(folio, blocksize, gfp);
1649 bh = head;1651 bh = head;
1650 do {1652 do {
1651 bh->b_state |= b_state;1653 bh->b_state |= b_state;
@@ -1667,13 +1669,8 @@ void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize, · 15 unmodified lines
1667 }1669 }
1668 folio_attach_private(folio, head);1670 folio_attach_private(folio, head);
1669 spin_unlock(&folio->mapping->private_lock);1671 spin_unlock(&folio->mapping->private_lock);
1670}
1671EXPORT_SYMBOL(folio_create_empty_buffers);
1672 1672
1673void create_empty_buffers(struct page *page,1673 return head;
1674 unsigned long blocksize, unsigned long b_state)
1675{
1676 folio_create_empty_buffers(page_folio(page), blocksize, b_state);
1677}1674}
1678EXPORT_SYMBOL(create_empty_buffers);1675EXPORT_SYMBOL(create_empty_buffers);
1679 1676
@@ -1768,13 +1765,15 @@ static struct buffer_head *folio_create_buffers(struct folio *folio, · 88 unmodified lines
1768 struct inode *inode,1765 struct inode *inode,
1769 unsigned int b_state)1766 unsigned int b_state)
1770{1767{
1768 struct buffer_head *bh;
1769
1771 BUG_ON(!folio_test_locked(folio));1770 BUG_ON(!folio_test_locked(folio));
1772 1771
1773 if (!folio_buffers(folio))1772 bh = folio_buffers(folio);
1774 folio_create_empty_buffers(folio,1773 if (!bh)
1775 1 << READ_ONCE(inode->i_blkbits),1774 bh = create_empty_buffers(folio,
1776 b_state);1775 1 << READ_ONCE(inode->i_blkbits), b_state);
1777 return folio_buffers(folio);1776 return bh;
1778}1777}
1779 1778
1780/*1779/*
@@ -2425,12 +2424,10 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) · 644 unmodified lines
2425 2424
2426 if (!nr) {2425 if (!nr) {
2427 /*2426 /*
2428 * All buffers are uptodate - we can set the folio uptodate2427 * All buffers are uptodate or get_block() returned an
2429 * as well. But not if get_block() returned an error.2428 * error when trying to map them - we can finish the read.
2430 */2429 */
2431 if (!page_error)2430 folio_end_read(folio, !page_error);
2432 folio_mark_uptodate(folio);
2433 folio_unlock(folio);
2434 return 0;2431 return 0;
2435 }2432 }
2436 2433
@@ -2676,10 +2673,8 @@ int block_truncate_page(struct address_space *mapping, · 239 unmodified lines
2676 return PTR_ERR(folio);2673 return PTR_ERR(folio);
2677 2674
2678 bh = folio_buffers(folio);2675 bh = folio_buffers(folio);
2679 if (!bh) {2676 if (!bh)
2680 folio_create_empty_buffers(folio, blocksize, 0);2677 bh = create_empty_buffers(folio, blocksize, 0);
2681 bh = folio_buffers(folio);
2682 }
2683 2678
2684 /* Find the buffer that contains "offset" */2679 /* Find the buffer that contains "offset" */
2685 offset = offset_in_folio(folio, from);2680 offset = offset_in_folio(folio, from);
fs/dax.c+12−12
@@ -412,23 +412,23 @@ static struct page *dax_busy_page(void *entry) · 411 unmodified lines
412 return NULL;412 return NULL;
413}413}
414 414
415/*415/**
416 * dax_lock_page - Lock the DAX entry corresponding to a page416 * dax_lock_folio - Lock the DAX entry corresponding to a folio
417 * @page: The page whose entry we want to lock417 * @folio: The folio whose entry we want to lock
418 *418 *
419 * Context: Process context.419 * Context: Process context.
420 * Return: A cookie to pass to dax_unlock_page() or 0 if the entry could420 * Return: A cookie to pass to dax_unlock_folio() or 0 if the entry could
421 * not be locked.421 * not be locked.
422 */422 */
423dax_entry_t dax_lock_page(struct page *page)423dax_entry_t dax_lock_folio(struct folio *folio)
424{424{
425 XA_STATE(xas, NULL, 0);425 XA_STATE(xas, NULL, 0);
426 void *entry;426 void *entry;
427 427
428 /* Ensure page->mapping isn't freed while we look at it */428 /* Ensure folio->mapping isn't freed while we look at it */
429 rcu_read_lock();429 rcu_read_lock();
430 for (;;) {430 for (;;) {
431 struct address_space *mapping = READ_ONCE(page->mapping);431 struct address_space *mapping = READ_ONCE(folio->mapping);
432 432
433 entry = NULL;433 entry = NULL;
434 if (!mapping || !dax_mapping(mapping))434 if (!mapping || !dax_mapping(mapping))
@@ -447,11 +447,11 @@ dax_entry_t dax_lock_page(struct page *page) · 12 unmodified lines
447 447
448 xas.xa = &mapping->i_pages;448 xas.xa = &mapping->i_pages;
449 xas_lock_irq(&xas);449 xas_lock_irq(&xas);
450 if (mapping != page->mapping) {450 if (mapping != folio->mapping) {
451 xas_unlock_irq(&xas);451 xas_unlock_irq(&xas);
452 continue;452 continue;
453 }453 }
454 xas_set(&xas, page->index);454 xas_set(&xas, folio->index);
455 entry = xas_load(&xas);455 entry = xas_load(&xas);
456 if (dax_is_locked(entry)) {456 if (dax_is_locked(entry)) {
457 rcu_read_unlock();457 rcu_read_unlock();
@@ -467,10 +467,10 @@ dax_entry_t dax_lock_page(struct page *page) · 9 unmodified lines
467 return (dax_entry_t)entry;467 return (dax_entry_t)entry;
468}468}
469 469
470void dax_unlock_page(struct page *page, dax_entry_t cookie)470void dax_unlock_folio(struct folio *folio, dax_entry_t cookie)
471{471{
472 struct address_space *mapping = page->mapping;472 struct address_space *mapping = folio->mapping;
473 XA_STATE(xas, &mapping->i_pages, page->index);473 XA_STATE(xas, &mapping->i_pages, folio->index);
474 474
475 if (S_ISCHR(mapping->host->i_mode))475 if (S_ISCHR(mapping->host->i_mode))
476 return;476 return;
fs/erofs/utils.c+12−7
@@ -264,19 +264,24 @@ static unsigned long erofs_shrink_scan(struct shrinker *shrink, · 263 unmodified lines
264 return freed;264 return freed;
265}265}
266 266
267static struct shrinker erofs_shrinker_info = {267static struct shrinker *erofs_shrinker_info;
268 .scan_objects = erofs_shrink_scan,
269 .count_objects = erofs_shrink_count,
270 .seeks = DEFAULT_SEEKS,
271};
272 268
273int __init erofs_init_shrinker(void)269int __init erofs_init_shrinker(void)
274{270{
275 return register_shrinker(&erofs_shrinker_info, "erofs-shrinker");271 erofs_shrinker_info = shrinker_alloc(0, "erofs-shrinker");
272 if (!erofs_shrinker_info)
273 return -ENOMEM;
274
275 erofs_shrinker_info->count_objects = erofs_shrink_count;
276 erofs_shrinker_info->scan_objects = erofs_shrink_scan;
277
278 shrinker_register(erofs_shrinker_info);
279
280 return 0;
276}281}
277 282
278void erofs_exit_shrinker(void)283void erofs_exit_shrinker(void)
279{284{
280 unregister_shrinker(&erofs_shrinker_info);285 shrinker_free(erofs_shrinker_info);
281}286}
282#endif /* !CONFIG_EROFS_FS_ZIP */287#endif /* !CONFIG_EROFS_FS_ZIP */
fs/exec.c+1−3
@@ -713,7 +713,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 712 unmodified lines
713 * process cleanup to remove whatever mess we made.713 * process cleanup to remove whatever mess we made.
714 */714 */
715 if (length != move_page_tables(vma, old_start,715 if (length != move_page_tables(vma, old_start,
716 vma, new_start, length, false))716 vma, new_start, length, false, true))
717 return -ENOMEM;717 return -ENOMEM;
718 718
719 lru_add_drain();719 lru_add_drain();
@@ -986,8 +986,6 @@ static int exec_mmap(struct mm_struct *mm) · 266 unmodified lines
986 tsk = current;986 tsk = current;
987 old_mm = current->mm;987 old_mm = current->mm;
988 exec_mm_release(tsk, old_mm);988 exec_mm_release(tsk, old_mm);
989 if (old_mm)
990 sync_mm_rss(old_mm);
991 989
992 ret = down_write_killable(&tsk->signal->exec_update_lock);990 ret = down_write_killable(&tsk->signal->exec_update_lock);
993 if (ret)991 if (ret)
fs/ext4/ext4.h+1−1
@@ -1664,7 +1664,7 @@ struct ext4_sb_info { · 1663 unmodified lines
1664 __u32 s_csum_seed;1664 __u32 s_csum_seed;
1665 1665
1666 /* Reclaim extents from extent status tree */1666 /* Reclaim extents from extent status tree */
1667 struct shrinker s_es_shrinker;1667 struct shrinker *s_es_shrinker;
1668 struct list_head s_es_list; /* List of inodes with reclaimable extents */1668 struct list_head s_es_list; /* List of inodes with reclaimable extents */
1669 long s_es_nr_inode;1669 long s_es_nr_inode;
1670 struct ext4_es_stats s_es_stats;1670 struct ext4_es_stats s_es_stats;
fs/ext4/extents_status.c+13−10
@@ -1632,7 +1632,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 1631 unmodified lines
1632 unsigned long nr;1632 unsigned long nr;
1633 struct ext4_sb_info *sbi;1633 struct ext4_sb_info *sbi;
1634 1634
1635 sbi = container_of(shrink, struct ext4_sb_info, s_es_shrinker);1635 sbi = shrink->private_data;
1636 nr = percpu_counter_read_positive(&sbi->s_es_stats.es_stats_shk_cnt);1636 nr = percpu_counter_read_positive(&sbi->s_es_stats.es_stats_shk_cnt);
1637 trace_ext4_es_shrink_count(sbi->s_sb, sc->nr_to_scan, nr);1637 trace_ext4_es_shrink_count(sbi->s_sb, sc->nr_to_scan, nr);
1638 return nr;1638 return nr;
@@ -1641,8 +1641,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 2 unmodified lines
1641static unsigned long ext4_es_scan(struct shrinker *shrink,1641static unsigned long ext4_es_scan(struct shrinker *shrink,
1642 struct shrink_control *sc)1642 struct shrink_control *sc)
1643{1643{
1644 struct ext4_sb_info *sbi = container_of(shrink,1644 struct ext4_sb_info *sbi = shrink->private_data;
1645 struct ext4_sb_info, s_es_shrinker);
1646 int nr_to_scan = sc->nr_to_scan;1645 int nr_to_scan = sc->nr_to_scan;
1647 int ret, nr_shrunk;1646 int ret, nr_shrunk;
1648 1647
@@ -1726,13 +1725,17 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi) · 77 unmodified lines
1726 if (err)1725 if (err)
1727 goto err3;1726 goto err3;
1728 1727
1729 sbi->s_es_shrinker.scan_objects = ext4_es_scan;1728 sbi->s_es_shrinker = shrinker_alloc(0, "ext4-es:%s", sbi->s_sb->s_id);
1730 sbi->s_es_shrinker.count_objects = ext4_es_count;1729 if (!sbi->s_es_shrinker) {
1731 sbi->s_es_shrinker.seeks = DEFAULT_SEEKS;1730 err = -ENOMEM;
1732 err = register_shrinker(&sbi->s_es_shrinker, "ext4-es:%s",
1733 sbi->s_sb->s_id);
1734 if (err)
1735 goto err4;1731 goto err4;
1732 }
1733
1734 sbi->s_es_shrinker->scan_objects = ext4_es_scan;
1735 sbi->s_es_shrinker->count_objects = ext4_es_count;
1736 sbi->s_es_shrinker->private_data = sbi;
1737
1738 shrinker_register(sbi->s_es_shrinker);
1736 1739
1737 return 0;1740 return 0;
1738err4:1741err4:
@@ -1752,7 +1755,7 @@ void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi) · 13 unmodified lines
1752 percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses);1755 percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses);
1753 percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt);1756 percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt);
1754 percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt);1757 percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt);
1755 unregister_shrinker(&sbi->s_es_shrinker);1758 shrinker_free(sbi->s_es_shrinker);
1756}1759}
1757 1760
1758/*1761/*
fs/ext4/inode.c+5−9
@@ -1032,10 +1032,8 @@ static int ext4_block_write_begin(struct folio *folio, loff_t pos, unsigned len, · 1031 unmodified lines
1032 BUG_ON(from > to);1032 BUG_ON(from > to);
1033 1033
1034 head = folio_buffers(folio);1034 head = folio_buffers(folio);
1035 if (!head) {1035 if (!head)
1036 create_empty_buffers(&folio->page, blocksize, 0);1036 head = create_empty_buffers(folio, blocksize, 0);
1037 head = folio_buffers(folio);
1038 }
1039 bbits = ilog2(blocksize);1037 bbits = ilog2(blocksize);
1040 block = (sector_t)folio->index << (PAGE_SHIFT - bbits);1038 block = (sector_t)folio->index << (PAGE_SHIFT - bbits);
1041 1039
@@ -1165,7 +1163,7 @@ retry_grab: · 123 unmodified lines
1165 * starting the handle.1163 * starting the handle.
1166 */1164 */
1167 if (!folio_buffers(folio))1165 if (!folio_buffers(folio))
1168 create_empty_buffers(&folio->page, inode->i_sb->s_blocksize, 0);1166 create_empty_buffers(folio, inode->i_sb->s_blocksize, 0);
1169 1167
1170 folio_unlock(folio);1168 folio_unlock(folio);
1171 1169
@@ -3655,10 +3653,8 @@ static int __ext4_block_zero_page_range(handle_t *handle, · 2483 unmodified lines
3655 iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);3653 iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits);
3656 3654
3657 bh = folio_buffers(folio);3655 bh = folio_buffers(folio);
3658 if (!bh) {3656 if (!bh)
3659 create_empty_buffers(&folio->page, blocksize, 0);3657 bh = create_empty_buffers(folio, blocksize, 0);
3660 bh = folio_buffers(folio);
3661 }
3662 3658
3663 /* Find the buffer that contains "offset" */3659 /* Find the buffer that contains "offset" */
3664 pos = blocksize;3660 pos = blocksize;
fs/ext4/move_extent.c+5−6
@@ -183,10 +183,8 @@ mext_page_mkuptodate(struct folio *folio, unsigned from, unsigned to) · 182 unmodified lines
183 183
184 blocksize = i_blocksize(inode);184 blocksize = i_blocksize(inode);
185 head = folio_buffers(folio);185 head = folio_buffers(folio);
186 if (!head) {186 if (!head)
187 create_empty_buffers(&folio->page, blocksize, 0);187 head = create_empty_buffers(folio, blocksize, 0);
188 head = folio_buffers(folio);
189 }
190 188
191 block = (sector_t)folio->index << (PAGE_SHIFT - inode->i_blkbits);189 block = (sector_t)folio->index << (PAGE_SHIFT - inode->i_blkbits);
192 for (bh = head, block_start = 0; bh != head || !block_start;190 for (bh = head, block_start = 0; bh != head || !block_start;
@@ -380,9 +378,10 @@ data_copy: · 187 unmodified lines
380 }378 }
381 /* Perform all necessary steps similar write_begin()/write_end()379 /* Perform all necessary steps similar write_begin()/write_end()
382 * but keeping in mind that i_size will not change */380 * but keeping in mind that i_size will not change */
383 if (!folio_buffers(folio[0]))
384 create_empty_buffers(&folio[0]->page, 1 << orig_inode->i_blkbits, 0);
385 bh = folio_buffers(folio[0]);381 bh = folio_buffers(folio[0]);
382 if (!bh)
383 bh = create_empty_buffers(folio[0],
384 1 << orig_inode->i_blkbits, 0);
386 for (i = 0; i < data_offset_in_page; i++)385 for (i = 0; i < data_offset_in_page; i++)
387 bh = bh->b_this_page;386 bh = bh->b_this_page;
388 for (i = 0; i < block_len_in_page; i++) {387 for (i = 0; i < block_len_in_page; i++) {
fs/ext4/readpage.c+3−11
@@ -70,15 +70,8 @@ static void __read_end_io(struct bio *bio) · 69 unmodified lines
70{70{
71 struct folio_iter fi;71 struct folio_iter fi;
72 72
73 bio_for_each_folio_all(fi, bio) {73 bio_for_each_folio_all(fi, bio)
74 struct folio *folio = fi.folio;74 folio_end_read(fi.folio, bio->bi_status == 0);
75
76 if (bio->bi_status)
77 folio_clear_uptodate(folio);
78 else
79 folio_mark_uptodate(folio);
80 folio_unlock(folio);
81 }
82 if (bio->bi_private)75 if (bio->bi_private)
83 mempool_free(bio->bi_private, bio_post_read_ctx_pool);76 mempool_free(bio->bi_private, bio_post_read_ctx_pool);
84 bio_put(bio);77 bio_put(bio);
@@ -336,8 +329,7 @@ int ext4_mpage_readpages(struct inode *inode, · 251 unmodified lines
336 if (ext4_need_verity(inode, folio->index) &&329 if (ext4_need_verity(inode, folio->index) &&
337 !fsverity_verify_folio(folio))330 !fsverity_verify_folio(folio))
338 goto set_error_page;331 goto set_error_page;
339 folio_mark_uptodate(folio);332 folio_end_read(folio, true);
340 folio_unlock(folio);
341 continue;333 continue;
342 }334 }
343 } else if (fully_mapped) {335 } else if (fully_mapped) {
fs/ext4/super.c+10−3
@@ -244,18 +244,25 @@ static struct buffer_head *__ext4_sb_bread_gfp(struct super_block *sb, · 243 unmodified lines
244struct buffer_head *ext4_sb_bread(struct super_block *sb, sector_t block,244struct buffer_head *ext4_sb_bread(struct super_block *sb, sector_t block,
245 blk_opf_t op_flags)245 blk_opf_t op_flags)
246{246{
247 return __ext4_sb_bread_gfp(sb, block, op_flags, __GFP_MOVABLE);247 gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping,
248 ~__GFP_FS) | __GFP_MOVABLE;
249
250 return __ext4_sb_bread_gfp(sb, block, op_flags, gfp);
248}251}
249 252
250struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb,253struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb,
251 sector_t block)254 sector_t block)
252{255{
253 return __ext4_sb_bread_gfp(sb, block, 0, 0);256 gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping,
257 ~__GFP_FS);
258
259 return __ext4_sb_bread_gfp(sb, block, 0, gfp);
254}260}
255 261
256void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block)262void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block)
257{263{
258 struct buffer_head *bh = sb_getblk_gfp(sb, block, 0);264 struct buffer_head *bh = bdev_getblk(sb->s_bdev, block,
265 sb->s_blocksize, GFP_NOWAIT | __GFP_NOWARN);
259 266
260 if (likely(bh)) {267 if (likely(bh)) {
261 if (trylock_buffer(bh))268 if (trylock_buffer(bh))
fs/f2fs/super.c+23−8
@@ -83,11 +83,26 @@ void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate, · 82 unmodified lines
83#endif83#endif
84 84
85/* f2fs-wide shrinker description */85/* f2fs-wide shrinker description */
86static struct shrinker f2fs_shrinker_info = {86static struct shrinker *f2fs_shrinker_info;
87 .scan_objects = f2fs_shrink_scan,87
88 .count_objects = f2fs_shrink_count,88static int __init f2fs_init_shrinker(void)
89 .seeks = DEFAULT_SEEKS,89{
90};90 f2fs_shrinker_info = shrinker_alloc(0, "f2fs-shrinker");
91 if (!f2fs_shrinker_info)
92 return -ENOMEM;
93
94 f2fs_shrinker_info->count_objects = f2fs_shrink_count;
95 f2fs_shrinker_info->scan_objects = f2fs_shrink_scan;
96
97 shrinker_register(f2fs_shrinker_info);
98
99 return 0;
100}
101
102static void f2fs_exit_shrinker(void)
103{
104 shrinker_free(f2fs_shrinker_info);
105}
91 106
92enum {107enum {
93 Opt_gc_background,108 Opt_gc_background,
@@ -4940,7 +4955,7 @@ static int __init init_f2fs_fs(void) · 4846 unmodified lines
4940 err = f2fs_init_sysfs();4955 err = f2fs_init_sysfs();
4941 if (err)4956 if (err)
4942 goto free_garbage_collection_cache;4957 goto free_garbage_collection_cache;
4943 err = register_shrinker(&f2fs_shrinker_info, "f2fs-shrinker");4958 err = f2fs_init_shrinker();
4944 if (err)4959 if (err)
4945 goto free_sysfs;4960 goto free_sysfs;
4946 err = register_filesystem(&f2fs_fs_type);4961 err = register_filesystem(&f2fs_fs_type);
@@ -4985,7 +5000,7 @@ free_root_stats: · 38 unmodified lines
4985 f2fs_destroy_root_stats();5000 f2fs_destroy_root_stats();
4986 unregister_filesystem(&f2fs_fs_type);5001 unregister_filesystem(&f2fs_fs_type);
4987free_shrinker:5002free_shrinker:
4988 unregister_shrinker(&f2fs_shrinker_info);5003 f2fs_exit_shrinker();
4989free_sysfs:5004free_sysfs:
4990 f2fs_exit_sysfs();5005 f2fs_exit_sysfs();
4991free_garbage_collection_cache:5006free_garbage_collection_cache:
@@ -5017,7 +5032,7 @@ static void __exit exit_f2fs_fs(void) · 25 unmodified lines
5017 f2fs_destroy_post_read_processing();5032 f2fs_destroy_post_read_processing();
5018 f2fs_destroy_root_stats();5033 f2fs_destroy_root_stats();
5019 unregister_filesystem(&f2fs_fs_type);5034 unregister_filesystem(&f2fs_fs_type);
5020 unregister_shrinker(&f2fs_shrinker_info);5035 f2fs_exit_shrinker();
5021 f2fs_exit_sysfs();5036 f2fs_exit_sysfs();
5022 f2fs_destroy_garbage_collection_cache();5037 f2fs_destroy_garbage_collection_cache();
5023 f2fs_destroy_extent_cache();5038 f2fs_destroy_extent_cache();
fs/gfs2/aops.c+1−1
@@ -130,7 +130,7 @@ static int __gfs2_jdata_write_folio(struct folio *folio, · 129 unmodified lines
130 if (folio_test_checked(folio)) {130 if (folio_test_checked(folio)) {
131 folio_clear_checked(folio);131 folio_clear_checked(folio);
132 if (!folio_buffers(folio)) {132 if (!folio_buffers(folio)) {
133 folio_create_empty_buffers(folio,133 create_empty_buffers(folio,
134 inode->i_sb->s_blocksize,134 inode->i_sb->s_blocksize,
135 BIT(BH_Dirty)|BIT(BH_Uptodate));135 BIT(BH_Dirty)|BIT(BH_Uptodate));
136 }136 }
fs/gfs2/bmap.c+23−25
@@ -43,53 +43,51 @@ struct metapath { · 42 unmodified lines
43static int punch_hole(struct gfs2_inode *ip, u64 offset, u64 length);43static int punch_hole(struct gfs2_inode *ip, u64 offset, u64 length);
44 44
45/**45/**
46 * gfs2_unstuffer_page - unstuff a stuffed inode into a block cached by a page46 * gfs2_unstuffer_folio - unstuff a stuffed inode into a block cached by a folio
47 * @ip: the inode47 * @ip: the inode
48 * @dibh: the dinode buffer48 * @dibh: the dinode buffer
49 * @block: the block number that was allocated49 * @block: the block number that was allocated
50 * @page: The (optional) page. This is looked up if @page is NULL50 * @folio: The folio.
51 *51 *
52 * Returns: errno52 * Returns: errno
53 */53 */
54 54static int gfs2_unstuffer_folio(struct gfs2_inode *ip, struct buffer_head *dibh,
55static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh,55 u64 block, struct folio *folio)
56 u64 block, struct page *page)
57{56{
58 struct inode *inode = &ip->i_inode;57 struct inode *inode = &ip->i_inode;
59 58
60 if (!PageUptodate(page)) {59 if (!folio_test_uptodate(folio)) {
61 void *kaddr = kmap(page);60 void *kaddr = kmap_local_folio(folio, 0);
62 u64 dsize = i_size_read(inode);61 u64 dsize = i_size_read(inode);
63 62
64 memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);63 memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);
65 memset(kaddr + dsize, 0, PAGE_SIZE - dsize);64 memset(kaddr + dsize, 0, folio_size(folio) - dsize);
66 kunmap(page);65 kunmap_local(kaddr);
67 66
68 SetPageUptodate(page);67 folio_mark_uptodate(folio);
69 }68 }
70 69
71 if (gfs2_is_jdata(ip)) {70 if (gfs2_is_jdata(ip)) {
72 struct buffer_head *bh;71 struct buffer_head *bh = folio_buffers(folio);
73 72
74 if (!page_has_buffers(page))73 if (!bh)
75 create_empty_buffers(page, BIT(inode->i_blkbits),74 bh = create_empty_buffers(folio,
76 BIT(BH_Uptodate));75 BIT(inode->i_blkbits), BIT(BH_Uptodate));
77 76
78 bh = page_buffers(page);
79 if (!buffer_mapped(bh))77 if (!buffer_mapped(bh))
80 map_bh(bh, inode->i_sb, block);78 map_bh(bh, inode->i_sb, block);
81 79
82 set_buffer_uptodate(bh);80 set_buffer_uptodate(bh);
83 gfs2_trans_add_data(ip->i_gl, bh);81 gfs2_trans_add_data(ip->i_gl, bh);
84 } else {82 } else {
85 set_page_dirty(page);83 folio_mark_dirty(folio);
86 gfs2_ordered_add_inode(ip);84 gfs2_ordered_add_inode(ip);
87 }85 }
88 86
89 return 0;87 return 0;
90}88}
91 89
92static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page)90static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct folio *folio)
93{91{
94 struct buffer_head *bh, *dibh;92 struct buffer_head *bh, *dibh;
95 struct gfs2_dinode *di;93 struct gfs2_dinode *di;
@@ -118,7 +116,7 @@ static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page) · 22 unmodified lines
118 dibh, sizeof(struct gfs2_dinode));116 dibh, sizeof(struct gfs2_dinode));
119 brelse(bh);117 brelse(bh);
120 } else {118 } else {
121 error = gfs2_unstuffer_page(ip, dibh, block, page);119 error = gfs2_unstuffer_folio(ip, dibh, block, folio);
122 if (error)120 if (error)
123 goto out_brelse;121 goto out_brelse;
124 }122 }
@@ -157,17 +155,17 @@ out_brelse: · 32 unmodified lines
157int gfs2_unstuff_dinode(struct gfs2_inode *ip)155int gfs2_unstuff_dinode(struct gfs2_inode *ip)
158{156{
159 struct inode *inode = &ip->i_inode;157 struct inode *inode = &ip->i_inode;
160 struct page *page;158 struct folio *folio;
161 int error;159 int error;
162 160
163 down_write(&ip->i_rw_mutex);161 down_write(&ip->i_rw_mutex);
164 page = grab_cache_page(inode->i_mapping, 0);162 folio = filemap_grab_folio(inode->i_mapping, 0);
165 error = -ENOMEM;163 error = PTR_ERR(folio);
166 if (!page)164 if (IS_ERR(folio))
167 goto out;165 goto out;
168 error = __gfs2_unstuff_inode(ip, page);166 error = __gfs2_unstuff_inode(ip, folio);
169 unlock_page(page);167 folio_unlock(folio);
170 put_page(page);168 folio_put(folio);
171out:169out:
172 up_write(&ip->i_rw_mutex);170 up_write(&ip->i_rw_mutex);
173 return error;171 return error;
fs/gfs2/glock.c+10−9
@@ -2041,11 +2041,7 @@ static unsigned long gfs2_glock_shrink_count(struct shrinker *shrink, · 2040 unmodified lines
2041 return vfs_pressure_ratio(atomic_read(&lru_count));2041 return vfs_pressure_ratio(atomic_read(&lru_count));
2042}2042}
2043 2043
2044static struct shrinker glock_shrinker = {2044static struct shrinker *glock_shrinker;
2045 .seeks = DEFAULT_SEEKS,
2046 .count_objects = gfs2_glock_shrink_count,
2047 .scan_objects = gfs2_glock_shrink_scan,
2048};
2049 2045
2050/**2046/**
2051 * glock_hash_walk - Call a function for glock in a hash bucket2047 * glock_hash_walk - Call a function for glock in a hash bucket
@@ -2465,13 +2461,18 @@ int __init gfs2_glock_init(void) · 413 unmodified lines
2465 return -ENOMEM;2461 return -ENOMEM;
2466 }2462 }
2467 2463
2468 ret = register_shrinker(&glock_shrinker, "gfs2-glock");2464 glock_shrinker = shrinker_alloc(0, "gfs2-glock");
2469 if (ret) {2465 if (!glock_shrinker) {
2470 destroy_workqueue(glock_workqueue);2466 destroy_workqueue(glock_workqueue);
2471 rhashtable_destroy(&gl_hash_table);2467 rhashtable_destroy(&gl_hash_table);
2472 return ret;2468 return -ENOMEM;
2473 }2469 }
2474 2470
2471 glock_shrinker->count_objects = gfs2_glock_shrink_count;
2472 glock_shrinker->scan_objects = gfs2_glock_shrink_scan;
2473
2474 shrinker_register(glock_shrinker);
2475
2475 for (i = 0; i < GLOCK_WAIT_TABLE_SIZE; i++)2476 for (i = 0; i < GLOCK_WAIT_TABLE_SIZE; i++)
2476 init_waitqueue_head(glock_wait_table + i);2477 init_waitqueue_head(glock_wait_table + i);
2477 2478
@@ -2480,7 +2481,7 @@ int __init gfs2_glock_init(void) · 2 unmodified lines
2480 2481
2481void gfs2_glock_exit(void)2482void gfs2_glock_exit(void)
2482{2483{
2483 unregister_shrinker(&glock_shrinker);2484 shrinker_free(glock_shrinker);
2484 rhashtable_destroy(&gl_hash_table);2485 rhashtable_destroy(&gl_hash_table);
2485 destroy_workqueue(glock_workqueue);2486 destroy_workqueue(glock_workqueue);
2486}2487}
fs/gfs2/main.c+3−3
@@ -147,7 +147,7 @@ static int __init init_gfs2_fs(void) · 146 unmodified lines
147 if (!gfs2_trans_cachep)147 if (!gfs2_trans_cachep)
148 goto fail_cachep8;148 goto fail_cachep8;
149 149
150 error = register_shrinker(&gfs2_qd_shrinker, "gfs2-qd");150 error = gfs2_qd_shrinker_init();
151 if (error)151 if (error)
152 goto fail_shrinker;152 goto fail_shrinker;
153 153
@@ -196,7 +196,7 @@ fail_wq3: · 42 unmodified lines
196fail_wq2:196fail_wq2:
197 destroy_workqueue(gfs2_recovery_wq);197 destroy_workqueue(gfs2_recovery_wq);
198fail_wq1:198fail_wq1:
199 unregister_shrinker(&gfs2_qd_shrinker);199 gfs2_qd_shrinker_exit();
200fail_shrinker:200fail_shrinker:
201 kmem_cache_destroy(gfs2_trans_cachep);201 kmem_cache_destroy(gfs2_trans_cachep);
202fail_cachep8:202fail_cachep8:
@@ -229,7 +229,7 @@ fail_lru: · 26 unmodified lines
229 229
230static void __exit exit_gfs2_fs(void)230static void __exit exit_gfs2_fs(void)
231{231{
232 unregister_shrinker(&gfs2_qd_shrinker);232 gfs2_qd_shrinker_exit();
233 gfs2_glock_exit();233 gfs2_glock_exit();
234 gfs2_unregister_debugfs();234 gfs2_unregister_debugfs();
235 unregister_filesystem(&gfs2_fs_type);235 unregister_filesystem(&gfs2_fs_type);
fs/gfs2/meta_io.c+25−36
@@ -115,7 +115,7 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 114 unmodified lines
115{115{
116 struct address_space *mapping = gfs2_glock2aspace(gl);116 struct address_space *mapping = gfs2_glock2aspace(gl);
117 struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;117 struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
118 struct page *page;118 struct folio *folio;
119 struct buffer_head *bh;119 struct buffer_head *bh;
120 unsigned int shift;120 unsigned int shift;
121 unsigned long index;121 unsigned long index;
@@ -129,36 +129,31 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 7 unmodified lines
129 bufnum = blkno - (index << shift); /* block buf index within page */129 bufnum = blkno - (index << shift); /* block buf index within page */
130 130
131 if (create) {131 if (create) {
132 for (;;) {132 folio = __filemap_get_folio(mapping, index,
133 page = grab_cache_page(mapping, index);133 FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
134 if (page)134 mapping_gfp_mask(mapping) | __GFP_NOFAIL);
135 break;135 bh = folio_buffers(folio);
136 yield();136 if (!bh)
137 }137 bh = create_empty_buffers(folio,
138 if (!page_has_buffers(page))138 sdp->sd_sb.sb_bsize, 0);
139 create_empty_buffers(page, sdp->sd_sb.sb_bsize, 0);
140 } else {139 } else {
141 page = find_get_page_flags(mapping, index,140 folio = __filemap_get_folio(mapping, index,
142 FGP_LOCK|FGP_ACCESSED);141 FGP_LOCK | FGP_ACCESSED, 0);
143 if (!page)142 if (IS_ERR(folio))
144 return NULL;143 return NULL;
145 if (!page_has_buffers(page)) {144 bh = folio_buffers(folio);
146 bh = NULL;
147 goto out_unlock;
148 }
149 }145 }
150 146
151 /* Locate header for our buffer within our page */147 if (!bh)
152 for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page)148 goto out_unlock;
153 /* Do nothing */;
154 get_bh(bh);
155 149
150 bh = get_nth_bh(bh, bufnum);
156 if (!buffer_mapped(bh))151 if (!buffer_mapped(bh))
157 map_bh(bh, sdp->sd_vfs, blkno);152 map_bh(bh, sdp->sd_vfs, blkno);
158 153
159out_unlock:154out_unlock:
160 unlock_page(page);155 folio_unlock(folio);
161 put_page(page);156 folio_put(folio);
162 157
163 return bh;158 return bh;
164}159}
@@ -405,26 +400,20 @@ static struct buffer_head *gfs2_getjdatabuf(struct gfs2_inode *ip, u64 blkno) · 240 unmodified lines
405{400{
406 struct address_space *mapping = ip->i_inode.i_mapping;401 struct address_space *mapping = ip->i_inode.i_mapping;
407 struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);402 struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
408 struct page *page;403 struct folio *folio;
409 struct buffer_head *bh;404 struct buffer_head *bh;
410 unsigned int shift = PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift;405 unsigned int shift = PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift;
411 unsigned long index = blkno >> shift; /* convert block to page */406 unsigned long index = blkno >> shift; /* convert block to page */
412 unsigned int bufnum = blkno - (index << shift);407 unsigned int bufnum = blkno - (index << shift);
413 408
414 page = find_get_page_flags(mapping, index, FGP_LOCK|FGP_ACCESSED);409 folio = __filemap_get_folio(mapping, index, FGP_LOCK | FGP_ACCESSED, 0);
415 if (!page)410 if (IS_ERR(folio))
416 return NULL;411 return NULL;
417 if (!page_has_buffers(page)) {412 bh = folio_buffers(folio);
418 unlock_page(page);413 if (bh)
419 put_page(page);414 bh = get_nth_bh(bh, bufnum);
420 return NULL;415 folio_unlock(folio);
421 }416 folio_put(folio);
422 /* Locate header for our buffer within our page */
423 for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page)
424 /* Do nothing */;
425 get_bh(bh);
426 unlock_page(page);
427 put_page(page);
428 return bh;417 return bh;
429}418}
430 419
fs/gfs2/quota.c+38−24
@@ -196,13 +196,26 @@ static unsigned long gfs2_qd_shrink_count(struct shrinker *shrink, · 195 unmodified lines
196 return vfs_pressure_ratio(list_lru_shrink_count(&gfs2_qd_lru, sc));196 return vfs_pressure_ratio(list_lru_shrink_count(&gfs2_qd_lru, sc));
197}197}
198 198
199struct shrinker gfs2_qd_shrinker = {199static struct shrinker *gfs2_qd_shrinker;
200 .count_objects = gfs2_qd_shrink_count,200
201 .scan_objects = gfs2_qd_shrink_scan,201int __init gfs2_qd_shrinker_init(void)
202 .seeks = DEFAULT_SEEKS,202{
203 .flags = SHRINKER_NUMA_AWARE,203 gfs2_qd_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "gfs2-qd");
204};204 if (!gfs2_qd_shrinker)
205 return -ENOMEM;
206
207 gfs2_qd_shrinker->count_objects = gfs2_qd_shrink_count;
208 gfs2_qd_shrinker->scan_objects = gfs2_qd_shrink_scan;
209
210 shrinker_register(gfs2_qd_shrinker);
205 211
212 return 0;
213}
214
215void gfs2_qd_shrinker_exit(void)
216{
217 shrinker_free(gfs2_qd_shrinker);
218}
206 219
207static u64 qd2index(struct gfs2_quota_data *qd)220static u64 qd2index(struct gfs2_quota_data *qd)
208{221{
@@ -736,7 +749,7 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 527 unmodified lines
736 struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode);749 struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode);
737 struct inode *inode = &ip->i_inode;750 struct inode *inode = &ip->i_inode;
738 struct address_space *mapping = inode->i_mapping;751 struct address_space *mapping = inode->i_mapping;
739 struct page *page;752 struct folio *folio;
740 struct buffer_head *bh;753 struct buffer_head *bh;
741 u64 blk;754 u64 blk;
742 unsigned bsize = sdp->sd_sb.sb_bsize, bnum = 0, boff = 0;755 unsigned bsize = sdp->sd_sb.sb_bsize, bnum = 0, boff = 0;
@@ -745,15 +758,15 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 2 unmodified lines
745 blk = index << (PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift);758 blk = index << (PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift);
746 boff = off % bsize;759 boff = off % bsize;
747 760
748 page = grab_cache_page(mapping, index);761 folio = filemap_grab_folio(mapping, index);
749 if (!page)762 if (IS_ERR(folio))
750 return -ENOMEM;763 return PTR_ERR(folio);
751 if (!page_has_buffers(page))764 bh = folio_buffers(folio);
752 create_empty_buffers(page, bsize, 0);765 if (!bh)
766 bh = create_empty_buffers(folio, bsize, 0);
753 767
754 bh = page_buffers(page);768 for (;;) {
755 for(;;) {769 /* Find the beginning block within the folio */
756 /* Find the beginning block within the page */
757 if (pg_off >= ((bnum * bsize) + bsize)) {770 if (pg_off >= ((bnum * bsize) + bsize)) {
758 bh = bh->b_this_page;771 bh = bh->b_this_page;
759 bnum++;772 bnum++;
@@ -766,9 +779,10 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 6 unmodified lines
766 goto unlock_out;779 goto unlock_out;
767 /* If it's a newly allocated disk block, zero it */780 /* If it's a newly allocated disk block, zero it */
768 if (buffer_new(bh))781 if (buffer_new(bh))
769 zero_user(page, bnum * bsize, bh->b_size);782 folio_zero_range(folio, bnum * bsize,
783 bh->b_size);
770 }784 }
771 if (PageUptodate(page))785 if (folio_test_uptodate(folio))
772 set_buffer_uptodate(bh);786 set_buffer_uptodate(bh);
773 if (bh_read(bh, REQ_META | REQ_PRIO) < 0)787 if (bh_read(bh, REQ_META | REQ_PRIO) < 0)
774 goto unlock_out;788 goto unlock_out;
@@ -784,17 +798,17 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 9 unmodified lines
784 break;798 break;
785 }799 }
786 800
787 /* Write to the page, now that we have setup the buffer(s) */801 /* Write to the folio, now that we have setup the buffer(s) */
788 memcpy_to_page(page, off, buf, bytes);802 memcpy_to_folio(folio, off, buf, bytes);
789 flush_dcache_page(page);803 flush_dcache_folio(folio);
790 unlock_page(page);804 folio_unlock(folio);
791 put_page(page);805 folio_put(folio);
792 806
793 return 0;807 return 0;
794 808
795unlock_out:809unlock_out:
796 unlock_page(page);810 folio_unlock(folio);
797 put_page(page);811 folio_put(folio);
798 return -EIO;812 return -EIO;
799}813}
800 814
fs/gfs2/quota.h+2−1
@@ -60,7 +60,8 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip, · 59 unmodified lines
60}60}
61 61
62extern const struct quotactl_ops gfs2_quotactl_ops;62extern const struct quotactl_ops gfs2_quotactl_ops;
63extern struct shrinker gfs2_qd_shrinker;63int __init gfs2_qd_shrinker_init(void);
64void gfs2_qd_shrinker_exit(void);
64extern struct list_lru gfs2_qd_lru;65extern struct list_lru gfs2_qd_lru;
65extern void __init gfs2_quota_hash_init(void);66extern void __init gfs2_quota_hash_init(void);
66 67
fs/hugetlbfs/inode.c+23−61
@@ -83,29 +83,6 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = { · 82 unmodified lines
83 {}83 {}
84};84};
85 85
86#ifdef CONFIG_NUMA
87static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma,
88 struct inode *inode, pgoff_t index)
89{
90 vma->vm_policy = mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy,
91 index);
92}
93
94static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma)
95{
96 mpol_cond_put(vma->vm_policy);
97}
98#else
99static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma,
100 struct inode *inode, pgoff_t index)
101{
102}
103
104static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma)
105{
106}
107#endif
108
109/*86/*
110 * Mask used when checking the page offset value passed in via system87 * Mask used when checking the page offset value passed in via system
111 * calls. This value will be converted to a loff_t which is signed.88 * calls. This value will be converted to a loff_t which is signed.
@@ -135,7 +112,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) · 23 unmodified lines
135 vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND);112 vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND);
136 vma->vm_ops = &hugetlb_vm_ops;113 vma->vm_ops = &hugetlb_vm_ops;
137 114
138 ret = seal_check_future_write(info->seals, vma);115 ret = seal_check_write(info->seals, vma);
139 if (ret)116 if (ret)
140 return ret;117 return ret;
141 118
@@ -295,7 +272,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 153 unmodified lines
295 size_t res = 0;272 size_t res = 0;
296 273
297 /* First subpage to start the loop. */274 /* First subpage to start the loop. */
298 page += offset / PAGE_SIZE;275 page = nth_page(page, offset / PAGE_SIZE);
299 offset %= PAGE_SIZE;276 offset %= PAGE_SIZE;
300 while (1) {277 while (1) {
301 if (is_raw_hwpoison_page_in_hugepage(page))278 if (is_raw_hwpoison_page_in_hugepage(page))
@@ -309,7 +286,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 7 unmodified lines
309 break;286 break;
310 offset += n;287 offset += n;
311 if (offset == PAGE_SIZE) {288 if (offset == PAGE_SIZE) {
312 page++;289 page = nth_page(page, 1);
313 offset = 0;290 offset = 0;
314 }291 }
315 }292 }
@@ -334,7 +311,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 18 unmodified lines
334 ssize_t retval = 0;311 ssize_t retval = 0;
335 312
336 while (iov_iter_count(to)) {313 while (iov_iter_count(to)) {
337 struct page *page;314 struct folio *folio;
338 size_t nr, copied, want;315 size_t nr, copied, want;
339 316
340 /* nr is the maximum number of bytes to copy from this page */317 /* nr is the maximum number of bytes to copy from this page */
@@ -352,18 +329,18 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 11 unmodified lines
352 }329 }
353 nr = nr - offset;330 nr = nr - offset;
354 331
355 /* Find the page */332 /* Find the folio */
356 page = find_lock_page(mapping, index);333 folio = filemap_lock_hugetlb_folio(h, mapping, index);
357 if (unlikely(page == NULL)) {334 if (IS_ERR(folio)) {
358 /*335 /*
359 * We have a HOLE, zero out the user-buffer for the336 * We have a HOLE, zero out the user-buffer for the
360 * length of the hole or request.337 * length of the hole or request.
361 */338 */
362 copied = iov_iter_zero(nr, to);339 copied = iov_iter_zero(nr, to);
363 } else {340 } else {
364 unlock_page(page);341 folio_unlock(folio);
365 342
366 if (!PageHWPoison(page))343 if (!folio_test_has_hwpoisoned(folio))
367 want = nr;344 want = nr;
368 else {345 else {
369 /*346 /*
@@ -371,19 +348,19 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 1 unmodified line
371 * touching the 1st raw HWPOISON subpage after348 * touching the 1st raw HWPOISON subpage after
372 * offset.349 * offset.
373 */350 */
374 want = adjust_range_hwpoison(page, offset, nr);351 want = adjust_range_hwpoison(&folio->page, offset, nr);
375 if (want == 0) {352 if (want == 0) {
376 put_page(page);353 folio_put(folio);
377 retval = -EIO;354 retval = -EIO;
378 break;355 break;
379 }356 }
380 }357 }
381 358
382 /*359 /*
383 * We have the page, copy it to user space buffer.360 * We have the folio, copy it to user space buffer.
384 */361 */
385 copied = copy_page_to_iter(page, offset, want, to);362 copied = copy_folio_to_iter(folio, offset, want, to);
386 put_page(page);363 folio_put(folio);
387 }364 }
388 offset += copied;365 offset += copied;
389 retval += copied;366 retval += copied;
@@ -661,21 +638,20 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 271 unmodified lines
661{638{
662 struct hstate *h = hstate_inode(inode);639 struct hstate *h = hstate_inode(inode);
663 struct address_space *mapping = &inode->i_data;640 struct address_space *mapping = &inode->i_data;
664 const pgoff_t start = lstart >> huge_page_shift(h);641 const pgoff_t end = lend >> PAGE_SHIFT;
665 const pgoff_t end = lend >> huge_page_shift(h);
666 struct folio_batch fbatch;642 struct folio_batch fbatch;
667 pgoff_t next, index;643 pgoff_t next, index;
668 int i, freed = 0;644 int i, freed = 0;
669 bool truncate_op = (lend == LLONG_MAX);645 bool truncate_op = (lend == LLONG_MAX);
670 646
671 folio_batch_init(&fbatch);647 folio_batch_init(&fbatch);
672 next = start;648 next = lstart >> PAGE_SHIFT;
673 while (filemap_get_folios(mapping, &next, end - 1, &fbatch)) {649 while (filemap_get_folios(mapping, &next, end - 1, &fbatch)) {
674 for (i = 0; i < folio_batch_count(&fbatch); ++i) {650 for (i = 0; i < folio_batch_count(&fbatch); ++i) {
675 struct folio *folio = fbatch.folios[i];651 struct folio *folio = fbatch.folios[i];
676 u32 hash = 0;652 u32 hash = 0;
677 653
678 index = folio->index;654 index = folio->index >> huge_page_order(h);
679 hash = hugetlb_fault_mutex_hash(mapping, index);655 hash = hugetlb_fault_mutex_hash(mapping, index);
680 mutex_lock(&hugetlb_fault_mutex_table[hash]);656 mutex_lock(&hugetlb_fault_mutex_table[hash]);
681 657
@@ -693,7 +669,9 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 11 unmodified lines
693 }669 }
694 670
695 if (truncate_op)671 if (truncate_op)
696 (void)hugetlb_unreserve_pages(inode, start, LONG_MAX, freed);672 (void)hugetlb_unreserve_pages(inode,
673 lstart >> huge_page_shift(h),
674 LONG_MAX, freed);
697}675}
698 676
699static void hugetlbfs_evict_inode(struct inode *inode)677static void hugetlbfs_evict_inode(struct inode *inode)
@@ -741,7 +719,7 @@ static void hugetlbfs_zero_partial_page(struct hstate *h, · 41 unmodified lines
741 pgoff_t idx = start >> huge_page_shift(h);719 pgoff_t idx = start >> huge_page_shift(h);
742 struct folio *folio;720 struct folio *folio;
743 721
744 folio = filemap_lock_folio(mapping, idx);722 folio = filemap_lock_hugetlb_folio(h, mapping, idx);
745 if (IS_ERR(folio))723 if (IS_ERR(folio))
746 return;724 return;
747 725
@@ -852,8 +830,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 104 unmodified lines
852 830
853 /*831 /*
854 * Initialize a pseudo vma as this is required by the huge page832 * Initialize a pseudo vma as this is required by the huge page
855 * allocation routines. If NUMA is configured, use page index833 * allocation routines.
856 * as input to create an allocation policy.
857 */834 */
858 vma_init(&pseudo_vma, mm);835 vma_init(&pseudo_vma, mm);
859 vm_flags_init(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED);836 vm_flags_init(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
@@ -886,7 +863,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 26 unmodified lines
886 mutex_lock(&hugetlb_fault_mutex_table[hash]);863 mutex_lock(&hugetlb_fault_mutex_table[hash]);
887 864
888 /* See if already present in mapping to avoid alloc/free */865 /* See if already present in mapping to avoid alloc/free */
889 folio = filemap_get_folio(mapping, index);866 folio = filemap_get_folio(mapping, index << huge_page_order(h));
890 if (!IS_ERR(folio)) {867 if (!IS_ERR(folio)) {
891 folio_put(folio);868 folio_put(folio);
892 mutex_unlock(&hugetlb_fault_mutex_table[hash]);869 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
@@ -901,9 +878,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 8 unmodified lines
901 * folios in these areas, we need to consume the reserves878 * folios in these areas, we need to consume the reserves
902 * to keep reservation accounting consistent.879 * to keep reservation accounting consistent.
903 */880 */
904 hugetlb_set_vma_policy(&pseudo_vma, inode, index);
905 folio = alloc_hugetlb_folio(&pseudo_vma, addr, 0);881 folio = alloc_hugetlb_folio(&pseudo_vma, addr, 0);
906 hugetlb_drop_vma_policy(&pseudo_vma);
907 if (IS_ERR(folio)) {882 if (IS_ERR(folio)) {
908 mutex_unlock(&hugetlb_fault_mutex_table[hash]);883 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
909 error = PTR_ERR(folio);884 error = PTR_ERR(folio);
@@ -1282,18 +1257,6 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) · 372 unmodified lines
1282 hugetlbfs_inc_free_inodes(sbinfo);1257 hugetlbfs_inc_free_inodes(sbinfo);
1283 return NULL;1258 return NULL;
1284 }1259 }
1285
1286 /*
1287 * Any time after allocation, hugetlbfs_destroy_inode can be called
1288 * for the inode. mpol_free_shared_policy is unconditionally called
1289 * as part of hugetlbfs_destroy_inode. So, initialize policy here
1290 * in case of a quick call to destroy.
1291 *
1292 * Note that the policy is initialized even if we are creating a
1293 * private inode. This simplifies hugetlbfs_destroy_inode.
1294 */
1295 mpol_shared_policy_init(&p->policy, NULL);
1296
1297 return &p->vfs_inode;1260 return &p->vfs_inode;
1298}1261}
1299 1262
@@ -1305,7 +1268,6 @@ static void hugetlbfs_free_inode(struct inode *inode) · 5 unmodified lines
1305static void hugetlbfs_destroy_inode(struct inode *inode)1268static void hugetlbfs_destroy_inode(struct inode *inode)
1306{1269{
1307 hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb));1270 hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb));
1308 mpol_free_shared_policy(&HUGETLBFS_I(inode)->policy);
1309}1271}
1310 1272
1311static const struct address_space_operations hugetlbfs_aops = {1273static const struct address_space_operations hugetlbfs_aops = {
fs/iomap/buffered-io.c+35−22
@@ -29,9 +29,9 @@ typedef int (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length); · 28 unmodified lines
29 * and I/O completions.29 * and I/O completions.
30 */30 */
31struct iomap_folio_state {31struct iomap_folio_state {
32 atomic_t read_bytes_pending;
33 atomic_t write_bytes_pending;
34 spinlock_t state_lock;32 spinlock_t state_lock;
33 unsigned int read_bytes_pending;
34 atomic_t write_bytes_pending;
35 35
36 /*36 /*
37 * Each block has two bits in this bitmap:37 * Each block has two bits in this bitmap:
@@ -57,30 +57,32 @@ static inline bool ifs_block_is_uptodate(struct iomap_folio_state *ifs, · 19 unmodified lines
57 return test_bit(block, ifs->state);57 return test_bit(block, ifs->state);
58}58}
59 59
60static void ifs_set_range_uptodate(struct folio *folio,60static bool ifs_set_range_uptodate(struct folio *folio,
61 struct iomap_folio_state *ifs, size_t off, size_t len)61 struct iomap_folio_state *ifs, size_t off, size_t len)
62{62{
63 struct inode *inode = folio->mapping->host;63 struct inode *inode = folio->mapping->host;
64 unsigned int first_blk = off >> inode->i_blkbits;64 unsigned int first_blk = off >> inode->i_blkbits;
65 unsigned int last_blk = (off + len - 1) >> inode->i_blkbits;65 unsigned int last_blk = (off + len - 1) >> inode->i_blkbits;
66 unsigned int nr_blks = last_blk - first_blk + 1;66 unsigned int nr_blks = last_blk - first_blk + 1;
67 unsigned long flags;
68 67
69 spin_lock_irqsave(&ifs->state_lock, flags);
70 bitmap_set(ifs->state, first_blk, nr_blks);68 bitmap_set(ifs->state, first_blk, nr_blks);
71 if (ifs_is_fully_uptodate(folio, ifs))69 return ifs_is_fully_uptodate(folio, ifs);
72 folio_mark_uptodate(folio);
73 spin_unlock_irqrestore(&ifs->state_lock, flags);
74}70}
75 71
76static void iomap_set_range_uptodate(struct folio *folio, size_t off,72static void iomap_set_range_uptodate(struct folio *folio, size_t off,
77 size_t len)73 size_t len)
78{74{
79 struct iomap_folio_state *ifs = folio->private;75 struct iomap_folio_state *ifs = folio->private;
76 unsigned long flags;
77 bool uptodate = true;
80 78
81 if (ifs)79 if (ifs) {
82 ifs_set_range_uptodate(folio, ifs, off, len);80 spin_lock_irqsave(&ifs->state_lock, flags);
83 else81 uptodate = ifs_set_range_uptodate(folio, ifs, off, len);
82 spin_unlock_irqrestore(&ifs->state_lock, flags);
83 }
84
85 if (uptodate)
84 folio_mark_uptodate(folio);86 folio_mark_uptodate(folio);
85}87}
86 88
@@ -181,7 +183,7 @@ static void ifs_free(struct folio *folio) · 94 unmodified lines
181 183
182 if (!ifs)184 if (!ifs)
183 return;185 return;
184 WARN_ON_ONCE(atomic_read(&ifs->read_bytes_pending));186 WARN_ON_ONCE(ifs->read_bytes_pending != 0);
185 WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending));187 WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending));
186 WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) !=188 WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) !=
187 folio_test_uptodate(folio));189 folio_test_uptodate(folio));
@@ -248,20 +250,28 @@ static void iomap_adjust_read_range(struct inode *inode, struct folio *folio, · 60 unmodified lines
248 *lenp = plen;250 *lenp = plen;
249}251}
250 252
251static void iomap_finish_folio_read(struct folio *folio, size_t offset,253static void iomap_finish_folio_read(struct folio *folio, size_t off,
252 size_t len, int error)254 size_t len, int error)
253{255{
254 struct iomap_folio_state *ifs = folio->private;256 struct iomap_folio_state *ifs = folio->private;
257 bool uptodate = !error;
258 bool finished = true;
255 259
256 if (unlikely(error)) {260 if (ifs) {
257 folio_clear_uptodate(folio);261 unsigned long flags;
258 folio_set_error(folio);262
259 } else {263 spin_lock_irqsave(&ifs->state_lock, flags);
260 iomap_set_range_uptodate(folio, offset, len);264 if (!error)
265 uptodate = ifs_set_range_uptodate(folio, ifs, off, len);
266 ifs->read_bytes_pending -= len;
267 finished = !ifs->read_bytes_pending;
268 spin_unlock_irqrestore(&ifs->state_lock, flags);
261 }269 }
262 270
263 if (!ifs || atomic_sub_and_test(len, &ifs->read_bytes_pending))271 if (error)
264 folio_unlock(folio);272 folio_set_error(folio);
273 if (finished)
274 folio_end_read(folio, uptodate);
265}275}
266 276
267static void iomap_read_end_io(struct bio *bio)277static void iomap_read_end_io(struct bio *bio)
@@ -358,8 +368,11 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, · 90 unmodified lines
358 }368 }
359 369
360 ctx->cur_folio_in_bio = true;370 ctx->cur_folio_in_bio = true;
361 if (ifs)371 if (ifs) {
362 atomic_add(plen, &ifs->read_bytes_pending);372 spin_lock_irq(&ifs->state_lock);
373 ifs->read_bytes_pending += plen;
374 spin_unlock_irq(&ifs->state_lock);
375 }
363 376
364 sector = iomap_sector(iomap, pos);377 sector = iomap_sector(iomap, pos);
365 if (!ctx->bio ||378 if (!ctx->bio ||
fs/jbd2/journal.c+18−11
@@ -1290,7 +1290,7 @@ static int jbd2_min_tag_size(void) · 1289 unmodified lines
1290static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink,1290static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink,
1291 struct shrink_control *sc)1291 struct shrink_control *sc)
1292{1292{
1293 journal_t *journal = container_of(shrink, journal_t, j_shrinker);1293 journal_t *journal = shrink->private_data;
1294 unsigned long nr_to_scan = sc->nr_to_scan;1294 unsigned long nr_to_scan = sc->nr_to_scan;
1295 unsigned long nr_shrunk;1295 unsigned long nr_shrunk;
1296 unsigned long count;1296 unsigned long count;
@@ -1316,7 +1316,7 @@ static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, · 19 unmodified lines
1316static unsigned long jbd2_journal_shrink_count(struct shrinker *shrink,1316static unsigned long jbd2_journal_shrink_count(struct shrinker *shrink,
1317 struct shrink_control *sc)1317 struct shrink_control *sc)
1318{1318{
1319 journal_t *journal = container_of(shrink, journal_t, j_shrinker);1319 journal_t *journal = shrink->private_data;
1320 unsigned long count;1320 unsigned long count;
1321 1321
1322 count = percpu_counter_read_positive(&journal->j_checkpoint_jh_count);1322 count = percpu_counter_read_positive(&journal->j_checkpoint_jh_count);
@@ -1588,14 +1588,21 @@ static journal_t *journal_init_common(struct block_device *bdev, · 265 unmodified lines
1588 goto err_cleanup;1588 goto err_cleanup;
1589 1589
1590 journal->j_shrink_transaction = NULL;1590 journal->j_shrink_transaction = NULL;
1591 journal->j_shrinker.scan_objects = jbd2_journal_shrink_scan;1591
1592 journal->j_shrinker.count_objects = jbd2_journal_shrink_count;1592 journal->j_shrinker = shrinker_alloc(0, "jbd2-journal:(%u:%u)",
1593 journal->j_shrinker.seeks = DEFAULT_SEEKS;1593 MAJOR(bdev->bd_dev),
1594 journal->j_shrinker.batch = journal->j_max_transaction_buffers;1594 MINOR(bdev->bd_dev));
1595 err = register_shrinker(&journal->j_shrinker, "jbd2-journal:(%u:%u)",1595 if (!journal->j_shrinker) {
1596 MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev));1596 err = -ENOMEM;
1597 if (err)
1598 goto err_cleanup;1597 goto err_cleanup;
1598 }
1599
1600 journal->j_shrinker->scan_objects = jbd2_journal_shrink_scan;
1601 journal->j_shrinker->count_objects = jbd2_journal_shrink_count;
1602 journal->j_shrinker->batch = journal->j_max_transaction_buffers;
1603 journal->j_shrinker->private_data = journal;
1604
1605 shrinker_register(journal->j_shrinker);
1599 1606
1600 return journal;1607 return journal;
1601 1608
@@ -2172,9 +2179,9 @@ int jbd2_journal_destroy(journal_t *journal) · 570 unmodified lines
2172 brelse(journal->j_sb_buffer);2179 brelse(journal->j_sb_buffer);
2173 }2180 }
2174 2181
2175 if (journal->j_shrinker.flags & SHRINKER_REGISTERED) {2182 if (journal->j_shrinker) {
2176 percpu_counter_destroy(&journal->j_checkpoint_jh_count);2183 percpu_counter_destroy(&journal->j_checkpoint_jh_count);
2177 unregister_shrinker(&journal->j_shrinker);2184 shrinker_free(journal->j_shrinker);
2178 }2185 }
2179 if (journal->j_proc_entry)2186 if (journal->j_proc_entry)
2180 jbd2_stats_proc_exit(journal);2187 jbd2_stats_proc_exit(journal);
fs/kernfs/file.c+0−49
@@ -429,60 +429,11 @@ static int kernfs_vma_access(struct vm_area_struct *vma, unsigned long addr, · 428 unmodified lines
429 return ret;429 return ret;
430}430}
431 431
432#ifdef CONFIG_NUMA
433static int kernfs_vma_set_policy(struct vm_area_struct *vma,
434 struct mempolicy *new)
435{
436 struct file *file = vma->vm_file;
437 struct kernfs_open_file *of = kernfs_of(file);
438 int ret;
439
440 if (!of->vm_ops)
441 return 0;
442
443 if (!kernfs_get_active(of->kn))
444 return -EINVAL;
445
446 ret = 0;
447 if (of->vm_ops->set_policy)
448 ret = of->vm_ops->set_policy(vma, new);
449
450 kernfs_put_active(of->kn);
451 return ret;
452}
453
454static struct mempolicy *kernfs_vma_get_policy(struct vm_area_struct *vma,
455 unsigned long addr)
456{
457 struct file *file = vma->vm_file;
458 struct kernfs_open_file *of = kernfs_of(file);
459 struct mempolicy *pol;
460
461 if (!of->vm_ops)
462 return vma->vm_policy;
463
464 if (!kernfs_get_active(of->kn))
465 return vma->vm_policy;
466
467 pol = vma->vm_policy;
468 if (of->vm_ops->get_policy)
469 pol = of->vm_ops->get_policy(vma, addr);
470
471 kernfs_put_active(of->kn);
472 return pol;
473}
474
475#endif
476
477static const struct vm_operations_struct kernfs_vm_ops = {432static const struct vm_operations_struct kernfs_vm_ops = {
478 .open = kernfs_vma_open,433 .open = kernfs_vma_open,
479 .fault = kernfs_vma_fault,434 .fault = kernfs_vma_fault,
480 .page_mkwrite = kernfs_vma_page_mkwrite,435 .page_mkwrite = kernfs_vma_page_mkwrite,
481 .access = kernfs_vma_access,436 .access = kernfs_vma_access,
482#ifdef CONFIG_NUMA
483 .set_policy = kernfs_vma_set_policy,
484 .get_policy = kernfs_vma_get_policy,
485#endif
486};437};
487 438
488static int kernfs_fop_mmap(struct file *file, struct vm_area_struct *vma)439static int kernfs_fop_mmap(struct file *file, struct vm_area_struct *vma)
fs/kernfs/mount.c+1−1
@@ -265,7 +265,7 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k · 264 unmodified lines
265 sb->s_time_gran = 1;265 sb->s_time_gran = 1;
266 266
267 /* sysfs dentries and inodes don't require IO to create */267 /* sysfs dentries and inodes don't require IO to create */
268 sb->s_shrink.seeks = 0;268 sb->s_shrink->seeks = 0;
269 269
270 /* get root inode, initialize and unlock it */270 /* get root inode, initialize and unlock it */
271 down_read(&kf_root->kernfs_rwsem);271 down_read(&kf_root->kernfs_rwsem);
fs/mbcache.c+12−10
@@ -37,7 +37,7 @@ struct mb_cache { · 36 unmodified lines
37 struct list_head c_list;37 struct list_head c_list;
38 /* Number of entries in cache */38 /* Number of entries in cache */
39 unsigned long c_entry_count;39 unsigned long c_entry_count;
40 struct shrinker c_shrink;40 struct shrinker *c_shrink;
41 /* Work for shrinking when the cache has too many entries */41 /* Work for shrinking when the cache has too many entries */
42 struct work_struct c_shrink_work;42 struct work_struct c_shrink_work;
43};43};
@@ -293,8 +293,7 @@ EXPORT_SYMBOL(mb_cache_entry_touch); · 249 unmodified lines
293static unsigned long mb_cache_count(struct shrinker *shrink,293static unsigned long mb_cache_count(struct shrinker *shrink,
294 struct shrink_control *sc)294 struct shrink_control *sc)
295{295{
296 struct mb_cache *cache = container_of(shrink, struct mb_cache,296 struct mb_cache *cache = shrink->private_data;
297 c_shrink);
298 297
299 return cache->c_entry_count;298 return cache->c_entry_count;
300}299}
@@ -333,8 +332,7 @@ static unsigned long mb_cache_shrink(struct mb_cache *cache, · 32 unmodified lines
333static unsigned long mb_cache_scan(struct shrinker *shrink,332static unsigned long mb_cache_scan(struct shrinker *shrink,
334 struct shrink_control *sc)333 struct shrink_control *sc)
335{334{
336 struct mb_cache *cache = container_of(shrink, struct mb_cache,335 struct mb_cache *cache = shrink->private_data;
337 c_shrink);
338 return mb_cache_shrink(cache, sc->nr_to_scan);336 return mb_cache_shrink(cache, sc->nr_to_scan);
339}337}
340 338
@@ -377,15 +375,19 @@ struct mb_cache *mb_cache_create(int bucket_bits) · 36 unmodified lines
377 for (i = 0; i < bucket_count; i++)375 for (i = 0; i < bucket_count; i++)
378 INIT_HLIST_BL_HEAD(&cache->c_hash[i]);376 INIT_HLIST_BL_HEAD(&cache->c_hash[i]);
379 377
380 cache->c_shrink.count_objects = mb_cache_count;378 cache->c_shrink = shrinker_alloc(0, "mbcache-shrinker");
381 cache->c_shrink.scan_objects = mb_cache_scan;379 if (!cache->c_shrink) {
382 cache->c_shrink.seeks = DEFAULT_SEEKS;
383 if (register_shrinker(&cache->c_shrink, "mbcache-shrinker")) {
384 kfree(cache->c_hash);380 kfree(cache->c_hash);
385 kfree(cache);381 kfree(cache);
386 goto err_out;382 goto err_out;
387 }383 }
388 384
385 cache->c_shrink->count_objects = mb_cache_count;
386 cache->c_shrink->scan_objects = mb_cache_scan;
387 cache->c_shrink->private_data = cache;
388
389 shrinker_register(cache->c_shrink);
390
389 INIT_WORK(&cache->c_shrink_work, mb_cache_shrink_worker);391 INIT_WORK(&cache->c_shrink_work, mb_cache_shrink_worker);
390 392
391 return cache;393 return cache;
@@ -406,7 +408,7 @@ void mb_cache_destroy(struct mb_cache *cache) · 14 unmodified lines
406{408{
407 struct mb_cache_entry *entry, *next;409 struct mb_cache_entry *entry, *next;
408 410
409 unregister_shrinker(&cache->c_shrink);411 shrinker_free(cache->c_shrink);
410 412
411 /*413 /*
412 * We don't bother with any locking. Cache must not be used at this414 * We don't bother with any locking. Cache must not be used at this
fs/mpage.c+1−2
@@ -119,8 +119,7 @@ static void map_buffer_to_folio(struct folio *folio, struct buffer_head *bh, · 118 unmodified lines
119 folio_mark_uptodate(folio);119 folio_mark_uptodate(folio);
120 return;120 return;
121 }121 }
122 create_empty_buffers(&folio->page, i_blocksize(inode), 0);122 head = create_empty_buffers(folio, i_blocksize(inode), 0);
123 head = folio_buffers(folio);
124 }123 }
125 124
126 page_bh = head;125 page_bh = head;
fs/nfs/nfs42xattr.c+44−43
@@ -796,28 +796,9 @@ static unsigned long nfs4_xattr_cache_scan(struct shrinker *shrink, · 795 unmodified lines
796static unsigned long nfs4_xattr_entry_scan(struct shrinker *shrink,796static unsigned long nfs4_xattr_entry_scan(struct shrinker *shrink,
797 struct shrink_control *sc);797 struct shrink_control *sc);
798 798
799static struct shrinker nfs4_xattr_cache_shrinker = {799static struct shrinker *nfs4_xattr_cache_shrinker;
800 .count_objects = nfs4_xattr_cache_count,800static struct shrinker *nfs4_xattr_entry_shrinker;
801 .scan_objects = nfs4_xattr_cache_scan,801static struct shrinker *nfs4_xattr_large_entry_shrinker;
802 .seeks = DEFAULT_SEEKS,
803 .flags = SHRINKER_MEMCG_AWARE,
804};
805
806static struct shrinker nfs4_xattr_entry_shrinker = {
807 .count_objects = nfs4_xattr_entry_count,
808 .scan_objects = nfs4_xattr_entry_scan,
809 .seeks = DEFAULT_SEEKS,
810 .batch = 512,
811 .flags = SHRINKER_MEMCG_AWARE,
812};
813
814static struct shrinker nfs4_xattr_large_entry_shrinker = {
815 .count_objects = nfs4_xattr_entry_count,
816 .scan_objects = nfs4_xattr_entry_scan,
817 .seeks = 1,
818 .batch = 512,
819 .flags = SHRINKER_MEMCG_AWARE,
820};
821 802
822static enum lru_status803static enum lru_status
823cache_lru_isolate(struct list_head *item,804cache_lru_isolate(struct list_head *item,
@@ -943,7 +924,7 @@ nfs4_xattr_entry_scan(struct shrinker *shrink, struct shrink_control *sc) · 119 unmodified lines
943 struct nfs4_xattr_entry *entry;924 struct nfs4_xattr_entry *entry;
944 struct list_lru *lru;925 struct list_lru *lru;
945 926
946 lru = (shrink == &nfs4_xattr_large_entry_shrinker) ?927 lru = (shrink == nfs4_xattr_large_entry_shrinker) ?
947 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;928 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;
948 929
949 freed = list_lru_shrink_walk(lru, sc, entry_lru_isolate, &dispose);930 freed = list_lru_shrink_walk(lru, sc, entry_lru_isolate, &dispose);
@@ -971,7 +952,7 @@ nfs4_xattr_entry_count(struct shrinker *shrink, struct shrink_control *sc) · 21 unmodified lines
971 unsigned long count;952 unsigned long count;
972 struct list_lru *lru;953 struct list_lru *lru;
973 954
974 lru = (shrink == &nfs4_xattr_large_entry_shrinker) ?955 lru = (shrink == nfs4_xattr_large_entry_shrinker) ?
975 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;956 &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru;
976 957
977 count = list_lru_shrink_count(lru, sc);958 count = list_lru_shrink_count(lru, sc);
@@ -991,18 +972,34 @@ static void nfs4_xattr_cache_init_once(void *p) · 13 unmodified lines
991 INIT_LIST_HEAD(&cache->dispose);972 INIT_LIST_HEAD(&cache->dispose);
992}973}
993 974
994static int nfs4_xattr_shrinker_init(struct shrinker *shrinker,975typedef unsigned long (*count_objects_cb)(struct shrinker *s,
995 struct list_lru *lru, const char *name)976 struct shrink_control *sc);
977typedef unsigned long (*scan_objects_cb)(struct shrinker *s,
978 struct shrink_control *sc);
979
980static int __init nfs4_xattr_shrinker_init(struct shrinker **shrinker,
981 struct list_lru *lru, const char *name,
982 count_objects_cb count,
983 scan_objects_cb scan, long batch, int seeks)
996{984{
997 int ret = 0;985 int ret;
998 986
999 ret = register_shrinker(shrinker, name);987 *shrinker = shrinker_alloc(SHRINKER_MEMCG_AWARE, name);
1000 if (ret)988 if (!*shrinker)
989 return -ENOMEM;
990
991 ret = list_lru_init_memcg(lru, *shrinker);
992 if (ret) {
993 shrinker_free(*shrinker);
1001 return ret;994 return ret;
995 }
1002 996
1003 ret = list_lru_init_memcg(lru, shrinker);997 (*shrinker)->count_objects = count;
1004 if (ret)998 (*shrinker)->scan_objects = scan;
1005 unregister_shrinker(shrinker);999 (*shrinker)->batch = batch;
1000 (*shrinker)->seeks = seeks;
1001
1002 shrinker_register(*shrinker);
1006 1003
1007 return ret;1004 return ret;
1008}1005}
@@ -1010,7 +1007,7 @@ static int nfs4_xattr_shrinker_init(struct shrinker *shrinker, · 1 unmodified line
1010static void nfs4_xattr_shrinker_destroy(struct shrinker *shrinker,1007static void nfs4_xattr_shrinker_destroy(struct shrinker *shrinker,
1011 struct list_lru *lru)1008 struct list_lru *lru)
1012{1009{
1013 unregister_shrinker(shrinker);1010 shrinker_free(shrinker);
1014 list_lru_destroy(lru);1011 list_lru_destroy(lru);
1015}1012}
1016 1013
@@ -1026,27 +1023,31 @@ int __init nfs4_xattr_cache_init(void) · 9 unmodified lines
1026 return -ENOMEM;1023 return -ENOMEM;
1027 1024
1028 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_cache_shrinker,1025 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_cache_shrinker,
1029 &nfs4_xattr_cache_lru,1026 &nfs4_xattr_cache_lru, "nfs-xattr_cache",
1030 "nfs-xattr_cache");1027 nfs4_xattr_cache_count,
1028 nfs4_xattr_cache_scan, 0, DEFAULT_SEEKS);
1031 if (ret)1029 if (ret)
1032 goto out1;1030 goto out1;
1033 1031
1034 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_entry_shrinker,1032 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_entry_shrinker,
1035 &nfs4_xattr_entry_lru,1033 &nfs4_xattr_entry_lru, "nfs-xattr_entry",
1036 "nfs-xattr_entry");1034 nfs4_xattr_entry_count,
1035 nfs4_xattr_entry_scan, 512, DEFAULT_SEEKS);
1037 if (ret)1036 if (ret)
1038 goto out2;1037 goto out2;
1039 1038
1040 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_large_entry_shrinker,1039 ret = nfs4_xattr_shrinker_init(&nfs4_xattr_large_entry_shrinker,
1041 &nfs4_xattr_large_entry_lru,1040 &nfs4_xattr_large_entry_lru,
1042 "nfs-xattr_large_entry");1041 "nfs-xattr_large_entry",
1042 nfs4_xattr_entry_count,
1043 nfs4_xattr_entry_scan, 512, 1);
1043 if (!ret)1044 if (!ret)
1044 return 0;1045 return 0;
1045 1046
1046 nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker,1047 nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker,
1047 &nfs4_xattr_entry_lru);1048 &nfs4_xattr_entry_lru);
1048out2:1049out2:
1049 nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker,1050 nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker,
1050 &nfs4_xattr_cache_lru);1051 &nfs4_xattr_cache_lru);
1051out1:1052out1:
1052 kmem_cache_destroy(nfs4_xattr_cache_cachep);1053 kmem_cache_destroy(nfs4_xattr_cache_cachep);
@@ -1056,11 +1057,11 @@ out1: · 3 unmodified lines
1056 1057
1057void nfs4_xattr_cache_exit(void)1058void nfs4_xattr_cache_exit(void)
1058{1059{
1059 nfs4_xattr_shrinker_destroy(&nfs4_xattr_large_entry_shrinker,1060 nfs4_xattr_shrinker_destroy(nfs4_xattr_large_entry_shrinker,
1060 &nfs4_xattr_large_entry_lru);1061 &nfs4_xattr_large_entry_lru);
1061 nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker,1062 nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker,
1062 &nfs4_xattr_entry_lru);1063 &nfs4_xattr_entry_lru);
1063 nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker,1064 nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker,
1064 &nfs4_xattr_cache_lru);1065 &nfs4_xattr_cache_lru);
1065 kmem_cache_destroy(nfs4_xattr_cache_cachep);1066 kmem_cache_destroy(nfs4_xattr_cache_cachep);
1066}1067}
fs/nfs/super.c+13−8
@@ -129,11 +129,7 @@ static void nfs_ssc_unregister_ops(void) · 128 unmodified lines
129}129}
130#endif /* CONFIG_NFS_V4_2 */130#endif /* CONFIG_NFS_V4_2 */
131 131
132static struct shrinker acl_shrinker = {132static struct shrinker *acl_shrinker;
133 .count_objects = nfs_access_cache_count,
134 .scan_objects = nfs_access_cache_scan,
135 .seeks = DEFAULT_SEEKS,
136};
137 133
138/*134/*
139 * Register the NFS filesystems135 * Register the NFS filesystems
@@ -153,9 +149,18 @@ int __init register_nfs_fs(void) · 13 unmodified lines
153 ret = nfs_register_sysctl();149 ret = nfs_register_sysctl();
154 if (ret < 0)150 if (ret < 0)
155 goto error_2;151 goto error_2;
156 ret = register_shrinker(&acl_shrinker, "nfs-acl");152
157 if (ret < 0)153 acl_shrinker = shrinker_alloc(0, "nfs-acl");
154 if (!acl_shrinker) {
155 ret = -ENOMEM;
158 goto error_3;156 goto error_3;
157 }
158
159 acl_shrinker->count_objects = nfs_access_cache_count;
160 acl_shrinker->scan_objects = nfs_access_cache_scan;
161
162 shrinker_register(acl_shrinker);
163
159#ifdef CONFIG_NFS_V4_2164#ifdef CONFIG_NFS_V4_2
160 nfs_ssc_register_ops();165 nfs_ssc_register_ops();
161#endif166#endif
@@ -175,7 +180,7 @@ error_0: · 13 unmodified lines
175 */180 */
176void __exit unregister_nfs_fs(void)181void __exit unregister_nfs_fs(void)
177{182{
178 unregister_shrinker(&acl_shrinker);183 shrinker_free(acl_shrinker);
179 nfs_unregister_sysctl();184 nfs_unregister_sysctl();
180 unregister_nfs4_fs();185 unregister_nfs4_fs();
181#ifdef CONFIG_NFS_V4_2186#ifdef CONFIG_NFS_V4_2
fs/nfsd/filecache.c+13−10
@@ -521,11 +521,7 @@ nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) · 520 unmodified lines
521 return ret;521 return ret;
522}522}
523 523
524static struct shrinker nfsd_file_shrinker = {524static struct shrinker *nfsd_file_shrinker;
525 .scan_objects = nfsd_file_lru_scan,
526 .count_objects = nfsd_file_lru_count,
527 .seeks = 1,
528};
529 525
530/**526/**
531 * nfsd_file_cond_queue - conditionally unhash and queue a nfsd_file527 * nfsd_file_cond_queue - conditionally unhash and queue a nfsd_file
@@ -746,12 +742,19 @@ nfsd_file_cache_init(void) · 214 unmodified lines
746 goto out_err;742 goto out_err;
747 }743 }
748 744
749 ret = register_shrinker(&nfsd_file_shrinker, "nfsd-filecache");745 nfsd_file_shrinker = shrinker_alloc(0, "nfsd-filecache");
750 if (ret) {746 if (!nfsd_file_shrinker) {
751 pr_err("nfsd: failed to register nfsd_file_shrinker: %d\n", ret);747 ret = -ENOMEM;
748 pr_err("nfsd: failed to allocate nfsd_file_shrinker\n");
752 goto out_lru;749 goto out_lru;
753 }750 }
754 751
752 nfsd_file_shrinker->count_objects = nfsd_file_lru_count;
753 nfsd_file_shrinker->scan_objects = nfsd_file_lru_scan;
754 nfsd_file_shrinker->seeks = 1;
755
756 shrinker_register(nfsd_file_shrinker);
757
755 ret = lease_register_notifier(&nfsd_file_lease_notifier);758 ret = lease_register_notifier(&nfsd_file_lease_notifier);
756 if (ret) {759 if (ret) {
757 pr_err("nfsd: unable to register lease notifier: %d\n", ret);760 pr_err("nfsd: unable to register lease notifier: %d\n", ret);
@@ -774,7 +777,7 @@ out: · 16 unmodified lines
774out_notifier:777out_notifier:
775 lease_unregister_notifier(&nfsd_file_lease_notifier);778 lease_unregister_notifier(&nfsd_file_lease_notifier);
776out_shrinker:779out_shrinker:
777 unregister_shrinker(&nfsd_file_shrinker);780 shrinker_free(nfsd_file_shrinker);
778out_lru:781out_lru:
779 list_lru_destroy(&nfsd_file_lru);782 list_lru_destroy(&nfsd_file_lru);
780out_err:783out_err:
@@ -891,7 +894,7 @@ nfsd_file_cache_shutdown(void) · 110 unmodified lines
891 return;894 return;
892 895
893 lease_unregister_notifier(&nfsd_file_lease_notifier);896 lease_unregister_notifier(&nfsd_file_lease_notifier);
894 unregister_shrinker(&nfsd_file_shrinker);897 shrinker_free(nfsd_file_shrinker);
895 /*898 /*
896 * make sure all callers of nfsd_file_lru_cb are done before899 * make sure all callers of nfsd_file_lru_cb are done before
897 * calling nfsd_file_cache_purge900 * calling nfsd_file_cache_purge
fs/nfsd/netns.h+2−2
@@ -177,7 +177,7 @@ struct nfsd_net { · 176 unmodified lines
177 /* size of cache when we saw the longest hash chain */177 /* size of cache when we saw the longest hash chain */
178 unsigned int longest_chain_cachesize;178 unsigned int longest_chain_cachesize;
179 179
180 struct shrinker nfsd_reply_cache_shrinker;180 struct shrinker *nfsd_reply_cache_shrinker;
181 181
182 /* tracking server-to-server copy mounts */182 /* tracking server-to-server copy mounts */
183 spinlock_t nfsd_ssc_lock;183 spinlock_t nfsd_ssc_lock;
@@ -195,7 +195,7 @@ struct nfsd_net { · 11 unmodified lines
195 int nfs4_max_clients;195 int nfs4_max_clients;
196 196
197 atomic_t nfsd_courtesy_clients;197 atomic_t nfsd_courtesy_clients;
198 struct shrinker nfsd_client_shrinker;198 struct shrinker *nfsd_client_shrinker;
199 struct work_struct nfsd_shrinker_work;199 struct work_struct nfsd_shrinker_work;
200};200};
201 201
fs/nfsd/nfs4state.c+11−8
@@ -4452,8 +4452,7 @@ static unsigned long · 4451 unmodified lines
4452nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)4452nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc)
4453{4453{
4454 int count;4454 int count;
4455 struct nfsd_net *nn = container_of(shrink,4455 struct nfsd_net *nn = shrink->private_data;
4456 struct nfsd_net, nfsd_client_shrinker);
4457 4456
4458 count = atomic_read(&nn->nfsd_courtesy_clients);4457 count = atomic_read(&nn->nfsd_courtesy_clients);
4459 if (!count)4458 if (!count)
@@ -8235,12 +8234,16 @@ static int nfs4_state_create_net(struct net *net) · 3775 unmodified lines
8235 INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker);8234 INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker);
8236 get_net(net);8235 get_net(net);
8237 8236
8238 nn->nfsd_client_shrinker.scan_objects = nfsd4_state_shrinker_scan;8237 nn->nfsd_client_shrinker = shrinker_alloc(0, "nfsd-client");
8239 nn->nfsd_client_shrinker.count_objects = nfsd4_state_shrinker_count;8238 if (!nn->nfsd_client_shrinker)
8240 nn->nfsd_client_shrinker.seeks = DEFAULT_SEEKS;
8241
8242 if (register_shrinker(&nn->nfsd_client_shrinker, "nfsd-client"))
8243 goto err_shrinker;8239 goto err_shrinker;
8240
8241 nn->nfsd_client_shrinker->scan_objects = nfsd4_state_shrinker_scan;
8242 nn->nfsd_client_shrinker->count_objects = nfsd4_state_shrinker_count;
8243 nn->nfsd_client_shrinker->private_data = nn;
8244
8245 shrinker_register(nn->nfsd_client_shrinker);
8246
8244 return 0;8247 return 0;
8245 8248
8246err_shrinker:8249err_shrinker:
@@ -8338,7 +8341,7 @@ nfs4_state_shutdown_net(struct net *net) · 91 unmodified lines
8338 struct list_head *pos, *next, reaplist;8341 struct list_head *pos, *next, reaplist;
8339 struct nfsd_net *nn = net_generic(net, nfsd_net_id);8342 struct nfsd_net *nn = net_generic(net, nfsd_net_id);
8340 8343
8341 unregister_shrinker(&nn->nfsd_client_shrinker);8344 shrinker_free(nn->nfsd_client_shrinker);
8342 cancel_work(&nn->nfsd_shrinker_work);8345 cancel_work(&nn->nfsd_shrinker_work);
8343 cancel_delayed_work_sync(&nn->laundromat_work);8346 cancel_delayed_work_sync(&nn->laundromat_work);
8344 locks_end_grace(&nn->nfsd4_manager);8347 locks_end_grace(&nn->nfsd4_manager);
fs/nfsd/nfscache.c+16−15
@@ -201,26 +201,29 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 200 unmodified lines
201{201{
202 unsigned int hashsize;202 unsigned int hashsize;
203 unsigned int i;203 unsigned int i;
204 int status = 0;
205 204
206 nn->max_drc_entries = nfsd_cache_size_limit();205 nn->max_drc_entries = nfsd_cache_size_limit();
207 atomic_set(&nn->num_drc_entries, 0);206 atomic_set(&nn->num_drc_entries, 0);
208 hashsize = nfsd_hashsize(nn->max_drc_entries);207 hashsize = nfsd_hashsize(nn->max_drc_entries);
209 nn->maskbits = ilog2(hashsize);208 nn->maskbits = ilog2(hashsize);
210 209
211 nn->nfsd_reply_cache_shrinker.scan_objects = nfsd_reply_cache_scan;
212 nn->nfsd_reply_cache_shrinker.count_objects = nfsd_reply_cache_count;
213 nn->nfsd_reply_cache_shrinker.seeks = 1;
214 status = register_shrinker(&nn->nfsd_reply_cache_shrinker,
215 "nfsd-reply:%s", nn->nfsd_name);
216 if (status)
217 return status;
218
219 nn->drc_hashtbl = kvzalloc(array_size(hashsize,210 nn->drc_hashtbl = kvzalloc(array_size(hashsize,
220 sizeof(*nn->drc_hashtbl)), GFP_KERNEL);211 sizeof(*nn->drc_hashtbl)), GFP_KERNEL);
221 if (!nn->drc_hashtbl)212 if (!nn->drc_hashtbl)
213 return -ENOMEM;
214
215 nn->nfsd_reply_cache_shrinker = shrinker_alloc(0, "nfsd-reply:%s",
216 nn->nfsd_name);
217 if (!nn->nfsd_reply_cache_shrinker)
222 goto out_shrinker;218 goto out_shrinker;
223 219
220 nn->nfsd_reply_cache_shrinker->scan_objects = nfsd_reply_cache_scan;
221 nn->nfsd_reply_cache_shrinker->count_objects = nfsd_reply_cache_count;
222 nn->nfsd_reply_cache_shrinker->seeks = 1;
223 nn->nfsd_reply_cache_shrinker->private_data = nn;
224
225 shrinker_register(nn->nfsd_reply_cache_shrinker);
226
224 for (i = 0; i < hashsize; i++) {227 for (i = 0; i < hashsize; i++) {
225 INIT_LIST_HEAD(&nn->drc_hashtbl[i].lru_head);228 INIT_LIST_HEAD(&nn->drc_hashtbl[i].lru_head);
226 spin_lock_init(&nn->drc_hashtbl[i].cache_lock);229 spin_lock_init(&nn->drc_hashtbl[i].cache_lock);
@@ -229,7 +232,7 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 2 unmodified lines
229 232
230 return 0;233 return 0;
231out_shrinker:234out_shrinker:
232 unregister_shrinker(&nn->nfsd_reply_cache_shrinker);235 kvfree(nn->drc_hashtbl);
233 printk(KERN_ERR "nfsd: failed to allocate reply cache\n");236 printk(KERN_ERR "nfsd: failed to allocate reply cache\n");
234 return -ENOMEM;237 return -ENOMEM;
235}238}
@@ -239,7 +242,7 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn) · 3 unmodified lines
239 struct nfsd_cacherep *rp;242 struct nfsd_cacherep *rp;
240 unsigned int i;243 unsigned int i;
241 244
242 unregister_shrinker(&nn->nfsd_reply_cache_shrinker);245 shrinker_free(nn->nfsd_reply_cache_shrinker);
243 246
244 for (i = 0; i < nn->drc_hashsize; i++) {247 for (i = 0; i < nn->drc_hashsize; i++) {
245 struct list_head *head = &nn->drc_hashtbl[i].lru_head;248 struct list_head *head = &nn->drc_hashtbl[i].lru_head;
@@ -323,8 +326,7 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, · 77 unmodified lines
323static unsigned long326static unsigned long
324nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)327nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)
325{328{
326 struct nfsd_net *nn = container_of(shrink,329 struct nfsd_net *nn = shrink->private_data;
327 struct nfsd_net, nfsd_reply_cache_shrinker);
328 330
329 return atomic_read(&nn->num_drc_entries);331 return atomic_read(&nn->num_drc_entries);
330}332}
@@ -343,8 +345,7 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) · 12 unmodified lines
343static unsigned long345static unsigned long
344nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)346nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
345{347{
346 struct nfsd_net *nn = container_of(shrink,348 struct nfsd_net *nn = shrink->private_data;
347 struct nfsd_net, nfsd_reply_cache_shrinker);
348 unsigned long freed = 0;349 unsigned long freed = 0;
349 LIST_HEAD(dispose);350 LIST_HEAD(dispose);
350 unsigned int i;351 unsigned int i;
fs/nilfs2/mdt.c+34−32
@@ -356,30 +356,28 @@ int nilfs_mdt_delete_block(struct inode *inode, unsigned long block) · 355 unmodified lines
356 */356 */
357int nilfs_mdt_forget_block(struct inode *inode, unsigned long block)357int nilfs_mdt_forget_block(struct inode *inode, unsigned long block)
358{358{
359 pgoff_t index = (pgoff_t)block >>359 pgoff_t index = block >> (PAGE_SHIFT - inode->i_blkbits);
360 (PAGE_SHIFT - inode->i_blkbits);360 struct folio *folio;
361 struct page *page;361 struct buffer_head *bh;
362 unsigned long first_block;
363 int ret = 0;362 int ret = 0;
364 int still_dirty;363 int still_dirty;
365 364
366 page = find_lock_page(inode->i_mapping, index);365 folio = filemap_lock_folio(inode->i_mapping, index);
367 if (!page)366 if (IS_ERR(folio))
368 return -ENOENT;367 return -ENOENT;
369 368
370 wait_on_page_writeback(page);369 folio_wait_writeback(folio);
371
372 first_block = (unsigned long)index <<
373 (PAGE_SHIFT - inode->i_blkbits);
374 if (page_has_buffers(page)) {
375 struct buffer_head *bh;
376 370
377 bh = nilfs_page_get_nth_block(page, block - first_block);371 bh = folio_buffers(folio);
372 if (bh) {
373 unsigned long first_block = index <<
374 (PAGE_SHIFT - inode->i_blkbits);
375 bh = get_nth_bh(bh, block - first_block);
378 nilfs_forget_buffer(bh);376 nilfs_forget_buffer(bh);
379 }377 }
380 still_dirty = PageDirty(page);378 still_dirty = folio_test_dirty(folio);
381 unlock_page(page);379 folio_unlock(folio);
382 put_page(page);380 folio_put(folio);
383 381
384 if (still_dirty ||382 if (still_dirty ||
385 invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0)383 invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0)
@@ -560,17 +558,19 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 174 unmodified lines
560{558{
561 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;559 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;
562 struct buffer_head *bh_frozen;560 struct buffer_head *bh_frozen;
563 struct page *page;561 struct folio *folio;
564 int blkbits = inode->i_blkbits;562 int blkbits = inode->i_blkbits;
565 563
566 page = grab_cache_page(shadow->inode->i_mapping, bh->b_folio->index);564 folio = filemap_grab_folio(shadow->inode->i_mapping,
567 if (!page)565 bh->b_folio->index);
568 return -ENOMEM;566 if (IS_ERR(folio))
567 return PTR_ERR(folio);
569 568
570 if (!page_has_buffers(page))569 bh_frozen = folio_buffers(folio);
571 create_empty_buffers(page, 1 << blkbits, 0);570 if (!bh_frozen)
571 bh_frozen = create_empty_buffers(folio, 1 << blkbits, 0);
572 572
573 bh_frozen = nilfs_page_get_nth_block(page, bh_offset(bh) >> blkbits);573 bh_frozen = get_nth_bh(bh_frozen, bh_offset(bh) >> blkbits);
574 574
575 if (!buffer_uptodate(bh_frozen))575 if (!buffer_uptodate(bh_frozen))
576 nilfs_copy_buffer(bh_frozen, bh);576 nilfs_copy_buffer(bh_frozen, bh);
@@ -582,8 +582,8 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 5 unmodified lines
582 brelse(bh_frozen); /* already frozen */582 brelse(bh_frozen); /* already frozen */
583 }583 }
584 584
585 unlock_page(page);585 folio_unlock(folio);
586 put_page(page);586 folio_put(folio);
587 return 0;587 return 0;
588}588}
589 589
@@ -592,17 +592,19 @@ nilfs_mdt_get_frozen_buffer(struct inode *inode, struct buffer_head *bh) · 2 unmodified lines
592{592{
593 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;593 struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow;
594 struct buffer_head *bh_frozen = NULL;594 struct buffer_head *bh_frozen = NULL;
595 struct page *page;595 struct folio *folio;
596 int n;596 int n;
597 597
598 page = find_lock_page(shadow->inode->i_mapping, bh->b_folio->index);598 folio = filemap_lock_folio(shadow->inode->i_mapping,
599 if (page) {599 bh->b_folio->index);
600 if (page_has_buffers(page)) {600 if (!IS_ERR(folio)) {
601 bh_frozen = folio_buffers(folio);
602 if (bh_frozen) {
601 n = bh_offset(bh) >> inode->i_blkbits;603 n = bh_offset(bh) >> inode->i_blkbits;
602 bh_frozen = nilfs_page_get_nth_block(page, n);604 bh_frozen = get_nth_bh(bh_frozen, n);
603 }605 }
604 unlock_page(page);606 folio_unlock(folio);
605 put_page(page);607 folio_put(folio);
606 }608 }
607 return bh_frozen;609 return bh_frozen;
608}610}
fs/nilfs2/page.c+39−37
@@ -25,19 +25,19 @@ · 24 unmodified lines
25 (BIT(BH_Uptodate) | BIT(BH_Mapped) | BIT(BH_NILFS_Node) | \25 (BIT(BH_Uptodate) | BIT(BH_Mapped) | BIT(BH_NILFS_Node) | \
26 BIT(BH_NILFS_Volatile) | BIT(BH_NILFS_Checked))26 BIT(BH_NILFS_Volatile) | BIT(BH_NILFS_Checked))
27 27
28static struct buffer_head *28static struct buffer_head *__nilfs_get_folio_block(struct folio *folio,
29__nilfs_get_page_block(struct page *page, unsigned long block, pgoff_t index,29 unsigned long block, pgoff_t index, int blkbits,
30 int blkbits, unsigned long b_state)30 unsigned long b_state)
31 31
32{32{
33 unsigned long first_block;33 unsigned long first_block;
34 struct buffer_head *bh;34 struct buffer_head *bh = folio_buffers(folio);
35 35
36 if (!page_has_buffers(page))36 if (!bh)
37 create_empty_buffers(page, 1 << blkbits, b_state);37 bh = create_empty_buffers(folio, 1 << blkbits, b_state);
38 38
39 first_block = (unsigned long)index << (PAGE_SHIFT - blkbits);39 first_block = (unsigned long)index << (PAGE_SHIFT - blkbits);
40 bh = nilfs_page_get_nth_block(page, block - first_block);40 bh = get_nth_bh(bh, block - first_block);
41 41
42 touch_buffer(bh);42 touch_buffer(bh);
43 wait_on_buffer(bh);43 wait_on_buffer(bh);
@@ -51,17 +51,17 @@ struct buffer_head *nilfs_grab_buffer(struct inode *inode, · 7 unmodified lines
51{51{
52 int blkbits = inode->i_blkbits;52 int blkbits = inode->i_blkbits;
53 pgoff_t index = blkoff >> (PAGE_SHIFT - blkbits);53 pgoff_t index = blkoff >> (PAGE_SHIFT - blkbits);
54 struct page *page;54 struct folio *folio;
55 struct buffer_head *bh;55 struct buffer_head *bh;
56 56
57 page = grab_cache_page(mapping, index);57 folio = filemap_grab_folio(mapping, index);
58 if (unlikely(!page))58 if (IS_ERR(folio))
59 return NULL;59 return NULL;
60 60
61 bh = __nilfs_get_page_block(page, blkoff, index, blkbits, b_state);61 bh = __nilfs_get_folio_block(folio, blkoff, index, blkbits, b_state);
62 if (unlikely(!bh)) {62 if (unlikely(!bh)) {
63 unlock_page(page);63 folio_unlock(folio);
64 put_page(page);64 folio_put(folio);
65 return NULL;65 return NULL;
66 }66 }
67 return bh;67 return bh;
@@ -184,30 +184,32 @@ void nilfs_page_bug(struct page *page) · 116 unmodified lines
184}184}
185 185
186/**186/**
187 * nilfs_copy_page -- copy the page with buffers187 * nilfs_copy_folio -- copy the folio with buffers
188 * @dst: destination page188 * @dst: destination folio
189 * @src: source page189 * @src: source folio
190 * @copy_dirty: flag whether to copy dirty states on the page's buffer heads.190 * @copy_dirty: flag whether to copy dirty states on the folio's buffer heads.
191 *191 *
192 * This function is for both data pages and btnode pages. The dirty flag192 * This function is for both data folios and btnode folios. The dirty flag
193 * should be treated by caller. The page must not be under i/o.193 * should be treated by caller. The folio must not be under i/o.
194 * Both src and dst page must be locked194 * Both src and dst folio must be locked
195 */195 */
196static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty)196static void nilfs_copy_folio(struct folio *dst, struct folio *src,
197 bool copy_dirty)
197{198{
198 struct buffer_head *dbh, *dbufs, *sbh;199 struct buffer_head *dbh, *dbufs, *sbh;
199 unsigned long mask = NILFS_BUFFER_INHERENT_BITS;200 unsigned long mask = NILFS_BUFFER_INHERENT_BITS;
200 201
201 BUG_ON(PageWriteback(dst));202 BUG_ON(folio_test_writeback(dst));
202 203
203 sbh = page_buffers(src);204 sbh = folio_buffers(src);
204 if (!page_has_buffers(dst))205 dbh = folio_buffers(dst);
205 create_empty_buffers(dst, sbh->b_size, 0);206 if (!dbh)
207 dbh = create_empty_buffers(dst, sbh->b_size, 0);
206 208
207 if (copy_dirty)209 if (copy_dirty)
208 mask |= BIT(BH_Dirty);210 mask |= BIT(BH_Dirty);
209 211
210 dbh = dbufs = page_buffers(dst);212 dbufs = dbh;
211 do {213 do {
212 lock_buffer(sbh);214 lock_buffer(sbh);
213 lock_buffer(dbh);215 lock_buffer(dbh);
@@ -218,16 +220,16 @@ static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty) · 4 unmodified lines
218 dbh = dbh->b_this_page;220 dbh = dbh->b_this_page;
219 } while (dbh != dbufs);221 } while (dbh != dbufs);
220 222
221 copy_highpage(dst, src);223 folio_copy(dst, src);
222 224
223 if (PageUptodate(src) && !PageUptodate(dst))225 if (folio_test_uptodate(src) && !folio_test_uptodate(dst))
224 SetPageUptodate(dst);226 folio_mark_uptodate(dst);
225 else if (!PageUptodate(src) && PageUptodate(dst))227 else if (!folio_test_uptodate(src) && folio_test_uptodate(dst))
226 ClearPageUptodate(dst);228 folio_clear_uptodate(dst);
227 if (PageMappedToDisk(src) && !PageMappedToDisk(dst))229 if (folio_test_mappedtodisk(src) && !folio_test_mappedtodisk(dst))
228 SetPageMappedToDisk(dst);230 folio_set_mappedtodisk(dst);
229 else if (!PageMappedToDisk(src) && PageMappedToDisk(dst))231 else if (!folio_test_mappedtodisk(src) && folio_test_mappedtodisk(dst))
230 ClearPageMappedToDisk(dst);232 folio_clear_mappedtodisk(dst);
231 233
232 do {234 do {
233 unlock_buffer(sbh);235 unlock_buffer(sbh);
@@ -269,7 +271,7 @@ repeat: · 35 unmodified lines
269 NILFS_PAGE_BUG(&folio->page,271 NILFS_PAGE_BUG(&folio->page,
270 "found empty page in dat page cache");272 "found empty page in dat page cache");
271 273
272 nilfs_copy_page(&dfolio->page, &folio->page, 1);274 nilfs_copy_folio(dfolio, folio, true);
273 filemap_dirty_folio(folio_mapping(dfolio), dfolio);275 filemap_dirty_folio(folio_mapping(dfolio), dfolio);
274 276
275 folio_unlock(dfolio);277 folio_unlock(dfolio);
@@ -314,7 +316,7 @@ repeat: · 38 unmodified lines
314 if (!IS_ERR(dfolio)) {316 if (!IS_ERR(dfolio)) {
315 /* overwrite existing folio in the destination cache */317 /* overwrite existing folio in the destination cache */
316 WARN_ON(folio_test_dirty(dfolio));318 WARN_ON(folio_test_dirty(dfolio));
317 nilfs_copy_page(&dfolio->page, &folio->page, 0);319 nilfs_copy_folio(dfolio, folio, false);
318 folio_unlock(dfolio);320 folio_unlock(dfolio);
319 folio_put(dfolio);321 folio_put(dfolio);
320 /* Do we not need to remove folio from smap here? */322 /* Do we not need to remove folio from smap here? */
fs/nilfs2/page.h+0−11
@@ -52,15 +52,4 @@ unsigned long nilfs_find_uncommitted_extent(struct inode *inode, · 51 unmodified lines
52#define NILFS_PAGE_BUG(page, m, a...) \52#define NILFS_PAGE_BUG(page, m, a...) \
53 do { nilfs_page_bug(page); BUG(); } while (0)53 do { nilfs_page_bug(page); BUG(); } while (0)
54 54
55static inline struct buffer_head *
56nilfs_page_get_nth_block(struct page *page, unsigned int count)
57{
58 struct buffer_head *bh = page_buffers(page);
59
60 while (count-- > 0)
61 bh = bh->b_this_page;
62 get_bh(bh);
63 return bh;
64}
65
66#endif /* _NILFS_PAGE_H */55#endif /* _NILFS_PAGE_H */
fs/nilfs2/segment.c+3−4
@@ -731,10 +731,9 @@ static size_t nilfs_lookup_dirty_data_buffers(struct inode *inode, · 730 unmodified lines
731 continue;731 continue;
732 }732 }
733 head = folio_buffers(folio);733 head = folio_buffers(folio);
734 if (!head) {734 if (!head)
735 create_empty_buffers(&folio->page, i_blocksize(inode), 0);735 head = create_empty_buffers(folio,
736 head = folio_buffers(folio);736 i_blocksize(inode), 0);
737 }
738 folio_unlock(folio);737 folio_unlock(folio);
739 738
740 bh = head;739 bh = head;
fs/ntfs/aops.c+119−136
@@ -145,13 +145,12 @@ still_busy: · 144 unmodified lines
145}145}
146 146
147/**147/**
148 * ntfs_read_block - fill a @page of an address space with data148 * ntfs_read_block - fill a @folio of an address space with data
149 * @page: page cache page to fill with data149 * @folio: page cache folio to fill with data
150 *150 *
151 * Fill the page @page of the address space belonging to the @page->host inode.
152 * We read each buffer asynchronously and when all buffers are read in, our io151 * We read each buffer asynchronously and when all buffers are read in, our io
153 * completion handler ntfs_end_buffer_read_async(), if required, automatically152 * completion handler ntfs_end_buffer_read_async(), if required, automatically
154 * applies the mst fixups to the page before finally marking it uptodate and153 * applies the mst fixups to the folio before finally marking it uptodate and
155 * unlocking it.154 * unlocking it.
156 *155 *
157 * We only enforce allocated_size limit because i_size is checked for in156 * We only enforce allocated_size limit because i_size is checked for in
@@ -161,7 +160,7 @@ still_busy: · 3 unmodified lines
161 *160 *
162 * Contains an adapted version of fs/buffer.c::block_read_full_folio().161 * Contains an adapted version of fs/buffer.c::block_read_full_folio().
163 */162 */
164static int ntfs_read_block(struct page *page)163static int ntfs_read_block(struct folio *folio)
165{164{
166 loff_t i_size;165 loff_t i_size;
167 VCN vcn;166 VCN vcn;
@@ -178,7 +177,7 @@ static int ntfs_read_block(struct page *page) · 10 unmodified lines
178 int i, nr;177 int i, nr;
179 unsigned char blocksize_bits;178 unsigned char blocksize_bits;
180 179
181 vi = page->mapping->host;180 vi = folio->mapping->host;
182 ni = NTFS_I(vi);181 ni = NTFS_I(vi);
183 vol = ni->vol;182 vol = ni->vol;
184 183
@@ -188,15 +187,10 @@ static int ntfs_read_block(struct page *page) · 3 unmodified lines
188 blocksize = vol->sb->s_blocksize;187 blocksize = vol->sb->s_blocksize;
189 blocksize_bits = vol->sb->s_blocksize_bits;188 blocksize_bits = vol->sb->s_blocksize_bits;
190 189
191 if (!page_has_buffers(page)) {190 head = folio_buffers(folio);
192 create_empty_buffers(page, blocksize, 0);191 if (!head)
193 if (unlikely(!page_has_buffers(page))) {192 head = create_empty_buffers(folio, blocksize, 0);
194 unlock_page(page);193 bh = head;
195 return -ENOMEM;
196 }
197 }
198 bh = head = page_buffers(page);
199 BUG_ON(!bh);
200 194
201 /*195 /*
202 * We may be racing with truncate. To avoid some of the problems we196 * We may be racing with truncate. To avoid some of the problems we
@@ -205,11 +199,11 @@ static int ntfs_read_block(struct page *page) · 2 unmodified lines
205 * may leave some buffers unmapped which are now allocated. This is199 * may leave some buffers unmapped which are now allocated. This is
206 * not a problem since these buffers will just get mapped when a write200 * not a problem since these buffers will just get mapped when a write
207 * occurs. In case of a shrinking truncate, we will detect this later201 * occurs. In case of a shrinking truncate, we will detect this later
208 * on due to the runlist being incomplete and if the page is being202 * on due to the runlist being incomplete and if the folio is being
209 * fully truncated, truncate will throw it away as soon as we unlock203 * fully truncated, truncate will throw it away as soon as we unlock
210 * it so no need to worry what we do with it.204 * it so no need to worry what we do with it.
211 */205 */
212 iblock = (s64)page->index << (PAGE_SHIFT - blocksize_bits);206 iblock = (s64)folio->index << (PAGE_SHIFT - blocksize_bits);
213 read_lock_irqsave(&ni->size_lock, flags);207 read_lock_irqsave(&ni->size_lock, flags);
214 lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits;208 lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits;
215 init_size = ni->initialized_size;209 init_size = ni->initialized_size;
@@ -221,7 +215,7 @@ static int ntfs_read_block(struct page *page) · 5 unmodified lines
221 }215 }
222 zblock = (init_size + blocksize - 1) >> blocksize_bits;216 zblock = (init_size + blocksize - 1) >> blocksize_bits;
223 217
224 /* Loop through all the buffers in the page. */218 /* Loop through all the buffers in the folio. */
225 rl = NULL;219 rl = NULL;
226 nr = i = 0;220 nr = i = 0;
227 do {221 do {
@@ -299,7 +293,7 @@ lock_retry_remap: · 71 unmodified lines
299 if (!err)293 if (!err)
300 err = -EIO;294 err = -EIO;
301 bh->b_blocknr = -1;295 bh->b_blocknr = -1;
302 SetPageError(page);296 folio_set_error(folio);
303 ntfs_error(vol->sb, "Failed to read from inode 0x%lx, "297 ntfs_error(vol->sb, "Failed to read from inode 0x%lx, "
304 "attribute type 0x%x, vcn 0x%llx, "298 "attribute type 0x%x, vcn 0x%llx, "
305 "offset 0x%x because its location on "299 "offset 0x%x because its location on "
@@ -312,13 +306,13 @@ lock_retry_remap: · 6 unmodified lines
312 /*306 /*
313 * Either iblock was outside lblock limits or307 * Either iblock was outside lblock limits or
314 * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion308 * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion
315 * of the page and set the buffer uptodate.309 * of the folio and set the buffer uptodate.
316 */310 */
317handle_hole:311handle_hole:
318 bh->b_blocknr = -1UL;312 bh->b_blocknr = -1UL;
319 clear_buffer_mapped(bh);313 clear_buffer_mapped(bh);
320handle_zblock:314handle_zblock:
321 zero_user(page, i * blocksize, blocksize);315 folio_zero_range(folio, i * blocksize, blocksize);
322 if (likely(!err))316 if (likely(!err))
323 set_buffer_uptodate(bh);317 set_buffer_uptodate(bh);
324 } while (i++, iblock++, (bh = bh->b_this_page) != head);318 } while (i++, iblock++, (bh = bh->b_this_page) != head);
@@ -349,11 +343,11 @@ handle_zblock: · 24 unmodified lines
349 return 0;343 return 0;
350 }344 }
351 /* No i/o was scheduled on any of the buffers. */345 /* No i/o was scheduled on any of the buffers. */
352 if (likely(!PageError(page)))346 if (likely(!folio_test_error(folio)))
353 SetPageUptodate(page);347 folio_mark_uptodate(folio);
354 else /* Signal synchronous i/o error. */348 else /* Signal synchronous i/o error. */
355 nr = -EIO;349 nr = -EIO;
356 unlock_page(page);350 folio_unlock(folio);
357 return nr;351 return nr;
358}352}
359 353
@@ -433,7 +427,7 @@ retry_readpage: · 73 unmodified lines
433 /* NInoNonResident() == NInoIndexAllocPresent() */427 /* NInoNonResident() == NInoIndexAllocPresent() */
434 if (NInoNonResident(ni)) {428 if (NInoNonResident(ni)) {
435 /* Normal, non-resident data stream. */429 /* Normal, non-resident data stream. */
436 return ntfs_read_block(page);430 return ntfs_read_block(folio);
437 }431 }
438 /*432 /*
439 * Attribute is resident, implying it is not compressed or encrypted.433 * Attribute is resident, implying it is not compressed or encrypted.
@@ -507,28 +501,29 @@ err_out: · 67 unmodified lines
507#ifdef NTFS_RW501#ifdef NTFS_RW
508 502
509/**503/**
510 * ntfs_write_block - write a @page to the backing store504 * ntfs_write_block - write a @folio to the backing store
511 * @page: page cache page to write out505 * @folio: page cache folio to write out
512 * @wbc: writeback control structure506 * @wbc: writeback control structure
513 *507 *
514 * This function is for writing pages belonging to non-resident, non-mst508 * This function is for writing folios belonging to non-resident, non-mst
515 * protected attributes to their backing store.509 * protected attributes to their backing store.
516 *510 *
517 * For a page with buffers, map and write the dirty buffers asynchronously511 * For a folio with buffers, map and write the dirty buffers asynchronously
518 * under page writeback. For a page without buffers, create buffers for the512 * under folio writeback. For a folio without buffers, create buffers for the
519 * page, then proceed as above.513 * folio, then proceed as above.
520 *514 *
521 * If a page doesn't have buffers the page dirty state is definitive. If a page515 * If a folio doesn't have buffers the folio dirty state is definitive. If
522 * does have buffers, the page dirty state is just a hint, and the buffer dirty516 * a folio does have buffers, the folio dirty state is just a hint,
523 * state is definitive. (A hint which has rules: dirty buffers against a clean517 * and the buffer dirty state is definitive. (A hint which has rules:
524 * page is illegal. Other combinations are legal and need to be handled. In518 * dirty buffers against a clean folio is illegal. Other combinations are
525 * particular a dirty page containing clean buffers for example.)519 * legal and need to be handled. In particular a dirty folio containing
520 * clean buffers for example.)
526 *521 *
527 * Return 0 on success and -errno on error.522 * Return 0 on success and -errno on error.
528 *523 *
529 * Based on ntfs_read_block() and __block_write_full_folio().524 * Based on ntfs_read_block() and __block_write_full_folio().
530 */525 */
531static int ntfs_write_block(struct page *page, struct writeback_control *wbc)526static int ntfs_write_block(struct folio *folio, struct writeback_control *wbc)
532{527{
533 VCN vcn;528 VCN vcn;
534 LCN lcn;529 LCN lcn;
@@ -546,41 +541,29 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 11 unmodified lines
546 bool need_end_writeback;541 bool need_end_writeback;
547 unsigned char blocksize_bits;542 unsigned char blocksize_bits;
548 543
549 vi = page->mapping->host;544 vi = folio->mapping->host;
550 ni = NTFS_I(vi);545 ni = NTFS_I(vi);
551 vol = ni->vol;546 vol = ni->vol;
552 547
553 ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index "548 ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index "
554 "0x%lx.", ni->mft_no, ni->type, page->index);549 "0x%lx.", ni->mft_no, ni->type, folio->index);
555 550
556 BUG_ON(!NInoNonResident(ni));551 BUG_ON(!NInoNonResident(ni));
557 BUG_ON(NInoMstProtected(ni));552 BUG_ON(NInoMstProtected(ni));
558 blocksize = vol->sb->s_blocksize;553 blocksize = vol->sb->s_blocksize;
559 blocksize_bits = vol->sb->s_blocksize_bits;554 blocksize_bits = vol->sb->s_blocksize_bits;
560 if (!page_has_buffers(page)) {555 head = folio_buffers(folio);
561 BUG_ON(!PageUptodate(page));556 if (!head) {
562 create_empty_buffers(page, blocksize,557 BUG_ON(!folio_test_uptodate(folio));
558 head = create_empty_buffers(folio, blocksize,
563 (1 << BH_Uptodate) | (1 << BH_Dirty));559 (1 << BH_Uptodate) | (1 << BH_Dirty));
564 if (unlikely(!page_has_buffers(page))) {
565 ntfs_warning(vol->sb, "Error allocating page "
566 "buffers. Redirtying page so we try "
567 "again later.");
568 /*
569 * Put the page back on mapping->dirty_pages, but leave
570 * its buffers' dirty state as-is.
571 */
572 redirty_page_for_writepage(wbc, page);
573 unlock_page(page);
574 return 0;
575 }
576 }560 }
577 bh = head = page_buffers(page);561 bh = head;
578 BUG_ON(!bh);
579 562
580 /* NOTE: Different naming scheme to ntfs_read_block()! */563 /* NOTE: Different naming scheme to ntfs_read_block()! */
581 564
582 /* The first block in the page. */565 /* The first block in the folio. */
583 block = (s64)page->index << (PAGE_SHIFT - blocksize_bits);566 block = (s64)folio->index << (PAGE_SHIFT - blocksize_bits);
584 567
585 read_lock_irqsave(&ni->size_lock, flags);568 read_lock_irqsave(&ni->size_lock, flags);
586 i_size = i_size_read(vi);569 i_size = i_size_read(vi);
@@ -597,14 +580,14 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 10 unmodified lines
597 * Be very careful. We have no exclusion from block_dirty_folio580 * Be very careful. We have no exclusion from block_dirty_folio
598 * here, and the (potentially unmapped) buffers may become dirty at581 * here, and the (potentially unmapped) buffers may become dirty at
599 * any time. If a buffer becomes dirty here after we've inspected it582 * any time. If a buffer becomes dirty here after we've inspected it
600 * then we just miss that fact, and the page stays dirty.583 * then we just miss that fact, and the folio stays dirty.
601 *584 *
602 * Buffers outside i_size may be dirtied by block_dirty_folio;585 * Buffers outside i_size may be dirtied by block_dirty_folio;
603 * handle that here by just cleaning them.586 * handle that here by just cleaning them.
604 */587 */
605 588
606 /*589 /*
607 * Loop through all the buffers in the page, mapping all the dirty590 * Loop through all the buffers in the folio, mapping all the dirty
608 * buffers to disk addresses and handling any aliases from the591 * buffers to disk addresses and handling any aliases from the
609 * underlying block device's mapping.592 * underlying block device's mapping.
610 */593 */
@@ -616,13 +599,13 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 5 unmodified lines
616 if (unlikely(block >= dblock)) {599 if (unlikely(block >= dblock)) {
617 /*600 /*
618 * Mapped buffers outside i_size will occur, because601 * Mapped buffers outside i_size will occur, because
619 * this page can be outside i_size when there is a602 * this folio can be outside i_size when there is a
620 * truncate in progress. The contents of such buffers603 * truncate in progress. The contents of such buffers
621 * were zeroed by ntfs_writepage().604 * were zeroed by ntfs_writepage().
622 *605 *
623 * FIXME: What about the small race window where606 * FIXME: What about the small race window where
624 * ntfs_writepage() has not done any clearing because607 * ntfs_writepage() has not done any clearing because
625 * the page was within i_size but before we get here,608 * the folio was within i_size but before we get here,
626 * vmtruncate() modifies i_size?609 * vmtruncate() modifies i_size?
627 */610 */
628 clear_buffer_dirty(bh);611 clear_buffer_dirty(bh);
@@ -638,38 +621,38 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 9 unmodified lines
638 if (unlikely((block >= iblock) &&621 if (unlikely((block >= iblock) &&
639 (initialized_size < i_size))) {622 (initialized_size < i_size))) {
640 /*623 /*
641 * If this page is fully outside initialized624 * If this folio is fully outside initialized
642 * size, zero out all pages between the current625 * size, zero out all folios between the current
643 * initialized size and the current page. Just626 * initialized size and the current folio. Just
644 * use ntfs_read_folio() to do the zeroing627 * use ntfs_read_folio() to do the zeroing
645 * transparently.628 * transparently.
646 */629 */
647 if (block > iblock) {630 if (block > iblock) {
648 // TODO:631 // TODO:
649 // For each page do:632 // For each folio do:
650 // - read_cache_page()633 // - read_cache_folio()
651 // Again for each page do:634 // Again for each folio do:
652 // - wait_on_page_locked()635 // - wait_on_folio_locked()
653 // - Check (PageUptodate(page) &&636 // - Check (folio_test_uptodate(folio) &&
654 // !PageError(page))637 // !folio_test_error(folio))
655 // Update initialized size in the attribute and638 // Update initialized size in the attribute and
656 // in the inode.639 // in the inode.
657 // Again, for each page do:640 // Again, for each folio do:
658 // block_dirty_folio();641 // block_dirty_folio();
659 // put_page()642 // folio_put()
660 // We don't need to wait on the writes.643 // We don't need to wait on the writes.
661 // Update iblock.644 // Update iblock.
662 }645 }
663 /*646 /*
664 * The current page straddles initialized size. Zero647 * The current folio straddles initialized size. Zero
665 * all non-uptodate buffers and set them uptodate (and648 * all non-uptodate buffers and set them uptodate (and
666 * dirty?). Note, there aren't any non-uptodate buffers649 * dirty?). Note, there aren't any non-uptodate buffers
667 * if the page is uptodate.650 * if the folio is uptodate.
668 * FIXME: For an uptodate page, the buffers may need to651 * FIXME: For an uptodate folio, the buffers may need to
669 * be written out because they were not initialized on652 * be written out because they were not initialized on
670 * disk before.653 * disk before.
671 */654 */
672 if (!PageUptodate(page)) {655 if (!folio_test_uptodate(folio)) {
673 // TODO:656 // TODO:
674 // Zero any non-uptodate buffers up to i_size.657 // Zero any non-uptodate buffers up to i_size.
675 // Set them uptodate and dirty.658 // Set them uptodate and dirty.
@@ -727,14 +710,14 @@ lock_retry_remap: · 51 unmodified lines
727 unsigned long *bpos, *bend;710 unsigned long *bpos, *bend;
728 711
729 /* Check if the buffer is zero. */712 /* Check if the buffer is zero. */
730 kaddr = kmap_atomic(page);713 kaddr = kmap_local_folio(folio, bh_offset(bh));
731 bpos = (unsigned long *)(kaddr + bh_offset(bh));714 bpos = (unsigned long *)kaddr;
732 bend = (unsigned long *)((u8*)bpos + blocksize);715 bend = (unsigned long *)(kaddr + blocksize);
733 do {716 do {
734 if (unlikely(*bpos))717 if (unlikely(*bpos))
735 break;718 break;
736 } while (likely(++bpos < bend));719 } while (likely(++bpos < bend));
737 kunmap_atomic(kaddr);720 kunmap_local(kaddr);
738 if (bpos == bend) {721 if (bpos == bend) {
739 /*722 /*
740 * Buffer is zero and sparse, no need to write723 * Buffer is zero and sparse, no need to write
@@ -774,7 +757,7 @@ lock_retry_remap: · 33 unmodified lines
774 if (err == -ENOENT || lcn == LCN_ENOENT) {757 if (err == -ENOENT || lcn == LCN_ENOENT) {
775 bh->b_blocknr = -1;758 bh->b_blocknr = -1;
776 clear_buffer_dirty(bh);759 clear_buffer_dirty(bh);
777 zero_user(page, bh_offset(bh), blocksize);760 folio_zero_range(folio, bh_offset(bh), blocksize);
778 set_buffer_uptodate(bh);761 set_buffer_uptodate(bh);
779 err = 0;762 err = 0;
780 continue;763 continue;
@@ -801,7 +784,7 @@ lock_retry_remap: · 20 unmodified lines
801 bh = head;784 bh = head;
802 785
803 /* Just an optimization, so ->read_folio() is not called later. */786 /* Just an optimization, so ->read_folio() is not called later. */
804 if (unlikely(!PageUptodate(page))) {787 if (unlikely(!folio_test_uptodate(folio))) {
805 int uptodate = 1;788 int uptodate = 1;
806 do {789 do {
807 if (!buffer_uptodate(bh)) {790 if (!buffer_uptodate(bh)) {
@@ -811,7 +794,7 @@ lock_retry_remap: · 3 unmodified lines
811 }794 }
812 } while ((bh = bh->b_this_page) != head);795 } while ((bh = bh->b_this_page) != head);
813 if (uptodate)796 if (uptodate)
814 SetPageUptodate(page);797 folio_mark_uptodate(folio);
815 }798 }
816 799
817 /* Setup all mapped, dirty buffers for async write i/o. */800 /* Setup all mapped, dirty buffers for async write i/o. */
@@ -826,7 +809,7 @@ lock_retry_remap: · 8 unmodified lines
826 } else if (unlikely(err)) {809 } else if (unlikely(err)) {
827 /*810 /*
828 * For the error case. The buffer may have been set811 * For the error case. The buffer may have been set
829 * dirty during attachment to a dirty page.812 * dirty during attachment to a dirty folio.
830 */813 */
831 if (err != -ENOMEM)814 if (err != -ENOMEM)
832 clear_buffer_dirty(bh);815 clear_buffer_dirty(bh);
@@ -839,20 +822,20 @@ lock_retry_remap: · 6 unmodified lines
839 err = 0;822 err = 0;
840 else if (err == -ENOMEM) {823 else if (err == -ENOMEM) {
841 ntfs_warning(vol->sb, "Error allocating memory. "824 ntfs_warning(vol->sb, "Error allocating memory. "
842 "Redirtying page so we try again "825 "Redirtying folio so we try again "
843 "later.");826 "later.");
844 /*827 /*
845 * Put the page back on mapping->dirty_pages, but828 * Put the folio back on mapping->dirty_pages, but
846 * leave its buffer's dirty state as-is.829 * leave its buffer's dirty state as-is.
847 */830 */
848 redirty_page_for_writepage(wbc, page);831 folio_redirty_for_writepage(wbc, folio);
849 err = 0;832 err = 0;
850 } else833 } else
851 SetPageError(page);834 folio_set_error(folio);
852 }835 }
853 836
854 BUG_ON(PageWriteback(page));837 BUG_ON(folio_test_writeback(folio));
855 set_page_writeback(page); /* Keeps try_to_free_buffers() away. */838 folio_start_writeback(folio); /* Keeps try_to_free_buffers() away. */
856 839
857 /* Submit the prepared buffers for i/o. */840 /* Submit the prepared buffers for i/o. */
858 need_end_writeback = true;841 need_end_writeback = true;
@@ -864,11 +847,11 @@ lock_retry_remap: · 5 unmodified lines
864 }847 }
865 bh = next;848 bh = next;
866 } while (bh != head);849 } while (bh != head);
867 unlock_page(page);850 folio_unlock(folio);
868 851
869 /* If no i/o was started, need to end_page_writeback(). */852 /* If no i/o was started, need to end writeback here. */
870 if (unlikely(need_end_writeback))853 if (unlikely(need_end_writeback))
871 end_page_writeback(page);854 folio_end_writeback(folio);
872 855
873 ntfs_debug("Done.");856 ntfs_debug("Done.");
874 return err;857 return err;
@@ -1337,8 +1320,9 @@ done: · 462 unmodified lines
1337 */1320 */
1338static int ntfs_writepage(struct page *page, struct writeback_control *wbc)1321static int ntfs_writepage(struct page *page, struct writeback_control *wbc)
1339{1322{
1323 struct folio *folio = page_folio(page);
1340 loff_t i_size;1324 loff_t i_size;
1341 struct inode *vi = page->mapping->host;1325 struct inode *vi = folio->mapping->host;
1342 ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi);1326 ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi);
1343 char *addr;1327 char *addr;
1344 ntfs_attr_search_ctx *ctx = NULL;1328 ntfs_attr_search_ctx *ctx = NULL;
@@ -1347,14 +1331,13 @@ static int ntfs_writepage(struct page *page, struct writeback_control *wbc) · 2 unmodified lines
1347 int err;1331 int err;
1348 1332
1349retry_writepage:1333retry_writepage:
1350 BUG_ON(!PageLocked(page));1334 BUG_ON(!folio_test_locked(folio));
1351 i_size = i_size_read(vi);1335 i_size = i_size_read(vi);
1352 /* Is the page fully outside i_size? (truncate in progress) */1336 /* Is the folio fully outside i_size? (truncate in progress) */
1353 if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >>1337 if (unlikely(folio->index >= (i_size + PAGE_SIZE - 1) >>
1354 PAGE_SHIFT)) {1338 PAGE_SHIFT)) {
1355 struct folio *folio = page_folio(page);
1356 /*1339 /*
1357 * The page may have dirty, unmapped buffers. Make them1340 * The folio may have dirty, unmapped buffers. Make them
1358 * freeable here, so the page does not leak.1341 * freeable here, so the page does not leak.
1359 */1342 */
1360 block_invalidate_folio(folio, 0, folio_size(folio));1343 block_invalidate_folio(folio, 0, folio_size(folio));
@@ -1373,7 +1356,7 @@ retry_writepage: · 12 unmodified lines
1373 if (ni->type != AT_INDEX_ALLOCATION) {1356 if (ni->type != AT_INDEX_ALLOCATION) {
1374 /* If file is encrypted, deny access, just like NT4. */1357 /* If file is encrypted, deny access, just like NT4. */
1375 if (NInoEncrypted(ni)) {1358 if (NInoEncrypted(ni)) {
1376 unlock_page(page);1359 folio_unlock(folio);
1377 BUG_ON(ni->type != AT_DATA);1360 BUG_ON(ni->type != AT_DATA);
1378 ntfs_debug("Denying write access to encrypted file.");1361 ntfs_debug("Denying write access to encrypted file.");
1379 return -EACCES;1362 return -EACCES;
@@ -1384,14 +1367,14 @@ retry_writepage: · 4 unmodified lines
1384 BUG_ON(ni->name_len);1367 BUG_ON(ni->name_len);
1385 // TODO: Implement and replace this with1368 // TODO: Implement and replace this with
1386 // return ntfs_write_compressed_block(page);1369 // return ntfs_write_compressed_block(page);
1387 unlock_page(page);1370 folio_unlock(folio);
1388 ntfs_error(vi->i_sb, "Writing to compressed files is "1371 ntfs_error(vi->i_sb, "Writing to compressed files is "
1389 "not supported yet. Sorry.");1372 "not supported yet. Sorry.");
1390 return -EOPNOTSUPP;1373 return -EOPNOTSUPP;
1391 }1374 }
1392 // TODO: Implement and remove this check.1375 // TODO: Implement and remove this check.
1393 if (NInoNonResident(ni) && NInoSparse(ni)) {1376 if (NInoNonResident(ni) && NInoSparse(ni)) {
1394 unlock_page(page);1377 folio_unlock(folio);
1395 ntfs_error(vi->i_sb, "Writing to sparse files is not "1378 ntfs_error(vi->i_sb, "Writing to sparse files is not "
1396 "supported yet. Sorry.");1379 "supported yet. Sorry.");
1397 return -EOPNOTSUPP;1380 return -EOPNOTSUPP;
@@ -1400,34 +1383,34 @@ retry_writepage: · 2 unmodified lines
1400 /* NInoNonResident() == NInoIndexAllocPresent() */1383 /* NInoNonResident() == NInoIndexAllocPresent() */
1401 if (NInoNonResident(ni)) {1384 if (NInoNonResident(ni)) {
1402 /* We have to zero every time due to mmap-at-end-of-file. */1385 /* We have to zero every time due to mmap-at-end-of-file. */
1403 if (page->index >= (i_size >> PAGE_SHIFT)) {1386 if (folio->index >= (i_size >> PAGE_SHIFT)) {
1404 /* The page straddles i_size. */1387 /* The folio straddles i_size. */
1405 unsigned int ofs = i_size & ~PAGE_MASK;1388 unsigned int ofs = i_size & (folio_size(folio) - 1);
1406 zero_user_segment(page, ofs, PAGE_SIZE);1389 folio_zero_segment(folio, ofs, folio_size(folio));
1407 }1390 }
1408 /* Handle mst protected attributes. */1391 /* Handle mst protected attributes. */
1409 if (NInoMstProtected(ni))1392 if (NInoMstProtected(ni))
1410 return ntfs_write_mst_block(page, wbc);1393 return ntfs_write_mst_block(page, wbc);
1411 /* Normal, non-resident data stream. */1394 /* Normal, non-resident data stream. */
1412 return ntfs_write_block(page, wbc);1395 return ntfs_write_block(folio, wbc);
1413 }1396 }
1414 /*1397 /*
1415 * Attribute is resident, implying it is not compressed, encrypted, or1398 * Attribute is resident, implying it is not compressed, encrypted, or
1416 * mst protected. This also means the attribute is smaller than an mft1399 * mst protected. This also means the attribute is smaller than an mft
1417 * record and hence smaller than a page, so can simply return error on1400 * record and hence smaller than a folio, so can simply return error on
1418 * any pages with index above 0. Note the attribute can actually be1401 * any folios with index above 0. Note the attribute can actually be
1419 * marked compressed but if it is resident the actual data is not1402 * marked compressed but if it is resident the actual data is not
1420 * compressed so we are ok to ignore the compressed flag here.1403 * compressed so we are ok to ignore the compressed flag here.
1421 */1404 */
1422 BUG_ON(page_has_buffers(page));1405 BUG_ON(folio_buffers(folio));
1423 BUG_ON(!PageUptodate(page));1406 BUG_ON(!folio_test_uptodate(folio));
1424 if (unlikely(page->index > 0)) {1407 if (unlikely(folio->index > 0)) {
1425 ntfs_error(vi->i_sb, "BUG()! page->index (0x%lx) > 0. "1408 ntfs_error(vi->i_sb, "BUG()! folio->index (0x%lx) > 0. "
1426 "Aborting write.", page->index);1409 "Aborting write.", folio->index);
1427 BUG_ON(PageWriteback(page));1410 BUG_ON(folio_test_writeback(folio));
1428 set_page_writeback(page);1411 folio_start_writeback(folio);
1429 unlock_page(page);1412 folio_unlock(folio);
1430 end_page_writeback(page);1413 folio_end_writeback(folio);
1431 return -EIO;1414 return -EIO;
1432 }1415 }
1433 if (!NInoAttr(ni))1416 if (!NInoAttr(ni))
@@ -1460,12 +1443,12 @@ retry_writepage: · 26 unmodified lines
1460 if (unlikely(err))1443 if (unlikely(err))
1461 goto err_out;1444 goto err_out;
1462 /*1445 /*
1463 * Keep the VM happy. This must be done otherwise the radix-tree tag1446 * Keep the VM happy. This must be done otherwise
1464 * PAGECACHE_TAG_DIRTY remains set even though the page is clean.1447 * PAGECACHE_TAG_DIRTY remains set even though the folio is clean.
1465 */1448 */
1466 BUG_ON(PageWriteback(page));1449 BUG_ON(folio_test_writeback(folio));
1467 set_page_writeback(page);1450 folio_start_writeback(folio);
1468 unlock_page(page);1451 folio_unlock(folio);
1469 attr_len = le32_to_cpu(ctx->attr->data.resident.value_length);1452 attr_len = le32_to_cpu(ctx->attr->data.resident.value_length);
1470 i_size = i_size_read(vi);1453 i_size = i_size_read(vi);
1471 if (unlikely(attr_len > i_size)) {1454 if (unlikely(attr_len > i_size)) {
@@ -1480,18 +1463,18 @@ retry_writepage: · 8 unmodified lines
1480 /* Shrinking cannot fail. */1463 /* Shrinking cannot fail. */
1481 BUG_ON(err);1464 BUG_ON(err);
1482 }1465 }
1483 addr = kmap_atomic(page);1466 addr = kmap_local_folio(folio, 0);
1484 /* Copy the data from the page to the mft record. */1467 /* Copy the data from the folio to the mft record. */
1485 memcpy((u8*)ctx->attr +1468 memcpy((u8*)ctx->attr +
1486 le16_to_cpu(ctx->attr->data.resident.value_offset),1469 le16_to_cpu(ctx->attr->data.resident.value_offset),
1487 addr, attr_len);1470 addr, attr_len);
1488 /* Zero out of bounds area in the page cache page. */1471 /* Zero out of bounds area in the page cache folio. */
1489 memset(addr + attr_len, 0, PAGE_SIZE - attr_len);1472 memset(addr + attr_len, 0, folio_size(folio) - attr_len);
1490 kunmap_atomic(addr);1473 kunmap_local(addr);
1491 flush_dcache_page(page);1474 flush_dcache_folio(folio);
1492 flush_dcache_mft_record_page(ctx->ntfs_ino);1475 flush_dcache_mft_record_page(ctx->ntfs_ino);
1493 /* We are done with the page. */1476 /* We are done with the folio. */
1494 end_page_writeback(page);1477 folio_end_writeback(folio);
1495 /* Finally, mark the mft record dirty, so it gets written back. */1478 /* Finally, mark the mft record dirty, so it gets written back. */
1496 mark_mft_record_dirty(ctx->ntfs_ino);1479 mark_mft_record_dirty(ctx->ntfs_ino);
1497 ntfs_attr_put_search_ctx(ctx);1480 ntfs_attr_put_search_ctx(ctx);
@@ -1502,18 +1485,18 @@ err_out: · 4 unmodified lines
1502 ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying "1485 ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying "
1503 "page so we try again later.");1486 "page so we try again later.");
1504 /*1487 /*
1505 * Put the page back on mapping->dirty_pages, but leave its1488 * Put the folio back on mapping->dirty_pages, but leave its
1506 * buffers' dirty state as-is.1489 * buffers' dirty state as-is.
1507 */1490 */
1508 redirty_page_for_writepage(wbc, page);1491 folio_redirty_for_writepage(wbc, folio);
1509 err = 0;1492 err = 0;
1510 } else {1493 } else {
1511 ntfs_error(vi->i_sb, "Resident attribute write failed with "1494 ntfs_error(vi->i_sb, "Resident attribute write failed with "
1512 "error %i.", err);1495 "error %i.", err);
1513 SetPageError(page);1496 folio_set_error(folio);
1514 NVolSetErrors(ni->vol);1497 NVolSetErrors(ni->vol);
1515 }1498 }
1516 unlock_page(page);1499 folio_unlock(folio);
1517 if (ctx)1500 if (ctx)
1518 ntfs_attr_put_search_ctx(ctx);1501 ntfs_attr_put_search_ctx(ctx);
1519 if (m)1502 if (m)
fs/ntfs/file.c+41−48
@@ -567,7 +567,7 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 566 unmodified lines
567 LCN lcn;567 LCN lcn;
568 s64 bh_pos, vcn_len, end, initialized_size;568 s64 bh_pos, vcn_len, end, initialized_size;
569 sector_t lcn_block;569 sector_t lcn_block;
570 struct page *page;570 struct folio *folio;
571 struct inode *vi;571 struct inode *vi;
572 ntfs_inode *ni, *base_ni = NULL;572 ntfs_inode *ni, *base_ni = NULL;
573 ntfs_volume *vol;573 ntfs_volume *vol;
@@ -601,20 +601,6 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 27 unmodified lines
601 (long long)pos, bytes);601 (long long)pos, bytes);
602 blocksize = vol->sb->s_blocksize;602 blocksize = vol->sb->s_blocksize;
603 blocksize_bits = vol->sb->s_blocksize_bits;603 blocksize_bits = vol->sb->s_blocksize_bits;
604 u = 0;
605 do {
606 page = pages[u];
607 BUG_ON(!page);
608 /*
609 * create_empty_buffers() will create uptodate/dirty buffers if
610 * the page is uptodate/dirty.
611 */
612 if (!page_has_buffers(page)) {
613 create_empty_buffers(page, blocksize, 0);
614 if (unlikely(!page_has_buffers(page)))
615 return -ENOMEM;
616 }
617 } while (++u < nr_pages);
618 rl_write_locked = false;604 rl_write_locked = false;
619 rl = NULL;605 rl = NULL;
620 err = 0;606 err = 0;
@@ -626,14 +612,21 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 5 unmodified lines
626 end = pos + bytes;612 end = pos + bytes;
627 cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits;613 cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits;
628 /*614 /*
629 * Loop over each page and for each page over each buffer. Use goto to615 * Loop over each buffer in each folio. Use goto to
630 * reduce indentation.616 * reduce indentation.
631 */617 */
632 u = 0;618 u = 0;
633do_next_page:619do_next_folio:
634 page = pages[u];620 folio = page_folio(pages[u]);
635 bh_pos = (s64)page->index << PAGE_SHIFT;621 bh_pos = folio_pos(folio);
636 bh = head = page_buffers(page);622 head = folio_buffers(folio);
623 if (!head)
624 /*
625 * create_empty_buffers() will create uptodate/dirty
626 * buffers if the folio is uptodate/dirty.
627 */
628 head = create_empty_buffers(folio, blocksize, 0);
629 bh = head;
637 do {630 do {
638 VCN cdelta;631 VCN cdelta;
639 s64 bh_end;632 s64 bh_end;
@@ -653,15 +646,15 @@ do_next_page: · 13 unmodified lines
653 if (buffer_uptodate(bh))646 if (buffer_uptodate(bh))
654 continue;647 continue;
655 /*648 /*
656 * The buffer is not uptodate. If the page is uptodate649 * The buffer is not uptodate. If the folio is uptodate
657 * set the buffer uptodate and otherwise ignore it.650 * set the buffer uptodate and otherwise ignore it.
658 */651 */
659 if (PageUptodate(page)) {652 if (folio_test_uptodate(folio)) {
660 set_buffer_uptodate(bh);653 set_buffer_uptodate(bh);
661 continue;654 continue;
662 }655 }
663 /*656 /*
664 * Neither the page nor the buffer are uptodate. If657 * Neither the folio nor the buffer are uptodate. If
665 * the buffer is only partially being written to, we658 * the buffer is only partially being written to, we
666 * need to read it in before the write, i.e. now.659 * need to read it in before the write, i.e. now.
667 */660 */
@@ -679,7 +672,7 @@ do_next_page: · 11 unmodified lines
679 ntfs_submit_bh_for_read(bh);672 ntfs_submit_bh_for_read(bh);
680 *wait_bh++ = bh;673 *wait_bh++ = bh;
681 } else {674 } else {
682 zero_user(page, bh_offset(bh),675 folio_zero_range(folio, bh_offset(bh),
683 blocksize);676 blocksize);
684 set_buffer_uptodate(bh);677 set_buffer_uptodate(bh);
685 }678 }
@@ -706,7 +699,7 @@ map_buffer_cached: · 20 unmodified lines
706 (bh_cofs >> blocksize_bits);699 (bh_cofs >> blocksize_bits);
707 set_buffer_mapped(bh);700 set_buffer_mapped(bh);
708 /*701 /*
709 * If the page is uptodate so is the buffer. If the702 * If the folio is uptodate so is the buffer. If the
710 * buffer is fully outside the write, we ignore it if703 * buffer is fully outside the write, we ignore it if
711 * it was already allocated and we mark it dirty so it704 * it was already allocated and we mark it dirty so it
712 * gets written out if we allocated it. On the other705 * gets written out if we allocated it. On the other
@@ -714,7 +707,7 @@ map_buffer_cached: · 1 unmodified line
714 * marking it dirty we set buffer_new so we can do707 * marking it dirty we set buffer_new so we can do
715 * error recovery.708 * error recovery.
716 */709 */
717 if (PageUptodate(page)) {710 if (folio_test_uptodate(folio)) {
718 if (!buffer_uptodate(bh))711 if (!buffer_uptodate(bh))
719 set_buffer_uptodate(bh);712 set_buffer_uptodate(bh);
720 if (unlikely(was_hole)) {713 if (unlikely(was_hole)) {
@@ -754,7 +747,8 @@ map_buffer_cached: · 33 unmodified lines
754 ntfs_submit_bh_for_read(bh);747 ntfs_submit_bh_for_read(bh);
755 *wait_bh++ = bh;748 *wait_bh++ = bh;
756 } else {749 } else {
757 zero_user(page, bh_offset(bh),750 folio_zero_range(folio,
751 bh_offset(bh),
758 blocksize);752 blocksize);
759 set_buffer_uptodate(bh);753 set_buffer_uptodate(bh);
760 }754 }
@@ -773,7 +767,7 @@ map_buffer_cached: · 12 unmodified lines
773 */767 */
774 if (bh_end <= pos || bh_pos >= end) {768 if (bh_end <= pos || bh_pos >= end) {
775 if (!buffer_uptodate(bh)) {769 if (!buffer_uptodate(bh)) {
776 zero_user(page, bh_offset(bh),770 folio_zero_range(folio, bh_offset(bh),
777 blocksize);771 blocksize);
778 set_buffer_uptodate(bh);772 set_buffer_uptodate(bh);
779 }773 }
@@ -786,7 +780,7 @@ map_buffer_cached: · 6 unmodified lines
786 u8 *kaddr;780 u8 *kaddr;
787 unsigned pofs;781 unsigned pofs;
788 782
789 kaddr = kmap_atomic(page);783 kaddr = kmap_local_folio(folio, 0);
790 if (bh_pos < pos) {784 if (bh_pos < pos) {
791 pofs = bh_pos & ~PAGE_MASK;785 pofs = bh_pos & ~PAGE_MASK;
792 memset(kaddr + pofs, 0, pos - bh_pos);786 memset(kaddr + pofs, 0, pos - bh_pos);
@@ -795,8 +789,8 @@ map_buffer_cached: · 2 unmodified lines
795 pofs = end & ~PAGE_MASK;789 pofs = end & ~PAGE_MASK;
796 memset(kaddr + pofs, 0, bh_end - end);790 memset(kaddr + pofs, 0, bh_end - end);
797 }791 }
798 kunmap_atomic(kaddr);792 kunmap_local(kaddr);
799 flush_dcache_page(page);793 flush_dcache_folio(folio);
800 }794 }
801 continue;795 continue;
802 }796 }
@@ -809,11 +803,12 @@ map_buffer_cached: · 6 unmodified lines
809 initialized_size = ni->allocated_size;803 initialized_size = ni->allocated_size;
810 read_unlock_irqrestore(&ni->size_lock, flags);804 read_unlock_irqrestore(&ni->size_lock, flags);
811 if (bh_pos > initialized_size) {805 if (bh_pos > initialized_size) {
812 if (PageUptodate(page)) {806 if (folio_test_uptodate(folio)) {
813 if (!buffer_uptodate(bh))807 if (!buffer_uptodate(bh))
814 set_buffer_uptodate(bh);808 set_buffer_uptodate(bh);
815 } else if (!buffer_uptodate(bh)) {809 } else if (!buffer_uptodate(bh)) {
816 zero_user(page, bh_offset(bh), blocksize);810 folio_zero_range(folio, bh_offset(bh),
811 blocksize);
817 set_buffer_uptodate(bh);812 set_buffer_uptodate(bh);
818 }813 }
819 continue;814 continue;
@@ -927,17 +922,17 @@ rl_not_mapped_enoent: · 107 unmodified lines
927 bh->b_blocknr = -1;922 bh->b_blocknr = -1;
928 /*923 /*
929 * If the buffer is uptodate we skip it. If it924 * If the buffer is uptodate we skip it. If it
930 * is not but the page is uptodate, we can set925 * is not but the folio is uptodate, we can set
931 * the buffer uptodate. If the page is not926 * the buffer uptodate. If the folio is not
932 * uptodate, we can clear the buffer and set it927 * uptodate, we can clear the buffer and set it
933 * uptodate. Whether this is worthwhile is928 * uptodate. Whether this is worthwhile is
934 * debatable and this could be removed.929 * debatable and this could be removed.
935 */930 */
936 if (PageUptodate(page)) {931 if (folio_test_uptodate(folio)) {
937 if (!buffer_uptodate(bh))932 if (!buffer_uptodate(bh))
938 set_buffer_uptodate(bh);933 set_buffer_uptodate(bh);
939 } else if (!buffer_uptodate(bh)) {934 } else if (!buffer_uptodate(bh)) {
940 zero_user(page, bh_offset(bh),935 folio_zero_range(folio, bh_offset(bh),
941 blocksize);936 blocksize);
942 set_buffer_uptodate(bh);937 set_buffer_uptodate(bh);
943 }938 }
@@ -1167,7 +1162,7 @@ rl_not_mapped_enoent: · 223 unmodified lines
1167 } while (bh_pos += blocksize, (bh = bh->b_this_page) != head);1162 } while (bh_pos += blocksize, (bh = bh->b_this_page) != head);
1168 /* If there are no errors, do the next page. */1163 /* If there are no errors, do the next page. */
1169 if (likely(!err && ++u < nr_pages))1164 if (likely(!err && ++u < nr_pages))
1170 goto do_next_page;1165 goto do_next_folio;
1171 /* If there are no errors, release the runlist lock if we took it. */1166 /* If there are no errors, release the runlist lock if we took it. */
1172 if (likely(!err)) {1167 if (likely(!err)) {
1173 if (unlikely(rl_write_locked)) {1168 if (unlikely(rl_write_locked)) {
@@ -1185,9 +1180,8 @@ rl_not_mapped_enoent: · 11 unmodified lines
1185 bh = *--wait_bh;1180 bh = *--wait_bh;
1186 wait_on_buffer(bh);1181 wait_on_buffer(bh);
1187 if (likely(buffer_uptodate(bh))) {1182 if (likely(buffer_uptodate(bh))) {
1188 page = bh->b_page;1183 folio = bh->b_folio;
1189 bh_pos = ((s64)page->index << PAGE_SHIFT) +1184 bh_pos = folio_pos(folio) + bh_offset(bh);
1190 bh_offset(bh);
1191 /*1185 /*
1192 * If the buffer overflows the initialized size, need1186 * If the buffer overflows the initialized size, need
1193 * to zero the overflowing region.1187 * to zero the overflowing region.
@@ -1197,7 +1191,7 @@ rl_not_mapped_enoent: · 3 unmodified lines
1197 1191
1198 if (likely(bh_pos < initialized_size))1192 if (likely(bh_pos < initialized_size))
1199 ofs = initialized_size - bh_pos;1193 ofs = initialized_size - bh_pos;
1200 zero_user_segment(page, bh_offset(bh) + ofs,1194 folio_zero_segment(folio, bh_offset(bh) + ofs,
1201 blocksize);1195 blocksize);
1202 }1196 }
1203 } else /* if (unlikely(!buffer_uptodate(bh))) */1197 } else /* if (unlikely(!buffer_uptodate(bh))) */
@@ -1324,21 +1318,20 @@ rl_not_mapped_enoent: · 120 unmodified lines
1324 u = 0;1318 u = 0;
1325 end = bh_cpos << vol->cluster_size_bits;1319 end = bh_cpos << vol->cluster_size_bits;
1326 do {1320 do {
1327 page = pages[u];1321 folio = page_folio(pages[u]);
1328 bh = head = page_buffers(page);1322 bh = head = folio_buffers(folio);
1329 do {1323 do {
1330 if (u == nr_pages &&1324 if (u == nr_pages &&
1331 ((s64)page->index << PAGE_SHIFT) +1325 folio_pos(folio) + bh_offset(bh) >= end)
1332 bh_offset(bh) >= end)
1333 break;1326 break;
1334 if (!buffer_new(bh))1327 if (!buffer_new(bh))
1335 continue;1328 continue;
1336 clear_buffer_new(bh);1329 clear_buffer_new(bh);
1337 if (!buffer_uptodate(bh)) {1330 if (!buffer_uptodate(bh)) {
1338 if (PageUptodate(page))1331 if (folio_test_uptodate(folio))
1339 set_buffer_uptodate(bh);1332 set_buffer_uptodate(bh);
1340 else {1333 else {
1341 zero_user(page, bh_offset(bh),1334 folio_zero_range(folio, bh_offset(bh),
1342 blocksize);1335 blocksize);
1343 set_buffer_uptodate(bh);1336 set_buffer_uptodate(bh);
1344 }1337 }
fs/ntfs3/file.c+16−15
@@ -187,7 +187,7 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 186 unmodified lines
187 struct buffer_head *head, *bh;187 struct buffer_head *head, *bh;
188 u32 bh_next, bh_off, to;188 u32 bh_next, bh_off, to;
189 sector_t iblock;189 sector_t iblock;
190 struct page *page;190 struct folio *folio;
191 191
192 for (; idx < idx_end; idx += 1, from = 0) {192 for (; idx < idx_end; idx += 1, from = 0) {
193 page_off = (loff_t)idx << PAGE_SHIFT;193 page_off = (loff_t)idx << PAGE_SHIFT;
@@ -195,16 +195,17 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 1 unmodified line
195 PAGE_SIZE;195 PAGE_SIZE;
196 iblock = page_off >> inode->i_blkbits;196 iblock = page_off >> inode->i_blkbits;
197 197
198 page = find_or_create_page(mapping, idx,198 folio = __filemap_get_folio(mapping, idx,
199 mapping_gfp_constraint(mapping,199 FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
200 ~__GFP_FS));200 mapping_gfp_constraint(mapping, ~__GFP_FS));
201 if (!page)201 if (IS_ERR(folio))
202 return -ENOMEM;202 return PTR_ERR(folio);
203 203
204 if (!page_has_buffers(page))204 head = folio_buffers(folio);
205 create_empty_buffers(page, blocksize, 0);205 if (!head)
206 head = create_empty_buffers(folio, blocksize, 0);
206 207
207 bh = head = page_buffers(page);208 bh = head;
208 bh_off = 0;209 bh_off = 0;
209 do {210 do {
210 bh_next = bh_off + blocksize;211 bh_next = bh_off + blocksize;
@@ -220,14 +221,14 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 9 unmodified lines
220 }221 }
221 222
222 /* Ok, it's mapped. Make sure it's up-to-date. */223 /* Ok, it's mapped. Make sure it's up-to-date. */
223 if (PageUptodate(page))224 if (folio_test_uptodate(folio))
224 set_buffer_uptodate(bh);225 set_buffer_uptodate(bh);
225 226
226 if (!buffer_uptodate(bh)) {227 if (!buffer_uptodate(bh)) {
227 err = bh_read(bh, 0);228 err = bh_read(bh, 0);
228 if (err < 0) {229 if (err < 0) {
229 unlock_page(page);230 folio_unlock(folio);
230 put_page(page);231 folio_put(folio);
231 goto out;232 goto out;
232 }233 }
233 }234 }
@@ -237,10 +238,10 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 3 unmodified lines
237 } while (bh_off = bh_next, iblock += 1,238 } while (bh_off = bh_next, iblock += 1,
238 head != (bh = bh->b_this_page));239 head != (bh = bh->b_this_page));
239 240
240 zero_user_segment(page, from, to);241 folio_zero_segment(folio, from, to);
241 242
242 unlock_page(page);243 folio_unlock(folio);
243 put_page(page);244 folio_put(folio);
244 cond_resched();245 cond_resched();
245 }246 }
246out:247out:
fs/ocfs2/aops.c+10−9
@@ -568,10 +568,10 @@ static void ocfs2_clear_page_regions(struct page *page, · 567 unmodified lines
568 * read-in the blocks at the tail of our file. Avoid reading them by568 * read-in the blocks at the tail of our file. Avoid reading them by
569 * testing i_size against each block offset.569 * testing i_size against each block offset.
570 */570 */
571static int ocfs2_should_read_blk(struct inode *inode, struct page *page,571static int ocfs2_should_read_blk(struct inode *inode, struct folio *folio,
572 unsigned int block_start)572 unsigned int block_start)
573{573{
574 u64 offset = page_offset(page) + block_start;574 u64 offset = folio_pos(folio) + block_start;
575 575
576 if (ocfs2_sparse_alloc(OCFS2_SB(inode->i_sb)))576 if (ocfs2_sparse_alloc(OCFS2_SB(inode->i_sb)))
577 return 1;577 return 1;
@@ -593,15 +593,16 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 15 unmodified lines
593 struct inode *inode, unsigned int from,593 struct inode *inode, unsigned int from,
594 unsigned int to, int new)594 unsigned int to, int new)
595{595{
596 struct folio *folio = page_folio(page);
596 int ret = 0;597 int ret = 0;
597 struct buffer_head *head, *bh, *wait[2], **wait_bh = wait;598 struct buffer_head *head, *bh, *wait[2], **wait_bh = wait;
598 unsigned int block_end, block_start;599 unsigned int block_end, block_start;
599 unsigned int bsize = i_blocksize(inode);600 unsigned int bsize = i_blocksize(inode);
600 601
601 if (!page_has_buffers(page))602 head = folio_buffers(folio);
602 create_empty_buffers(page, bsize, 0);603 if (!head)
604 head = create_empty_buffers(folio, bsize, 0);
603 605
604 head = page_buffers(page);
605 for (bh = head, block_start = 0; bh != head || !block_start;606 for (bh = head, block_start = 0; bh != head || !block_start;
606 bh = bh->b_this_page, block_start += bsize) {607 bh = bh->b_this_page, block_start += bsize) {
607 block_end = block_start + bsize;608 block_end = block_start + bsize;
@@ -613,7 +614,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 5 unmodified lines
613 * they may belong to unallocated clusters.614 * they may belong to unallocated clusters.
614 */615 */
615 if (block_start >= to || block_end <= from) {616 if (block_start >= to || block_end <= from) {
616 if (PageUptodate(page))617 if (folio_test_uptodate(folio))
617 set_buffer_uptodate(bh);618 set_buffer_uptodate(bh);
618 continue;619 continue;
619 }620 }
@@ -630,11 +631,11 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 10 unmodified lines
630 clean_bdev_bh_alias(bh);631 clean_bdev_bh_alias(bh);
631 }632 }
632 633
633 if (PageUptodate(page)) {634 if (folio_test_uptodate(folio)) {
634 set_buffer_uptodate(bh);635 set_buffer_uptodate(bh);
635 } else if (!buffer_uptodate(bh) && !buffer_delay(bh) &&636 } else if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
636 !buffer_new(bh) &&637 !buffer_new(bh) &&
637 ocfs2_should_read_blk(inode, page, block_start) &&638 ocfs2_should_read_blk(inode, folio, block_start) &&
638 (block_start < from || block_end > to)) {639 (block_start < from || block_end > to)) {
639 bh_read_nowait(bh, 0);640 bh_read_nowait(bh, 0);
640 *wait_bh++=bh;641 *wait_bh++=bh;
@@ -668,7 +669,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 27 unmodified lines
668 if (block_start >= to)669 if (block_start >= to)
669 break;670 break;
670 671
671 zero_user(page, block_start, bh->b_size);672 folio_zero_range(folio, block_start, bh->b_size);
672 set_buffer_uptodate(bh);673 set_buffer_uptodate(bh);
673 mark_buffer_dirty(bh);674 mark_buffer_dirty(bh);
674 675
fs/proc/kcore.c+2−1
@@ -546,7 +546,8 @@ static ssize_t read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) · 545 unmodified lines
546 * and explicitly excluded physical ranges.546 * and explicitly excluded physical ranges.
547 */547 */
548 if (!page || PageOffline(page) ||548 if (!page || PageOffline(page) ||
549 is_page_hwpoison(page) || !pfn_is_ram(pfn)) {549 is_page_hwpoison(page) || !pfn_is_ram(pfn) ||
550 pfn_is_unaccepted_memory(pfn)) {
550 if (iov_iter_zero(tsz, iter) != tsz) {551 if (iov_iter_zero(tsz, iter) != tsz) {
551 ret = -EFAULT;552 ret = -EFAULT;
552 goto out;553 goto out;
fs/proc/root.c+1−1
@@ -188,7 +188,7 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc) · 187 unmodified lines
188 s->s_stack_depth = FILESYSTEM_MAX_STACK_DEPTH;188 s->s_stack_depth = FILESYSTEM_MAX_STACK_DEPTH;
189 189
190 /* procfs dentries and inodes don't require IO to create */190 /* procfs dentries and inodes don't require IO to create */
191 s->s_shrink.seeks = 0;191 s->s_shrink->seeks = 0;
192 192
193 pde_get(&proc_root);193 pde_get(&proc_root);
194 root_inode = proc_get_inode(s, &proc_root);194 root_inode = proc_get_inode(s, &proc_root);
fs/proc/task_mmu.c+731−2
@@ -20,6 +20,8 @@ · 19 unmodified lines
20#include <linux/shmem_fs.h>20#include <linux/shmem_fs.h>
21#include <linux/uaccess.h>21#include <linux/uaccess.h>
22#include <linux/pkeys.h>22#include <linux/pkeys.h>
23#include <linux/minmax.h>
24#include <linux/overflow.h>
23 25
24#include <asm/elf.h>26#include <asm/elf.h>
25#include <asm/tlb.h>27#include <asm/tlb.h>
@@ -1761,11 +1763,737 @@ static int pagemap_release(struct inode *inode, struct file *file) · 1735 unmodified lines
1761 return 0;1763 return 0;
1762}1764}
1763 1765
1766#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \
1767 PAGE_IS_FILE | PAGE_IS_PRESENT | \
1768 PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \
1769 PAGE_IS_HUGE)
1770#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
1771
1772struct pagemap_scan_private {
1773 struct pm_scan_arg arg;
1774 unsigned long masks_of_interest, cur_vma_category;
1775 struct page_region *vec_buf;
1776 unsigned long vec_buf_len, vec_buf_index, found_pages;
1777 struct page_region __user *vec_out;
1778};
1779
1780static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
1781 struct vm_area_struct *vma,
1782 unsigned long addr, pte_t pte)
1783{
1784 unsigned long categories = 0;
1785
1786 if (pte_present(pte)) {
1787 struct page *page;
1788
1789 categories |= PAGE_IS_PRESENT;
1790 if (!pte_uffd_wp(pte))
1791 categories |= PAGE_IS_WRITTEN;
1792
1793 if (p->masks_of_interest & PAGE_IS_FILE) {
1794 page = vm_normal_page(vma, addr, pte);
1795 if (page && !PageAnon(page))
1796 categories |= PAGE_IS_FILE;
1797 }
1798
1799 if (is_zero_pfn(pte_pfn(pte)))
1800 categories |= PAGE_IS_PFNZERO;
1801 } else if (is_swap_pte(pte)) {
1802 swp_entry_t swp;
1803
1804 categories |= PAGE_IS_SWAPPED;
1805 if (!pte_swp_uffd_wp_any(pte))
1806 categories |= PAGE_IS_WRITTEN;
1807
1808 if (p->masks_of_interest & PAGE_IS_FILE) {
1809 swp = pte_to_swp_entry(pte);
1810 if (is_pfn_swap_entry(swp) &&
1811 !PageAnon(pfn_swap_entry_to_page(swp)))
1812 categories |= PAGE_IS_FILE;
1813 }
1814 }
1815
1816 return categories;
1817}
1818
1819static void make_uffd_wp_pte(struct vm_area_struct *vma,
1820 unsigned long addr, pte_t *pte)
1821{
1822 pte_t ptent = ptep_get(pte);
1823
1824 if (pte_present(ptent)) {
1825 pte_t old_pte;
1826
1827 old_pte = ptep_modify_prot_start(vma, addr, pte);
1828 ptent = pte_mkuffd_wp(ptent);
1829 ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
1830 } else if (is_swap_pte(ptent)) {
1831 ptent = pte_swp_mkuffd_wp(ptent);
1832 set_pte_at(vma->vm_mm, addr, pte, ptent);
1833 } else {
1834 set_pte_at(vma->vm_mm, addr, pte,
1835 make_pte_marker(PTE_MARKER_UFFD_WP));
1836 }
1837}
1838
1839#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1840static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
1841 struct vm_area_struct *vma,
1842 unsigned long addr, pmd_t pmd)
1843{
1844 unsigned long categories = PAGE_IS_HUGE;
1845
1846 if (pmd_present(pmd)) {
1847 struct page *page;
1848
1849 categories |= PAGE_IS_PRESENT;
1850 if (!pmd_uffd_wp(pmd))
1851 categories |= PAGE_IS_WRITTEN;
1852
1853 if (p->masks_of_interest & PAGE_IS_FILE) {
1854 page = vm_normal_page_pmd(vma, addr, pmd);
1855 if (page && !PageAnon(page))
1856 categories |= PAGE_IS_FILE;
1857 }
1858
1859 if (is_zero_pfn(pmd_pfn(pmd)))
1860 categories |= PAGE_IS_PFNZERO;
1861 } else if (is_swap_pmd(pmd)) {
1862 swp_entry_t swp;
1863
1864 categories |= PAGE_IS_SWAPPED;
1865 if (!pmd_swp_uffd_wp(pmd))
1866 categories |= PAGE_IS_WRITTEN;
1867
1868 if (p->masks_of_interest & PAGE_IS_FILE) {
1869 swp = pmd_to_swp_entry(pmd);
1870 if (is_pfn_swap_entry(swp) &&
1871 !PageAnon(pfn_swap_entry_to_page(swp)))
1872 categories |= PAGE_IS_FILE;
1873 }
1874 }
1875
1876 return categories;
1877}
1878
1879static void make_uffd_wp_pmd(struct vm_area_struct *vma,
1880 unsigned long addr, pmd_t *pmdp)
1881{
1882 pmd_t old, pmd = *pmdp;
1883
1884 if (pmd_present(pmd)) {
1885 old = pmdp_invalidate_ad(vma, addr, pmdp);
1886 pmd = pmd_mkuffd_wp(old);
1887 set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
1888 } else if (is_migration_entry(pmd_to_swp_entry(pmd))) {
1889 pmd = pmd_swp_mkuffd_wp(pmd);
1890 set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
1891 }
1892}
1893#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
1894
1895#ifdef CONFIG_HUGETLB_PAGE
1896static unsigned long pagemap_hugetlb_category(pte_t pte)
1897{
1898 unsigned long categories = PAGE_IS_HUGE;
1899
1900 /*
1901 * According to pagemap_hugetlb_range(), file-backed HugeTLB
1902 * page cannot be swapped. So PAGE_IS_FILE is not checked for
1903 * swapped pages.
1904 */
1905 if (pte_present(pte)) {
1906 categories |= PAGE_IS_PRESENT;
1907 if (!huge_pte_uffd_wp(pte))
1908 categories |= PAGE_IS_WRITTEN;
1909 if (!PageAnon(pte_page(pte)))
1910 categories |= PAGE_IS_FILE;
1911 if (is_zero_pfn(pte_pfn(pte)))
1912 categories |= PAGE_IS_PFNZERO;
1913 } else if (is_swap_pte(pte)) {
1914 categories |= PAGE_IS_SWAPPED;
1915 if (!pte_swp_uffd_wp_any(pte))
1916 categories |= PAGE_IS_WRITTEN;
1917 }
1918
1919 return categories;
1920}
1921
1922static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
1923 unsigned long addr, pte_t *ptep,
1924 pte_t ptent)
1925{
1926 unsigned long psize;
1927
1928 if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent))
1929 return;
1930
1931 psize = huge_page_size(hstate_vma(vma));
1932
1933 if (is_hugetlb_entry_migration(ptent))
1934 set_huge_pte_at(vma->vm_mm, addr, ptep,
1935 pte_swp_mkuffd_wp(ptent), psize);
1936 else if (!huge_pte_none(ptent))
1937 huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
1938 huge_pte_mkuffd_wp(ptent));
1939 else
1940 set_huge_pte_at(vma->vm_mm, addr, ptep,
1941 make_pte_marker(PTE_MARKER_UFFD_WP), psize);
1942}
1943#endif /* CONFIG_HUGETLB_PAGE */
1944
1945#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE)
1946static void pagemap_scan_backout_range(struct pagemap_scan_private *p,
1947 unsigned long addr, unsigned long end)
1948{
1949 struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index];
1950
1951 if (cur_buf->start != addr)
1952 cur_buf->end = addr;
1953 else
1954 cur_buf->start = cur_buf->end = 0;
1955
1956 p->found_pages -= (end - addr) / PAGE_SIZE;
1957}
1958#endif
1959
1960static bool pagemap_scan_is_interesting_page(unsigned long categories,
1961 const struct pagemap_scan_private *p)
1962{
1963 categories ^= p->arg.category_inverted;
1964 if ((categories & p->arg.category_mask) != p->arg.category_mask)
1965 return false;
1966 if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask))
1967 return false;
1968
1969 return true;
1970}
1971
1972static bool pagemap_scan_is_interesting_vma(unsigned long categories,
1973 const struct pagemap_scan_private *p)
1974{
1975 unsigned long required = p->arg.category_mask & PAGE_IS_WPALLOWED;
1976
1977 categories ^= p->arg.category_inverted;
1978 if ((categories & required) != required)
1979 return false;
1980
1981 return true;
1982}
1983
1984static int pagemap_scan_test_walk(unsigned long start, unsigned long end,
1985 struct mm_walk *walk)
1986{
1987 struct pagemap_scan_private *p = walk->private;
1988 struct vm_area_struct *vma = walk->vma;
1989 unsigned long vma_category = 0;
1990
1991 if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma))
1992 vma_category |= PAGE_IS_WPALLOWED;
1993 else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC)
1994 return -EPERM;
1995
1996 if (vma->vm_flags & VM_PFNMAP)
1997 return 1;
1998
1999 if (!pagemap_scan_is_interesting_vma(vma_category, p))
2000 return 1;
2001
2002 p->cur_vma_category = vma_category;
2003
2004 return 0;
2005}
2006
2007static bool pagemap_scan_push_range(unsigned long categories,
2008 struct pagemap_scan_private *p,
2009 unsigned long addr, unsigned long end)
2010{
2011 struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index];
2012
2013 /*
2014 * When there is no output buffer provided at all, the sentinel values
2015 * won't match here. There is no other way for `cur_buf->end` to be
2016 * non-zero other than it being non-empty.
2017 */
2018 if (addr == cur_buf->end && categories == cur_buf->categories) {
2019 cur_buf->end = end;
2020 return true;
2021 }
2022
2023 if (cur_buf->end) {
2024 if (p->vec_buf_index >= p->vec_buf_len - 1)
2025 return false;
2026
2027 cur_buf = &p->vec_buf[++p->vec_buf_index];
2028 }
2029
2030 cur_buf->start = addr;
2031 cur_buf->end = end;
2032 cur_buf->categories = categories;
2033
2034 return true;
2035}
2036
2037static int pagemap_scan_output(unsigned long categories,
2038 struct pagemap_scan_private *p,
2039 unsigned long addr, unsigned long *end)
2040{
2041 unsigned long n_pages, total_pages;
2042 int ret = 0;
2043
2044 if (!p->vec_buf)
2045 return 0;
2046
2047 categories &= p->arg.return_mask;
2048
2049 n_pages = (*end - addr) / PAGE_SIZE;
2050 if (check_add_overflow(p->found_pages, n_pages, &total_pages) ||
2051 total_pages > p->arg.max_pages) {
2052 size_t n_too_much = total_pages - p->arg.max_pages;
2053 *end -= n_too_much * PAGE_SIZE;
2054 n_pages -= n_too_much;
2055 ret = -ENOSPC;
2056 }
2057
2058 if (!pagemap_scan_push_range(categories, p, addr, *end)) {
2059 *end = addr;
2060 n_pages = 0;
2061 ret = -ENOSPC;
2062 }
2063
2064 p->found_pages += n_pages;
2065 if (ret)
2066 p->arg.walk_end = *end;
2067
2068 return ret;
2069}
2070
2071static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start,
2072 unsigned long end, struct mm_walk *walk)
2073{
2074#ifdef CONFIG_TRANSPARENT_HUGEPAGE
2075 struct pagemap_scan_private *p = walk->private;
2076 struct vm_area_struct *vma = walk->vma;
2077 unsigned long categories;
2078 spinlock_t *ptl;
2079 int ret = 0;
2080
2081 ptl = pmd_trans_huge_lock(pmd, vma);
2082 if (!ptl)
2083 return -ENOENT;
2084
2085 categories = p->cur_vma_category |
2086 pagemap_thp_category(p, vma, start, *pmd);
2087
2088 if (!pagemap_scan_is_interesting_page(categories, p))
2089 goto out_unlock;
2090
2091 ret = pagemap_scan_output(categories, p, start, &end);
2092 if (start == end)
2093 goto out_unlock;
2094
2095 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2096 goto out_unlock;
2097 if (~categories & PAGE_IS_WRITTEN)
2098 goto out_unlock;
2099
2100 /*
2101 * Break huge page into small pages if the WP operation
2102 * needs to be performed on a portion of the huge page.
2103 */
2104 if (end != start + HPAGE_SIZE) {
2105 spin_unlock(ptl);
2106 split_huge_pmd(vma, pmd, start);
2107 pagemap_scan_backout_range(p, start, end);
2108 /* Report as if there was no THP */
2109 return -ENOENT;
2110 }
2111
2112 make_uffd_wp_pmd(vma, start, pmd);
2113 flush_tlb_range(vma, start, end);
2114out_unlock:
2115 spin_unlock(ptl);
2116 return ret;
2117#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
2118 return -ENOENT;
2119#endif
2120}
2121
2122static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
2123 unsigned long end, struct mm_walk *walk)
2124{
2125 struct pagemap_scan_private *p = walk->private;
2126 struct vm_area_struct *vma = walk->vma;
2127 unsigned long addr, flush_end = 0;
2128 pte_t *pte, *start_pte;
2129 spinlock_t *ptl;
2130 int ret;
2131
2132 arch_enter_lazy_mmu_mode();
2133
2134 ret = pagemap_scan_thp_entry(pmd, start, end, walk);
2135 if (ret != -ENOENT) {
2136 arch_leave_lazy_mmu_mode();
2137 return ret;
2138 }
2139
2140 ret = 0;
2141 start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl);
2142 if (!pte) {
2143 arch_leave_lazy_mmu_mode();
2144 walk->action = ACTION_AGAIN;
2145 return 0;
2146 }
2147
2148 if (!p->vec_out) {
2149 /* Fast path for performing exclusive WP */
2150 for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
2151 if (pte_uffd_wp(ptep_get(pte)))
2152 continue;
2153 make_uffd_wp_pte(vma, addr, pte);
2154 if (!flush_end)
2155 start = addr;
2156 flush_end = addr + PAGE_SIZE;
2157 }
2158 goto flush_and_return;
2159 }
2160
2161 if (!p->arg.category_anyof_mask && !p->arg.category_inverted &&
2162 p->arg.category_mask == PAGE_IS_WRITTEN &&
2163 p->arg.return_mask == PAGE_IS_WRITTEN) {
2164 for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
2165 unsigned long next = addr + PAGE_SIZE;
2166
2167 if (pte_uffd_wp(ptep_get(pte)))
2168 continue;
2169 ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN,
2170 p, addr, &next);
2171 if (next == addr)
2172 break;
2173 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2174 continue;
2175 make_uffd_wp_pte(vma, addr, pte);
2176 if (!flush_end)
2177 start = addr;
2178 flush_end = next;
2179 }
2180 goto flush_and_return;
2181 }
2182
2183 for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
2184 unsigned long categories = p->cur_vma_category |
2185 pagemap_page_category(p, vma, addr, ptep_get(pte));
2186 unsigned long next = addr + PAGE_SIZE;
2187
2188 if (!pagemap_scan_is_interesting_page(categories, p))
2189 continue;
2190
2191 ret = pagemap_scan_output(categories, p, addr, &next);
2192 if (next == addr)
2193 break;
2194
2195 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2196 continue;
2197 if (~categories & PAGE_IS_WRITTEN)
2198 continue;
2199
2200 make_uffd_wp_pte(vma, addr, pte);
2201 if (!flush_end)
2202 start = addr;
2203 flush_end = next;
2204 }
2205
2206flush_and_return:
2207 if (flush_end)
2208 flush_tlb_range(vma, start, addr);
2209
2210 pte_unmap_unlock(start_pte, ptl);
2211 arch_leave_lazy_mmu_mode();
2212
2213 cond_resched();
2214 return ret;
2215}
2216
2217#ifdef CONFIG_HUGETLB_PAGE
2218static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
2219 unsigned long start, unsigned long end,
2220 struct mm_walk *walk)
2221{
2222 struct pagemap_scan_private *p = walk->private;
2223 struct vm_area_struct *vma = walk->vma;
2224 unsigned long categories;
2225 spinlock_t *ptl;
2226 int ret = 0;
2227 pte_t pte;
2228
2229 if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
2230 /* Go the short route when not write-protecting pages. */
2231
2232 pte = huge_ptep_get(ptep);
2233 categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
2234
2235 if (!pagemap_scan_is_interesting_page(categories, p))
2236 return 0;
2237
2238 return pagemap_scan_output(categories, p, start, &end);
2239 }
2240
2241 i_mmap_lock_write(vma->vm_file->f_mapping);
2242 ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
2243
2244 pte = huge_ptep_get(ptep);
2245 categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
2246
2247 if (!pagemap_scan_is_interesting_page(categories, p))
2248 goto out_unlock;
2249
2250 ret = pagemap_scan_output(categories, p, start, &end);
2251 if (start == end)
2252 goto out_unlock;
2253
2254 if (~categories & PAGE_IS_WRITTEN)
2255 goto out_unlock;
2256
2257 if (end != start + HPAGE_SIZE) {
2258 /* Partial HugeTLB page WP isn't possible. */
2259 pagemap_scan_backout_range(p, start, end);
2260 p->arg.walk_end = start;
2261 ret = 0;
2262 goto out_unlock;
2263 }
2264
2265 make_uffd_wp_huge_pte(vma, start, ptep, pte);
2266 flush_hugetlb_tlb_range(vma, start, end);
2267
2268out_unlock:
2269 spin_unlock(ptl);
2270 i_mmap_unlock_write(vma->vm_file->f_mapping);
2271
2272 return ret;
2273}
2274#else
2275#define pagemap_scan_hugetlb_entry NULL
2276#endif
2277
2278static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
2279 int depth, struct mm_walk *walk)
2280{
2281 struct pagemap_scan_private *p = walk->private;
2282 struct vm_area_struct *vma = walk->vma;
2283 int ret, err;
2284
2285 if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p))
2286 return 0;
2287
2288 ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end);
2289 if (addr == end)
2290 return ret;
2291
2292 if (~p->arg.flags & PM_SCAN_WP_MATCHING)
2293 return ret;
2294
2295 err = uffd_wp_range(vma, addr, end - addr, true);
2296 if (err < 0)
2297 ret = err;
2298
2299 return ret;
2300}
2301
2302static const struct mm_walk_ops pagemap_scan_ops = {
2303 .test_walk = pagemap_scan_test_walk,
2304 .pmd_entry = pagemap_scan_pmd_entry,
2305 .pte_hole = pagemap_scan_pte_hole,
2306 .hugetlb_entry = pagemap_scan_hugetlb_entry,
2307};
2308
2309static int pagemap_scan_get_args(struct pm_scan_arg *arg,
2310 unsigned long uarg)
2311{
2312 if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg)))
2313 return -EFAULT;
2314
2315 if (arg->size != sizeof(struct pm_scan_arg))
2316 return -EINVAL;
2317
2318 /* Validate requested features */
2319 if (arg->flags & ~PM_SCAN_FLAGS)
2320 return -EINVAL;
2321 if ((arg->category_inverted | arg->category_mask |
2322 arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES)
2323 return -EINVAL;
2324
2325 arg->start = untagged_addr((unsigned long)arg->start);
2326 arg->end = untagged_addr((unsigned long)arg->end);
2327 arg->vec = untagged_addr((unsigned long)arg->vec);
2328
2329 /* Validate memory pointers */
2330 if (!IS_ALIGNED(arg->start, PAGE_SIZE))
2331 return -EINVAL;
2332 if (!access_ok((void __user *)(long)arg->start, arg->end - arg->start))
2333 return -EFAULT;
2334 if (!arg->vec && arg->vec_len)
2335 return -EINVAL;
2336 if (arg->vec && !access_ok((void __user *)(long)arg->vec,
2337 arg->vec_len * sizeof(struct page_region)))
2338 return -EFAULT;
2339
2340 /* Fixup default values */
2341 arg->end = ALIGN(arg->end, PAGE_SIZE);
2342 arg->walk_end = 0;
2343 if (!arg->max_pages)
2344 arg->max_pages = ULONG_MAX;
2345
2346 return 0;
2347}
2348
2349static int pagemap_scan_writeback_args(struct pm_scan_arg *arg,
2350 unsigned long uargl)
2351{
2352 struct pm_scan_arg __user *uarg = (void __user *)uargl;
2353
2354 if (copy_to_user(&uarg->walk_end, &arg->walk_end, sizeof(arg->walk_end)))
2355 return -EFAULT;
2356
2357 return 0;
2358}
2359
2360static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p)
2361{
2362 if (!p->arg.vec_len)
2363 return 0;
2364
2365 p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT,
2366 p->arg.vec_len);
2367 p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf),
2368 GFP_KERNEL);
2369 if (!p->vec_buf)
2370 return -ENOMEM;
2371
2372 p->vec_buf->start = p->vec_buf->end = 0;
2373 p->vec_out = (struct page_region __user *)(long)p->arg.vec;
2374
2375 return 0;
2376}
2377
2378static long pagemap_scan_flush_buffer(struct pagemap_scan_private *p)
2379{
2380 const struct page_region *buf = p->vec_buf;
2381 long n = p->vec_buf_index;
2382
2383 if (!p->vec_buf)
2384 return 0;
2385
2386 if (buf[n].end != buf[n].start)
2387 n++;
2388
2389 if (!n)
2390 return 0;
2391
2392 if (copy_to_user(p->vec_out, buf, n * sizeof(*buf)))
2393 return -EFAULT;
2394
2395 p->arg.vec_len -= n;
2396 p->vec_out += n;
2397
2398 p->vec_buf_index = 0;
2399 p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len);
2400 p->vec_buf->start = p->vec_buf->end = 0;
2401
2402 return n;
2403}
2404
2405static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg)
2406{
2407 struct mmu_notifier_range range;
2408 struct pagemap_scan_private p = {0};
2409 unsigned long walk_start;
2410 size_t n_ranges_out = 0;
2411 int ret;
2412
2413 ret = pagemap_scan_get_args(&p.arg, uarg);
2414 if (ret)
2415 return ret;
2416
2417 p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask |
2418 p.arg.return_mask;
2419 ret = pagemap_scan_init_bounce_buffer(&p);
2420 if (ret)
2421 return ret;
2422
2423 /* Protection change for the range is going to happen. */
2424 if (p.arg.flags & PM_SCAN_WP_MATCHING) {
2425 mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
2426 mm, p.arg.start, p.arg.end);
2427 mmu_notifier_invalidate_range_start(&range);
2428 }
2429
2430 for (walk_start = p.arg.start; walk_start < p.arg.end;
2431 walk_start = p.arg.walk_end) {
2432 long n_out;
2433
2434 if (fatal_signal_pending(current)) {
2435 ret = -EINTR;
2436 break;
2437 }
2438
2439 ret = mmap_read_lock_killable(mm);
2440 if (ret)
2441 break;
2442 ret = walk_page_range(mm, walk_start, p.arg.end,
2443 &pagemap_scan_ops, &p);
2444 mmap_read_unlock(mm);
2445
2446 n_out = pagemap_scan_flush_buffer(&p);
2447 if (n_out < 0)
2448 ret = n_out;
2449 else
2450 n_ranges_out += n_out;
2451
2452 if (ret != -ENOSPC)
2453 break;
2454
2455 if (p.arg.vec_len == 0 || p.found_pages == p.arg.max_pages)
2456 break;
2457 }
2458
2459 /* ENOSPC signifies early stop (buffer full) from the walk. */
2460 if (!ret || ret == -ENOSPC)
2461 ret = n_ranges_out;
2462
2463 /* The walk_end isn't set when ret is zero */
2464 if (!p.arg.walk_end)
2465 p.arg.walk_end = p.arg.end;
2466 if (pagemap_scan_writeback_args(&p.arg, uarg))
2467 ret = -EFAULT;
2468
2469 if (p.arg.flags & PM_SCAN_WP_MATCHING)
2470 mmu_notifier_invalidate_range_end(&range);
2471
2472 kfree(p.vec_buf);
2473 return ret;
2474}
2475
2476static long do_pagemap_cmd(struct file *file, unsigned int cmd,
2477 unsigned long arg)
2478{
2479 struct mm_struct *mm = file->private_data;
2480
2481 switch (cmd) {
2482 case PAGEMAP_SCAN:
2483 return do_pagemap_scan(mm, arg);
2484
2485 default:
2486 return -EINVAL;
2487 }
2488}
2489
1764const struct file_operations proc_pagemap_operations = {2490const struct file_operations proc_pagemap_operations = {
1765 .llseek = mem_lseek, /* borrow this */2491 .llseek = mem_lseek, /* borrow this */
1766 .read = pagemap_read,2492 .read = pagemap_read,
1767 .open = pagemap_open,2493 .open = pagemap_open,
1768 .release = pagemap_release,2494 .release = pagemap_release,
2495 .unlocked_ioctl = do_pagemap_cmd,
2496 .compat_ioctl = do_pagemap_cmd,
1769};2497};
1770#endif /* CONFIG_PROC_PAGE_MONITOR */2498#endif /* CONFIG_PROC_PAGE_MONITOR */
1771 2499
@@ -1945,8 +2673,9 @@ static int show_numa_map(struct seq_file *m, void *v) · 173 unmodified lines
1945 struct numa_maps *md = &numa_priv->md;2673 struct numa_maps *md = &numa_priv->md;
1946 struct file *file = vma->vm_file;2674 struct file *file = vma->vm_file;
1947 struct mm_struct *mm = vma->vm_mm;2675 struct mm_struct *mm = vma->vm_mm;
1948 struct mempolicy *pol;
1949 char buffer[64];2676 char buffer[64];
2677 struct mempolicy *pol;
2678 pgoff_t ilx;
1950 int nid;2679 int nid;
1951 2680
1952 if (!mm)2681 if (!mm)
@@ -1955,7 +2684,7 @@ static int show_numa_map(struct seq_file *m, void *v) · 2 unmodified lines
1955 /* Ensure we start with an empty set of numa_maps statistics. */2684 /* Ensure we start with an empty set of numa_maps statistics. */
1956 memset(md, 0, sizeof(*md));2685 memset(md, 0, sizeof(*md));
1957 2686
1958 pol = __get_vma_policy(vma, vma->vm_start);2687 pol = __get_vma_policy(vma, vma->vm_start, &ilx);
1959 if (pol) {2688 if (pol) {
1960 mpol_to_str(buffer, sizeof(buffer), pol);2689 mpol_to_str(buffer, sizeof(buffer), pol);
1961 mpol_cond_put(pol);2690 mpol_cond_put(pol);
fs/quota/dquot.c+9−8
@@ -803,12 +803,6 @@ dqcache_shrink_count(struct shrinker *shrink, struct shrink_control *sc) · 802 unmodified lines
803 percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));803 percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));
804}804}
805 805
806static struct shrinker dqcache_shrinker = {
807 .count_objects = dqcache_shrink_count,
808 .scan_objects = dqcache_shrink_scan,
809 .seeks = DEFAULT_SEEKS,
810};
811
812/*806/*
813 * Safely release dquot and put reference to dquot.807 * Safely release dquot and put reference to dquot.
814 */808 */
@@ -2982,6 +2976,7 @@ static int __init dquot_init(void) · 2167 unmodified lines
2982{2976{
2983 int i, ret;2977 int i, ret;
2984 unsigned long nr_hash, order;2978 unsigned long nr_hash, order;
2979 struct shrinker *dqcache_shrinker;
2985 2980
2986 printk(KERN_NOTICE "VFS: Disk quotas %s\n", __DQUOT_VERSION__);2981 printk(KERN_NOTICE "VFS: Disk quotas %s\n", __DQUOT_VERSION__);
2987 2982
@@ -3016,8 +3011,14 @@ static int __init dquot_init(void) · 28 unmodified lines
3016 pr_info("VFS: Dquot-cache hash table entries: %ld (order %ld,"3011 pr_info("VFS: Dquot-cache hash table entries: %ld (order %ld,"
3017 " %ld bytes)\n", nr_hash, order, (PAGE_SIZE << order));3012 " %ld bytes)\n", nr_hash, order, (PAGE_SIZE << order));
3018 3013
3019 if (register_shrinker(&dqcache_shrinker, "dquota-cache"))3014 dqcache_shrinker = shrinker_alloc(0, "dquota-cache");
3020 panic("Cannot register dquot shrinker");3015 if (!dqcache_shrinker)
3016 panic("Cannot allocate dquot shrinker");
3017
3018 dqcache_shrinker->count_objects = dqcache_shrink_count;
3019 dqcache_shrinker->scan_objects = dqcache_shrink_scan;
3020
3021 shrinker_register(dqcache_shrinker);
3021 3022
3022 return 0;3023 return 0;
3023}3024}
fs/reiserfs/inode.c+40−40
@@ -2503,10 +2503,10 @@ out: · 2502 unmodified lines
2503 * start/recovery path as __block_write_full_folio, along with special2503 * start/recovery path as __block_write_full_folio, along with special
2504 * code to handle reiserfs tails.2504 * code to handle reiserfs tails.
2505 */2505 */
2506static int reiserfs_write_full_page(struct page *page,2506static int reiserfs_write_full_folio(struct folio *folio,
2507 struct writeback_control *wbc)2507 struct writeback_control *wbc)
2508{2508{
2509 struct inode *inode = page->mapping->host;2509 struct inode *inode = folio->mapping->host;
2510 unsigned long end_index = inode->i_size >> PAGE_SHIFT;2510 unsigned long end_index = inode->i_size >> PAGE_SHIFT;
2511 int error = 0;2511 int error = 0;
2512 unsigned long block;2512 unsigned long block;
@@ -2514,7 +2514,7 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line
2514 struct buffer_head *head, *bh;2514 struct buffer_head *head, *bh;
2515 int partial = 0;2515 int partial = 0;
2516 int nr = 0;2516 int nr = 0;
2517 int checked = PageChecked(page);2517 int checked = folio_test_checked(folio);
2518 struct reiserfs_transaction_handle th;2518 struct reiserfs_transaction_handle th;
2519 struct super_block *s = inode->i_sb;2519 struct super_block *s = inode->i_sb;
2520 int bh_per_page = PAGE_SIZE / s->s_blocksize;2520 int bh_per_page = PAGE_SIZE / s->s_blocksize;
@@ -2522,47 +2522,46 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line
2522 2522
2523 /* no logging allowed when nonblocking or from PF_MEMALLOC */2523 /* no logging allowed when nonblocking or from PF_MEMALLOC */
2524 if (checked && (current->flags & PF_MEMALLOC)) {2524 if (checked && (current->flags & PF_MEMALLOC)) {
2525 redirty_page_for_writepage(wbc, page);2525 folio_redirty_for_writepage(wbc, folio);
2526 unlock_page(page);2526 folio_unlock(folio);
2527 return 0;2527 return 0;
2528 }2528 }
2529 2529
2530 /*2530 /*
2531 * The page dirty bit is cleared before writepage is called, which2531 * The folio dirty bit is cleared before writepage is called, which
2532 * means we have to tell create_empty_buffers to make dirty buffers2532 * means we have to tell create_empty_buffers to make dirty buffers
2533 * The page really should be up to date at this point, so tossing2533 * The folio really should be up to date at this point, so tossing
2534 * in the BH_Uptodate is just a sanity check.2534 * in the BH_Uptodate is just a sanity check.
2535 */2535 */
2536 if (!page_has_buffers(page)) {2536 head = folio_buffers(folio);
2537 create_empty_buffers(page, s->s_blocksize,2537 if (!head)
2538 head = create_empty_buffers(folio, s->s_blocksize,
2538 (1 << BH_Dirty) | (1 << BH_Uptodate));2539 (1 << BH_Dirty) | (1 << BH_Uptodate));
2539 }
2540 head = page_buffers(page);
2541 2540
2542 /*2541 /*
2543 * last page in the file, zero out any contents past the2542 * last folio in the file, zero out any contents past the
2544 * last byte in the file2543 * last byte in the file
2545 */2544 */
2546 if (page->index >= end_index) {2545 if (folio->index >= end_index) {
2547 unsigned last_offset;2546 unsigned last_offset;
2548 2547
2549 last_offset = inode->i_size & (PAGE_SIZE - 1);2548 last_offset = inode->i_size & (PAGE_SIZE - 1);
2550 /* no file contents in this page */2549 /* no file contents in this folio */
2551 if (page->index >= end_index + 1 || !last_offset) {2550 if (folio->index >= end_index + 1 || !last_offset) {
2552 unlock_page(page);2551 folio_unlock(folio);
2553 return 0;2552 return 0;
2554 }2553 }
2555 zero_user_segment(page, last_offset, PAGE_SIZE);2554 folio_zero_segment(folio, last_offset, folio_size(folio));
2556 }2555 }
2557 bh = head;2556 bh = head;
2558 block = page->index << (PAGE_SHIFT - s->s_blocksize_bits);2557 block = folio->index << (PAGE_SHIFT - s->s_blocksize_bits);
2559 last_block = (i_size_read(inode) - 1) >> inode->i_blkbits;2558 last_block = (i_size_read(inode) - 1) >> inode->i_blkbits;
2560 /* first map all the buffers, logging any direct items we find */2559 /* first map all the buffers, logging any direct items we find */
2561 do {2560 do {
2562 if (block > last_block) {2561 if (block > last_block) {
2563 /*2562 /*
2564 * This can happen when the block size is less than2563 * This can happen when the block size is less than
2565 * the page size. The corresponding bytes in the page2564 * the folio size. The corresponding bytes in the folio
2566 * were zero filled above2565 * were zero filled above
2567 */2566 */
2568 clear_buffer_dirty(bh);2567 clear_buffer_dirty(bh);
@@ -2589,7 +2588,7 @@ static int reiserfs_write_full_page(struct page *page, · 20 unmodified lines
2589 * blocks we're going to log2588 * blocks we're going to log
2590 */2589 */
2591 if (checked) {2590 if (checked) {
2592 ClearPageChecked(page);2591 folio_clear_checked(folio);
2593 reiserfs_write_lock(s);2592 reiserfs_write_lock(s);
2594 error = journal_begin(&th, s, bh_per_page + 1);2593 error = journal_begin(&th, s, bh_per_page + 1);
2595 if (error) {2594 if (error) {
@@ -2598,7 +2597,7 @@ static int reiserfs_write_full_page(struct page *page, · 2 unmodified lines
2598 }2597 }
2599 reiserfs_update_inode_transaction(inode);2598 reiserfs_update_inode_transaction(inode);
2600 }2599 }
2601 /* now go through and lock any dirty buffers on the page */2600 /* now go through and lock any dirty buffers on the folio */
2602 do {2601 do {
2603 get_bh(bh);2602 get_bh(bh);
2604 if (!buffer_mapped(bh))2603 if (!buffer_mapped(bh))
@@ -2619,7 +2618,7 @@ static int reiserfs_write_full_page(struct page *page, · 14 unmodified lines
2619 lock_buffer(bh);2618 lock_buffer(bh);
2620 } else {2619 } else {
2621 if (!trylock_buffer(bh)) {2620 if (!trylock_buffer(bh)) {
2622 redirty_page_for_writepage(wbc, page);2621 folio_redirty_for_writepage(wbc, folio);
2623 continue;2622 continue;
2624 }2623 }
2625 }2624 }
@@ -2636,13 +2635,13 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines
2636 if (error)2635 if (error)
2637 goto fail;2636 goto fail;
2638 }2637 }
2639 BUG_ON(PageWriteback(page));2638 BUG_ON(folio_test_writeback(folio));
2640 set_page_writeback(page);2639 folio_start_writeback(folio);
2641 unlock_page(page);2640 folio_unlock(folio);
2642 2641
2643 /*2642 /*
2644 * since any buffer might be the only dirty buffer on the page,2643 * since any buffer might be the only dirty buffer on the folio,
2645 * the first submit_bh can bring the page out of writeback.2644 * the first submit_bh can bring the folio out of writeback.
2646 * be careful with the buffers.2645 * be careful with the buffers.
2647 */2646 */
2648 do {2647 do {
@@ -2659,10 +2658,10 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines
2659done:2658done:
2660 if (nr == 0) {2659 if (nr == 0) {
2661 /*2660 /*
2662 * if this page only had a direct item, it is very possible for2661 * if this folio only had a direct item, it is very possible for
2663 * no io to be required without there being an error. Or,2662 * no io to be required without there being an error. Or,
2664 * someone else could have locked them and sent them down the2663 * someone else could have locked them and sent them down the
2665 * pipe without locking the page2664 * pipe without locking the folio
2666 */2665 */
2667 bh = head;2666 bh = head;
2668 do {2667 do {
@@ -2673,18 +2672,18 @@ done: · 4 unmodified lines
2673 bh = bh->b_this_page;2672 bh = bh->b_this_page;
2674 } while (bh != head);2673 } while (bh != head);
2675 if (!partial)2674 if (!partial)
2676 SetPageUptodate(page);2675 folio_mark_uptodate(folio);
2677 end_page_writeback(page);2676 folio_end_writeback(folio);
2678 }2677 }
2679 return error;2678 return error;
2680 2679
2681fail:2680fail:
2682 /*2681 /*
2683 * catches various errors, we need to make sure any valid dirty blocks2682 * catches various errors, we need to make sure any valid dirty blocks
2684 * get to the media. The page is currently locked and not marked for2683 * get to the media. The folio is currently locked and not marked for
2685 * writeback2684 * writeback
2686 */2685 */
2687 ClearPageUptodate(page);2686 folio_clear_uptodate(folio);
2688 bh = head;2687 bh = head;
2689 do {2688 do {
2690 get_bh(bh);2689 get_bh(bh);
@@ -2694,16 +2693,16 @@ fail: · 3 unmodified lines
2694 } else {2693 } else {
2695 /*2694 /*
2696 * clear any dirty bits that might have come from2695 * clear any dirty bits that might have come from
2697 * getting attached to a dirty page2696 * getting attached to a dirty folio
2698 */2697 */
2699 clear_buffer_dirty(bh);2698 clear_buffer_dirty(bh);
2700 }2699 }
2701 bh = bh->b_this_page;2700 bh = bh->b_this_page;
2702 } while (bh != head);2701 } while (bh != head);
2703 SetPageError(page);2702 folio_set_error(folio);
2704 BUG_ON(PageWriteback(page));2703 BUG_ON(folio_test_writeback(folio));
2705 set_page_writeback(page);2704 folio_start_writeback(folio);
2706 unlock_page(page);2705 folio_unlock(folio);
2707 do {2706 do {
2708 struct buffer_head *next = bh->b_this_page;2707 struct buffer_head *next = bh->b_this_page;
2709 if (buffer_async_write(bh)) {2708 if (buffer_async_write(bh)) {
@@ -2724,9 +2723,10 @@ static int reiserfs_read_folio(struct file *f, struct folio *folio) · 14 unmodified lines
2724 2723
2725static int reiserfs_writepage(struct page *page, struct writeback_control *wbc)2724static int reiserfs_writepage(struct page *page, struct writeback_control *wbc)
2726{2725{
2727 struct inode *inode = page->mapping->host;2726 struct folio *folio = page_folio(page);
2727 struct inode *inode = folio->mapping->host;
2728 reiserfs_wait_on_write_block(inode->i_sb);2728 reiserfs_wait_on_write_block(inode->i_sb);
2729 return reiserfs_write_full_page(page, wbc);2729 return reiserfs_write_full_folio(folio, wbc);
2730}2730}
2731 2731
2732static void reiserfs_truncate_failed_write(struct inode *inode)2732static void reiserfs_truncate_failed_write(struct inode *inode)
fs/super.c+19−16
@@ -178,7 +178,7 @@ static void super_wake(struct super_block *sb, unsigned int flag) · 177 unmodified lines
178 * One thing we have to be careful of with a per-sb shrinker is that we don't178 * One thing we have to be careful of with a per-sb shrinker is that we don't
179 * drop the last active reference to the superblock from within the shrinker.179 * drop the last active reference to the superblock from within the shrinker.
180 * If that happens we could trigger unregistering the shrinker from within the180 * If that happens we could trigger unregistering the shrinker from within the
181 * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we181 * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we
182 * take a passive reference to the superblock to avoid this from occurring.182 * take a passive reference to the superblock to avoid this from occurring.
183 */183 */
184static unsigned long super_cache_scan(struct shrinker *shrink,184static unsigned long super_cache_scan(struct shrinker *shrink,
@@ -191,7 +191,7 @@ static unsigned long super_cache_scan(struct shrinker *shrink, · 6 unmodified lines
191 long dentries;191 long dentries;
192 long inodes;192 long inodes;
193 193
194 sb = container_of(shrink, struct super_block, s_shrink);194 sb = shrink->private_data;
195 195
196 /*196 /*
197 * Deadlock avoidance. We may hold various FS locks, and we don't want197 * Deadlock avoidance. We may hold various FS locks, and we don't want
@@ -244,7 +244,7 @@ static unsigned long super_cache_count(struct shrinker *shrink, · 46 unmodified lines
244 struct super_block *sb;244 struct super_block *sb;
245 long total_objects = 0;245 long total_objects = 0;
246 246
247 sb = container_of(shrink, struct super_block, s_shrink);247 sb = shrink->private_data;
248 248
249 /*249 /*
250 * We don't call super_trylock_shared() here as it is a scalability250 * We don't call super_trylock_shared() here as it is a scalability
@@ -306,7 +306,7 @@ static void destroy_unused_super(struct super_block *s) · 55 unmodified lines
306 security_sb_free(s);306 security_sb_free(s);
307 put_user_ns(s->s_user_ns);307 put_user_ns(s->s_user_ns);
308 kfree(s->s_subtype);308 kfree(s->s_subtype);
309 free_prealloced_shrinker(&s->s_shrink);309 shrinker_free(s->s_shrink);
310 /* no delays needed */310 /* no delays needed */
311 destroy_super_work(&s->destroy_work);311 destroy_super_work(&s->destroy_work);
312}312}
@@ -383,16 +383,19 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags, · 70 unmodified lines
383 s->s_time_min = TIME64_MIN;383 s->s_time_min = TIME64_MIN;
384 s->s_time_max = TIME64_MAX;384 s->s_time_max = TIME64_MAX;
385 385
386 s->s_shrink.seeks = DEFAULT_SEEKS;386 s->s_shrink = shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE,
387 s->s_shrink.scan_objects = super_cache_scan;387 "sb-%s", type->name);
388 s->s_shrink.count_objects = super_cache_count;388 if (!s->s_shrink)
389 s->s_shrink.batch = 1024;
390 s->s_shrink.flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE;
391 if (prealloc_shrinker(&s->s_shrink, "sb-%s", type->name))
392 goto fail;389 goto fail;
393 if (list_lru_init_memcg(&s->s_dentry_lru, &s->s_shrink))390
391 s->s_shrink->scan_objects = super_cache_scan;
392 s->s_shrink->count_objects = super_cache_count;
393 s->s_shrink->batch = 1024;
394 s->s_shrink->private_data = s;
395
396 if (list_lru_init_memcg(&s->s_dentry_lru, s->s_shrink))
394 goto fail;397 goto fail;
395 if (list_lru_init_memcg(&s->s_inode_lru, &s->s_shrink))398 if (list_lru_init_memcg(&s->s_inode_lru, s->s_shrink))
396 goto fail;399 goto fail;
397 return s;400 return s;
398 401
@@ -477,7 +480,7 @@ void deactivate_locked_super(struct super_block *s) · 78 unmodified lines
477{480{
478 struct file_system_type *fs = s->s_type;481 struct file_system_type *fs = s->s_type;
479 if (atomic_dec_and_test(&s->s_active)) {482 if (atomic_dec_and_test(&s->s_active)) {
480 unregister_shrinker(&s->s_shrink);483 shrinker_free(s->s_shrink);
481 fs->kill_sb(s);484 fs->kill_sb(s);
482 485
483 kill_super_notify(s);486 kill_super_notify(s);
@@ -818,7 +821,7 @@ retry: · 334 unmodified lines
818 hlist_add_head(&s->s_instances, &s->s_type->fs_supers);821 hlist_add_head(&s->s_instances, &s->s_type->fs_supers);
819 spin_unlock(&sb_lock);822 spin_unlock(&sb_lock);
820 get_filesystem(s->s_type);823 get_filesystem(s->s_type);
821 register_shrinker_prepared(&s->s_shrink);824 shrinker_register(s->s_shrink);
822 return s;825 return s;
823 826
824share_extant_sb:827share_extant_sb:
@@ -901,7 +904,7 @@ retry: · 76 unmodified lines
901 hlist_add_head(&s->s_instances, &type->fs_supers);904 hlist_add_head(&s->s_instances, &type->fs_supers);
902 spin_unlock(&sb_lock);905 spin_unlock(&sb_lock);
903 get_filesystem(type);906 get_filesystem(type);
904 register_shrinker_prepared(&s->s_shrink);907 shrinker_register(s->s_shrink);
905 return s;908 return s;
906}909}
907EXPORT_SYMBOL(sget);910EXPORT_SYMBOL(sget);
@@ -1540,7 +1543,7 @@ int setup_bdev_super(struct super_block *sb, int sb_flags, · 632 unmodified lines
1540 mutex_unlock(&bdev->bd_fsfreeze_mutex);1543 mutex_unlock(&bdev->bd_fsfreeze_mutex);
1541 1544
1542 snprintf(sb->s_id, sizeof(sb->s_id), "%pg", bdev);1545 snprintf(sb->s_id, sizeof(sb->s_id), "%pg", bdev);
1543 shrinker_debugfs_rename(&sb->s_shrink, "sb-%s:%s", sb->s_type->name,1546 shrinker_debugfs_rename(sb->s_shrink, "sb-%s:%s", sb->s_type->name,
1544 sb->s_id);1547 sb->s_id);
1545 sb_set_blocksize(sb, block_size(bdev));1548 sb_set_blocksize(sb, block_size(bdev));
1546 return 0;1549 return 0;
fs/ubifs/super.c+11−10
@@ -54,11 +54,7 @@ module_param_cb(default_version, &ubifs_default_version_ops, &ubifs_default_vers · 53 unmodified lines
54static struct kmem_cache *ubifs_inode_slab;54static struct kmem_cache *ubifs_inode_slab;
55 55
56/* UBIFS TNC shrinker description */56/* UBIFS TNC shrinker description */
57static struct shrinker ubifs_shrinker_info = {57static struct shrinker *ubifs_shrinker_info;
58 .scan_objects = ubifs_shrink_scan,
59 .count_objects = ubifs_shrink_count,
60 .seeks = DEFAULT_SEEKS,
61};
62 58
63/**59/**
64 * validate_inode - validate inode.60 * validate_inode - validate inode.
@@ -2373,7 +2369,7 @@ static void inode_slab_ctor(void *obj) · 2308 unmodified lines
2373 2369
2374static int __init ubifs_init(void)2370static int __init ubifs_init(void)
2375{2371{
2376 int err;2372 int err = -ENOMEM;
2377 2373
2378 BUILD_BUG_ON(sizeof(struct ubifs_ch) != 24);2374 BUILD_BUG_ON(sizeof(struct ubifs_ch) != 24);
2379 2375
@@ -2439,10 +2435,15 @@ static int __init ubifs_init(void) · 59 unmodified lines
2439 if (!ubifs_inode_slab)2435 if (!ubifs_inode_slab)
2440 return -ENOMEM;2436 return -ENOMEM;
2441 2437
2442 err = register_shrinker(&ubifs_shrinker_info, "ubifs-slab");2438 ubifs_shrinker_info = shrinker_alloc(0, "ubifs-slab");
2443 if (err)2439 if (!ubifs_shrinker_info)
2444 goto out_slab;2440 goto out_slab;
2445 2441
2442 ubifs_shrinker_info->count_objects = ubifs_shrink_count;
2443 ubifs_shrinker_info->scan_objects = ubifs_shrink_scan;
2444
2445 shrinker_register(ubifs_shrinker_info);
2446
2446 err = ubifs_compressors_init();2447 err = ubifs_compressors_init();
2447 if (err)2448 if (err)
2448 goto out_shrinker;2449 goto out_shrinker;
@@ -2467,7 +2468,7 @@ out_dbg: · 18 unmodified lines
2467 dbg_debugfs_exit();2468 dbg_debugfs_exit();
2468 ubifs_compressors_exit();2469 ubifs_compressors_exit();
2469out_shrinker:2470out_shrinker:
2470 unregister_shrinker(&ubifs_shrinker_info);2471 shrinker_free(ubifs_shrinker_info);
2471out_slab:2472out_slab:
2472 kmem_cache_destroy(ubifs_inode_slab);2473 kmem_cache_destroy(ubifs_inode_slab);
2473 return err;2474 return err;
@@ -2483,7 +2484,7 @@ static void __exit ubifs_exit(void) · 9 unmodified lines
2483 dbg_debugfs_exit();2484 dbg_debugfs_exit();
2484 ubifs_sysfs_exit();2485 ubifs_sysfs_exit();
2485 ubifs_compressors_exit();2486 ubifs_compressors_exit();
2486 unregister_shrinker(&ubifs_shrinker_info);2487 shrinker_free(ubifs_shrinker_info);
2487 2488
2488 /*2489 /*
2489 * Make sure all delayed rcu free inodes are flushed before we2490 * Make sure all delayed rcu free inodes are flushed before we
fs/ufs/balloc.c+9−11
@@ -240,6 +240,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 239 unmodified lines
240 unsigned int count, sector_t oldb,240 unsigned int count, sector_t oldb,
241 sector_t newb, struct page *locked_page)241 sector_t newb, struct page *locked_page)
242{242{
243 struct folio *folio, *locked_folio = page_folio(locked_page);
243 const unsigned blks_per_page =244 const unsigned blks_per_page =
244 1 << (PAGE_SHIFT - inode->i_blkbits);245 1 << (PAGE_SHIFT - inode->i_blkbits);
245 const unsigned mask = blks_per_page - 1;246 const unsigned mask = blks_per_page - 1;
@@ -247,42 +248,39 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 1 unmodified line
247 pgoff_t index, cur_index, last_index;248 pgoff_t index, cur_index, last_index;
248 unsigned pos, j, lblock;249 unsigned pos, j, lblock;
249 sector_t end, i;250 sector_t end, i;
250 struct page *page;
251 struct buffer_head *head, *bh;251 struct buffer_head *head, *bh;
252 252
253 UFSD("ENTER, ino %lu, count %u, oldb %llu, newb %llu\n",253 UFSD("ENTER, ino %lu, count %u, oldb %llu, newb %llu\n",
254 inode->i_ino, count,254 inode->i_ino, count,
255 (unsigned long long)oldb, (unsigned long long)newb);255 (unsigned long long)oldb, (unsigned long long)newb);
256 256
257 BUG_ON(!locked_page);257 BUG_ON(!folio_test_locked(locked_folio));
258 BUG_ON(!PageLocked(locked_page));
259 258
260 cur_index = locked_page->index;259 cur_index = locked_folio->index;
261 end = count + beg;260 end = count + beg;
262 last_index = end >> (PAGE_SHIFT - inode->i_blkbits);261 last_index = end >> (PAGE_SHIFT - inode->i_blkbits);
263 for (i = beg; i < end; i = (i | mask) + 1) {262 for (i = beg; i < end; i = (i | mask) + 1) {
264 index = i >> (PAGE_SHIFT - inode->i_blkbits);263 index = i >> (PAGE_SHIFT - inode->i_blkbits);
265 264
266 if (likely(cur_index != index)) {265 if (likely(cur_index != index)) {
267 page = ufs_get_locked_page(mapping, index);266 folio = ufs_get_locked_folio(mapping, index);
268 if (!page)/* it was truncated */267 if (!folio) /* it was truncated */
269 continue;268 continue;
270 if (IS_ERR(page)) {/* or EIO */269 if (IS_ERR(folio)) {/* or EIO */
271 ufs_error(inode->i_sb, __func__,270 ufs_error(inode->i_sb, __func__,
272 "read of page %llu failed\n",271 "read of page %llu failed\n",
273 (unsigned long long)index);272 (unsigned long long)index);
274 continue;273 continue;
275 }274 }
276 } else275 } else
277 page = locked_page;276 folio = locked_folio;
278 277
279 head = page_buffers(page);278 head = folio_buffers(folio);
280 bh = head;279 bh = head;
281 pos = i & mask;280 pos = i & mask;
282 for (j = 0; j < pos; ++j)281 for (j = 0; j < pos; ++j)
283 bh = bh->b_this_page;282 bh = bh->b_this_page;
284 283
285
286 if (unlikely(index == last_index))284 if (unlikely(index == last_index))
287 lblock = end & mask;285 lblock = end & mask;
288 else286 else
@@ -313,7 +311,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 24 unmodified lines
313 } while (bh != head);311 } while (bh != head);
314 312
315 if (likely(cur_index != index))313 if (likely(cur_index != index))
316 ufs_put_locked_page(page);314 ufs_put_locked_folio(folio);
317 }315 }
318 UFSD("EXIT\n");316 UFSD("EXIT\n");
319}317}
fs/ufs/inode.c+12−13
@@ -1063,7 +1063,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 1062 unmodified lines
1063 struct ufs_sb_private_info *uspi = UFS_SB(sb)->s_uspi;1063 struct ufs_sb_private_info *uspi = UFS_SB(sb)->s_uspi;
1064 unsigned i, end;1064 unsigned i, end;
1065 sector_t lastfrag;1065 sector_t lastfrag;
1066 struct page *lastpage;1066 struct folio *folio;
1067 struct buffer_head *bh;1067 struct buffer_head *bh;
1068 u64 phys64;1068 u64 phys64;
1069 1069
@@ -1074,18 +1074,17 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 4 unmodified lines
1074 1074
1075 lastfrag--;1075 lastfrag--;
1076 1076
1077 lastpage = ufs_get_locked_page(mapping, lastfrag >>1077 folio = ufs_get_locked_folio(mapping, lastfrag >>
1078 (PAGE_SHIFT - inode->i_blkbits));1078 (PAGE_SHIFT - inode->i_blkbits));
1079 if (IS_ERR(lastpage)) {1079 if (IS_ERR(folio)) {
1080 err = -EIO;1080 err = -EIO;
1081 goto out;1081 goto out;
1082 }1082 }
1083
1084 end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1);
1085 bh = page_buffers(lastpage);
1086 for (i = 0; i < end; ++i)
1087 bh = bh->b_this_page;
1088 1083
1084 end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1);
1085 bh = folio_buffers(folio);
1086 for (i = 0; i < end; ++i)
1087 bh = bh->b_this_page;
1089 1088
1090 err = ufs_getfrag_block(inode, lastfrag, bh, 1);1089 err = ufs_getfrag_block(inode, lastfrag, bh, 1);
1091 1090
@@ -1101,7 +1100,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 9 unmodified lines
1101 */1100 */
1102 set_buffer_uptodate(bh);1101 set_buffer_uptodate(bh);
1103 mark_buffer_dirty(bh);1102 mark_buffer_dirty(bh);
1104 set_page_dirty(lastpage);1103 folio_mark_dirty(folio);
1105 }1104 }
1106 1105
1107 if (lastfrag >= UFS_IND_FRAGMENT) {1106 if (lastfrag >= UFS_IND_FRAGMENT) {
@@ -1119,7 +1118,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 11 unmodified lines
1119 }1118 }
1120 }1119 }
1121out_unlock:1120out_unlock:
1122 ufs_put_locked_page(lastpage);1121 ufs_put_locked_folio(folio);
1123out:1122out:
1124 return err;1123 return err;
1125}1124}
fs/ufs/util.c+16−18
@@ -230,42 +230,40 @@ ufs_set_inode_dev(struct super_block *sb, struct ufs_inode_info *ufsi, dev_t dev · 229 unmodified lines
230}230}
231 231
232/**232/**
233 * ufs_get_locked_page() - locate, pin and lock a pagecache page, if not exist233 * ufs_get_locked_folio() - locate, pin and lock a pagecache folio, if not exist
234 * read it from disk.234 * read it from disk.
235 * @mapping: the address_space to search235 * @mapping: the address_space to search
236 * @index: the page index236 * @index: the page index
237 *237 *
238 * Locates the desired pagecache page, if not exist we'll read it,238 * Locates the desired pagecache folio, if not exist we'll read it,
239 * locks it, increments its reference239 * locks it, increments its reference
240 * count and returns its address.240 * count and returns its address.
241 *241 *
242 */242 */
243 243struct folio *ufs_get_locked_folio(struct address_space *mapping,
244struct page *ufs_get_locked_page(struct address_space *mapping,
245 pgoff_t index)244 pgoff_t index)
246{245{
247 struct inode *inode = mapping->host;246 struct inode *inode = mapping->host;
248 struct page *page = find_lock_page(mapping, index);247 struct folio *folio = filemap_lock_folio(mapping, index);
249 if (!page) {248 if (!folio) {
250 page = read_mapping_page(mapping, index, NULL);249 folio = read_mapping_folio(mapping, index, NULL);
251 250
252 if (IS_ERR(page)) {251 if (IS_ERR(folio)) {
253 printk(KERN_ERR "ufs_change_blocknr: "252 printk(KERN_ERR "ufs_change_blocknr: read_mapping_folio error: ino %lu, index: %lu\n",
254 "read_mapping_page error: ino %lu, index: %lu\n",
255 mapping->host->i_ino, index);253 mapping->host->i_ino, index);
256 return page;254 return folio;
257 }255 }
258 256
259 lock_page(page);257 folio_lock(folio);
260 258
261 if (unlikely(page->mapping == NULL)) {259 if (unlikely(folio->mapping == NULL)) {
262 /* Truncate got there first */260 /* Truncate got there first */
263 unlock_page(page);261 folio_unlock(folio);
264 put_page(page);262 folio_put(folio);
265 return NULL;263 return NULL;
266 }264 }
267 }265 }
268 if (!page_has_buffers(page))266 if (!folio_buffers(folio))
269 create_empty_buffers(page, 1 << inode->i_blkbits, 0);267 create_empty_buffers(folio, 1 << inode->i_blkbits, 0);
270 return page;268 return folio;
271}269}
fs/ufs/util.h+4−6
@@ -273,15 +273,13 @@ extern void _ubh_ubhcpymem_(struct ufs_sb_private_info *, unsigned char *, struc · 272 unmodified lines
273extern void _ubh_memcpyubh_(struct ufs_sb_private_info *, struct ufs_buffer_head *, unsigned char *, unsigned);273extern void _ubh_memcpyubh_(struct ufs_sb_private_info *, struct ufs_buffer_head *, unsigned char *, unsigned);
274 274
275/* This functions works with cache pages*/275/* This functions works with cache pages*/
276extern struct page *ufs_get_locked_page(struct address_space *mapping,276struct folio *ufs_get_locked_folio(struct address_space *mapping, pgoff_t index);
277 pgoff_t index);277static inline void ufs_put_locked_folio(struct folio *folio)
278static inline void ufs_put_locked_page(struct page *page)
279{278{
280 unlock_page(page);279 folio_unlock(folio);
281 put_page(page);280 folio_put(folio);
282}281}
283 282
284
285/*283/*
286 * macros and inline function to get important structures from ufs_sb_private_info284 * macros and inline function to get important structures from ufs_sb_private_info
287 */285 */
fs/userfaultfd.c+40−56
@@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx) · 122 unmodified lines
123 return ctx->features & UFFD_FEATURE_INITIALIZED;123 return ctx->features & UFFD_FEATURE_INITIALIZED;
124}124}
125 125
126static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
127{
128 return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
129}
130
126/*131/*
127 * Whether WP_UNPOPULATED is enabled on the uffd context. It is only132 * Whether WP_UNPOPULATED is enabled on the uffd context. It is only
128 * meaningful when userfaultfd_wp()==true on the vma and when it's133 * meaningful when userfaultfd_wp()==true on the vma and when it's
@@ -922,20 +927,15 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 793 unmodified lines
922 continue;927 continue;
923 }928 }
924 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;929 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;
925 prev = vma_merge(&vmi, mm, prev, vma->vm_start, vma->vm_end,930 vma = vma_modify_flags_uffd(&vmi, prev, vma, vma->vm_start,
926 new_flags, vma->anon_vma,931 vma->vm_end, new_flags,
927 vma->vm_file, vma->vm_pgoff,932 NULL_VM_UFFD_CTX);
928 vma_policy(vma),
929 NULL_VM_UFFD_CTX, anon_vma_name(vma));
930 if (prev) {
931 vma = prev;
932 } else {
933 prev = vma;
934 }
935 933
936 vma_start_write(vma);934 vma_start_write(vma);
937 userfaultfd_set_vm_flags(vma, new_flags);935 userfaultfd_set_vm_flags(vma, new_flags);
938 vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX;936 vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX;
937
938 prev = vma;
939 }939 }
940 mmap_write_unlock(mm);940 mmap_write_unlock(mm);
941 mmput(mm);941 mmput(mm);
@@ -1325,7 +1325,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 383 unmodified lines
1325 bool basic_ioctls;1325 bool basic_ioctls;
1326 unsigned long start, end, vma_end;1326 unsigned long start, end, vma_end;
1327 struct vma_iterator vmi;1327 struct vma_iterator vmi;
1328 pgoff_t pgoff;1328 bool wp_async = userfaultfd_wp_async_ctx(ctx);
1329 1329
1330 user_uffdio_register = (struct uffdio_register __user *) arg;1330 user_uffdio_register = (struct uffdio_register __user *) arg;
1331 1331
@@ -1399,7 +1399,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 67 unmodified lines
1399 1399
1400 /* check not compatible vmas */1400 /* check not compatible vmas */
1401 ret = -EINVAL;1401 ret = -EINVAL;
1402 if (!vma_can_userfault(cur, vm_flags))1402 if (!vma_can_userfault(cur, vm_flags, wp_async))
1403 goto out_unlock;1403 goto out_unlock;
1404 1404
1405 /*1405 /*
@@ -1460,7 +1460,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 54 unmodified lines
1460 for_each_vma_range(vmi, vma, end) {1460 for_each_vma_range(vmi, vma, end) {
1461 cond_resched();1461 cond_resched();
1462 1462
1463 BUG_ON(!vma_can_userfault(vma, vm_flags));1463 BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async));
1464 BUG_ON(vma->vm_userfaultfd_ctx.ctx &&1464 BUG_ON(vma->vm_userfaultfd_ctx.ctx &&
1465 vma->vm_userfaultfd_ctx.ctx != ctx);1465 vma->vm_userfaultfd_ctx.ctx != ctx);
1466 WARN_ON(!(vma->vm_flags & VM_MAYWRITE));1466 WARN_ON(!(vma->vm_flags & VM_MAYWRITE));
@@ -1478,28 +1478,14 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 11 unmodified lines
1478 vma_end = min(end, vma->vm_end);1478 vma_end = min(end, vma->vm_end);
1479 1479
1480 new_flags = (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags;1480 new_flags = (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags;
1481 pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);1481 vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end,
1482 prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags,1482 new_flags,
1483 vma->anon_vma, vma->vm_file, pgoff,1483 (struct vm_userfaultfd_ctx){ctx});
1484 vma_policy(vma),1484 if (IS_ERR(vma)) {
1485 ((struct vm_userfaultfd_ctx){ ctx }),1485 ret = PTR_ERR(vma);
1486 anon_vma_name(vma));1486 break;
1487 if (prev) {
1488 /* vma_merge() invalidated the mas */
1489 vma = prev;
1490 goto next;
1491 }
1492 if (vma->vm_start < start) {
1493 ret = split_vma(&vmi, vma, start, 1);
1494 if (ret)
1495 break;
1496 }
1497 if (vma->vm_end > end) {
1498 ret = split_vma(&vmi, vma, end, 0);
1499 if (ret)
1500 break;
1501 }1487 }
1502 next:1488
1503 /*1489 /*
1504 * In the vma_merge() successful mprotect-like case 8:1490 * In the vma_merge() successful mprotect-like case 8:
1505 * the next vma was merged into the current one and1491 * the next vma was merged into the current one and
@@ -1561,7 +1547,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 55 unmodified lines
1561 unsigned long start, end, vma_end;1547 unsigned long start, end, vma_end;
1562 const void __user *buf = (void __user *)arg;1548 const void __user *buf = (void __user *)arg;
1563 struct vma_iterator vmi;1549 struct vma_iterator vmi;
1564 pgoff_t pgoff;1550 bool wp_async = userfaultfd_wp_async_ctx(ctx);
1565 1551
1566 ret = -EFAULT;1552 ret = -EFAULT;
1567 if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))1553 if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
@@ -1615,7 +1601,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 47 unmodified lines
1615 * provides for more strict behavior to notice1601 * provides for more strict behavior to notice
1616 * unregistration errors.1602 * unregistration errors.
1617 */1603 */
1618 if (!vma_can_userfault(cur, cur->vm_flags))1604 if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
1619 goto out_unlock;1605 goto out_unlock;
1620 1606
1621 found = true;1607 found = true;
@@ -1631,7 +1617,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 9 unmodified lines
1631 for_each_vma_range(vmi, vma, end) {1617 for_each_vma_range(vmi, vma, end) {
1632 cond_resched();1618 cond_resched();
1633 1619
1634 BUG_ON(!vma_can_userfault(vma, vma->vm_flags));1620 BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async));
1635 1621
1636 /*1622 /*
1637 * Nothing to do: this vma is already registered into this1623 * Nothing to do: this vma is already registered into this
@@ -1664,26 +1650,13 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 26 unmodified lines
1664 uffd_wp_range(vma, start, vma_end - start, false);1650 uffd_wp_range(vma, start, vma_end - start, false);
1665 1651
1666 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;1652 new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;
1667 pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);1653 vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end,
1668 prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags,1654 new_flags, NULL_VM_UFFD_CTX);
1669 vma->anon_vma, vma->vm_file, pgoff,1655 if (IS_ERR(vma)) {
1670 vma_policy(vma),1656 ret = PTR_ERR(vma);
1671 NULL_VM_UFFD_CTX, anon_vma_name(vma));1657 break;
1672 if (prev) {
1673 vma = prev;
1674 goto next;
1675 }
1676 if (vma->vm_start < start) {
1677 ret = split_vma(&vmi, vma, start, 1);
1678 if (ret)
1679 break;
1680 }
1681 if (vma->vm_end > end) {
1682 ret = split_vma(&vmi, vma, end, 0);
1683 if (ret)
1684 break;
1685 }1658 }
1686 next:1659
1687 /*1660 /*
1688 * In the vma_merge() successful mprotect-like case 8:1661 * In the vma_merge() successful mprotect-like case 8:
1689 * the next vma was merged into the current one and1662 * the next vma was merged into the current one and
@@ -2018,6 +1991,11 @@ out: · 328 unmodified lines
2018 return ret;1991 return ret;
2019}1992}
2020 1993
1994bool userfaultfd_wp_async(struct vm_area_struct *vma)
1995{
1996 return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
1997}
1998
2021static inline unsigned int uffd_ctx_features(__u64 user_features)1999static inline unsigned int uffd_ctx_features(__u64 user_features)
2022{2000{
2023 /*2001 /*
@@ -2051,6 +2029,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 27 unmodified lines
2051 ret = -EPERM;2029 ret = -EPERM;
2052 if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))2030 if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE))
2053 goto err_out;2031 goto err_out;
2032
2033 /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */
2034 if (features & UFFD_FEATURE_WP_ASYNC)
2035 features |= UFFD_FEATURE_WP_UNPOPULATED;
2036
2054 /* report all available features and ioctls to userland */2037 /* report all available features and ioctls to userland */
2055 uffdio_api.features = UFFD_API_FEATURES;2038 uffdio_api.features = UFFD_API_FEATURES;
2056#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR2039#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
@@ -2063,6 +2046,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 6 unmodified lines
2063#ifndef CONFIG_PTE_MARKER_UFFD_WP2046#ifndef CONFIG_PTE_MARKER_UFFD_WP
2064 uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;2047 uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
2065 uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;2048 uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
2049 uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
2066#endif2050#endif
2067 uffdio_api.ioctls = UFFD_API_IOCTLS;2051 uffdio_api.ioctls = UFFD_API_IOCTLS;
2068 ret = -EFAULT;2052 ret = -EFAULT;
fs/xfs/xfs_buf.c+13−11
@@ -1913,8 +1913,7 @@ xfs_buftarg_shrink_scan( · 1912 unmodified lines
1913 struct shrinker *shrink,1913 struct shrinker *shrink,
1914 struct shrink_control *sc)1914 struct shrink_control *sc)
1915{1915{
1916 struct xfs_buftarg *btp = container_of(shrink,1916 struct xfs_buftarg *btp = shrink->private_data;
1917 struct xfs_buftarg, bt_shrinker);
1918 LIST_HEAD(dispose);1917 LIST_HEAD(dispose);
1919 unsigned long freed;1918 unsigned long freed;
1920 1919
@@ -1936,8 +1935,7 @@ xfs_buftarg_shrink_count( · 15 unmodified lines
1936 struct shrinker *shrink,1935 struct shrinker *shrink,
1937 struct shrink_control *sc)1936 struct shrink_control *sc)
1938{1937{
1939 struct xfs_buftarg *btp = container_of(shrink,1938 struct xfs_buftarg *btp = shrink->private_data;
1940 struct xfs_buftarg, bt_shrinker);
1941 return list_lru_shrink_count(&btp->bt_lru, sc);1939 return list_lru_shrink_count(&btp->bt_lru, sc);
1942}1940}
1943 1941
@@ -1945,7 +1943,7 @@ void · 1 unmodified line
1945xfs_free_buftarg(1943xfs_free_buftarg(
1946 struct xfs_buftarg *btp)1944 struct xfs_buftarg *btp)
1947{1945{
1948 unregister_shrinker(&btp->bt_shrinker);1946 shrinker_free(btp->bt_shrinker);
1949 ASSERT(percpu_counter_sum(&btp->bt_io_count) == 0);1947 ASSERT(percpu_counter_sum(&btp->bt_io_count) == 0);
1950 percpu_counter_destroy(&btp->bt_io_count);1948 percpu_counter_destroy(&btp->bt_io_count);
1951 list_lru_destroy(&btp->bt_lru);1949 list_lru_destroy(&btp->bt_lru);
@@ -2029,13 +2027,17 @@ xfs_alloc_buftarg( · 77 unmodified lines
2029 if (percpu_counter_init(&btp->bt_io_count, 0, GFP_KERNEL))2027 if (percpu_counter_init(&btp->bt_io_count, 0, GFP_KERNEL))
2030 goto error_lru;2028 goto error_lru;
2031 2029
2032 btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count;2030 btp->bt_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-buf:%s",
2033 btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan;2031 mp->m_super->s_id);
2034 btp->bt_shrinker.seeks = DEFAULT_SEEKS;2032 if (!btp->bt_shrinker)
2035 btp->bt_shrinker.flags = SHRINKER_NUMA_AWARE;
2036 if (register_shrinker(&btp->bt_shrinker, "xfs-buf:%s",
2037 mp->m_super->s_id))
2038 goto error_pcpu;2033 goto error_pcpu;
2034
2035 btp->bt_shrinker->count_objects = xfs_buftarg_shrink_count;
2036 btp->bt_shrinker->scan_objects = xfs_buftarg_shrink_scan;
2037 btp->bt_shrinker->private_data = btp;
2038
2039 shrinker_register(btp->bt_shrinker);
2040
2039 return btp;2041 return btp;
2040 2042
2041error_pcpu:2043error_pcpu:
fs/xfs/xfs_buf.h+1−1
@@ -109,7 +109,7 @@ typedef struct xfs_buftarg { · 108 unmodified lines
109 size_t bt_logical_sectormask;109 size_t bt_logical_sectormask;
110 110
111 /* LRU control structures */111 /* LRU control structures */
112 struct shrinker bt_shrinker;112 struct shrinker *bt_shrinker;
113 struct list_lru bt_lru;113 struct list_lru bt_lru;
114 114
115 struct percpu_counter bt_io_count;115 struct percpu_counter bt_io_count;
fs/xfs/xfs_icache.c+15−11
@@ -2165,8 +2165,7 @@ xfs_inodegc_shrinker_count( · 2164 unmodified lines
2165 struct shrinker *shrink,2165 struct shrinker *shrink,
2166 struct shrink_control *sc)2166 struct shrink_control *sc)
2167{2167{
2168 struct xfs_mount *mp = container_of(shrink, struct xfs_mount,2168 struct xfs_mount *mp = shrink->private_data;
2169 m_inodegc_shrinker);
2170 struct xfs_inodegc *gc;2169 struct xfs_inodegc *gc;
2171 int cpu;2170 int cpu;
2172 2171
@@ -2187,8 +2186,7 @@ xfs_inodegc_shrinker_scan( · 14 unmodified lines
2187 struct shrinker *shrink,2186 struct shrinker *shrink,
2188 struct shrink_control *sc)2187 struct shrink_control *sc)
2189{2188{
2190 struct xfs_mount *mp = container_of(shrink, struct xfs_mount,2189 struct xfs_mount *mp = shrink->private_data;
2191 m_inodegc_shrinker);
2192 struct xfs_inodegc *gc;2190 struct xfs_inodegc *gc;
2193 int cpu;2191 int cpu;
2194 bool no_items = true;2192 bool no_items = true;
@@ -2224,13 +2222,19 @@ int · 29 unmodified lines
2224xfs_inodegc_register_shrinker(2222xfs_inodegc_register_shrinker(
2225 struct xfs_mount *mp)2223 struct xfs_mount *mp)
2226{2224{
2227 struct shrinker *shrink = &mp->m_inodegc_shrinker;2225 mp->m_inodegc_shrinker = shrinker_alloc(SHRINKER_NONSLAB,
2226 "xfs-inodegc:%s",
2227 mp->m_super->s_id);
2228 if (!mp->m_inodegc_shrinker)
2229 return -ENOMEM;
2230
2231 mp->m_inodegc_shrinker->count_objects = xfs_inodegc_shrinker_count;
2232 mp->m_inodegc_shrinker->scan_objects = xfs_inodegc_shrinker_scan;
2233 mp->m_inodegc_shrinker->seeks = 0;
2234 mp->m_inodegc_shrinker->batch = XFS_INODEGC_SHRINKER_BATCH;
2235 mp->m_inodegc_shrinker->private_data = mp;
2228 2236
2229 shrink->count_objects = xfs_inodegc_shrinker_count;2237 shrinker_register(mp->m_inodegc_shrinker);
2230 shrink->scan_objects = xfs_inodegc_shrinker_scan;
2231 shrink->seeks = 0;
2232 shrink->flags = SHRINKER_NONSLAB;
2233 shrink->batch = XFS_INODEGC_SHRINKER_BATCH;
2234 2238
2235 return register_shrinker(shrink, "xfs-inodegc:%s", mp->m_super->s_id);2239 return 0;
2236}2240}
fs/xfs/xfs_mount.c+2−2
@@ -1021,7 +1021,7 @@ xfs_mountfs( · 1020 unmodified lines
1021 out_log_dealloc:1021 out_log_dealloc:
1022 xfs_log_mount_cancel(mp);1022 xfs_log_mount_cancel(mp);
1023 out_inodegc_shrinker:1023 out_inodegc_shrinker:
1024 unregister_shrinker(&mp->m_inodegc_shrinker);1024 shrinker_free(mp->m_inodegc_shrinker);
1025 out_fail_wait:1025 out_fail_wait:
1026 if (mp->m_logdev_targp && mp->m_logdev_targp != mp->m_ddev_targp)1026 if (mp->m_logdev_targp && mp->m_logdev_targp != mp->m_ddev_targp)
1027 xfs_buftarg_drain(mp->m_logdev_targp);1027 xfs_buftarg_drain(mp->m_logdev_targp);
@@ -1104,7 +1104,7 @@ xfs_unmountfs( · 76 unmodified lines
1104#if defined(DEBUG)1104#if defined(DEBUG)
1105 xfs_errortag_clearall(mp);1105 xfs_errortag_clearall(mp);
1106#endif1106#endif
1107 unregister_shrinker(&mp->m_inodegc_shrinker);1107 shrinker_free(mp->m_inodegc_shrinker);
1108 xfs_free_perag(mp);1108 xfs_free_perag(mp);
1109 1109
1110 xfs_errortag_del(mp);1110 xfs_errortag_del(mp);
fs/xfs/xfs_mount.h+1−1
@@ -219,7 +219,7 @@ typedef struct xfs_mount { · 218 unmodified lines
219 atomic_t m_agirotor; /* last ag dir inode alloced */219 atomic_t m_agirotor; /* last ag dir inode alloced */
220 220
221 /* Memory shrinker to throttle and reprioritize inodegc */221 /* Memory shrinker to throttle and reprioritize inodegc */
222 struct shrinker m_inodegc_shrinker;222 struct shrinker *m_inodegc_shrinker;
223 /*223 /*
224 * Workqueue item so that we can coalesce multiple inode flush attempts224 * Workqueue item so that we can coalesce multiple inode flush attempts
225 * into a single flush.225 * into a single flush.
fs/xfs/xfs_qm.c+14−13
@@ -504,8 +504,7 @@ xfs_qm_shrink_scan( · 503 unmodified lines
504 struct shrinker *shrink,504 struct shrinker *shrink,
505 struct shrink_control *sc)505 struct shrink_control *sc)
506{506{
507 struct xfs_quotainfo *qi = container_of(shrink,507 struct xfs_quotainfo *qi = shrink->private_data;
508 struct xfs_quotainfo, qi_shrinker);
509 struct xfs_qm_isolate isol;508 struct xfs_qm_isolate isol;
510 unsigned long freed;509 unsigned long freed;
511 int error;510 int error;
@@ -539,8 +538,7 @@ xfs_qm_shrink_count( · 27 unmodified lines
539 struct shrinker *shrink,538 struct shrinker *shrink,
540 struct shrink_control *sc)539 struct shrink_control *sc)
541{540{
542 struct xfs_quotainfo *qi = container_of(shrink,541 struct xfs_quotainfo *qi = shrink->private_data;
543 struct xfs_quotainfo, qi_shrinker);
544 542
545 return list_lru_shrink_count(&qi->qi_lru, sc);543 return list_lru_shrink_count(&qi->qi_lru, sc);
546}544}
@@ -680,15 +678,18 @@ xfs_qm_init_quotainfo( · 133 unmodified lines
680 if (XFS_IS_PQUOTA_ON(mp))678 if (XFS_IS_PQUOTA_ON(mp))
681 xfs_qm_set_defquota(mp, XFS_DQTYPE_PROJ, qinf);679 xfs_qm_set_defquota(mp, XFS_DQTYPE_PROJ, qinf);
682 680
683 qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;681 qinf->qi_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-qm:%s",
684 qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;682 mp->m_super->s_id);
685 qinf->qi_shrinker.seeks = DEFAULT_SEEKS;683 if (!qinf->qi_shrinker) {
686 qinf->qi_shrinker.flags = SHRINKER_NUMA_AWARE;684 error = -ENOMEM;
687
688 error = register_shrinker(&qinf->qi_shrinker, "xfs-qm:%s",
689 mp->m_super->s_id);
690 if (error)
691 goto out_free_inos;685 goto out_free_inos;
686 }
687
688 qinf->qi_shrinker->count_objects = xfs_qm_shrink_count;
689 qinf->qi_shrinker->scan_objects = xfs_qm_shrink_scan;
690 qinf->qi_shrinker->private_data = qinf;
691
692 shrinker_register(qinf->qi_shrinker);
692 693
693 return 0;694 return 0;
694 695
@@ -718,7 +719,7 @@ xfs_qm_destroy_quotainfo( · 23 unmodified lines
718 qi = mp->m_quotainfo;719 qi = mp->m_quotainfo;
719 ASSERT(qi != NULL);720 ASSERT(qi != NULL);
720 721
721 unregister_shrinker(&qi->qi_shrinker);722 shrinker_free(qi->qi_shrinker);
722 list_lru_destroy(&qi->qi_lru);723 list_lru_destroy(&qi->qi_lru);
723 xfs_qm_destroy_quotainos(qi);724 xfs_qm_destroy_quotainos(qi);
724 mutex_destroy(&qi->qi_tree_lock);725 mutex_destroy(&qi->qi_tree_lock);
fs/xfs/xfs_qm.h+1−1
@@ -63,7 +63,7 @@ struct xfs_quotainfo { · 62 unmodified lines
63 struct xfs_def_quota qi_usr_default;63 struct xfs_def_quota qi_usr_default;
64 struct xfs_def_quota qi_grp_default;64 struct xfs_def_quota qi_grp_default;
65 struct xfs_def_quota qi_prj_default;65 struct xfs_def_quota qi_prj_default;
66 struct shrinker qi_shrinker;66 struct shrinker *qi_shrinker;
67 67
68 /* Minimum and maximum quota expiration timestamp values. */68 /* Minimum and maximum quota expiration timestamp values. */
69 time64_t qi_expiry_min;69 time64_t qi_expiry_min;
include/asm-generic/bitops/instrumented-lock.h+13−15
@@ -58,27 +58,25 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr) · 57 unmodified lines
58 return arch_test_and_set_bit_lock(nr, addr);58 return arch_test_and_set_bit_lock(nr, addr);
59}59}
60 60
61#if defined(arch_clear_bit_unlock_is_negative_byte)
62/**61/**
63 * clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom62 * xor_unlock_is_negative_byte - XOR a single byte in memory and test if
64 * byte is negative, for unlock.63 * it is negative, for unlock.
65 * @nr: the bit to clear64 * @mask: Change the bits which are set in this mask.
66 * @addr: the address to start counting from65 * @addr: The address of the word containing the byte to change.
67 *66 *
67 * Changes some of bits 0-6 in the word pointed to by @addr.
68 * This operation is atomic and provides release barrier semantics.68 * This operation is atomic and provides release barrier semantics.
69 * Used to optimise some folio operations which are commonly paired
70 * with an unlock or end of writeback. Bit 7 is used as PG_waiters to
71 * indicate whether anybody is waiting for the unlock.
69 *72 *
70 * This is a bit of a one-trick-pony for the filemap code, which clears73 * Return: Whether the top bit of the byte is set.
71 * PG_locked and tests PG_waiters,
72 */74 */
73static inline bool75static inline bool xor_unlock_is_negative_byte(unsigned long mask,
74clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr)76 volatile unsigned long *addr)
75{77{
76 kcsan_release();78 kcsan_release();
77 instrument_atomic_write(addr + BIT_WORD(nr), sizeof(long));79 instrument_atomic_write(addr, sizeof(long));
78 return arch_clear_bit_unlock_is_negative_byte(nr, addr);80 return arch_xor_unlock_is_negative_byte(mask, addr);
79}81}
80/* Let everybody know we have it. */
81#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte
82#endif
83
84#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */82#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */
include/asm-generic/bitops/lock.h+4−16
@@ -66,27 +66,15 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p) · 65 unmodified lines
66 raw_atomic_long_set_release((atomic_long_t *)p, old);66 raw_atomic_long_set_release((atomic_long_t *)p, old);
67}67}
68 68
69/**69#ifndef arch_xor_unlock_is_negative_byte
70 * arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom70static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
71 * byte is negative, for unlock.71 volatile unsigned long *p)
72 * @nr: the bit to clear
73 * @addr: the address to start counting from
74 *
75 * This is a bit of a one-trick-pony for the filemap code, which clears
76 * PG_locked and tests PG_waiters,
77 */
78#ifndef arch_clear_bit_unlock_is_negative_byte
79static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr,
80 volatile unsigned long *p)
81{72{
82 long old;73 long old;
83 unsigned long mask = BIT_MASK(nr);
84 74
85 p += BIT_WORD(nr);75 old = raw_atomic_long_fetch_xor_release(mask, (atomic_long_t *)p);
86 old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
87 return !!(old & BIT(7));76 return !!(old & BIT(7));
88}77}
89#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte
90#endif78#endif
91 79
92#include <asm-generic/bitops/instrumented-lock.h>80#include <asm-generic/bitops/instrumented-lock.h>
include/asm-generic/pgalloc.h+6−1
@@ -169,6 +169,8 @@ static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 168 unmodified lines
169 ptdesc = pagetable_alloc(gfp, 0);169 ptdesc = pagetable_alloc(gfp, 0);
170 if (!ptdesc)170 if (!ptdesc)
171 return NULL;171 return NULL;
172
173 pagetable_pud_ctor(ptdesc);
172 return ptdesc_address(ptdesc);174 return ptdesc_address(ptdesc);
173}175}
174 176
@@ -190,8 +192,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 15 unmodified lines
190 192
191static inline void __pud_free(struct mm_struct *mm, pud_t *pud)193static inline void __pud_free(struct mm_struct *mm, pud_t *pud)
192{194{
195 struct ptdesc *ptdesc = virt_to_ptdesc(pud);
196
193 BUG_ON((unsigned long)pud & (PAGE_SIZE-1));197 BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
194 pagetable_free(virt_to_ptdesc(pud));198 pagetable_pud_dtor(ptdesc);
199 pagetable_free(ptdesc);
195}200}
196 201
197#ifndef __HAVE_ARCH_PUD_FREE202#ifndef __HAVE_ARCH_PUD_FREE
include/linux/bootmem_info.h+1−1
@@ -60,7 +60,7 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 59 unmodified lines
60 60
61static inline void free_bootmem_page(struct page *page)61static inline void free_bootmem_page(struct page *page)
62{62{
63 kmemleak_free_part(page_to_virt(page), PAGE_SIZE);63 kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
64 free_reserved_page(page);64 free_reserved_page(page);
65}65}
66#endif66#endif
include/linux/buffer_head.h+54−27
@@ -198,13 +198,11 @@ void touch_buffer(struct buffer_head *bh); · 197 unmodified lines
198void folio_set_bh(struct buffer_head *bh, struct folio *folio,198void folio_set_bh(struct buffer_head *bh, struct folio *folio,
199 unsigned long offset);199 unsigned long offset);
200struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,200struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
201 bool retry);201 gfp_t gfp);
202struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,202struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
203 bool retry);203 bool retry);
204void create_empty_buffers(struct page *, unsigned long,204struct buffer_head *create_empty_buffers(struct folio *folio,
205 unsigned long b_state);205 unsigned long blocksize, unsigned long b_state);
206void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize,
207 unsigned long b_state);
208void end_buffer_read_sync(struct buffer_head *bh, int uptodate);206void end_buffer_read_sync(struct buffer_head *bh, int uptodate);
209void end_buffer_write_sync(struct buffer_head *bh, int uptodate);207void end_buffer_write_sync(struct buffer_head *bh, int uptodate);
210void end_buffer_async_write(struct buffer_head *bh, int uptodate);208void end_buffer_async_write(struct buffer_head *bh, int uptodate);
@@ -227,8 +225,8 @@ void __wait_on_buffer(struct buffer_head *); · 16 unmodified lines
227wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);225wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
228struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,226struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
229 unsigned size);227 unsigned size);
230struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block,228struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
231 unsigned size, gfp_t gfp);229 unsigned size, gfp_t gfp);
232void __brelse(struct buffer_head *);230void __brelse(struct buffer_head *);
233void __bforget(struct buffer_head *);231void __bforget(struct buffer_head *);
234void __breadahead(struct block_device *, sector_t block, unsigned int size);232void __breadahead(struct block_device *, sector_t block, unsigned int size);
@@ -338,17 +336,38 @@ sb_breadahead(struct super_block *sb, sector_t block) · 103 unmodified lines
338 __breadahead(sb->s_bdev, block, sb->s_blocksize);336 __breadahead(sb->s_bdev, block, sb->s_blocksize);
339}337}
340 338
341static inline struct buffer_head *339static inline struct buffer_head *getblk_unmovable(struct block_device *bdev,
342sb_getblk(struct super_block *sb, sector_t block)340 sector_t block, unsigned size)
343{341{
344 return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, __GFP_MOVABLE);342 gfp_t gfp;
343
344 gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
345 gfp |= __GFP_NOFAIL;
346
347 return bdev_getblk(bdev, block, size, gfp);
345}348}
346 349
350static inline struct buffer_head *__getblk(struct block_device *bdev,
351 sector_t block, unsigned size)
352{
353 gfp_t gfp;
347 354
348static inline struct buffer_head *355 gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
349sb_getblk_gfp(struct super_block *sb, sector_t block, gfp_t gfp)356 gfp |= __GFP_MOVABLE | __GFP_NOFAIL;
357
358 return bdev_getblk(bdev, block, size, gfp);
359}
360
361static inline struct buffer_head *sb_getblk(struct super_block *sb,
362 sector_t block)
350{363{
351 return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, gfp);364 return __getblk(sb->s_bdev, block, sb->s_blocksize);
365}
366
367static inline struct buffer_head *sb_getblk_gfp(struct super_block *sb,
368 sector_t block, gfp_t gfp)
369{
370 return bdev_getblk(sb->s_bdev, block, sb->s_blocksize, gfp);
352}371}
353 372
354static inline struct buffer_head *373static inline struct buffer_head *
@@ -385,20 +404,6 @@ static inline void lock_buffer(struct buffer_head *bh) · 30 unmodified lines
385 __lock_buffer(bh);404 __lock_buffer(bh);
386}405}
387 406
388static inline struct buffer_head *getblk_unmovable(struct block_device *bdev,
389 sector_t block,
390 unsigned size)
391{
392 return __getblk_gfp(bdev, block, size, 0);
393}
394
395static inline struct buffer_head *__getblk(struct block_device *bdev,
396 sector_t block,
397 unsigned size)
398{
399 return __getblk_gfp(bdev, block, size, __GFP_MOVABLE);
400}
401
402static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags)407static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags)
403{408{
404 if (!buffer_uptodate(bh) && trylock_buffer(bh)) {409 if (!buffer_uptodate(bh) && trylock_buffer(bh)) {
@@ -450,6 +455,28 @@ __bread(struct block_device *bdev, sector_t block, unsigned size) · 45 unmodified lines
450 return __bread_gfp(bdev, block, size, __GFP_MOVABLE);455 return __bread_gfp(bdev, block, size, __GFP_MOVABLE);
451}456}
452 457
458/**
459 * get_nth_bh - Get a reference on the n'th buffer after this one.
460 * @bh: The buffer to start counting from.
461 * @count: How many buffers to skip.
462 *
463 * This is primarily useful for finding the nth buffer in a folio; in
464 * that case you pass the head buffer and the byte offset in the folio
465 * divided by the block size. It can be used for other purposes, but
466 * it will wrap at the end of the folio rather than returning NULL or
467 * proceeding to the next folio for you.
468 *
469 * Return: The requested buffer with an elevated refcount.
470 */
471static inline __must_check
472struct buffer_head *get_nth_bh(struct buffer_head *bh, unsigned int count)
473{
474 while (count--)
475 bh = bh->b_this_page;
476 get_bh(bh);
477 return bh;
478}
479
453bool block_dirty_folio(struct address_space *mapping, struct folio *folio);480bool block_dirty_folio(struct address_space *mapping, struct folio *folio);
454 481
455#ifdef CONFIG_BUFFER_HEAD482#ifdef CONFIG_BUFFER_HEAD
include/linux/cacheinfo.h+1−0
@@ -73,6 +73,7 @@ struct cacheinfo { · 72 unmodified lines
73 73
74struct cpu_cacheinfo {74struct cpu_cacheinfo {
75 struct cacheinfo *info_list;75 struct cacheinfo *info_list;
76 unsigned int per_cpu_data_slice_size;
76 unsigned int num_levels;77 unsigned int num_levels;
77 unsigned int num_leaves;78 unsigned int num_leaves;
78 bool cpu_map_populated;79 bool cpu_map_populated;
include/linux/cgroup-defs.h+5−0
@@ -115,6 +115,11 @@ enum { · 114 unmodified lines
115 * Enable recursive subtree protection115 * Enable recursive subtree protection
116 */116 */
117 CGRP_ROOT_MEMORY_RECURSIVE_PROT = (1 << 18),117 CGRP_ROOT_MEMORY_RECURSIVE_PROT = (1 << 18),
118
119 /*
120 * Enable hugetlb accounting for the memory controller.
121 */
122 CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING = (1 << 19),
118};123};
119 124
120/* cftype->flags */125/* cftype->flags */
include/linux/damon.h+60−14
@@ -40,9 +40,24 @@ struct damon_addr_range { · 39 unmodified lines
40 * @ar: The address range of the region.40 * @ar: The address range of the region.
41 * @sampling_addr: Address of the sample for the next access check.41 * @sampling_addr: Address of the sample for the next access check.
42 * @nr_accesses: Access frequency of this region.42 * @nr_accesses: Access frequency of this region.
43 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
44 * each sampling interval.
43 * @list: List head for siblings.45 * @list: List head for siblings.
44 * @age: Age of this region.46 * @age: Age of this region.
45 *47 *
48 * @nr_accesses is reset to zero for every &damon_attrs->aggr_interval and be
49 * increased for every &damon_attrs->sample_interval if an access to the region
50 * during the last sampling interval is found. The update of this field should
51 * not be done with direct access but with the helper function,
52 * damon_update_region_access_rate().
53 *
54 * @nr_accesses_bp is another representation of @nr_accesses in basis point
55 * (1 in 10,000) that updated for every &damon_attrs->sample_interval in a
56 * manner similar to moving sum. By the algorithm, this value becomes
57 * @nr_accesses * 10000 for every &struct damon_attrs->aggr_interval. This can
58 * be used when the aggregation interval is too huge and therefore cannot wait
59 * for it before getting the access monitoring results.
60 *
46 * @age is initially zero, increased for each aggregation interval, and reset61 * @age is initially zero, increased for each aggregation interval, and reset
47 * to zero again if the access frequency is significantly changed. If two62 * to zero again if the access frequency is significantly changed. If two
48 * regions are merged into a new region, both @nr_accesses and @age of the new63 * regions are merged into a new region, both @nr_accesses and @age of the new
@@ -52,6 +67,7 @@ struct damon_region { · 3 unmodified lines
52 struct damon_addr_range ar;67 struct damon_addr_range ar;
53 unsigned long sampling_addr;68 unsigned long sampling_addr;
54 unsigned int nr_accesses;69 unsigned int nr_accesses;
70 unsigned int nr_accesses_bp;
55 struct list_head list;71 struct list_head list;
56 72
57 unsigned int age;73 unsigned int age;
@@ -298,24 +314,24 @@ struct damos_access_pattern { · 240 unmodified lines
298 * struct damos - Represents a Data Access Monitoring-based Operation Scheme.314 * struct damos - Represents a Data Access Monitoring-based Operation Scheme.
299 * @pattern: Access pattern of target regions.315 * @pattern: Access pattern of target regions.
300 * @action: &damo_action to be applied to the target regions.316 * @action: &damo_action to be applied to the target regions.
317 * @apply_interval_us: The time between applying the @action.
301 * @quota: Control the aggressiveness of this scheme.318 * @quota: Control the aggressiveness of this scheme.
302 * @wmarks: Watermarks for automated (in)activation of this scheme.319 * @wmarks: Watermarks for automated (in)activation of this scheme.
303 * @filters: Additional set of &struct damos_filter for &action.320 * @filters: Additional set of &struct damos_filter for &action.
304 * @stat: Statistics of this scheme.321 * @stat: Statistics of this scheme.
305 * @list: List head for siblings.322 * @list: List head for siblings.
306 *323 *
307 * For each aggregation interval, DAMON finds regions which fit in the324 * For each @apply_interval_us, DAMON finds regions which fit in the
308 * &pattern and applies &action to those. To avoid consuming too much325 * &pattern and applies &action to those. To avoid consuming too much
309 * CPU time or IO resources for the &action, &quota is used.326 * CPU time or IO resources for the &action, &quota is used.
310 *327 *
328 * If @apply_interval_us is zero, &damon_attrs->aggr_interval is used instead.
329 *
311 * To do the work only when needed, schemes can be activated for specific330 * To do the work only when needed, schemes can be activated for specific
312 * system situations using &wmarks. If all schemes that registered to the331 * system situations using &wmarks. If all schemes that registered to the
313 * monitoring context are inactive, DAMON stops monitoring either, and just332 * monitoring context are inactive, DAMON stops monitoring either, and just
314 * repeatedly checks the watermarks.333 * repeatedly checks the watermarks.
315 *334 *
316 * If all schemes that registered to a &struct damon_ctx are inactive, DAMON
317 * stops monitoring and just repeatedly checks the watermarks.
318 *
319 * Before applying the &action to a memory region, &struct damon_operations335 * Before applying the &action to a memory region, &struct damon_operations
320 * implementation could check pages of the region and skip &action to respect336 * implementation could check pages of the region and skip &action to respect
321 * &filters337 * &filters
@@ -327,6 +343,14 @@ struct damos_access_pattern { · 5 unmodified lines
327struct damos {343struct damos {
328 struct damos_access_pattern pattern;344 struct damos_access_pattern pattern;
329 enum damos_action action;345 enum damos_action action;
346 unsigned long apply_interval_us;
347/* private: internal use only */
348 /*
349 * number of sample intervals that should be passed before applying
350 * @action
351 */
352 unsigned long next_apply_sis;
353/* public: */
330 struct damos_quota quota;354 struct damos_quota quota;
331 struct damos_watermarks wmarks;355 struct damos_watermarks wmarks;
332 struct list_head filters;356 struct list_head filters;
@@ -472,13 +496,14 @@ struct damon_callback { · 139 unmodified lines
472 * regions.496 * regions.
473 *497 *
474 * For each @sample_interval, DAMON checks whether each region is accessed or498 * For each @sample_interval, DAMON checks whether each region is accessed or
475 * not. It aggregates and keeps the access information (number of accesses to499 * not during the last @sample_interval. If such access is found, DAMON
476 * each region) for @aggr_interval time. DAMON also checks whether the target500 * aggregates the information by increasing &damon_region->nr_accesses for
477 * memory regions need update (e.g., by ``mmap()`` calls from the application,501 * @aggr_interval time. For each @aggr_interval, the count is reset. DAMON
478 * in case of virtual memory monitoring) and applies the changes for each502 * also checks whether the target memory regions need update (e.g., by
479 * @ops_update_interval. All time intervals are in micro-seconds.503 * ``mmap()`` calls from the application, in case of virtual memory monitoring)
480 * Please refer to &struct damon_operations and &struct damon_callback for more504 * and applies the changes for each @ops_update_interval. All time intervals
481 * detail.505 * are in micro-seconds. Please refer to &struct damon_operations and &struct
506 * damon_callback for more detail.
482 */507 */
483struct damon_attrs {508struct damon_attrs {
484 unsigned long sample_interval;509 unsigned long sample_interval;
@@ -522,8 +547,18 @@ struct damon_ctx { · 37 unmodified lines
522 struct damon_attrs attrs;547 struct damon_attrs attrs;
523 548
524/* private: internal use only */549/* private: internal use only */
525 struct timespec64 last_aggregation;550 /* number of sample intervals that passed since this context started */
526 struct timespec64 last_ops_update;551 unsigned long passed_sample_intervals;
552 /*
553 * number of sample intervals that should be passed before next
554 * aggregation
555 */
556 unsigned long next_aggregation_sis;
557 /*
558 * number of sample intervals that should be passed before next ops
559 * update
560 */
561 unsigned long next_ops_update_sis;
527 562
528/* public: */563/* public: */
529 struct task_struct *kdamond;564 struct task_struct *kdamond;
@@ -608,6 +643,8 @@ void damon_add_region(struct damon_region *r, struct damon_target *t); · 78 unmodified lines
608void damon_destroy_region(struct damon_region *r, struct damon_target *t);643void damon_destroy_region(struct damon_region *r, struct damon_target *t);
609int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,644int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
610 unsigned int nr_ranges);645 unsigned int nr_ranges);
646void damon_update_region_access_rate(struct damon_region *r, bool accessed,
647 struct damon_attrs *attrs);
611 648
612struct damos_filter *damos_new_filter(enum damos_filter_type type,649struct damos_filter *damos_new_filter(enum damos_filter_type type,
613 bool matching);650 bool matching);
@@ -615,7 +652,9 @@ void damos_add_filter(struct damos *s, struct damos_filter *f); · 1 unmodified line
615void damos_destroy_filter(struct damos_filter *f);652void damos_destroy_filter(struct damos_filter *f);
616 653
617struct damos *damon_new_scheme(struct damos_access_pattern *pattern,654struct damos *damon_new_scheme(struct damos_access_pattern *pattern,
618 enum damos_action action, struct damos_quota *quota,655 enum damos_action action,
656 unsigned long apply_interval_us,
657 struct damos_quota *quota,
619 struct damos_watermarks *wmarks);658 struct damos_watermarks *wmarks);
620void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);659void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);
621void damon_destroy_scheme(struct damos *s);660void damon_destroy_scheme(struct damos *s);
@@ -642,6 +681,13 @@ static inline bool damon_target_has_pid(const struct damon_ctx *ctx) · 20 unmodified lines
642 return ctx->ops.id == DAMON_OPS_VADDR || ctx->ops.id == DAMON_OPS_FVADDR;681 return ctx->ops.id == DAMON_OPS_VADDR || ctx->ops.id == DAMON_OPS_FVADDR;
643}682}
644 683
684static inline unsigned int damon_max_nr_accesses(const struct damon_attrs *attrs)
685{
686 /* {aggr,sample}_interval are unsigned long, hence could overflow */
687 return min(attrs->aggr_interval / attrs->sample_interval,
688 (unsigned long)UINT_MAX);
689}
690
645 691
646int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive);692int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive);
647int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);693int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
include/linux/dax.h+5−5
@@ -159,8 +159,8 @@ int dax_writeback_mapping_range(struct address_space *mapping, · 158 unmodified lines
159 159
160struct page *dax_layout_busy_page(struct address_space *mapping);160struct page *dax_layout_busy_page(struct address_space *mapping);
161struct page *dax_layout_busy_page_range(struct address_space *mapping, loff_t start, loff_t end);161struct page *dax_layout_busy_page_range(struct address_space *mapping, loff_t start, loff_t end);
162dax_entry_t dax_lock_page(struct page *page);162dax_entry_t dax_lock_folio(struct folio *folio);
163void dax_unlock_page(struct page *page, dax_entry_t cookie);163void dax_unlock_folio(struct folio *folio, dax_entry_t cookie);
164dax_entry_t dax_lock_mapping_entry(struct address_space *mapping,164dax_entry_t dax_lock_mapping_entry(struct address_space *mapping,
165 unsigned long index, struct page **page);165 unsigned long index, struct page **page);
166void dax_unlock_mapping_entry(struct address_space *mapping,166void dax_unlock_mapping_entry(struct address_space *mapping,
@@ -182,14 +182,14 @@ static inline int dax_writeback_mapping_range(struct address_space *mapping, · 15 unmodified lines
182 return -EOPNOTSUPP;182 return -EOPNOTSUPP;
183}183}
184 184
185static inline dax_entry_t dax_lock_page(struct page *page)185static inline dax_entry_t dax_lock_folio(struct folio *folio)
186{186{
187 if (IS_DAX(page->mapping->host))187 if (IS_DAX(folio->mapping->host))
188 return ~0UL;188 return ~0UL;
189 return 0;189 return 0;
190}190}
191 191
192static inline void dax_unlock_page(struct page *page, dax_entry_t cookie)192static inline void dax_unlock_folio(struct folio *folio, dax_entry_t cookie)
193{193{
194}194}
195 195
include/linux/fs.h+3−3
@@ -454,7 +454,7 @@ extern const struct address_space_operations empty_aops; · 453 unmodified lines
454 * It is also used to block modification of page cache contents through454 * It is also used to block modification of page cache contents through
455 * memory mappings.455 * memory mappings.
456 * @gfp_mask: Memory allocation flags to use for allocating pages.456 * @gfp_mask: Memory allocation flags to use for allocating pages.
457 * @i_mmap_writable: Number of VM_SHARED mappings.457 * @i_mmap_writable: Number of VM_SHARED, VM_MAYWRITE mappings.
458 * @nr_thps: Number of THPs in the pagecache (non-shmem only).458 * @nr_thps: Number of THPs in the pagecache (non-shmem only).
459 * @i_mmap: Tree of private and shared mappings.459 * @i_mmap: Tree of private and shared mappings.
460 * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable.460 * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable.
@@ -557,7 +557,7 @@ static inline int mapping_mapped(struct address_space *mapping) · 96 unmodified lines
557 557
558/*558/*
559 * Might pages of this file have been modified in userspace?559 * Might pages of this file have been modified in userspace?
560 * Note that i_mmap_writable counts all VM_SHARED vmas: do_mmap560 * Note that i_mmap_writable counts all VM_SHARED, VM_MAYWRITE vmas: do_mmap
561 * marks vma as VM_SHARED if it is shared, and the file was opened for561 * marks vma as VM_SHARED if it is shared, and the file was opened for
562 * writing i.e. vma may be mprotected writable even if now readonly.562 * writing i.e. vma may be mprotected writable even if now readonly.
563 *563 *
@@ -1270,7 +1270,7 @@ struct super_block { · 706 unmodified lines
1270 1270
1271 const struct dentry_operations *s_d_op; /* default d_op for dentries */1271 const struct dentry_operations *s_d_op; /* default d_op for dentries */
1272 1272
1273 struct shrinker s_shrink; /* per-sb shrinker handle */1273 struct shrinker *s_shrink; /* per-sb shrinker handle */
1274 1274
1275 /* Number of inodes with nlink == 0 but still referenced */1275 /* Number of inodes with nlink == 0 but still referenced */
1276 atomic_long_t s_remove_count;1276 atomic_long_t s_remove_count;
include/linux/gfp.h+11−1
@@ -8,6 +8,7 @@ · 7 unmodified lines
8#include <linux/topology.h>8#include <linux/topology.h>
9 9
10struct vm_area_struct;10struct vm_area_struct;
11struct mempolicy;
11 12
12/* Convert GFP flags to their corresponding migrate type */13/* Convert GFP flags to their corresponding migrate type */
13#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)14#define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
@@ -262,7 +263,9 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, · 248 unmodified lines
262 263
263#ifdef CONFIG_NUMA264#ifdef CONFIG_NUMA
264struct page *alloc_pages(gfp_t gfp, unsigned int order);265struct page *alloc_pages(gfp_t gfp, unsigned int order);
265struct folio *folio_alloc(gfp_t gfp, unsigned order);266struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
267 struct mempolicy *mpol, pgoff_t ilx, int nid);
268struct folio *folio_alloc(gfp_t gfp, unsigned int order);
266struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,269struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,
267 unsigned long addr, bool hugepage);270 unsigned long addr, bool hugepage);
268#else271#else
@@ -270,6 +273,11 @@ static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) · 1 unmodified line
270{273{
271 return alloc_pages_node(numa_node_id(), gfp_mask, order);274 return alloc_pages_node(numa_node_id(), gfp_mask, order);
272}275}
276static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
277 struct mempolicy *mpol, pgoff_t ilx, int nid)
278{
279 return alloc_pages(gfp, order);
280}
273static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order)281static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order)
274{282{
275 return __folio_alloc_node(gfp, order, numa_node_id());283 return __folio_alloc_node(gfp, order, numa_node_id());
@@ -320,11 +328,13 @@ extern void page_frag_free(void *addr); · 44 unmodified lines
320#define free_page(addr) free_pages((addr), 0)328#define free_page(addr) free_pages((addr), 0)
321 329
322void page_alloc_init_cpuhp(void);330void page_alloc_init_cpuhp(void);
331int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp);
323void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp);332void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp);
324void drain_all_pages(struct zone *zone);333void drain_all_pages(struct zone *zone);
325void drain_local_pages(struct zone *zone);334void drain_local_pages(struct zone *zone);
326 335
327void page_alloc_init_late(void);336void page_alloc_init_late(void);
337void setup_pcp_cacheinfo(void);
328 338
329/*339/*
330 * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what340 * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what
include/linux/hugetlb.h+14−11
@@ -30,7 +30,7 @@ void free_huge_folio(struct folio *folio); · 29 unmodified lines
30 30
31#ifdef CONFIG_HUGETLB_PAGE31#ifdef CONFIG_HUGETLB_PAGE
32 32
33#include <linux/mempolicy.h>33#include <linux/pagemap.h>
34#include <linux/shm.h>34#include <linux/shm.h>
35#include <asm/tlbflush.h>35#include <asm/tlbflush.h>
36 36
@@ -280,6 +280,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, · 243 unmodified lines
280 unsigned long cp_flags);280 unsigned long cp_flags);
281 281
282bool is_hugetlb_entry_migration(pte_t pte);282bool is_hugetlb_entry_migration(pte_t pte);
283bool is_hugetlb_entry_hwpoisoned(pte_t pte);
283void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);284void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
284 285
285#else /* !CONFIG_HUGETLB_PAGE */286#else /* !CONFIG_HUGETLB_PAGE */
@@ -544,7 +545,6 @@ static inline struct hugetlbfs_sb_info *HUGETLBFS_SB(struct super_block *sb) · 258 unmodified lines
544}545}
545 546
546struct hugetlbfs_inode_info {547struct hugetlbfs_inode_info {
547 struct shared_policy policy;
548 struct inode vfs_inode;548 struct inode vfs_inode;
549 unsigned int seals;549 unsigned int seals;
550};550};
@@ -748,8 +748,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 197 unmodified lines
748 unsigned long addr, int avoid_reserve);748 unsigned long addr, int avoid_reserve);
749struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,749struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
750 nodemask_t *nmask, gfp_t gfp_mask);750 nodemask_t *nmask, gfp_t gfp_mask);
751struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma,
752 unsigned long address);
753int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping,751int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping,
754 pgoff_t idx);752 pgoff_t idx);
755void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,753void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
@@ -844,6 +842,12 @@ static inline unsigned int blocks_per_huge_page(struct hstate *h) · 88 unmodified lines
844 return huge_page_size(h) / 512;842 return huge_page_size(h) / 512;
845}843}
846 844
845static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h,
846 struct address_space *mapping, pgoff_t idx)
847{
848 return filemap_lock_folio(mapping, idx << huge_page_order(h));
849}
850
847#include <asm/hugetlb.h>851#include <asm/hugetlb.h>
848 852
849#ifndef is_hugepage_only_range853#ifndef is_hugepage_only_range
@@ -1040,6 +1044,12 @@ static inline struct hugepage_subpool *hugetlb_folio_subpool(struct folio *folio · 190 unmodified lines
1040 return NULL;1044 return NULL;
1041}1045}
1042 1046
1047static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h,
1048 struct address_space *mapping, pgoff_t idx)
1049{
1050 return NULL;
1051}
1052
1043static inline int isolate_or_dissolve_huge_page(struct page *page,1053static inline int isolate_or_dissolve_huge_page(struct page *page,
1044 struct list_head *list)1054 struct list_head *list)
1045{1055{
@@ -1060,13 +1070,6 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 14 unmodified lines
1060 return NULL;1070 return NULL;
1061}1071}
1062 1072
1063static inline struct folio *alloc_hugetlb_folio_vma(struct hstate *h,
1064 struct vm_area_struct *vma,
1065 unsigned long address)
1066{
1067 return NULL;
1068}
1069
1070static inline int __alloc_bootmem_huge_page(struct hstate *h)1073static inline int __alloc_bootmem_huge_page(struct hstate *h)
1071{1074{
1072 return 0;1075 return 0;
include/linux/hugetlb_cgroup.h+0−11
@@ -22,13 +22,6 @@ struct resv_map; · 21 unmodified lines
22struct file_region;22struct file_region;
23 23
24#ifdef CONFIG_CGROUP_HUGETLB24#ifdef CONFIG_CGROUP_HUGETLB
25/*
26 * Minimum page order trackable by hugetlb cgroup.
27 * At least 3 pages are necessary for all the tracking information.
28 * The second tail page contains all of the hugetlb-specific fields.
29 */
30#define HUGETLB_CGROUP_MIN_ORDER order_base_2(__NR_USED_SUBPAGE)
31
32enum hugetlb_memory_event {25enum hugetlb_memory_event {
33 HUGETLB_MAX,26 HUGETLB_MAX,
34 HUGETLB_NR_MEMORY_EVENTS,27 HUGETLB_NR_MEMORY_EVENTS,
@@ -68,8 +61,6 @@ static inline struct hugetlb_cgroup * · 33 unmodified lines
68__hugetlb_cgroup_from_folio(struct folio *folio, bool rsvd)61__hugetlb_cgroup_from_folio(struct folio *folio, bool rsvd)
69{62{
70 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);63 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);
71 if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER)
72 return NULL;
73 if (rsvd)64 if (rsvd)
74 return folio->_hugetlb_cgroup_rsvd;65 return folio->_hugetlb_cgroup_rsvd;
75 else66 else
@@ -91,8 +82,6 @@ static inline void __set_hugetlb_cgroup(struct folio *folio, · 15 unmodified lines
91 struct hugetlb_cgroup *h_cg, bool rsvd)82 struct hugetlb_cgroup *h_cg, bool rsvd)
92{83{
93 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);84 VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio);
94 if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER)
95 return;
96 if (rsvd)85 if (rsvd)
97 folio->_hugetlb_cgroup_rsvd = h_cg;86 folio->_hugetlb_cgroup_rsvd = h_cg;
98 else87 else
include/linux/jbd2.h+1−1
@@ -886,7 +886,7 @@ struct journal_s · 885 unmodified lines
886 * Journal head shrinker, reclaim buffer's journal head which886 * Journal head shrinker, reclaim buffer's journal head which
887 * has been written back.887 * has been written back.
888 */888 */
889 struct shrinker j_shrinker;889 struct shrinker *j_shrinker;
890 890
891 /**891 /**
892 * @j_checkpoint_jh_count:892 * @j_checkpoint_jh_count:
include/linux/memblock.h+9−0
@@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; · 39 unmodified lines
40 * via a driver, and never indicated in the firmware-provided memory map as40 * via a driver, and never indicated in the firmware-provided memory map as
41 * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the41 * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the
42 * kernel resource tree.42 * kernel resource tree.
43 * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are
44 * not initialized (only for reserved regions).
43 */45 */
44enum memblock_flags {46enum memblock_flags {
45 MEMBLOCK_NONE = 0x0, /* No special request */47 MEMBLOCK_NONE = 0x0, /* No special request */
@@ -47,6 +49,7 @@ enum memblock_flags { · 1 unmodified line
47 MEMBLOCK_MIRROR = 0x2, /* mirrored region */49 MEMBLOCK_MIRROR = 0x2, /* mirrored region */
48 MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */50 MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */
49 MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */51 MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */
52 MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */
50};53};
51 54
52/**55/**
@@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); · 72 unmodified lines
125int memblock_mark_mirror(phys_addr_t base, phys_addr_t size);128int memblock_mark_mirror(phys_addr_t base, phys_addr_t size);
126int memblock_mark_nomap(phys_addr_t base, phys_addr_t size);129int memblock_mark_nomap(phys_addr_t base, phys_addr_t size);
127int memblock_clear_nomap(phys_addr_t base, phys_addr_t size);130int memblock_clear_nomap(phys_addr_t base, phys_addr_t size);
131int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size);
128 132
129void memblock_free_all(void);133void memblock_free_all(void);
130void memblock_free(void *ptr, size_t size);134void memblock_free(void *ptr, size_t size);
@@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) · 128 unmodified lines
259 return m->flags & MEMBLOCK_NOMAP;263 return m->flags & MEMBLOCK_NOMAP;
260}264}
261 265
266static inline bool memblock_is_reserved_noinit(struct memblock_region *m)
267{
268 return m->flags & MEMBLOCK_RSRV_NOINIT;
269}
270
262static inline bool memblock_is_driver_managed(struct memblock_region *m)271static inline bool memblock_is_driver_managed(struct memblock_region *m)
263{272{
264 return m->flags & MEMBLOCK_DRIVER_MANAGED;273 return m->flags & MEMBLOCK_DRIVER_MANAGED;
include/linux/memcontrol.h+65−28
@@ -21,6 +21,7 @@ · 20 unmodified lines
21#include <linux/vmstat.h>21#include <linux/vmstat.h>
22#include <linux/writeback.h>22#include <linux/writeback.h>
23#include <linux/page-flags.h>23#include <linux/page-flags.h>
24#include <linux/shrinker.h>
24 25
25struct mem_cgroup;26struct mem_cgroup;
26struct obj_cgroup;27struct obj_cgroup;
@@ -88,17 +89,6 @@ struct mem_cgroup_reclaim_iter { · 61 unmodified lines
88 unsigned int generation;89 unsigned int generation;
89};90};
90 91
91/*
92 * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
93 * shrinkers, which have elements charged to this memcg.
94 */
95struct shrinker_info {
96 struct rcu_head rcu;
97 atomic_long_t *nr_deferred;
98 unsigned long *map;
99 int map_nr_max;
100};
101
102struct lruvec_stats_percpu {92struct lruvec_stats_percpu {
103 /* Local (CPU and cgroup) state */93 /* Local (CPU and cgroup) state */
104 long state[NR_VM_NODE_STAT_ITEMS];94 long state[NR_VM_NODE_STAT_ITEMS];
@@ -153,7 +143,7 @@ struct mem_cgroup_threshold_ary { · 48 unmodified lines
153 /* Size of entries[] */143 /* Size of entries[] */
154 unsigned int size;144 unsigned int size;
155 /* Array of thresholds */145 /* Array of thresholds */
156 struct mem_cgroup_threshold entries[];146 struct mem_cgroup_threshold entries[] __counted_by(size);
157};147};
158 148
159struct mem_cgroup_thresholds {149struct mem_cgroup_thresholds {
@@ -299,7 +289,13 @@ struct mem_cgroup { · 139 unmodified lines
299 289
300#ifdef CONFIG_MEMCG_KMEM290#ifdef CONFIG_MEMCG_KMEM
301 int kmemcg_id;291 int kmemcg_id;
302 struct obj_cgroup __rcu *objcg;292 /*
293 * memcg->objcg is wiped out as a part of the objcg repaprenting
294 * process. memcg->orig_objcg preserves a pointer (and a reference)
295 * to the original objcg until the end of live of memcg.
296 */
297 struct obj_cgroup __rcu *objcg;
298 struct obj_cgroup *orig_objcg;
303 /* list of inherited objcgs, protected by objcg_lock */299 /* list of inherited objcgs, protected by objcg_lock */
304 struct list_head objcg_list;300 struct list_head objcg_list;
305#endif301#endif
@@ -662,6 +658,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 356 unmodified lines
662 page_counter_read(&memcg->memory);658 page_counter_read(&memcg->memory);
663}659}
664 660
661void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg);
662
665int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);663int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp);
666 664
667/**665/**
@@ -686,6 +684,9 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 18 unmodified lines
686 return __mem_cgroup_charge(folio, mm, gfp);684 return __mem_cgroup_charge(folio, mm, gfp);
687}685}
688 686
687int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
688 long nr_pages);
689
689int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,690int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
690 gfp_t gfp, swp_entry_t entry);691 gfp_t gfp, swp_entry_t entry);
691void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);692void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
@@ -713,6 +714,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 21 unmodified lines
713 __mem_cgroup_uncharge_list(page_list);714 __mem_cgroup_uncharge_list(page_list);
714}715}
715 716
717void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages);
718
719void mem_cgroup_replace_folio(struct folio *old, struct folio *new);
720
716void mem_cgroup_migrate(struct folio *old, struct folio *new);721void mem_cgroup_migrate(struct folio *old, struct folio *new);
717 722
718/**723/**
@@ -769,6 +774,8 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); · 50 unmodified lines
769 774
770struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);775struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);
771 776
777struct mem_cgroup *get_mem_cgroup_from_current(void);
778
772struct lruvec *folio_lruvec_lock(struct folio *folio);779struct lruvec *folio_lruvec_lock(struct folio *folio);
773struct lruvec *folio_lruvec_lock_irq(struct folio *folio);780struct lruvec *folio_lruvec_lock_irq(struct folio *folio);
774struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,781struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
@@ -1080,15 +1087,6 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, · 305 unmodified lines
1080 local_irq_restore(flags);1087 local_irq_restore(flags);
1081}1088}
1082 1089
1083static inline void count_memcg_page_event(struct page *page,
1084 enum vm_event_item idx)
1085{
1086 struct mem_cgroup *memcg = page_memcg(page);
1087
1088 if (memcg)
1089 count_memcg_events(memcg, idx, 1);
1090}
1091
1092static inline void count_memcg_folio_events(struct folio *folio,1090static inline void count_memcg_folio_events(struct folio *folio,
1093 enum vm_event_item idx, unsigned long nr)1091 enum vm_event_item idx, unsigned long nr)
1094{1092{
@@ -1249,12 +1247,23 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 154 unmodified lines
1249 return false;1247 return false;
1250}1248}
1251 1249
1250static inline void mem_cgroup_commit_charge(struct folio *folio,
1251 struct mem_cgroup *memcg)
1252{
1253}
1254
1252static inline int mem_cgroup_charge(struct folio *folio,1255static inline int mem_cgroup_charge(struct folio *folio,
1253 struct mm_struct *mm, gfp_t gfp)1256 struct mm_struct *mm, gfp_t gfp)
1254{1257{
1255 return 0;1258 return 0;
1256}1259}
1257 1260
1261static inline int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg,
1262 gfp_t gfp, long nr_pages)
1263{
1264 return 0;
1265}
1266
1258static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,1267static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
1259 struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)1268 struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
1260{1269{
@@ -1273,6 +1282,16 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 12 unmodified lines
1273{1282{
1274}1283}
1275 1284
1285static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg,
1286 unsigned int nr_pages)
1287{
1288}
1289
1290static inline void mem_cgroup_replace_folio(struct folio *old,
1291 struct folio *new)
1292{
1293}
1294
1276static inline void mem_cgroup_migrate(struct folio *old, struct folio *new)1295static inline void mem_cgroup_migrate(struct folio *old, struct folio *new)
1277{1296{
1278}1297}
@@ -1310,6 +1329,11 @@ static inline struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 31 unmodified lines
1310 return NULL;1329 return NULL;
1311}1330}
1312 1331
1332static inline struct mem_cgroup *get_mem_cgroup_from_current(void)
1333{
1334 return NULL;
1335}
1336
1313static inline1337static inline
1314struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)1338struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)
1315{1339{
@@ -1565,11 +1589,6 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, · 249 unmodified lines
1565{1589{
1566}1590}
1567 1591
1568static inline void count_memcg_page_event(struct page *page,
1569 int idx)
1570{
1571}
1572
1573static inline void count_memcg_folio_events(struct folio *folio,1592static inline void count_memcg_folio_events(struct folio *folio,
1574 enum vm_event_item idx, unsigned long nr)1593 enum vm_event_item idx, unsigned long nr)
1575{1594{
@@ -1763,9 +1782,27 @@ bool mem_cgroup_kmem_disabled(void); · 187 unmodified lines
1763int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);1782int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
1764void __memcg_kmem_uncharge_page(struct page *page, int order);1783void __memcg_kmem_uncharge_page(struct page *page, int order);
1765 1784
1766struct obj_cgroup *get_obj_cgroup_from_current(void);1785/*
1786 * The returned objcg pointer is safe to use without additional
1787 * protection within a scope. The scope is defined either by
1788 * the current task (similar to the "current" global variable)
1789 * or by set_active_memcg() pair.
1790 * Please, use obj_cgroup_get() to get a reference if the pointer
1791 * needs to be used outside of the local scope.
1792 */
1793struct obj_cgroup *current_obj_cgroup(void);
1767struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio);1794struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio);
1768 1795
1796static inline struct obj_cgroup *get_obj_cgroup_from_current(void)
1797{
1798 struct obj_cgroup *objcg = current_obj_cgroup();
1799
1800 if (objcg)
1801 obj_cgroup_get(objcg);
1802
1803 return objcg;
1804}
1805
1769int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size);1806int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size);
1770void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size);1807void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size);
1771 1808
include/linux/memory-tiers.h+40−1
@@ -6,6 +6,7 @@ · 5 unmodified lines
6#include <linux/nodemask.h>6#include <linux/nodemask.h>
7#include <linux/kref.h>7#include <linux/kref.h>
8#include <linux/mmzone.h>8#include <linux/mmzone.h>
9#include <linux/notifier.h>
9/*10/*
10 * Each tier cover a abstrace distance chunk size of 12811 * Each tier cover a abstrace distance chunk size of 128
11 */12 */
@@ -22,7 +23,9 @@ · 10 unmodified lines
22struct memory_tier;23struct memory_tier;
23struct memory_dev_type {24struct memory_dev_type {
24 /* list of memory types that are part of same tier as this type */25 /* list of memory types that are part of same tier as this type */
25 struct list_head tier_sibiling;26 struct list_head tier_sibling;
27 /* list of memory types that are managed by one driver */
28 struct list_head list;
26 /* abstract distance for this specific memory type */29 /* abstract distance for this specific memory type */
27 int adistance;30 int adistance;
28 /* Nodes of same abstract distance */31 /* Nodes of same abstract distance */
@@ -30,12 +33,21 @@ struct memory_dev_type { · 1 unmodified line
30 struct kref kref;33 struct kref kref;
31};34};
32 35
36struct node_hmem_attrs;
37
33#ifdef CONFIG_NUMA38#ifdef CONFIG_NUMA
34extern bool numa_demotion_enabled;39extern bool numa_demotion_enabled;
40extern struct memory_dev_type *default_dram_type;
35struct memory_dev_type *alloc_memory_type(int adistance);41struct memory_dev_type *alloc_memory_type(int adistance);
36void put_memory_type(struct memory_dev_type *memtype);42void put_memory_type(struct memory_dev_type *memtype);
37void init_node_memory_type(int node, struct memory_dev_type *default_type);43void init_node_memory_type(int node, struct memory_dev_type *default_type);
38void clear_node_memory_type(int node, struct memory_dev_type *memtype);44void clear_node_memory_type(int node, struct memory_dev_type *memtype);
45int register_mt_adistance_algorithm(struct notifier_block *nb);
46int unregister_mt_adistance_algorithm(struct notifier_block *nb);
47int mt_calc_adistance(int node, int *adist);
48int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
49 const char *source);
50int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist);
39#ifdef CONFIG_MIGRATION51#ifdef CONFIG_MIGRATION
40int next_demotion_node(int node);52int next_demotion_node(int node);
41void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);53void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
@@ -60,6 +72,7 @@ static inline bool node_is_toptier(int node) · 18 unmodified lines
60#else72#else
61 73
62#define numa_demotion_enabled false74#define numa_demotion_enabled false
75#define default_dram_type NULL
63/*76/*
64 * CONFIG_NUMA implementation returns non NULL error.77 * CONFIG_NUMA implementation returns non NULL error.
65 */78 */
@@ -97,5 +110,31 @@ static inline bool node_is_toptier(int node) · 31 unmodified lines
97{110{
98 return true;111 return true;
99}112}
113
114static inline int register_mt_adistance_algorithm(struct notifier_block *nb)
115{
116 return 0;
117}
118
119static inline int unregister_mt_adistance_algorithm(struct notifier_block *nb)
120{
121 return 0;
122}
123
124static inline int mt_calc_adistance(int node, int *adist)
125{
126 return NOTIFY_DONE;
127}
128
129static inline int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
130 const char *source)
131{
132 return -EIO;
133}
134
135static inline int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist)
136{
137 return -EIO;
138}
100#endif /* CONFIG_NUMA */139#endif /* CONFIG_NUMA */
101#endif /* _LINUX_MEMORY_TIERS_H */140#endif /* _LINUX_MEMORY_TIERS_H */
include/linux/mempolicy.h+31−27
@@ -17,6 +17,8 @@ · 16 unmodified lines
17 17
18struct mm_struct;18struct mm_struct;
19 19
20#define NO_INTERLEAVE_INDEX (-1UL) /* use task il_prev for interleaving */
21
20#ifdef CONFIG_NUMA22#ifdef CONFIG_NUMA
21 23
22/*24/*
@@ -89,8 +91,6 @@ static inline struct mempolicy *mpol_dup(struct mempolicy *pol) · 66 unmodified lines
89 return pol;91 return pol;
90}92}
91 93
92#define vma_policy(vma) ((vma)->vm_policy)
93
94static inline void mpol_get(struct mempolicy *pol)94static inline void mpol_get(struct mempolicy *pol)
95{95{
96 if (pol)96 if (pol)
@@ -107,35 +107,30 @@ static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) · 10 unmodified lines
107 107
108/*108/*
109 * Tree of shared policies for a shared memory region.109 * Tree of shared policies for a shared memory region.
110 * Maintain the policies in a pseudo mm that contains vmas. The vmas
111 * carry the policy. As a special twist the pseudo mm is indexed in pages, not
112 * bytes, so that we can work with shared memory segments bigger than
113 * unsigned long.
114 */110 */
115
116struct sp_node {
117 struct rb_node nd;
118 unsigned long start, end;
119 struct mempolicy *policy;
120};
121
122struct shared_policy {111struct shared_policy {
123 struct rb_root root;112 struct rb_root root;
124 rwlock_t lock;113 rwlock_t lock;
125};114};
115struct sp_node {
116 struct rb_node nd;
117 pgoff_t start, end;
118 struct mempolicy *policy;
119};
126 120
127int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst);121int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst);
128void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol);122void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol);
129int mpol_set_shared_policy(struct shared_policy *info,123int mpol_set_shared_policy(struct shared_policy *sp,
130 struct vm_area_struct *vma,124 struct vm_area_struct *vma, struct mempolicy *mpol);
131 struct mempolicy *new);125void mpol_free_shared_policy(struct shared_policy *sp);
132void mpol_free_shared_policy(struct shared_policy *p);
133struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,126struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
134 unsigned long idx);127 pgoff_t idx);
135 128
136struct mempolicy *get_task_policy(struct task_struct *p);129struct mempolicy *get_task_policy(struct task_struct *p);
137struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,130struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
138 unsigned long addr);131 unsigned long addr, pgoff_t *ilx);
132struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
133 unsigned long addr, int order, pgoff_t *ilx);
139bool vma_policy_mof(struct vm_area_struct *vma);134bool vma_policy_mof(struct vm_area_struct *vma);
140 135
141extern void numa_default_policy(void);136extern void numa_default_policy(void);
@@ -149,8 +144,6 @@ extern int huge_node(struct vm_area_struct *vma, · 7 unmodified lines
149extern bool init_nodemask_of_mempolicy(nodemask_t *mask);144extern bool init_nodemask_of_mempolicy(nodemask_t *mask);
150extern bool mempolicy_in_oom_domain(struct task_struct *tsk,145extern bool mempolicy_in_oom_domain(struct task_struct *tsk,
151 const nodemask_t *mask);146 const nodemask_t *mask);
152extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy);
153
154extern unsigned int mempolicy_slab_node(void);147extern unsigned int mempolicy_slab_node(void);
155 148
156extern enum zone_type policy_zone;149extern enum zone_type policy_zone;
@@ -174,7 +167,7 @@ extern void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol); · 17 unmodified lines
174/* Check if a vma is migratable */167/* Check if a vma is migratable */
175extern bool vma_migratable(struct vm_area_struct *vma);168extern bool vma_migratable(struct vm_area_struct *vma);
176 169
177extern int mpol_misplaced(struct page *, struct vm_area_struct *, unsigned long);170int mpol_misplaced(struct folio *, struct vm_area_struct *, unsigned long);
178extern void mpol_put_task_policy(struct task_struct *);171extern void mpol_put_task_policy(struct task_struct *);
179 172
180static inline bool mpol_is_preferred_many(struct mempolicy *pol)173static inline bool mpol_is_preferred_many(struct mempolicy *pol)
@@ -188,12 +181,17 @@ extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone); · 7 unmodified lines
188 181
189struct mempolicy {};182struct mempolicy {};
190 183
184static inline struct mempolicy *get_task_policy(struct task_struct *p)
185{
186 return NULL;
187}
188
191static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b)189static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b)
192{190{
193 return true;191 return true;
194}192}
195 193
196static inline void mpol_put(struct mempolicy *p)194static inline void mpol_put(struct mempolicy *pol)
197{195{
198}196}
199 197
@@ -212,17 +210,22 @@ static inline void mpol_shared_policy_init(struct shared_policy *sp, · 12 unmodified lines
212{210{
213}211}
214 212
215static inline void mpol_free_shared_policy(struct shared_policy *p)213static inline void mpol_free_shared_policy(struct shared_policy *sp)
216{214{
217}215}
218 216
219static inline struct mempolicy *217static inline struct mempolicy *
220mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx)218mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx)
221{219{
222 return NULL;220 return NULL;
223}221}
224 222
225#define vma_policy(vma) NULL223static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
224 unsigned long addr, int order, pgoff_t *ilx)
225{
226 *ilx = 0;
227 return NULL;
228}
226 229
227static inline int230static inline int
228vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)231vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
@@ -278,7 +281,8 @@ static inline int mpol_parse_str(char *str, struct mempolicy **mpol) · 49 unmodified lines
278}281}
279#endif282#endif
280 283
281static inline int mpol_misplaced(struct page *page, struct vm_area_struct *vma,284static inline int mpol_misplaced(struct folio *folio,
285 struct vm_area_struct *vma,
282 unsigned long address)286 unsigned long address)
283{287{
284 return -1; /* no node preference */288 return -1; /* no node preference */
include/linux/migrate.h+2−2
@@ -142,10 +142,10 @@ const struct movable_operations *page_movable_ops(struct page *page) · 141 unmodified lines
142}142}
143 143
144#ifdef CONFIG_NUMA_BALANCING144#ifdef CONFIG_NUMA_BALANCING
145int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,145int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
146 int node);146 int node);
147#else147#else
148static inline int migrate_misplaced_page(struct page *page,148static inline int migrate_misplaced_folio(struct folio *folio,
149 struct vm_area_struct *vma, int node)149 struct vm_area_struct *vma, int node)
150{150{
151 return -EAGAIN; /* can't migrate now */151 return -EAGAIN; /* can't migrate now */
include/linux/mm.h+130−46
@@ -617,7 +617,7 @@ struct vm_operations_struct { · 616 unmodified lines
617 * policy.617 * policy.
618 */618 */
619 struct mempolicy *(*get_policy)(struct vm_area_struct *vma,619 struct mempolicy *(*get_policy)(struct vm_area_struct *vma,
620 unsigned long addr);620 unsigned long addr, pgoff_t *ilx);
621#endif621#endif
622 /*622 /*
623 * Called by vm_normal_page() for special PTEs to find the623 * Called by vm_normal_page() for special PTEs to find the
@@ -935,6 +935,17 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) · 311 unmodified lines
935 return vma->vm_flags & VM_ACCESS_FLAGS;935 return vma->vm_flags & VM_ACCESS_FLAGS;
936}936}
937 937
938static inline bool is_shared_maywrite(vm_flags_t vm_flags)
939{
940 return (vm_flags & (VM_SHARED | VM_MAYWRITE)) ==
941 (VM_SHARED | VM_MAYWRITE);
942}
943
944static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma)
945{
946 return is_shared_maywrite(vma->vm_flags);
947}
948
938static inline949static inline
939struct vm_area_struct *vma_find(struct vma_iterator *vmi, unsigned long max)950struct vm_area_struct *vma_find(struct vma_iterator *vmi, unsigned long max)
940{951{
@@ -1335,7 +1346,6 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, · 394 unmodified lines
1335 struct page *page, unsigned int nr, unsigned long addr);1346 struct page *page, unsigned int nr, unsigned long addr);
1336 1347
1337vm_fault_t finish_fault(struct vm_fault *vmf);1348vm_fault_t finish_fault(struct vm_fault *vmf);
1338vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);
1339#endif1349#endif
1340 1350
1341/*1351/*
@@ -1684,26 +1694,26 @@ static inline bool __cpupid_match_pid(pid_t task_pid, int cpupid) · 342 unmodified lines
1684 1694
1685#define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid)1695#define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid)
1686#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS1696#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
1687static inline int page_cpupid_xchg_last(struct page *page, int cpupid)1697static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
1688{1698{
1689 return xchg(&page->_last_cpupid, cpupid & LAST_CPUPID_MASK);1699 return xchg(&folio->_last_cpupid, cpupid & LAST_CPUPID_MASK);
1690}1700}
1691 1701
1692static inline int page_cpupid_last(struct page *page)1702static inline int folio_last_cpupid(struct folio *folio)
1693{1703{
1694 return page->_last_cpupid;1704 return folio->_last_cpupid;
1695}1705}
1696static inline void page_cpupid_reset_last(struct page *page)1706static inline void page_cpupid_reset_last(struct page *page)
1697{1707{
1698 page->_last_cpupid = -1 & LAST_CPUPID_MASK;1708 page->_last_cpupid = -1 & LAST_CPUPID_MASK;
1699}1709}
1700#else1710#else
1701static inline int page_cpupid_last(struct page *page)1711static inline int folio_last_cpupid(struct folio *folio)
1702{1712{
1703 return (page->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;1713 return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK;
1704}1714}
1705 1715
1706extern int page_cpupid_xchg_last(struct page *page, int cpupid);1716int folio_xchg_last_cpupid(struct folio *folio, int cpupid);
1707 1717
1708static inline void page_cpupid_reset_last(struct page *page)1718static inline void page_cpupid_reset_last(struct page *page)
1709{1719{
@@ -1711,11 +1721,12 @@ static inline void page_cpupid_reset_last(struct page *page) · 1 unmodified line
1711}1721}
1712#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */1722#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
1713 1723
1714static inline int xchg_page_access_time(struct page *page, int time)1724static inline int folio_xchg_access_time(struct folio *folio, int time)
1715{1725{
1716 int last_time;1726 int last_time;
1717 1727
1718 last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS);1728 last_time = folio_xchg_last_cpupid(folio,
1729 time >> PAGE_ACCESS_TIME_BUCKETS);
1719 return last_time << PAGE_ACCESS_TIME_BUCKETS;1730 return last_time << PAGE_ACCESS_TIME_BUCKETS;
1720}1731}
1721 1732
@@ -1729,19 +1740,19 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) · 7 unmodified lines
1729 }1740 }
1730}1741}
1731#else /* !CONFIG_NUMA_BALANCING */1742#else /* !CONFIG_NUMA_BALANCING */
1732static inline int page_cpupid_xchg_last(struct page *page, int cpupid)1743static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
1733{1744{
1734 return page_to_nid(page); /* XXX */1745 return folio_nid(folio); /* XXX */
1735}1746}
1736 1747
1737static inline int xchg_page_access_time(struct page *page, int time)1748static inline int folio_xchg_access_time(struct folio *folio, int time)
1738{1749{
1739 return 0;1750 return 0;
1740}1751}
1741 1752
1742static inline int page_cpupid_last(struct page *page)1753static inline int folio_last_cpupid(struct folio *folio)
1743{1754{
1744 return page_to_nid(page); /* XXX */1755 return folio_nid(folio); /* XXX */
1745}1756}
1746 1757
1747static inline int cpupid_to_nid(int cpupid)1758static inline int cpupid_to_nid(int cpupid)
@@ -2325,6 +2336,8 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, · 577 unmodified lines
2325 pte_t pte);2336 pte_t pte);
2326struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,2337struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
2327 pte_t pte);2338 pte_t pte);
2339struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
2340 unsigned long addr, pmd_t pmd);
2328struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,2341struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
2329 pmd_t pmd);2342 pmd_t pmd);
2330 2343
@@ -2411,8 +2424,6 @@ extern int access_process_vm(struct task_struct *tsk, unsigned long addr, · 80 unmodified lines
2411 void *buf, int len, unsigned int gup_flags);2424 void *buf, int len, unsigned int gup_flags);
2412extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,2425extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,
2413 void *buf, int len, unsigned int gup_flags);2426 void *buf, int len, unsigned int gup_flags);
2414extern int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
2415 void *buf, int len, unsigned int gup_flags);
2416 2427
2417long get_user_pages_remote(struct mm_struct *mm,2428long get_user_pages_remote(struct mm_struct *mm,
2418 unsigned long start, unsigned long nr_pages,2429 unsigned long start, unsigned long nr_pages,
@@ -2423,6 +2434,9 @@ long pin_user_pages_remote(struct mm_struct *mm, · 4 unmodified lines
2423 unsigned int gup_flags, struct page **pages,2434 unsigned int gup_flags, struct page **pages,
2424 int *locked);2435 int *locked);
2425 2436
2437/*
2438 * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
2439 */
2426static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,2440static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,
2427 unsigned long addr,2441 unsigned long addr,
2428 int gup_flags,2442 int gup_flags,
@@ -2430,12 +2444,15 @@ static inline struct page *get_user_page_vma_remote(struct mm_struct *mm, · 1 unmodified line
2430{2444{
2431 struct page *page;2445 struct page *page;
2432 struct vm_area_struct *vma;2446 struct vm_area_struct *vma;
2433 int got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);2447 int got;
2448
2449 if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT)))
2450 return ERR_PTR(-EINVAL);
2451
2452 got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);
2434 2453
2435 if (got < 0)2454 if (got < 0)
2436 return ERR_PTR(got);2455 return ERR_PTR(got);
2437 if (got == 0)
2438 return NULL;
2439 2456
2440 vma = vma_lookup(mm, addr);2457 vma = vma_lookup(mm, addr);
2441 if (WARN_ON_ONCE(!vma)) {2458 if (WARN_ON_ONCE(!vma)) {
@@ -2478,7 +2495,7 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen); · 36 unmodified lines
2478extern unsigned long move_page_tables(struct vm_area_struct *vma,2495extern unsigned long move_page_tables(struct vm_area_struct *vma,
2479 unsigned long old_addr, struct vm_area_struct *new_vma,2496 unsigned long old_addr, struct vm_area_struct *new_vma,
2480 unsigned long new_addr, unsigned long len,2497 unsigned long new_addr, unsigned long len,
2481 bool need_rmap_locks);2498 bool need_rmap_locks, bool for_stack);
2482 2499
2483/*2500/*
2484 * Flags used by change_protection(). For now we make it a bitmap so2501 * Flags used by change_protection(). For now we make it a bitmap so
@@ -2626,14 +2643,6 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, · 141 unmodified lines
2626 *maxrss = hiwater_rss;2643 *maxrss = hiwater_rss;
2627}2644}
2628 2645
2629#if defined(SPLIT_RSS_COUNTING)
2630void sync_mm_rss(struct mm_struct *mm);
2631#else
2632static inline void sync_mm_rss(struct mm_struct *mm)
2633{
2634}
2635#endif
2636
2637#ifndef CONFIG_ARCH_HAS_PTE_SPECIAL2646#ifndef CONFIG_ARCH_HAS_PTE_SPECIAL
2638static inline int pte_special(pte_t pte)2647static inline int pte_special(pte_t pte)
2639{2648{
@@ -3055,6 +3064,22 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) · 415 unmodified lines
3055 return ptl;3064 return ptl;
3056}3065}
3057 3066
3067static inline void pagetable_pud_ctor(struct ptdesc *ptdesc)
3068{
3069 struct folio *folio = ptdesc_folio(ptdesc);
3070
3071 __folio_set_pgtable(folio);
3072 lruvec_stat_add_folio(folio, NR_PAGETABLE);
3073}
3074
3075static inline void pagetable_pud_dtor(struct ptdesc *ptdesc)
3076{
3077 struct folio *folio = ptdesc_folio(ptdesc);
3078
3079 __folio_clear_pgtable(folio);
3080 lruvec_stat_sub_folio(folio, NR_PAGETABLE);
3081}
3082
3058extern void __init pagecache_init(void);3083extern void __init pagecache_init(void);
3059extern void free_initmem(void);3084extern void free_initmem(void);
3060 3085
@@ -3219,22 +3244,73 @@ extern int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, · 158 unmodified lines
3219 struct vm_area_struct *next);3244 struct vm_area_struct *next);
3220extern int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,3245extern int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
3221 unsigned long start, unsigned long end, pgoff_t pgoff);3246 unsigned long start, unsigned long end, pgoff_t pgoff);
3222extern struct vm_area_struct *vma_merge(struct vma_iterator *vmi,
3223 struct mm_struct *, struct vm_area_struct *prev, unsigned long addr,
3224 unsigned long end, unsigned long vm_flags, struct anon_vma *,
3225 struct file *, pgoff_t, struct mempolicy *, struct vm_userfaultfd_ctx,
3226 struct anon_vma_name *);
3227extern struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *);3247extern struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *);
3228extern int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *,
3229 unsigned long addr, int new_below);
3230extern int split_vma(struct vma_iterator *vmi, struct vm_area_struct *,
3231 unsigned long addr, int new_below);
3232extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);3248extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);
3233extern void unlink_file_vma(struct vm_area_struct *);3249extern void unlink_file_vma(struct vm_area_struct *);
3234extern struct vm_area_struct *copy_vma(struct vm_area_struct **,3250extern struct vm_area_struct *copy_vma(struct vm_area_struct **,
3235 unsigned long addr, unsigned long len, pgoff_t pgoff,3251 unsigned long addr, unsigned long len, pgoff_t pgoff,
3236 bool *need_rmap_locks);3252 bool *need_rmap_locks);
3237extern void exit_mmap(struct mm_struct *);3253extern void exit_mmap(struct mm_struct *);
3254struct vm_area_struct *vma_modify(struct vma_iterator *vmi,
3255 struct vm_area_struct *prev,
3256 struct vm_area_struct *vma,
3257 unsigned long start, unsigned long end,
3258 unsigned long vm_flags,
3259 struct mempolicy *policy,
3260 struct vm_userfaultfd_ctx uffd_ctx,
3261 struct anon_vma_name *anon_name);
3262
3263/* We are about to modify the VMA's flags. */
3264static inline struct vm_area_struct
3265*vma_modify_flags(struct vma_iterator *vmi,
3266 struct vm_area_struct *prev,
3267 struct vm_area_struct *vma,
3268 unsigned long start, unsigned long end,
3269 unsigned long new_flags)
3270{
3271 return vma_modify(vmi, prev, vma, start, end, new_flags,
3272 vma_policy(vma), vma->vm_userfaultfd_ctx,
3273 anon_vma_name(vma));
3274}
3275
3276/* We are about to modify the VMA's flags and/or anon_name. */
3277static inline struct vm_area_struct
3278*vma_modify_flags_name(struct vma_iterator *vmi,
3279 struct vm_area_struct *prev,
3280 struct vm_area_struct *vma,
3281 unsigned long start,
3282 unsigned long end,
3283 unsigned long new_flags,
3284 struct anon_vma_name *new_name)
3285{
3286 return vma_modify(vmi, prev, vma, start, end, new_flags,
3287 vma_policy(vma), vma->vm_userfaultfd_ctx, new_name);
3288}
3289
3290/* We are about to modify the VMA's memory policy. */
3291static inline struct vm_area_struct
3292*vma_modify_policy(struct vma_iterator *vmi,
3293 struct vm_area_struct *prev,
3294 struct vm_area_struct *vma,
3295 unsigned long start, unsigned long end,
3296 struct mempolicy *new_pol)
3297{
3298 return vma_modify(vmi, prev, vma, start, end, vma->vm_flags,
3299 new_pol, vma->vm_userfaultfd_ctx, anon_vma_name(vma));
3300}
3301
3302/* We are about to modify the VMA's flags and/or uffd context. */
3303static inline struct vm_area_struct
3304*vma_modify_flags_uffd(struct vma_iterator *vmi,
3305 struct vm_area_struct *prev,
3306 struct vm_area_struct *vma,
3307 unsigned long start, unsigned long end,
3308 unsigned long new_flags,
3309 struct vm_userfaultfd_ctx new_ctx)
3310{
3311 return vma_modify(vmi, prev, vma, start, end, new_flags,
3312 vma_policy(vma), new_ctx, anon_vma_name(vma));
3313}
3238 3314
3239static inline int check_data_rlimit(unsigned long rlim,3315static inline int check_data_rlimit(unsigned long rlim,
3240 unsigned long new,3316 unsigned long new,
@@ -3997,25 +4073,26 @@ static inline void mem_dump_obj(void *object) {} · 756 unmodified lines
3997#endif4073#endif
3998 4074
3999/**4075/**
4000 * seal_check_future_write - Check for F_SEAL_FUTURE_WRITE flag and handle it4076 * seal_check_write - Check for F_SEAL_WRITE or F_SEAL_FUTURE_WRITE flags and
4077 * handle them.
4001 * @seals: the seals to check4078 * @seals: the seals to check
4002 * @vma: the vma to operate on4079 * @vma: the vma to operate on
4003 *4080 *
4004 * Check whether F_SEAL_FUTURE_WRITE is set; if so, do proper check/handling on4081 * Check whether F_SEAL_WRITE or F_SEAL_FUTURE_WRITE are set; if so, do proper
4005 * the vma flags. Return 0 if check pass, or <0 for errors.4082 * check/handling on the vma flags. Return 0 if check pass, or <0 for errors.
4006 */4083 */
4007static inline int seal_check_future_write(int seals, struct vm_area_struct *vma)4084static inline int seal_check_write(int seals, struct vm_area_struct *vma)
4008{4085{
4009 if (seals & F_SEAL_FUTURE_WRITE) {4086 if (seals & (F_SEAL_WRITE | F_SEAL_FUTURE_WRITE)) {
4010 /*4087 /*
4011 * New PROT_WRITE and MAP_SHARED mmaps are not allowed when4088 * New PROT_WRITE and MAP_SHARED mmaps are not allowed when
4012 * "future write" seal active.4089 * write seals are active.
4013 */4090 */
4014 if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE))4091 if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE))
4015 return -EPERM;4092 return -EPERM;
4016 4093
4017 /*4094 /*
4018 * Since an F_SEAL_FUTURE_WRITE sealed memfd can be mapped as4095 * Since an F_SEAL_[FUTURE_]WRITE sealed memfd can be mapped as
4019 * MAP_SHARED and read-only, take care to not allow mprotect to4096 * MAP_SHARED and read-only, take care to not allow mprotect to
4020 * revert protections on such mappings. Do this only for shared4097 * revert protections on such mappings. Do this only for shared
4021 * mappings. For private mappings, don't need to mask4098 * mappings. For private mappings, don't need to mask
@@ -4059,4 +4136,11 @@ static inline void accept_memory(phys_addr_t start, phys_addr_t end) · 37 unmodified lines
4059 4136
4060#endif4137#endif
4061 4138
4139static inline bool pfn_is_unaccepted_memory(unsigned long pfn)
4140{
4141 phys_addr_t paddr = pfn << PAGE_SHIFT;
4142
4143 return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE);
4144}
4145
4062#endif /* _LINUX_MM_H */4146#endif /* _LINUX_MM_H */
include/linux/mm_inline.h+1−19
@@ -4,6 +4,7 @@ · 3 unmodified lines
4 4
5#include <linux/atomic.h>5#include <linux/atomic.h>
6#include <linux/huge_mm.h>6#include <linux/huge_mm.h>
7#include <linux/mm_types.h>
7#include <linux/swap.h>8#include <linux/swap.h>
8#include <linux/string.h>9#include <linux/string.h>
9#include <linux/userfaultfd_k.h>10#include <linux/userfaultfd_k.h>
@@ -352,15 +353,6 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) · 342 unmodified lines
352}353}
353 354
354#ifdef CONFIG_ANON_VMA_NAME355#ifdef CONFIG_ANON_VMA_NAME
355/*
356 * mmap_lock should be read-locked when calling anon_vma_name(). Caller should
357 * either keep holding the lock while using the returned pointer or it should
358 * raise anon_vma_name refcount before releasing the lock.
359 */
360extern struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma);
361extern struct anon_vma_name *anon_vma_name_alloc(const char *name);
362extern void anon_vma_name_free(struct kref *kref);
363
364/* mmap_lock should be read-locked */356/* mmap_lock should be read-locked */
365static inline void anon_vma_name_get(struct anon_vma_name *anon_name)357static inline void anon_vma_name_get(struct anon_vma_name *anon_name)
366{358{
@@ -415,16 +407,6 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, · 48 unmodified lines
415}407}
416 408
417#else /* CONFIG_ANON_VMA_NAME */409#else /* CONFIG_ANON_VMA_NAME */
418static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma)
419{
420 return NULL;
421}
422
423static inline struct anon_vma_name *anon_vma_name_alloc(const char *name)
424{
425 return NULL;
426}
427
428static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {}410static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {}
429static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {}411static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {}
430static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma,412static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma,
include/linux/mm_types.h+45−4
@@ -188,6 +188,10 @@ struct page { · 187 unmodified lines
188 not kmapped, ie. highmem) */188 not kmapped, ie. highmem) */
189#endif /* WANT_PAGE_VIRTUAL */189#endif /* WANT_PAGE_VIRTUAL */
190 190
191#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
192 int _last_cpupid;
193#endif
194
191#ifdef CONFIG_KMSAN195#ifdef CONFIG_KMSAN
192 /*196 /*
193 * KMSAN metadata for this page:197 * KMSAN metadata for this page:
@@ -199,10 +203,6 @@ struct page { · 5 unmodified lines
199 struct page *kmsan_shadow;203 struct page *kmsan_shadow;
200 struct page *kmsan_origin;204 struct page *kmsan_origin;
201#endif205#endif
202
203#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
204 int _last_cpupid;
205#endif
206} _struct_page_alignment;206} _struct_page_alignment;
207 207
208/*208/*
@@ -261,6 +261,8 @@ typedef struct { · 52 unmodified lines
261 * @_refcount: Do not access this member directly. Use folio_ref_count()261 * @_refcount: Do not access this member directly. Use folio_ref_count()
262 * to find how many references there are to this folio.262 * to find how many references there are to this folio.
263 * @memcg_data: Memory Control Group data.263 * @memcg_data: Memory Control Group data.
264 * @virtual: Virtual address in the kernel direct map.
265 * @_last_cpupid: IDs of last CPU and last process that accessed the folio.
264 * @_entire_mapcount: Do not use directly, call folio_entire_mapcount().266 * @_entire_mapcount: Do not use directly, call folio_entire_mapcount().
265 * @_nr_pages_mapped: Do not use directly, call folio_mapcount().267 * @_nr_pages_mapped: Do not use directly, call folio_mapcount().
266 * @_pincount: Do not use directly, call folio_maybe_dma_pinned().268 * @_pincount: Do not use directly, call folio_maybe_dma_pinned().
@@ -306,6 +308,12 @@ struct folio { · 39 unmodified lines
306 atomic_t _refcount;308 atomic_t _refcount;
307#ifdef CONFIG_MEMCG309#ifdef CONFIG_MEMCG
308 unsigned long memcg_data;310 unsigned long memcg_data;
311#endif
312#if defined(WANT_PAGE_VIRTUAL)
313 void *virtual;
314#endif
315#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
316 int _last_cpupid;
309#endif317#endif
310 /* private: the union with struct page is transitional */318 /* private: the union with struct page is transitional */
311 };319 };
@@ -362,6 +370,12 @@ FOLIO_MATCH(_refcount, _refcount); · 50 unmodified lines
362#ifdef CONFIG_MEMCG370#ifdef CONFIG_MEMCG
363FOLIO_MATCH(memcg_data, memcg_data);371FOLIO_MATCH(memcg_data, memcg_data);
364#endif372#endif
373#if defined(WANT_PAGE_VIRTUAL)
374FOLIO_MATCH(virtual, virtual);
375#endif
376#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
377FOLIO_MATCH(_last_cpupid, _last_cpupid);
378#endif
365#undef FOLIO_MATCH379#undef FOLIO_MATCH
366#define FOLIO_MATCH(pg, fl) \380#define FOLIO_MATCH(pg, fl) \
367 static_assert(offsetof(struct folio, fl) == \381 static_assert(offsetof(struct folio, fl) == \
@@ -535,6 +549,27 @@ struct anon_vma_name { · 167 unmodified lines
535 char name[];549 char name[];
536};550};
537 551
552#ifdef CONFIG_ANON_VMA_NAME
553/*
554 * mmap_lock should be read-locked when calling anon_vma_name(). Caller should
555 * either keep holding the lock while using the returned pointer or it should
556 * raise anon_vma_name refcount before releasing the lock.
557 */
558struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma);
559struct anon_vma_name *anon_vma_name_alloc(const char *name);
560void anon_vma_name_free(struct kref *kref);
561#else /* CONFIG_ANON_VMA_NAME */
562static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma)
563{
564 return NULL;
565}
566
567static inline struct anon_vma_name *anon_vma_name_alloc(const char *name)
568{
569 return NULL;
570}
571#endif
572
538struct vma_lock {573struct vma_lock {
539 struct rw_semaphore lock;574 struct rw_semaphore lock;
540};575};
@@ -678,6 +713,12 @@ struct vm_area_struct { · 137 unmodified lines
678 struct vm_userfaultfd_ctx vm_userfaultfd_ctx;713 struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
679} __randomize_layout;714} __randomize_layout;
680 715
716#ifdef CONFIG_NUMA
717#define vma_policy(vma) ((vma)->vm_policy)
718#else
719#define vma_policy(vma) NULL
720#endif
721
681#ifdef CONFIG_SCHED_MM_CID722#ifdef CONFIG_SCHED_MM_CID
682struct mm_cid {723struct mm_cid {
683 u64 time;724 u64 time;
include/linux/mmu_notifier.h+8−1
@@ -459,7 +459,14 @@ mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) · 458 unmodified lines
459 lock_map_release(&__mmu_notifier_invalidate_range_start_map);459 lock_map_release(&__mmu_notifier_invalidate_range_start_map);
460}460}
461 461
462static inline int462/*
463 * This version of mmu_notifier_invalidate_range_start() avoids blocking, but it
464 * can return an error if a notifier can't proceed without blocking, in which
465 * case you're not allowed to modify PTEs in the specified range.
466 *
467 * This is mainly intended for OOM handling.
468 */
469static inline int __must_check
463mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range)470mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range)
464{471{
465 int ret = 0;472 int ret = 0;
include/linux/mmzone.h+24−5
@@ -639,8 +639,6 @@ struct lruvec { · 638 unmodified lines
639#endif639#endif
640};640};
641 641
642/* Isolate unmapped pages */
643#define ISOLATE_UNMAPPED ((__force isolate_mode_t)0x2)
644/* Isolate for asynchronous migration */642/* Isolate for asynchronous migration */
645#define ISOLATE_ASYNC_MIGRATE ((__force isolate_mode_t)0x4)643#define ISOLATE_ASYNC_MIGRATE ((__force isolate_mode_t)0x4)
646/* Isolate unevictable pages */644/* Isolate unevictable pages */
@@ -676,15 +674,34 @@ enum zone_watermarks { · 29 unmodified lines
676#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)674#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
677#define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)675#define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)
678 676
677/*
678 * Flags used in pcp->flags field.
679 *
680 * PCPF_PREV_FREE_HIGH_ORDER: a high-order page is freed in the
681 * previous page freeing. To avoid to drain PCP for an accident
682 * high-order page freeing.
683 *
684 * PCPF_FREE_HIGH_BATCH: preserve "pcp->batch" pages in PCP before
685 * draining PCP for consecutive high-order pages freeing without
686 * allocation if data cache slice of CPU is large enough. To reduce
687 * zone lock contention and keep cache-hot pages reusing.
688 */
689#define PCPF_PREV_FREE_HIGH_ORDER BIT(0)
690#define PCPF_FREE_HIGH_BATCH BIT(1)
691
679struct per_cpu_pages {692struct per_cpu_pages {
680 spinlock_t lock; /* Protects lists field */693 spinlock_t lock; /* Protects lists field */
681 int count; /* number of pages in the list */694 int count; /* number of pages in the list */
682 int high; /* high watermark, emptying needed */695 int high; /* high watermark, emptying needed */
696 int high_min; /* min high watermark */
697 int high_max; /* max high watermark */
683 int batch; /* chunk size for buddy add/remove */698 int batch; /* chunk size for buddy add/remove */
684 short free_factor; /* batch scaling factor during free */699 u8 flags; /* protected by pcp->lock */
700 u8 alloc_factor; /* batch scaling factor during allocate */
685#ifdef CONFIG_NUMA701#ifdef CONFIG_NUMA
686 short expire; /* When 0, remote pagesets are drained */702 u8 expire; /* When 0, remote pagesets are drained */
687#endif703#endif
704 short free_count; /* consecutive free count */
688 705
689 /* Lists of pages, one per migrate type stored on the pcp-lists */706 /* Lists of pages, one per migrate type stored on the pcp-lists */
690 struct list_head lists[NR_PCP_LISTS];707 struct list_head lists[NR_PCP_LISTS];
@@ -837,7 +854,8 @@ struct zone { · 146 unmodified lines
837 * the high and batch values are copied to individual pagesets for854 * the high and batch values are copied to individual pagesets for
838 * faster access855 * faster access
839 */856 */
840 int pageset_high;857 int pageset_high_min;
858 int pageset_high_max;
841 int pageset_batch;859 int pageset_batch;
842 860
843#ifndef CONFIG_SPARSEMEM861#ifndef CONFIG_SPARSEMEM
@@ -998,6 +1016,7 @@ enum zone_flags { · 154 unmodified lines
998 * Cleared when kswapd is woken.1016 * Cleared when kswapd is woken.
999 */1017 */
1000 ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */1018 ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */
1019 ZONE_BELOW_HIGH, /* zone is below high watermark. */
1001};1020};
1002 1021
1003static inline unsigned long zone_managed_pages(struct zone *zone)1022static inline unsigned long zone_managed_pages(struct zone *zone)
include/linux/page-flags.h+19−0
@@ -692,6 +692,25 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) · 691 unmodified lines
692 692
693u64 stable_page_flags(struct page *page);693u64 stable_page_flags(struct page *page);
694 694
695/**
696 * folio_xor_flags_has_waiters - Change some folio flags.
697 * @folio: The folio.
698 * @mask: Bits set in this word will be changed.
699 *
700 * This must only be used for flags which are changed with the folio
701 * lock held. For example, it is unsafe to use for PG_dirty as that
702 * can be set without the folio lock held. It can also only be used
703 * on flags which are in the range 0-6 as some of the implementations
704 * only affect those bits.
705 *
706 * Return: Whether there are tasks waiting on the folio.
707 */
708static inline bool folio_xor_flags_has_waiters(struct folio *folio,
709 unsigned long mask)
710{
711 return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0));
712}
713
695/**714/**
696 * folio_test_uptodate - Is this folio up to date?715 * folio_test_uptodate - Is this folio up to date?
697 * @folio: The folio.716 * @folio: The folio.
include/linux/pagemap.h+3−30
@@ -789,9 +789,6 @@ static inline pgoff_t folio_next_index(struct folio *folio) · 788 unmodified lines
789 */789 */
790static inline struct page *folio_file_page(struct folio *folio, pgoff_t index)790static inline struct page *folio_file_page(struct folio *folio, pgoff_t index)
791{791{
792 /* HugeTLBfs indexes the page cache in units of hpage_size */
793 if (folio_test_hugetlb(folio))
794 return &folio->page;
795 return folio_page(folio, index & (folio_nr_pages(folio) - 1));792 return folio_page(folio, index & (folio_nr_pages(folio) - 1));
796}793}
797 794
@@ -807,9 +804,6 @@ static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) · 9 unmodified lines
807 */804 */
808static inline bool folio_contains(struct folio *folio, pgoff_t index)805static inline bool folio_contains(struct folio *folio, pgoff_t index)
809{806{
810 /* HugeTLBfs indexes the page cache in units of hpage_size */
811 if (folio_test_hugetlb(folio))
812 return folio->index == index;
813 return index - folio_index(folio) < folio_nr_pages(folio);807 return index - folio_index(folio) < folio_nr_pages(folio);
814}808}
815 809
@@ -867,10 +861,9 @@ static inline struct folio *read_mapping_folio(struct address_space *mapping, · 51 unmodified lines
867}861}
868 862
869/*863/*
870 * Get index of the page within radix-tree (but not for hugetlb pages).864 * Get the offset in PAGE_SIZE (even for hugetlb pages).
871 * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE)
872 */865 */
873static inline pgoff_t page_to_index(struct page *page)866static inline pgoff_t page_to_pgoff(struct page *page)
874{867{
875 struct page *head;868 struct page *head;
876 869
@@ -885,19 +878,6 @@ static inline pgoff_t page_to_index(struct page *page) · 8 unmodified lines
885 return head->index + page - head;878 return head->index + page - head;
886}879}
887 880
888extern pgoff_t hugetlb_basepage_index(struct page *page);
889
890/*
891 * Get the offset in PAGE_SIZE (even for hugetlb pages).
892 * (TODO: hugetlb pages should have ->index in PAGE_SIZE)
893 */
894static inline pgoff_t page_to_pgoff(struct page *page)
895{
896 if (unlikely(PageHuge(page)))
897 return hugetlb_basepage_index(page);
898 return page_to_index(page);
899}
900
901/*881/*
902 * Return byte-offset into filesystem object for page.882 * Return byte-offset into filesystem object for page.
903 */883 */
@@ -934,24 +914,16 @@ static inline loff_t folio_file_pos(struct folio *folio) · 30 unmodified lines
934 914
935/*915/*
936 * Get the offset in PAGE_SIZE (even for hugetlb folios).916 * Get the offset in PAGE_SIZE (even for hugetlb folios).
937 * (TODO: hugetlb folios should have ->index in PAGE_SIZE)
938 */917 */
939static inline pgoff_t folio_pgoff(struct folio *folio)918static inline pgoff_t folio_pgoff(struct folio *folio)
940{919{
941 if (unlikely(folio_test_hugetlb(folio)))
942 return hugetlb_basepage_index(&folio->page);
943 return folio->index;920 return folio->index;
944}921}
945 922
946extern pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
947 unsigned long address);
948
949static inline pgoff_t linear_page_index(struct vm_area_struct *vma,923static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
950 unsigned long address)924 unsigned long address)
951{925{
952 pgoff_t pgoff;926 pgoff_t pgoff;
953 if (unlikely(is_vm_hugetlb_page(vma)))
954 return linear_hugepage_index(vma, address);
955 pgoff = (address - vma->vm_start) >> PAGE_SHIFT;927 pgoff = (address - vma->vm_start) >> PAGE_SHIFT;
956 pgoff += vma->vm_pgoff;928 pgoff += vma->vm_pgoff;
957 return pgoff;929 return pgoff;
@@ -1129,6 +1101,7 @@ static inline void wait_on_page_locked(struct page *page) · 171 unmodified lines
1129 folio_wait_locked(page_folio(page));1101 folio_wait_locked(page_folio(page));
1130}1102}
1131 1103
1104void folio_end_read(struct folio *folio, bool success);
1132void wait_on_page_writeback(struct page *page);1105void wait_on_page_writeback(struct page *page);
1133void folio_wait_writeback(struct folio *folio);1106void folio_wait_writeback(struct folio *folio);
1134int folio_wait_writeback_killable(struct folio *folio);1107int folio_wait_writeback_killable(struct folio *folio);
include/linux/percpu_counter.h+30−0
@@ -57,6 +57,8 @@ void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, · 56 unmodified lines
57 s32 batch);57 s32 batch);
58s64 __percpu_counter_sum(struct percpu_counter *fbc);58s64 __percpu_counter_sum(struct percpu_counter *fbc);
59int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch);59int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch);
60bool __percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit,
61 s64 amount, s32 batch);
60void percpu_counter_sync(struct percpu_counter *fbc);62void percpu_counter_sync(struct percpu_counter *fbc);
61 63
62static inline int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs)64static inline int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs)
@@ -69,6 +71,13 @@ static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 6 unmodified lines
69 percpu_counter_add_batch(fbc, amount, percpu_counter_batch);71 percpu_counter_add_batch(fbc, amount, percpu_counter_batch);
70}72}
71 73
74static inline bool
75percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount)
76{
77 return __percpu_counter_limited_add(fbc, limit, amount,
78 percpu_counter_batch);
79}
80
72/*81/*
73 * With percpu_counter_add_local() and percpu_counter_sub_local(), counts82 * With percpu_counter_add_local() and percpu_counter_sub_local(), counts
74 * are accumulated in local per cpu counter and not in fbc->count until83 * are accumulated in local per cpu counter and not in fbc->count until
@@ -185,6 +194,27 @@ percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 110 unmodified lines
185 local_irq_restore(flags);194 local_irq_restore(flags);
186}195}
187 196
197static inline bool
198percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount)
199{
200 unsigned long flags;
201 bool good = false;
202 s64 count;
203
204 if (amount == 0)
205 return true;
206
207 local_irq_save(flags);
208 count = fbc->count + amount;
209 if ((amount > 0 && count <= limit) ||
210 (amount < 0 && count >= limit)) {
211 fbc->count = count;
212 good = true;
213 }
214 local_irq_restore(flags);
215 return good;
216}
217
188/* non-SMP percpu_counter_add_local is the same with percpu_counter_add */218/* non-SMP percpu_counter_add_local is the same with percpu_counter_add */
189static inline void219static inline void
190percpu_counter_add_local(struct percpu_counter *fbc, s64 amount)220percpu_counter_add_local(struct percpu_counter *fbc, s64 amount)
include/linux/rmap.h+2−2
@@ -189,7 +189,7 @@ typedef int __bitwise rmap_t; · 188 unmodified lines
189/*189/*
190 * rmap interfaces called when adding or removing pte of page190 * rmap interfaces called when adding or removing pte of page
191 */191 */
192void page_move_anon_rmap(struct page *, struct vm_area_struct *);192void folio_move_anon_rmap(struct folio *, struct vm_area_struct *);
193void page_add_anon_rmap(struct page *, struct vm_area_struct *,193void page_add_anon_rmap(struct page *, struct vm_area_struct *,
194 unsigned long address, rmap_t flags);194 unsigned long address, rmap_t flags);
195void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,195void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
@@ -203,7 +203,7 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, · 7 unmodified lines
203void page_remove_rmap(struct page *, struct vm_area_struct *,203void page_remove_rmap(struct page *, struct vm_area_struct *,
204 bool compound);204 bool compound);
205 205
206void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *,206void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *,
207 unsigned long address, rmap_t flags);207 unsigned long address, rmap_t flags);
208void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *,208void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
209 unsigned long address);209 unsigned long address);
include/linux/sched.h+4−0
@@ -1448,6 +1448,10 @@ struct task_struct { · 1447 unmodified lines
1448 struct mem_cgroup *active_memcg;1448 struct mem_cgroup *active_memcg;
1449#endif1449#endif
1450 1450
1451#ifdef CONFIG_MEMCG_KMEM
1452 struct obj_cgroup *objcg;
1453#endif
1454
1451#ifdef CONFIG_BLK_CGROUP1455#ifdef CONFIG_BLK_CGROUP
1452 struct gendisk *throttle_disk;1456 struct gendisk *throttle_disk;
1453#endif1457#endif
include/linux/sched/coredump.h+16−3
@@ -71,6 +71,7 @@ static inline int get_dumpable(struct mm_struct *mm) · 70 unmodified lines
71#define MMF_UNSTABLE 22 /* mm is unstable for copy_from_user */71#define MMF_UNSTABLE 22 /* mm is unstable for copy_from_user */
72#define MMF_HUGE_ZERO_PAGE 23 /* mm has ever used the global huge zero page */72#define MMF_HUGE_ZERO_PAGE 23 /* mm has ever used the global huge zero page */
73#define MMF_DISABLE_THP 24 /* disable THP for all VMAs */73#define MMF_DISABLE_THP 24 /* disable THP for all VMAs */
74#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)
74#define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */75#define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */
75#define MMF_MULTIPROCESS 26 /* mm is shared between processes */76#define MMF_MULTIPROCESS 26 /* mm is shared between processes */
76/*77/*
@@ -85,10 +86,22 @@ static inline int get_dumpable(struct mm_struct *mm) · 8 unmodified lines
85#define MMF_HAS_MDWE 2886#define MMF_HAS_MDWE 28
86#define MMF_HAS_MDWE_MASK (1 << MMF_HAS_MDWE)87#define MMF_HAS_MDWE_MASK (1 << MMF_HAS_MDWE)
87 88
88#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP)89
90#define MMF_HAS_MDWE_NO_INHERIT 29
91
92#define MMF_VM_MERGE_ANY 30
93#define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY)
89 94
90#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\95#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\
91 MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK)96 MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\
97 MMF_VM_MERGE_ANY_MASK)
98
99static inline unsigned long mmf_init_flags(unsigned long flags)
100{
101 if (flags & (1UL << MMF_HAS_MDWE_NO_INHERIT))
102 flags &= ~((1UL << MMF_HAS_MDWE) |
103 (1UL << MMF_HAS_MDWE_NO_INHERIT));
104 return flags & MMF_INIT_MASK;
105}
92 106
93#define MMF_VM_MERGE_ANY 29
94#endif /* _LINUX_SCHED_COREDUMP_H */107#endif /* _LINUX_SCHED_COREDUMP_H */
include/linux/sched/mm.h+4−0
@@ -403,6 +403,10 @@ DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); · 402 unmodified lines
403 * __GFP_ACCOUNT allocations till the end of the scope will be charged to the403 * __GFP_ACCOUNT allocations till the end of the scope will be charged to the
404 * given memcg.404 * given memcg.
405 *405 *
406 * Please, make sure that caller has a reference to the passed memcg structure,
407 * so its lifetime is guaranteed to exceed the scope between two
408 * set_active_memcg() calls.
409 *
406 * NOTE: This function can nest. Users must save the return value and410 * NOTE: This function can nest. Users must save the return value and
407 * reset the previous value after their own charging scope is over.411 * reset the previous value after their own charging scope is over.
408 */412 */
include/linux/sched/numa_balancing.h+3−3
@@ -30,8 +30,8 @@ extern void task_numa_fault(int last_node, int node, int pages, int flags); · 29 unmodified lines
30extern pid_t task_numa_group_id(struct task_struct *p);30extern pid_t task_numa_group_id(struct task_struct *p);
31extern void set_numabalancing_state(bool enabled);31extern void set_numabalancing_state(bool enabled);
32extern void task_numa_free(struct task_struct *p, bool final);32extern void task_numa_free(struct task_struct *p, bool final);
33extern bool should_numa_migrate_memory(struct task_struct *p, struct page *page,33bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
34 int src_nid, int dst_cpu);34 int src_nid, int dst_cpu);
35#else35#else
36static inline void task_numa_fault(int last_node, int node, int pages,36static inline void task_numa_fault(int last_node, int node, int pages,
37 int flags)37 int flags)
@@ -48,7 +48,7 @@ static inline void task_numa_free(struct task_struct *p, bool final) · 10 unmodified lines
48{48{
49}49}
50static inline bool should_numa_migrate_memory(struct task_struct *p,50static inline bool should_numa_migrate_memory(struct task_struct *p,
51 struct page *page, int src_nid, int dst_cpu)51 struct folio *folio, int src_nid, int dst_cpu)
52{52{
53 return true;53 return true;
54}54}
include/linux/shmem_fs.h+10−6
@@ -23,18 +23,22 @@ struct shmem_inode_info { · 22 unmodified lines
23 unsigned long flags;23 unsigned long flags;
24 unsigned long alloced; /* data pages alloced to file */24 unsigned long alloced; /* data pages alloced to file */
25 unsigned long swapped; /* subtotal assigned to swap */25 unsigned long swapped; /* subtotal assigned to swap */
26 pgoff_t fallocend; /* highest fallocate endindex */26 union {
27 struct list_head shrinklist; /* shrinkable hpage inodes */27 struct offset_ctx dir_offsets; /* stable directory offsets */
28 struct list_head swaplist; /* chain of maybes on swap */28 struct {
29 struct list_head shrinklist; /* shrinkable hpage inodes */
30 struct list_head swaplist; /* chain of maybes on swap */
31 };
32 };
33 struct timespec64 i_crtime; /* file creation time */
29 struct shared_policy policy; /* NUMA memory alloc policy */34 struct shared_policy policy; /* NUMA memory alloc policy */
30 struct simple_xattrs xattrs; /* list of xattrs */35 struct simple_xattrs xattrs; /* list of xattrs */
36 pgoff_t fallocend; /* highest fallocate endindex */
37 unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */
31 atomic_t stop_eviction; /* hold when working on inode */38 atomic_t stop_eviction; /* hold when working on inode */
32 struct timespec64 i_crtime; /* file creation time */
33 unsigned int fsflags; /* flags for FS_IOC_[SG]ETFLAGS */
34#ifdef CONFIG_TMPFS_QUOTA39#ifdef CONFIG_TMPFS_QUOTA
35 struct dquot *i_dquot[MAXQUOTAS];40 struct dquot *i_dquot[MAXQUOTAS];
36#endif41#endif
37 struct offset_ctx dir_offsets; /* stable entry offsets */
38 struct inode vfs_inode;42 struct inode vfs_inode;
39};43};
40 44
include/linux/shrinker.h+55−32
@@ -4,6 +4,25 @@ · 3 unmodified lines
4 4
5#include <linux/atomic.h>5#include <linux/atomic.h>
6#include <linux/types.h>6#include <linux/types.h>
7#include <linux/refcount.h>
8#include <linux/completion.h>
9
10#define SHRINKER_UNIT_BITS BITS_PER_LONG
11
12/*
13 * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
14 * shrinkers, which have elements charged to the memcg.
15 */
16struct shrinker_info_unit {
17 atomic_long_t nr_deferred[SHRINKER_UNIT_BITS];
18 DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
19};
20
21struct shrinker_info {
22 struct rcu_head rcu;
23 int map_nr_max;
24 struct shrinker_info_unit *unit[];
25};
7 26
8/*27/*
9 * This struct is used to pass information from page reclaim to the shrinkers.28 * This struct is used to pass information from page reclaim to the shrinkers.
@@ -70,6 +89,19 @@ struct shrinker { · 60 unmodified lines
70 int seeks; /* seeks to recreate an obj */89 int seeks; /* seeks to recreate an obj */
71 unsigned flags;90 unsigned flags;
72 91
92 /*
93 * The reference count of this shrinker. Registered shrinker have an
94 * initial refcount of 1, then the lookup operations are now allowed
95 * to use it via shrinker_try_get(). Later in the unregistration step,
96 * the initial refcount will be discarded, and will free the shrinker
97 * asynchronously via RCU after its refcount reaches 0.
98 */
99 refcount_t refcount;
100 struct completion done; /* use to wait for refcount to reach 0 */
101 struct rcu_head rcu;
102
103 void *private_data;
104
73 /* These are for internal use */105 /* These are for internal use */
74 struct list_head list;106 struct list_head list;
75#ifdef CONFIG_MEMCG107#ifdef CONFIG_MEMCG
@@ -86,48 +118,39 @@ struct shrinker { · 10 unmodified lines
86};118};
87#define DEFAULT_SEEKS 2 /* A good number if you don't know better. */119#define DEFAULT_SEEKS 2 /* A good number if you don't know better. */
88 120
89/* Flags */121/* Internal flags */
90#define SHRINKER_REGISTERED (1 << 0)122#define SHRINKER_REGISTERED BIT(0)
91#define SHRINKER_NUMA_AWARE (1 << 1)123#define SHRINKER_ALLOCATED BIT(1)
92#define SHRINKER_MEMCG_AWARE (1 << 2)124
125/* Flags for users to use */
126#define SHRINKER_NUMA_AWARE BIT(2)
127#define SHRINKER_MEMCG_AWARE BIT(3)
93/*128/*
94 * It just makes sense when the shrinker is also MEMCG_AWARE for now,129 * It just makes sense when the shrinker is also MEMCG_AWARE for now,
95 * non-MEMCG_AWARE shrinker should not have this flag set.130 * non-MEMCG_AWARE shrinker should not have this flag set.
96 */131 */
97#define SHRINKER_NONSLAB (1 << 3)132#define SHRINKER_NONSLAB BIT(4)
98 133
99extern int __printf(2, 3) prealloc_shrinker(struct shrinker *shrinker,134__printf(2, 3)
100 const char *fmt, ...);135struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...);
101extern void register_shrinker_prepared(struct shrinker *shrinker);136void shrinker_register(struct shrinker *shrinker);
102extern int __printf(2, 3) register_shrinker(struct shrinker *shrinker,137void shrinker_free(struct shrinker *shrinker);
103 const char *fmt, ...);
104extern void unregister_shrinker(struct shrinker *shrinker);
105extern void free_prealloced_shrinker(struct shrinker *shrinker);
106extern void synchronize_shrinkers(void);
107 138
108#ifdef CONFIG_SHRINKER_DEBUG139static inline bool shrinker_try_get(struct shrinker *shrinker)
109extern int shrinker_debugfs_add(struct shrinker *shrinker);
110extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
111 int *debugfs_id);
112extern void shrinker_debugfs_remove(struct dentry *debugfs_entry,
113 int debugfs_id);
114extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker,
115 const char *fmt, ...);
116#else /* CONFIG_SHRINKER_DEBUG */
117static inline int shrinker_debugfs_add(struct shrinker *shrinker)
118{
119 return 0;
120}
121static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
122 int *debugfs_id)
123{140{
124 *debugfs_id = -1;141 return refcount_inc_not_zero(&shrinker->refcount);
125 return NULL;
126}142}
127static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,143
128 int debugfs_id)144static inline void shrinker_put(struct shrinker *shrinker)
129{145{
146 if (refcount_dec_and_test(&shrinker->refcount))
147 complete(&shrinker->done);
130}148}
149
150#ifdef CONFIG_SHRINKER_DEBUG
151extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker,
152 const char *fmt, ...);
153#else /* CONFIG_SHRINKER_DEBUG */
131static inline __printf(2, 3)154static inline __printf(2, 3)
132int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...)155int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...)
133{156{
include/linux/userfaultfd_k.h+27−1
@@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 160 unmodified lines
161}161}
162 162
163static inline bool vma_can_userfault(struct vm_area_struct *vma,163static inline bool vma_can_userfault(struct vm_area_struct *vma,
164 unsigned long vm_flags)164 unsigned long vm_flags,
165 bool wp_async)
165{166{
167 vm_flags &= __VM_UFFD_FLAGS;
168
166 if ((vm_flags & VM_UFFD_MINOR) &&169 if ((vm_flags & VM_UFFD_MINOR) &&
167 (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))170 (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
168 return false;171 return false;
172
173 /*
174 * If wp async enabled, and WP is the only mode enabled, allow any
175 * memory type.
176 */
177 if (wp_async && (vm_flags == VM_UFFD_WP))
178 return true;
179
169#ifndef CONFIG_PTE_MARKER_UFFD_WP180#ifndef CONFIG_PTE_MARKER_UFFD_WP
170 /*181 /*
171 * If user requested uffd-wp but not enabled pte markers for182 * If user requested uffd-wp but not enabled pte markers for
@@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, · 3 unmodified lines
175 if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))186 if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma))
176 return false;187 return false;
177#endif188#endif
189
190 /* By default, allow any of anon|shmem|hugetlb */
178 return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||191 return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) ||
179 vma_is_shmem(vma);192 vma_is_shmem(vma);
180}193}
@@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, · 16 unmodified lines
197extern void userfaultfd_unmap_complete(struct mm_struct *mm,210extern void userfaultfd_unmap_complete(struct mm_struct *mm,
198 struct list_head *uf);211 struct list_head *uf);
199extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);212extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
213extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
200 214
201#else /* CONFIG_USERFAULTFD */215#else /* CONFIG_USERFAULTFD */
202 216
@@ -207,6 +221,13 @@ static inline vm_fault_t handle_userfault(struct vm_fault *vmf, · 4 unmodified lines
207 return VM_FAULT_SIGBUS;221 return VM_FAULT_SIGBUS;
208}222}
209 223
224static inline long uffd_wp_range(struct vm_area_struct *vma,
225 unsigned long start, unsigned long len,
226 bool enable_wp)
227{
228 return false;
229}
230
210static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,231static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
211 struct vm_userfaultfd_ctx vm_ctx)232 struct vm_userfaultfd_ctx vm_ctx)
212{233{
@@ -297,6 +318,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) · 84 unmodified lines
297 return false;318 return false;
298}319}
299 320
321static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
322{
323 return false;
324}
325
300#endif /* CONFIG_USERFAULTFD */326#endif /* CONFIG_USERFAULTFD */
301 327
302static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)328static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
include/linux/wait.h+3−6
@@ -19,10 +19,9 @@ int default_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int · 18 unmodified lines
19/* wait_queue_entry::flags */19/* wait_queue_entry::flags */
20#define WQ_FLAG_EXCLUSIVE 0x0120#define WQ_FLAG_EXCLUSIVE 0x01
21#define WQ_FLAG_WOKEN 0x0221#define WQ_FLAG_WOKEN 0x02
22#define WQ_FLAG_BOOKMARK 0x0422#define WQ_FLAG_CUSTOM 0x04
23#define WQ_FLAG_CUSTOM 0x0823#define WQ_FLAG_DONE 0x08
24#define WQ_FLAG_DONE 0x1024#define WQ_FLAG_PRIORITY 0x10
25#define WQ_FLAG_PRIORITY 0x20
26 25
27/*26/*
28 * A single wait-queue entry structure:27 * A single wait-queue entry structure:
@@ -212,8 +211,6 @@ __remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq · 183 unmodified lines
212int __wake_up(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key);211int __wake_up(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key);
213void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode, void *key);212void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode, void *key);
214void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);213void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
215void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head,
216 unsigned int mode, void *key, wait_queue_entry_t *bookmark);
217void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);214void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
218void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);215void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key);
219void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr);216void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr);
include/trace/events/damon.h+42−3
@@ -9,12 +9,51 @@ · 8 unmodified lines
9#include <linux/types.h>9#include <linux/types.h>
10#include <linux/tracepoint.h>10#include <linux/tracepoint.h>
11 11
12TRACE_EVENT_CONDITION(damos_before_apply,
13
14 TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,
15 unsigned int target_idx, struct damon_region *r,
16 unsigned int nr_regions, bool do_trace),
17
18 TP_ARGS(context_idx, target_idx, scheme_idx, r, nr_regions, do_trace),
19
20 TP_CONDITION(do_trace),
21
22 TP_STRUCT__entry(
23 __field(unsigned int, context_idx)
24 __field(unsigned int, scheme_idx)
25 __field(unsigned long, target_idx)
26 __field(unsigned long, start)
27 __field(unsigned long, end)
28 __field(unsigned int, nr_accesses)
29 __field(unsigned int, age)
30 __field(unsigned int, nr_regions)
31 ),
32
33 TP_fast_assign(
34 __entry->context_idx = context_idx;
35 __entry->scheme_idx = scheme_idx;
36 __entry->target_idx = target_idx;
37 __entry->start = r->ar.start;
38 __entry->end = r->ar.end;
39 __entry->nr_accesses = r->nr_accesses_bp / 10000;
40 __entry->age = r->age;
41 __entry->nr_regions = nr_regions;
42 ),
43
44 TP_printk("ctx_idx=%u scheme_idx=%u target_idx=%lu nr_regions=%u %lu-%lu: %u %u",
45 __entry->context_idx, __entry->scheme_idx,
46 __entry->target_idx, __entry->nr_regions,
47 __entry->start, __entry->end,
48 __entry->nr_accesses, __entry->age)
49);
50
12TRACE_EVENT(damon_aggregated,51TRACE_EVENT(damon_aggregated,
13 52
14 TP_PROTO(struct damon_target *t, unsigned int target_id,53 TP_PROTO(unsigned int target_id, struct damon_region *r,
15 struct damon_region *r, unsigned int nr_regions),54 unsigned int nr_regions),
16 55
17 TP_ARGS(t, target_id, r, nr_regions),56 TP_ARGS(target_id, r, nr_regions),
18 57
19 TP_STRUCT__entry(58 TP_STRUCT__entry(
20 __field(unsigned long, target_id)59 __field(unsigned long, target_id)
include/trace/events/migrate.h+14−10
@@ -49,10 +49,11 @@ TRACE_EVENT(mm_migrate_pages, · 48 unmodified lines
49 49
50 TP_PROTO(unsigned long succeeded, unsigned long failed,50 TP_PROTO(unsigned long succeeded, unsigned long failed,
51 unsigned long thp_succeeded, unsigned long thp_failed,51 unsigned long thp_succeeded, unsigned long thp_failed,
52 unsigned long thp_split, enum migrate_mode mode, int reason),52 unsigned long thp_split, unsigned long large_folio_split,
53 enum migrate_mode mode, int reason),
53 54
54 TP_ARGS(succeeded, failed, thp_succeeded, thp_failed,55 TP_ARGS(succeeded, failed, thp_succeeded, thp_failed,
55 thp_split, mode, reason),56 thp_split, large_folio_split, mode, reason),
56 57
57 TP_STRUCT__entry(58 TP_STRUCT__entry(
58 __field( unsigned long, succeeded)59 __field( unsigned long, succeeded)
@@ -60,26 +61,29 @@ TRACE_EVENT(mm_migrate_pages, · 1 unmodified line
60 __field( unsigned long, thp_succeeded)61 __field( unsigned long, thp_succeeded)
61 __field( unsigned long, thp_failed)62 __field( unsigned long, thp_failed)
62 __field( unsigned long, thp_split)63 __field( unsigned long, thp_split)
64 __field( unsigned long, large_folio_split)
63 __field( enum migrate_mode, mode)65 __field( enum migrate_mode, mode)
64 __field( int, reason)66 __field( int, reason)
65 ),67 ),
66 68
67 TP_fast_assign(69 TP_fast_assign(
68 __entry->succeeded = succeeded;70 __entry->succeeded = succeeded;
69 __entry->failed = failed;71 __entry->failed = failed;
70 __entry->thp_succeeded = thp_succeeded;72 __entry->thp_succeeded = thp_succeeded;
71 __entry->thp_failed = thp_failed;73 __entry->thp_failed = thp_failed;
72 __entry->thp_split = thp_split;74 __entry->thp_split = thp_split;
73 __entry->mode = mode;75 __entry->large_folio_split = large_folio_split;
74 __entry->reason = reason;76 __entry->mode = mode;
77 __entry->reason = reason;
75 ),78 ),
76 79
77 TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu mode=%s reason=%s",80 TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu nr_split=%lu mode=%s reason=%s",
78 __entry->succeeded,81 __entry->succeeded,
79 __entry->failed,82 __entry->failed,
80 __entry->thp_succeeded,83 __entry->thp_succeeded,
81 __entry->thp_failed,84 __entry->thp_failed,
82 __entry->thp_split,85 __entry->thp_split,
86 __entry->large_folio_split,
83 __print_symbolic(__entry->mode, MIGRATE_MODE),87 __print_symbolic(__entry->mode, MIGRATE_MODE),
84 __print_symbolic(__entry->reason, MIGRATE_REASON))88 __print_symbolic(__entry->reason, MIGRATE_REASON))
85);89);
include/trace/events/vmscan.h+2−6
@@ -285,10 +285,9 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 284 unmodified lines
285 unsigned long nr_scanned,285 unsigned long nr_scanned,
286 unsigned long nr_skipped,286 unsigned long nr_skipped,
287 unsigned long nr_taken,287 unsigned long nr_taken,
288 isolate_mode_t isolate_mode,
289 int lru),288 int lru),
290 289
291 TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, isolate_mode, lru),290 TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, lru),
292 291
293 TP_STRUCT__entry(292 TP_STRUCT__entry(
294 __field(int, highest_zoneidx)293 __field(int, highest_zoneidx)
@@ -297,7 +296,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 2 unmodified lines
297 __field(unsigned long, nr_scanned)296 __field(unsigned long, nr_scanned)
298 __field(unsigned long, nr_skipped)297 __field(unsigned long, nr_skipped)
299 __field(unsigned long, nr_taken)298 __field(unsigned long, nr_taken)
300 __field(unsigned int, isolate_mode)
301 __field(int, lru)299 __field(int, lru)
302 ),300 ),
303 301
@@ -308,7 +306,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 4 unmodified lines
308 __entry->nr_scanned = nr_scanned;306 __entry->nr_scanned = nr_scanned;
309 __entry->nr_skipped = nr_skipped;307 __entry->nr_skipped = nr_skipped;
310 __entry->nr_taken = nr_taken;308 __entry->nr_taken = nr_taken;
311 __entry->isolate_mode = (__force unsigned int)isolate_mode;
312 __entry->lru = lru;309 __entry->lru = lru;
313 ),310 ),
314 311
@@ -316,8 +313,7 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 1 unmodified line
316 * classzone is previous name of the highest_zoneidx.313 * classzone is previous name of the highest_zoneidx.
317 * Reason not to change it is the ABI requirement of the tracepoint.314 * Reason not to change it is the ABI requirement of the tracepoint.
318 */315 */
319 TP_printk("isolate_mode=%d classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s",316 TP_printk("classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s",
320 __entry->isolate_mode,
321 __entry->highest_zoneidx,317 __entry->highest_zoneidx,
322 __entry->order,318 __entry->order,
323 __entry->nr_requested,319 __entry->nr_requested,
include/uapi/linux/fs.h+59−0
@@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; · 304 unmodified lines
305#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\305#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
306 RWF_APPEND)306 RWF_APPEND)
307 307
308/* Pagemap ioctl */
309#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)
310
311/* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */
312#define PAGE_IS_WPALLOWED (1 << 0)
313#define PAGE_IS_WRITTEN (1 << 1)
314#define PAGE_IS_FILE (1 << 2)
315#define PAGE_IS_PRESENT (1 << 3)
316#define PAGE_IS_SWAPPED (1 << 4)
317#define PAGE_IS_PFNZERO (1 << 5)
318#define PAGE_IS_HUGE (1 << 6)
319
320/*
321 * struct page_region - Page region with flags
322 * @start: Start of the region
323 * @end: End of the region (exclusive)
324 * @categories: PAGE_IS_* category bitmask for the region
325 */
326struct page_region {
327 __u64 start;
328 __u64 end;
329 __u64 categories;
330};
331
332/* Flags for PAGEMAP_SCAN ioctl */
333#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */
334#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */
335
336/*
337 * struct pm_scan_arg - Pagemap ioctl argument
338 * @size: Size of the structure
339 * @flags: Flags for the IOCTL
340 * @start: Starting address of the region
341 * @end: Ending address of the region
342 * @walk_end Address where the scan stopped (written by kernel).
343 * walk_end == end (address tags cleared) informs that the scan completed on entire range.
344 * @vec: Address of page_region struct array for output
345 * @vec_len: Length of the page_region struct array
346 * @max_pages: Optional limit for number of returned pages (0 = disabled)
347 * @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1
348 * @category_mask: Skip pages for which any category doesn't match
349 * @category_anyof_mask: Skip pages for which no category matches
350 * @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned
351 */
352struct pm_scan_arg {
353 __u64 size;
354 __u64 flags;
355 __u64 start;
356 __u64 end;
357 __u64 walk_end;
358 __u64 vec;
359 __u64 vec_len;
360 __u64 max_pages;
361 __u64 category_inverted;
362 __u64 category_mask;
363 __u64 category_anyof_mask;
364 __u64 return_mask;
365};
366
308#endif /* _UAPI_LINUX_FS_H */367#endif /* _UAPI_LINUX_FS_H */
include/uapi/linux/mempolicy.h+1−1
@@ -48,7 +48,7 @@ enum { · 47 unmodified lines
48#define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform48#define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform
49 to policy */49 to policy */
50#define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */50#define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */
51#define MPOL_MF_LAZY (1<<3) /* Modifies '_MOVE: lazy migrate on fault */51#define MPOL_MF_LAZY (1<<3) /* UNSUPPORTED FLAG: Lazy migrate on fault */
52#define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */52#define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */
53 53
54#define MPOL_MF_VALID (MPOL_MF_STRICT | \54#define MPOL_MF_VALID (MPOL_MF_STRICT | \
include/uapi/linux/prctl.h+2−1
@@ -283,7 +283,8 @@ struct prctl_mm_map { · 282 unmodified lines
283 283
284/* Memory deny write / execute */284/* Memory deny write / execute */
285#define PR_SET_MDWE 65285#define PR_SET_MDWE 65
286# define PR_MDWE_REFUSE_EXEC_GAIN 1286# define PR_MDWE_REFUSE_EXEC_GAIN (1UL << 0)
287# define PR_MDWE_NO_INHERIT (1UL << 1)
287 288
288#define PR_GET_MDWE 66289#define PR_GET_MDWE 66
289 290
include/uapi/linux/userfaultfd.h+8−1
@@ -40,7 +40,8 @@ · 39 unmodified lines
40 UFFD_FEATURE_EXACT_ADDRESS | \40 UFFD_FEATURE_EXACT_ADDRESS | \
41 UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \41 UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
42 UFFD_FEATURE_WP_UNPOPULATED | \42 UFFD_FEATURE_WP_UNPOPULATED | \
43 UFFD_FEATURE_POISON)43 UFFD_FEATURE_POISON | \
44 UFFD_FEATURE_WP_ASYNC)
44#define UFFD_API_IOCTLS \45#define UFFD_API_IOCTLS \
45 ((__u64)1 << _UFFDIO_REGISTER | \46 ((__u64)1 << _UFFDIO_REGISTER | \
46 (__u64)1 << _UFFDIO_UNREGISTER | \47 (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -216,6 +217,11 @@ struct uffdio_api { · 169 unmodified lines
216 * (i.e. empty ptes). This will be the default behavior for shmem217 * (i.e. empty ptes). This will be the default behavior for shmem
217 * & hugetlbfs, so this flag only affects anonymous memory behavior218 * & hugetlbfs, so this flag only affects anonymous memory behavior
218 * when userfault write-protection mode is registered.219 * when userfault write-protection mode is registered.
220 *
221 * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection
222 * asynchronous mode is supported in which the write fault is
223 * automatically resolved and write-protection is un-set.
224 * It implies UFFD_FEATURE_WP_UNPOPULATED.
219 */225 */
220#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)226#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
221#define UFFD_FEATURE_EVENT_FORK (1<<1)227#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -232,6 +238,7 @@ struct uffdio_api { · 10 unmodified lines
232#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)238#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
233#define UFFD_FEATURE_WP_UNPOPULATED (1<<13)239#define UFFD_FEATURE_WP_UNPOPULATED (1<<13)
234#define UFFD_FEATURE_POISON (1<<14)240#define UFFD_FEATURE_POISON (1<<14)
241#define UFFD_FEATURE_WP_ASYNC (1<<15)
235 __u64 features;242 __u64 features;
236 243
237 __u64 ioctls;244 __u64 ioctls;
ipc/shm.c+8−13
@@ -562,30 +562,25 @@ static unsigned long shm_pagesize(struct vm_area_struct *vma) · 561 unmodified lines
562}562}
563 563
564#ifdef CONFIG_NUMA564#ifdef CONFIG_NUMA
565static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *new)565static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol)
566{566{
567 struct file *file = vma->vm_file;567 struct shm_file_data *sfd = shm_file_data(vma->vm_file);
568 struct shm_file_data *sfd = shm_file_data(file);
569 int err = 0;568 int err = 0;
570 569
571 if (sfd->vm_ops->set_policy)570 if (sfd->vm_ops->set_policy)
572 err = sfd->vm_ops->set_policy(vma, new);571 err = sfd->vm_ops->set_policy(vma, mpol);
573 return err;572 return err;
574}573}
575 574
576static struct mempolicy *shm_get_policy(struct vm_area_struct *vma,575static struct mempolicy *shm_get_policy(struct vm_area_struct *vma,
577 unsigned long addr)576 unsigned long addr, pgoff_t *ilx)
578{577{
579 struct file *file = vma->vm_file;578 struct shm_file_data *sfd = shm_file_data(vma->vm_file);
580 struct shm_file_data *sfd = shm_file_data(file);579 struct mempolicy *mpol = vma->vm_policy;
581 struct mempolicy *pol = NULL;
582 580
583 if (sfd->vm_ops->get_policy)581 if (sfd->vm_ops->get_policy)
584 pol = sfd->vm_ops->get_policy(vma, addr);582 mpol = sfd->vm_ops->get_policy(vma, addr, ilx);
585 else if (vma->vm_policy)583 return mpol;
586 pol = vma->vm_policy;
587
588 return pol;
589}584}
590#endif585#endif
591 586
kernel/cgroup/cgroup.c+14−1
@@ -1908,6 +1908,7 @@ enum cgroup2_param { · 1907 unmodified lines
1908 Opt_favordynmods,1908 Opt_favordynmods,
1909 Opt_memory_localevents,1909 Opt_memory_localevents,
1910 Opt_memory_recursiveprot,1910 Opt_memory_recursiveprot,
1911 Opt_memory_hugetlb_accounting,
1911 nr__cgroup2_params1912 nr__cgroup2_params
1912};1913};
1913 1914
@@ -1916,6 +1917,7 @@ static const struct fs_parameter_spec cgroup2_fs_parameters[] = { · 2 unmodified lines
1916 fsparam_flag("favordynmods", Opt_favordynmods),1917 fsparam_flag("favordynmods", Opt_favordynmods),
1917 fsparam_flag("memory_localevents", Opt_memory_localevents),1918 fsparam_flag("memory_localevents", Opt_memory_localevents),
1918 fsparam_flag("memory_recursiveprot", Opt_memory_recursiveprot),1919 fsparam_flag("memory_recursiveprot", Opt_memory_recursiveprot),
1920 fsparam_flag("memory_hugetlb_accounting", Opt_memory_hugetlb_accounting),
1919 {}1921 {}
1920};1922};
1921 1923
@@ -1942,6 +1944,9 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *param · 20 unmodified lines
1942 case Opt_memory_recursiveprot:1944 case Opt_memory_recursiveprot:
1943 ctx->flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;1945 ctx->flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
1944 return 0;1946 return 0;
1947 case Opt_memory_hugetlb_accounting:
1948 ctx->flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
1949 return 0;
1945 }1950 }
1946 return -EINVAL;1951 return -EINVAL;
1947}1952}
@@ -1966,6 +1971,11 @@ static void apply_cgroup_root_flags(unsigned int root_flags) · 18 unmodified lines
1966 cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;1971 cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
1967 else1972 else
1968 cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_RECURSIVE_PROT;1973 cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_RECURSIVE_PROT;
1974
1975 if (root_flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING)
1976 cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
1977 else
1978 cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
1969 }1979 }
1970}1980}
1971 1981
@@ -1979,6 +1989,8 @@ static int cgroup_show_options(struct seq_file *seq, struct kernfs_root *kf_root · 7 unmodified lines
1979 seq_puts(seq, ",memory_localevents");1989 seq_puts(seq, ",memory_localevents");
1980 if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT)1990 if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT)
1981 seq_puts(seq, ",memory_recursiveprot");1991 seq_puts(seq, ",memory_recursiveprot");
1992 if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING)
1993 seq_puts(seq, ",memory_hugetlb_accounting");
1982 return 0;1994 return 0;
1983}1995}
1984 1996
@@ -7068,7 +7080,8 @@ static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, · 5083 unmodified lines
7068 "nsdelegate\n"7080 "nsdelegate\n"
7069 "favordynmods\n"7081 "favordynmods\n"
7070 "memory_localevents\n"7082 "memory_localevents\n"
7071 "memory_recursiveprot\n");7083 "memory_recursiveprot\n"
7084 "memory_hugetlb_accounting\n");
7072}7085}
7073static struct kobj_attribute cgroup_features_attr = __ATTR_RO(features);7086static struct kobj_attribute cgroup_features_attr = __ATTR_RO(features);
7074 7087
kernel/events/uprobes.c+2−2
@@ -474,8 +474,8 @@ retry: · 473 unmodified lines
474 gup_flags |= FOLL_SPLIT_PMD;474 gup_flags |= FOLL_SPLIT_PMD;
475 /* Read the page with vaddr into memory */475 /* Read the page with vaddr into memory */
476 old_page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);476 old_page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);
477 if (IS_ERR_OR_NULL(old_page))477 if (IS_ERR(old_page))
478 return old_page ? PTR_ERR(old_page) : 0;478 return PTR_ERR(old_page);
479 479
480 ret = verify_opcode(old_page, vaddr, &opcode);480 ret = verify_opcode(old_page, vaddr, &opcode);
481 if (ret <= 0)481 if (ret <= 0)
kernel/exit.c+0−4
@@ -541,7 +541,6 @@ static void exit_mm(void) · 540 unmodified lines
541 exit_mm_release(current, mm);541 exit_mm_release(current, mm);
542 if (!mm)542 if (!mm)
543 return;543 return;
544 sync_mm_rss(mm);
545 mmap_read_lock(mm);544 mmap_read_lock(mm);
546 mmgrab_lazy_tlb(mm);545 mmgrab_lazy_tlb(mm);
547 BUG_ON(mm != current->active_mm);546 BUG_ON(mm != current->active_mm);
@@ -831,9 +830,6 @@ void __noreturn do_exit(long code) · 283 unmodified lines
831 io_uring_files_cancel();830 io_uring_files_cancel();
832 exit_signals(tsk); /* sets PF_EXITING */831 exit_signals(tsk); /* sets PF_EXITING */
833 832
834 /* sync mm's RSS info before statistics gathering */
835 if (tsk->mm)
836 sync_mm_rss(tsk->mm);
837 acct_update_integrals(tsk);833 acct_update_integrals(tsk);
838 group_dead = atomic_dec_and_test(&tsk->signal->live);834 group_dead = atomic_dec_and_test(&tsk->signal->live);
839 if (group_dead) {835 if (group_dead) {
kernel/fork.c+2−6
@@ -733,7 +733,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 732 unmodified lines
733 733
734 get_file(file);734 get_file(file);
735 i_mmap_lock_write(mapping);735 i_mmap_lock_write(mapping);
736 if (tmp->vm_flags & VM_SHARED)736 if (vma_is_shared_maywrite(tmp))
737 mapping_allow_writable(mapping);737 mapping_allow_writable(mapping);
738 flush_dcache_mmap_lock(mapping);738 flush_dcache_mmap_lock(mapping);
739 /* insert tmp into the share list, just after mpnt */739 /* insert tmp into the share list, just after mpnt */
@@ -1288,7 +1288,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, · 548 unmodified lines
1288 hugetlb_count_init(mm);1288 hugetlb_count_init(mm);
1289 1289
1290 if (current->mm) {1290 if (current->mm) {
1291 mm->flags = current->mm->flags & MMF_INIT_MASK;1291 mm->flags = mmf_init_flags(current->mm->flags);
1292 mm->def_flags = current->mm->def_flags & VM_INIT_DEF_MASK;1292 mm->def_flags = current->mm->def_flags & VM_INIT_DEF_MASK;
1293 } else {1293 } else {
1294 mm->flags = default_dump_filter;1294 mm->flags = default_dump_filter;
@@ -2411,10 +2411,6 @@ __latent_entropy struct task_struct *copy_process( · 1116 unmodified lines
2411 p->io_uring = NULL;2411 p->io_uring = NULL;
2412#endif2412#endif
2413 2413
2414#if defined(SPLIT_RSS_COUNTING)
2415 memset(&p->rss_stat, 0, sizeof(p->rss_stat));
2416#endif
2417
2418 p->default_timer_slack_ns = current->timer_slack_ns;2414 p->default_timer_slack_ns = current->timer_slack_ns;
2419 2415
2420#ifdef CONFIG_PSI2416#ifdef CONFIG_PSI
kernel/kcsan/kcsan_test.c+3−6
@@ -699,12 +699,9 @@ static void test_barrier_nothreads(struct kunit *test) · 698 unmodified lines
699 KCSAN_EXPECT_RW_BARRIER(spin_unlock(&test_spinlock), true);699 KCSAN_EXPECT_RW_BARRIER(spin_unlock(&test_spinlock), true);
700 KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false);700 KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false);
701 KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true);701 KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true);
702 702 KCSAN_EXPECT_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
703#ifdef clear_bit_unlock_is_negative_byte703 KCSAN_EXPECT_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
704 KCSAN_EXPECT_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);704 KCSAN_EXPECT_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true);
705 KCSAN_EXPECT_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);
706 KCSAN_EXPECT_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true);
707#endif
708 kcsan_nestable_atomic_end();705 kcsan_nestable_atomic_end();
709}706}
710 707
kernel/kcsan/selftest.c+3−6
@@ -227,12 +227,9 @@ static bool __init test_barrier(void) · 226 unmodified lines
227 KCSAN_CHECK_RW_BARRIER(arch_spin_unlock(&arch_spinlock));227 KCSAN_CHECK_RW_BARRIER(arch_spin_unlock(&arch_spinlock));
228 spin_lock(&test_spinlock);228 spin_lock(&test_spinlock);
229 KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock));229 KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock));
230 230 KCSAN_CHECK_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
231#ifdef clear_bit_unlock_is_negative_byte231 KCSAN_CHECK_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
232 KCSAN_CHECK_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));232 KCSAN_CHECK_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var));
233 KCSAN_CHECK_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));
234 KCSAN_CHECK_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var));
235#endif
236 kcsan_nestable_atomic_end();233 kcsan_nestable_atomic_end();
237 234
238 return ret;235 return ret;
kernel/kthread.c+0−1
@@ -1469,7 +1469,6 @@ void kthread_unuse_mm(struct mm_struct *mm) · 1468 unmodified lines
1469 * clearing tsk->mm.1469 * clearing tsk->mm.
1470 */1470 */
1471 smp_mb__after_spinlock();1471 smp_mb__after_spinlock();
1472 sync_mm_rss(mm);
1473 local_irq_disable();1472 local_irq_disable();
1474 tsk->mm = NULL;1473 tsk->mm = NULL;
1475 membarrier_update_current_mm(NULL);1474 membarrier_update_current_mm(NULL);
kernel/ptrace.c+1−1
@@ -59,7 +59,7 @@ int ptrace_access_vm(struct task_struct *tsk, unsigned long addr, · 58 unmodified lines
59 return 0;59 return 0;
60 }60 }
61 61
62 ret = __access_remote_vm(mm, addr, buf, len, gup_flags);62 ret = access_remote_vm(mm, addr, buf, len, gup_flags);
63 mmput(mm);63 mmput(mm);
64 64
65 return ret;65 return ret;
kernel/rcu/tree.c+12−9
@@ -3471,13 +3471,6 @@ kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 3470 unmodified lines
3471 return freed == 0 ? SHRINK_STOP : freed;3471 return freed == 0 ? SHRINK_STOP : freed;
3472}3472}
3473 3473
3474static struct shrinker kfree_rcu_shrinker = {
3475 .count_objects = kfree_rcu_shrink_count,
3476 .scan_objects = kfree_rcu_shrink_scan,
3477 .batch = 0,
3478 .seeks = DEFAULT_SEEKS,
3479};
3480
3481void __init kfree_rcu_scheduler_running(void)3474void __init kfree_rcu_scheduler_running(void)
3482{3475{
3483 int cpu;3476 int cpu;
@@ -5029,6 +5022,7 @@ static void __init kfree_rcu_batch_init(void) · 1545 unmodified lines
5029{5022{
5030 int cpu;5023 int cpu;
5031 int i, j;5024 int i, j;
5025 struct shrinker *kfree_rcu_shrinker;
5032 5026
5033 /* Clamp it to [0:100] seconds interval. */5027 /* Clamp it to [0:100] seconds interval. */
5034 if (rcu_delay_page_cache_fill_msec < 0 ||5028 if (rcu_delay_page_cache_fill_msec < 0 ||
@@ -5060,8 +5054,17 @@ static void __init kfree_rcu_batch_init(void) · 25 unmodified lines
5060 INIT_DELAYED_WORK(&krcp->page_cache_work, fill_page_cache_func);5054 INIT_DELAYED_WORK(&krcp->page_cache_work, fill_page_cache_func);
5061 krcp->initialized = true;5055 krcp->initialized = true;
5062 }5056 }
5063 if (register_shrinker(&kfree_rcu_shrinker, "rcu-kfree"))5057
5064 pr_err("Failed to register kfree_rcu() shrinker!\n");5058 kfree_rcu_shrinker = shrinker_alloc(0, "rcu-kfree");
5059 if (!kfree_rcu_shrinker) {
5060 pr_err("Failed to allocate kfree_rcu() shrinker!\n");
5061 return;
5062 }
5063
5064 kfree_rcu_shrinker->count_objects = kfree_rcu_shrink_count;
5065 kfree_rcu_shrinker->scan_objects = kfree_rcu_shrink_scan;
5066
5067 shrinker_register(kfree_rcu_shrinker);
5065}5068}
5066 5069
5067void __init rcu_init(void)5070void __init rcu_init(void)
kernel/rcu/tree_nocb.h+10−9
@@ -1396,13 +1396,6 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1395 unmodified lines
1396 1396
1397 return count ? count : SHRINK_STOP;1397 return count ? count : SHRINK_STOP;
1398}1398}
1399
1400static struct shrinker lazy_rcu_shrinker = {
1401 .count_objects = lazy_rcu_shrink_count,
1402 .scan_objects = lazy_rcu_shrink_scan,
1403 .batch = 0,
1404 .seeks = DEFAULT_SEEKS,
1405};
1406#endif // #ifdef CONFIG_RCU_LAZY1399#endif // #ifdef CONFIG_RCU_LAZY
1407 1400
1408void __init rcu_init_nohz(void)1401void __init rcu_init_nohz(void)
@@ -1410,6 +1403,7 @@ void __init rcu_init_nohz(void) · 1 unmodified line
1410 int cpu;1403 int cpu;
1411 struct rcu_data *rdp;1404 struct rcu_data *rdp;
1412 const struct cpumask *cpumask = NULL;1405 const struct cpumask *cpumask = NULL;
1406 struct shrinker * __maybe_unused lazy_rcu_shrinker;
1413 1407
1414#if defined(CONFIG_NO_HZ_FULL)1408#if defined(CONFIG_NO_HZ_FULL)
1415 if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))1409 if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))
@@ -1436,8 +1430,15 @@ void __init rcu_init_nohz(void) · 20 unmodified lines
1436 return;1430 return;
1437 1431
1438#ifdef CONFIG_RCU_LAZY1432#ifdef CONFIG_RCU_LAZY
1439 if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy"))1433 lazy_rcu_shrinker = shrinker_alloc(0, "rcu-lazy");
1440 pr_err("Failed to register lazy_rcu shrinker!\n");1434 if (!lazy_rcu_shrinker) {
1435 pr_err("Failed to allocate lazy_rcu shrinker!\n");
1436 } else {
1437 lazy_rcu_shrinker->count_objects = lazy_rcu_shrink_count;
1438 lazy_rcu_shrinker->scan_objects = lazy_rcu_shrink_scan;
1439
1440 shrinker_register(lazy_rcu_shrinker);
1441 }
1441#endif // #ifdef CONFIG_RCU_LAZY1442#endif // #ifdef CONFIG_RCU_LAZY
1442 1443
1443 if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {1444 if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {
kernel/sched/fair.c+6−6
@@ -1759,12 +1759,12 @@ static bool pgdat_free_space_enough(struct pglist_data *pgdat) · 1758 unmodified lines
1759 * The smaller the hint page fault latency, the higher the possibility1759 * The smaller the hint page fault latency, the higher the possibility
1760 * for the page to be hot.1760 * for the page to be hot.
1761 */1761 */
1762static int numa_hint_fault_latency(struct page *page)1762static int numa_hint_fault_latency(struct folio *folio)
1763{1763{
1764 int last_time, time;1764 int last_time, time;
1765 1765
1766 time = jiffies_to_msecs(jiffies);1766 time = jiffies_to_msecs(jiffies);
1767 last_time = xchg_page_access_time(page, time);1767 last_time = folio_xchg_access_time(folio, time);
1768 1768
1769 return (time - last_time) & PAGE_ACCESS_TIME_MASK;1769 return (time - last_time) & PAGE_ACCESS_TIME_MASK;
1770}1770}
@@ -1821,7 +1821,7 @@ static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, · 50 unmodified lines
1821 }1821 }
1822}1822}
1823 1823
1824bool should_numa_migrate_memory(struct task_struct *p, struct page * page,1824bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
1825 int src_nid, int dst_cpu)1825 int src_nid, int dst_cpu)
1826{1826{
1827 struct numa_group *ng = deref_curr_numa_group(p);1827 struct numa_group *ng = deref_curr_numa_group(p);
@@ -1851,16 +1851,16 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, · 23 unmodified lines
1851 numa_promotion_adjust_threshold(pgdat, rate_limit, def_th);1851 numa_promotion_adjust_threshold(pgdat, rate_limit, def_th);
1852 1852
1853 th = pgdat->nbp_threshold ? : def_th;1853 th = pgdat->nbp_threshold ? : def_th;
1854 latency = numa_hint_fault_latency(page);1854 latency = numa_hint_fault_latency(folio);
1855 if (latency >= th)1855 if (latency >= th)
1856 return false;1856 return false;
1857 1857
1858 return !numa_promotion_rate_limit(pgdat, rate_limit,1858 return !numa_promotion_rate_limit(pgdat, rate_limit,
1859 thp_nr_pages(page));1859 folio_nr_pages(folio));
1860 }1860 }
1861 1861
1862 this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);1862 this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
1863 last_cpupid = page_cpupid_xchg_last(page, this_cpupid);1863 last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid);
1864 1864
1865 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&1865 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
1866 !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))1866 !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))
kernel/sched/wait.c+10−50
@@ -57,13 +57,6 @@ void remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry · 56 unmodified lines
57}57}
58EXPORT_SYMBOL(remove_wait_queue);58EXPORT_SYMBOL(remove_wait_queue);
59 59
60/*
61 * Scan threshold to break wait queue walk.
62 * This allows a waker to take a break from holding the
63 * wait queue lock during the wait queue walk.
64 */
65#define WAITQUEUE_WALK_BREAK_CNT 64
66
67/*60/*
68 * The core wakeup function. Non-exclusive wakeups (nr_exclusive == 0) just61 * The core wakeup function. Non-exclusive wakeups (nr_exclusive == 0) just
69 * wake everything up. If it's an exclusive wakeup (nr_exclusive == small +ve62 * wake everything up. If it's an exclusive wakeup (nr_exclusive == small +ve
@@ -78,21 +71,13 @@ EXPORT_SYMBOL(remove_wait_queue); · 8 unmodified lines
78 * zero in this (rare) case, and we handle it by continuing to scan the queue.71 * zero in this (rare) case, and we handle it by continuing to scan the queue.
79 */72 */
80static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode,73static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode,
81 int nr_exclusive, int wake_flags, void *key,74 int nr_exclusive, int wake_flags, void *key)
82 wait_queue_entry_t *bookmark)
83{75{
84 wait_queue_entry_t *curr, *next;76 wait_queue_entry_t *curr, *next;
85 int cnt = 0;
86 77
87 lockdep_assert_held(&wq_head->lock);78 lockdep_assert_held(&wq_head->lock);
88 79
89 if (bookmark && (bookmark->flags & WQ_FLAG_BOOKMARK)) {80 curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry);
90 curr = list_next_entry(bookmark, entry);
91
92 list_del(&bookmark->entry);
93 bookmark->flags = 0;
94 } else
95 curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry);
96 81
97 if (&curr->entry == &wq_head->head)82 if (&curr->entry == &wq_head->head)
98 return nr_exclusive;83 return nr_exclusive;
@@ -101,21 +86,11 @@ static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode, · 2 unmodified lines
101 unsigned flags = curr->flags;86 unsigned flags = curr->flags;
102 int ret;87 int ret;
103 88
104 if (flags & WQ_FLAG_BOOKMARK)
105 continue;
106
107 ret = curr->func(curr, mode, wake_flags, key);89 ret = curr->func(curr, mode, wake_flags, key);
108 if (ret < 0)90 if (ret < 0)
109 break;91 break;
110 if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)92 if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
111 break;93 break;
112
113 if (bookmark && (++cnt > WAITQUEUE_WALK_BREAK_CNT) &&
114 (&next->entry != &wq_head->head)) {
115 bookmark->flags = WQ_FLAG_BOOKMARK;
116 list_add_tail(&bookmark->entry, &next->entry);
117 break;
118 }
119 }94 }
120 95
121 return nr_exclusive;96 return nr_exclusive;
@@ -125,20 +100,12 @@ static int __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int m · 3 unmodified lines
125 int nr_exclusive, int wake_flags, void *key)100 int nr_exclusive, int wake_flags, void *key)
126{101{
127 unsigned long flags;102 unsigned long flags;
128 wait_queue_entry_t bookmark;103 int remaining;
129 int remaining = nr_exclusive;
130 104
131 bookmark.flags = 0;105 spin_lock_irqsave(&wq_head->lock, flags);
132 bookmark.private = NULL;106 remaining = __wake_up_common(wq_head, mode, nr_exclusive, wake_flags,
133 bookmark.func = NULL;107 key);
134 INIT_LIST_HEAD(&bookmark.entry);108 spin_unlock_irqrestore(&wq_head->lock, flags);
135
136 do {
137 spin_lock_irqsave(&wq_head->lock, flags);
138 remaining = __wake_up_common(wq_head, mode, remaining,
139 wake_flags, key, &bookmark);
140 spin_unlock_irqrestore(&wq_head->lock, flags);
141 } while (bookmark.flags & WQ_FLAG_BOOKMARK);
142 109
143 return nr_exclusive - remaining;110 return nr_exclusive - remaining;
144}111}
@@ -171,23 +138,16 @@ void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode · 26 unmodified lines
171 */138 */
172void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr)139void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr)
173{140{
174 __wake_up_common(wq_head, mode, nr, 0, NULL, NULL);141 __wake_up_common(wq_head, mode, nr, 0, NULL);
175}142}
176EXPORT_SYMBOL_GPL(__wake_up_locked);143EXPORT_SYMBOL_GPL(__wake_up_locked);
177 144
178void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key)145void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key)
179{146{
180 __wake_up_common(wq_head, mode, 1, 0, key, NULL);147 __wake_up_common(wq_head, mode, 1, 0, key);
181}148}
182EXPORT_SYMBOL_GPL(__wake_up_locked_key);149EXPORT_SYMBOL_GPL(__wake_up_locked_key);
183 150
184void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head,
185 unsigned int mode, void *key, wait_queue_entry_t *bookmark)
186{
187 __wake_up_common(wq_head, mode, 1, 0, key, bookmark);
188}
189EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark);
190
191/**151/**
192 * __wake_up_sync_key - wake up threads blocked on a waitqueue.152 * __wake_up_sync_key - wake up threads blocked on a waitqueue.
193 * @wq_head: the waitqueue153 * @wq_head: the waitqueue
@@ -233,7 +193,7 @@ EXPORT_SYMBOL_GPL(__wake_up_sync_key); · 39 unmodified lines
233void __wake_up_locked_sync_key(struct wait_queue_head *wq_head,193void __wake_up_locked_sync_key(struct wait_queue_head *wq_head,
234 unsigned int mode, void *key)194 unsigned int mode, void *key)
235{195{
236 __wake_up_common(wq_head, mode, 1, WF_SYNC, key, NULL);196 __wake_up_common(wq_head, mode, 1, WF_SYNC, key);
237}197}
238EXPORT_SYMBOL_GPL(__wake_up_locked_sync_key);198EXPORT_SYMBOL_GPL(__wake_up_locked_sync_key);
239 199
kernel/sys.c+26−6
@@ -2368,19 +2368,41 @@ static int prctl_set_vma(unsigned long opt, unsigned long start, · 2367 unmodified lines
2368}2368}
2369#endif /* CONFIG_ANON_VMA_NAME */2369#endif /* CONFIG_ANON_VMA_NAME */
2370 2370
2371static inline unsigned long get_current_mdwe(void)
2372{
2373 unsigned long ret = 0;
2374
2375 if (test_bit(MMF_HAS_MDWE, &current->mm->flags))
2376 ret |= PR_MDWE_REFUSE_EXEC_GAIN;
2377 if (test_bit(MMF_HAS_MDWE_NO_INHERIT, &current->mm->flags))
2378 ret |= PR_MDWE_NO_INHERIT;
2379
2380 return ret;
2381}
2382
2371static inline int prctl_set_mdwe(unsigned long bits, unsigned long arg3,2383static inline int prctl_set_mdwe(unsigned long bits, unsigned long arg3,
2372 unsigned long arg4, unsigned long arg5)2384 unsigned long arg4, unsigned long arg5)
2373{2385{
2386 unsigned long current_bits;
2387
2374 if (arg3 || arg4 || arg5)2388 if (arg3 || arg4 || arg5)
2375 return -EINVAL;2389 return -EINVAL;
2376 2390
2377 if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN))2391 if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN | PR_MDWE_NO_INHERIT))
2392 return -EINVAL;
2393
2394 /* NO_INHERIT only makes sense with REFUSE_EXEC_GAIN */
2395 if (bits & PR_MDWE_NO_INHERIT && !(bits & PR_MDWE_REFUSE_EXEC_GAIN))
2378 return -EINVAL;2396 return -EINVAL;
2379 2397
2398 current_bits = get_current_mdwe();
2399 if (current_bits && current_bits != bits)
2400 return -EPERM; /* Cannot unset the flags */
2401
2402 if (bits & PR_MDWE_NO_INHERIT)
2403 set_bit(MMF_HAS_MDWE_NO_INHERIT, &current->mm->flags);
2380 if (bits & PR_MDWE_REFUSE_EXEC_GAIN)2404 if (bits & PR_MDWE_REFUSE_EXEC_GAIN)
2381 set_bit(MMF_HAS_MDWE, &current->mm->flags);2405 set_bit(MMF_HAS_MDWE, &current->mm->flags);
2382 else if (test_bit(MMF_HAS_MDWE, &current->mm->flags))
2383 return -EPERM; /* Cannot unset the flag */
2384 2406
2385 return 0;2407 return 0;
2386}2408}
@@ -2390,9 +2412,7 @@ static inline int prctl_get_mdwe(unsigned long arg2, unsigned long arg3, · 3 unmodified lines
2390{2412{
2391 if (arg2 || arg3 || arg4 || arg5)2413 if (arg2 || arg3 || arg4 || arg5)
2392 return -EINVAL;2414 return -EINVAL;
2393 2415 return get_current_mdwe();
2394 return test_bit(MMF_HAS_MDWE, &current->mm->flags) ?
2395 PR_MDWE_REFUSE_EXEC_GAIN : 0;
2396}2416}
2397 2417
2398static int prctl_get_auxv(void __user *addr, unsigned long len)2418static int prctl_get_auxv(void __user *addr, unsigned long len)
lib/percpu_counter.c+79−0
@@ -278,6 +278,85 @@ int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch) · 277 unmodified lines
278}278}
279EXPORT_SYMBOL(__percpu_counter_compare);279EXPORT_SYMBOL(__percpu_counter_compare);
280 280
281/*
282 * Compare counter, and add amount if total is: less than or equal to limit if
283 * amount is positive, or greater than or equal to limit if amount is negative.
284 * Return true if amount is added, or false if total would be beyond the limit.
285 *
286 * Negative limit is allowed, but unusual.
287 * When negative amounts (subs) are given to percpu_counter_limited_add(),
288 * the limit would most naturally be 0 - but other limits are also allowed.
289 *
290 * Overflow beyond S64_MAX is not allowed for: counter, limit and amount
291 * are all assumed to be sane (far from S64_MIN and S64_MAX).
292 */
293bool __percpu_counter_limited_add(struct percpu_counter *fbc,
294 s64 limit, s64 amount, s32 batch)
295{
296 s64 count;
297 s64 unknown;
298 unsigned long flags;
299 bool good = false;
300
301 if (amount == 0)
302 return true;
303
304 local_irq_save(flags);
305 unknown = batch * num_online_cpus();
306 count = __this_cpu_read(*fbc->counters);
307
308 /* Skip taking the lock when safe */
309 if (abs(count + amount) <= batch &&
310 ((amount > 0 && fbc->count + unknown <= limit) ||
311 (amount < 0 && fbc->count - unknown >= limit))) {
312 this_cpu_add(*fbc->counters, amount);
313 local_irq_restore(flags);
314 return true;
315 }
316
317 raw_spin_lock(&fbc->lock);
318 count = fbc->count + amount;
319
320 /* Skip percpu_counter_sum() when safe */
321 if (amount > 0) {
322 if (count - unknown > limit)
323 goto out;
324 if (count + unknown <= limit)
325 good = true;
326 } else {
327 if (count + unknown < limit)
328 goto out;
329 if (count - unknown >= limit)
330 good = true;
331 }
332
333 if (!good) {
334 s32 *pcount;
335 int cpu;
336
337 for_each_cpu_or(cpu, cpu_online_mask, cpu_dying_mask) {
338 pcount = per_cpu_ptr(fbc->counters, cpu);
339 count += *pcount;
340 }
341 if (amount > 0) {
342 if (count > limit)
343 goto out;
344 } else {
345 if (count < limit)
346 goto out;
347 }
348 good = true;
349 }
350
351 count = __this_cpu_read(*fbc->counters);
352 fbc->count += count + amount;
353 __this_cpu_sub(*fbc->counters, count);
354out:
355 raw_spin_unlock(&fbc->lock);
356 local_irq_restore(flags);
357 return good;
358}
359
281static int __init percpu_counter_startup(void)360static int __init percpu_counter_startup(void)
282{361{
283 int ret;362 int ret;
mm/Kconfig+12−0
@@ -130,6 +130,7 @@ config ZSWAP_COMPRESSOR_DEFAULT · 129 unmodified lines
130choice130choice
131 prompt "Default allocator"131 prompt "Default allocator"
132 depends on ZSWAP132 depends on ZSWAP
133 default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if MMU
133 default ZSWAP_ZPOOL_DEFAULT_ZBUD134 default ZSWAP_ZPOOL_DEFAULT_ZBUD
134 help135 help
135 Selects the default allocator for the compressed cache for136 Selects the default allocator for the compressed cache for
@@ -704,6 +705,17 @@ config HUGETLB_PAGE_SIZE_VARIABLE · 568 unmodified lines
704config CONTIG_ALLOC705config CONTIG_ALLOC
705 def_bool (MEMORY_ISOLATION && COMPACTION) || CMA706 def_bool (MEMORY_ISOLATION && COMPACTION) || CMA
706 707
708config PCP_BATCH_SCALE_MAX
709 int "Maximum scale factor of PCP (Per-CPU pageset) batch allocate/free"
710 default 5
711 range 0 6
712 help
713 In page allocator, PCP (Per-CPU pageset) is refilled and drained in
714 batches. The batch number is scaled automatically to improve page
715 allocation/free throughput. But too large scale factor may hurt
716 latency. This option sets the upper limit of scale factor to limit
717 the maximum latency.
718
707config PHYS_ADDR_T_64BIT719config PHYS_ADDR_T_64BIT
708 def_bool 64BIT720 def_bool 64BIT
709 721
mm/Makefile+2−2
@@ -48,8 +48,8 @@ endif · 47 unmodified lines
48 48
49obj-y := filemap.o mempool.o oom_kill.o fadvise.o \49obj-y := filemap.o mempool.o oom_kill.o fadvise.o \
50 maccess.o page-writeback.o folio-compat.o \50 maccess.o page-writeback.o folio-compat.o \
51 readahead.o swap.o truncate.o vmscan.o shmem.o \51 readahead.o swap.o truncate.o vmscan.o shrinker.o \
52 util.o mmzone.o vmstat.o backing-dev.o \52 shmem.o util.o mmzone.o vmstat.o backing-dev.o \
53 mm_init.o percpu.o slab_common.o \53 mm_init.o percpu.o slab_common.o \
54 compaction.o show_mem.o shmem_quota.o\54 compaction.o show_mem.o shmem_quota.o\
55 interval_tree.o list_lru.o workingset.o \55 interval_tree.o list_lru.o workingset.o \
mm/bootmem_info.c+1−1
@@ -34,7 +34,7 @@ void put_page_bootmem(struct page *page) · 33 unmodified lines
34 ClearPagePrivate(page);34 ClearPagePrivate(page);
35 set_page_private(page, 0);35 set_page_private(page, 0);
36 INIT_LIST_HEAD(&page->lru);36 INIT_LIST_HEAD(&page->lru);
37 kmemleak_free_part(page_to_virt(page), PAGE_SIZE);37 kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
38 free_reserved_page(page);38 free_reserved_page(page);
39 }39 }
40}40}
mm/cma.c+1−1
@@ -505,7 +505,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, · 504 unmodified lines
505 */505 */
506 if (page) {506 if (page) {
507 for (i = 0; i < count; i++)507 for (i = 0; i < count; i++)
508 page_kasan_tag_reset(page + i);508 page_kasan_tag_reset(nth_page(page, i));
509 }509 }
510 510
511 if (ret && !no_warn) {511 if (ret && !no_warn) {
mm/compaction.c+52−39
@@ -382,6 +382,7 @@ static void __reset_isolation_suitable(struct zone *zone) · 381 unmodified lines
382 bool source_set = false;382 bool source_set = false;
383 bool free_set = false;383 bool free_set = false;
384 384
385 /* Only flush if a full compaction finished recently */
385 if (!zone->compact_blockskip_flush)386 if (!zone->compact_blockskip_flush)
386 return;387 return;
387 388
@@ -434,9 +435,7 @@ void reset_isolation_suitable(pg_data_t *pgdat) · 46 unmodified lines
434 if (!populated_zone(zone))435 if (!populated_zone(zone))
435 continue;436 continue;
436 437
437 /* Only flush if a full compaction finished recently */438 __reset_isolation_suitable(zone);
438 if (zone->compact_blockskip_flush)
439 __reset_isolation_suitable(zone);
440 }439 }
441}440}
442 441
@@ -626,11 +625,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, · 183 unmodified lines
626 if (PageCompound(page)) {625 if (PageCompound(page)) {
627 const unsigned int order = compound_order(page);626 const unsigned int order = compound_order(page);
628 627
629 if (likely(order <= MAX_ORDER)) {628 if (blockpfn + (1UL << order) <= end_pfn) {
630 blockpfn += (1UL << order) - 1;629 blockpfn += (1UL << order) - 1;
631 page += (1UL << order) - 1;630 page += (1UL << order) - 1;
632 nr_scanned += (1UL << order) - 1;631 nr_scanned += (1UL << order) - 1;
633 }632 }
633
634 goto isolate_fail;634 goto isolate_fail;
635 }635 }
636 636
@@ -678,8 +678,7 @@ isolate_fail: · 41 unmodified lines
678 spin_unlock_irqrestore(&cc->zone->lock, flags);678 spin_unlock_irqrestore(&cc->zone->lock, flags);
679 679
680 /*680 /*
681 * There is a tiny chance that we have read bogus compound_order(),681 * Be careful to not go outside of the pageblock.
682 * so be careful to not go outside of the pageblock.
683 */682 */
684 if (unlikely(blockpfn > end_pfn))683 if (unlikely(blockpfn > end_pfn))
685 blockpfn = end_pfn;684 blockpfn = end_pfn;
@@ -1395,8 +1394,8 @@ move_freelist_head(struct list_head *freelist, struct page *freepage) · 709 unmodified lines
1395{1394{
1396 LIST_HEAD(sublist);1395 LIST_HEAD(sublist);
1397 1396
1398 if (!list_is_last(freelist, &freepage->lru)) {1397 if (!list_is_first(&freepage->buddy_list, freelist)) {
1399 list_cut_before(&sublist, freelist, &freepage->lru);1398 list_cut_before(&sublist, freelist, &freepage->buddy_list);
1400 list_splice_tail(&sublist, freelist);1399 list_splice_tail(&sublist, freelist);
1401 }1400 }
1402}1401}
@@ -1412,8 +1411,8 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) · 9 unmodified lines
1412{1411{
1413 LIST_HEAD(sublist);1412 LIST_HEAD(sublist);
1414 1413
1415 if (!list_is_first(freelist, &freepage->lru)) {1414 if (!list_is_last(&freepage->buddy_list, freelist)) {
1416 list_cut_position(&sublist, freelist, &freepage->lru);1415 list_cut_position(&sublist, freelist, &freepage->buddy_list);
1417 list_splice_tail(&sublist, freelist);1416 list_splice_tail(&sublist, freelist);
1418 }1417 }
1419}1418}
@@ -2066,8 +2065,10 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) · 646 unmodified lines
2066}2065}
2067 2066
2068/*2067/*
2069 * order == -1 is expected when compacting via2068 * order == -1 is expected when compacting proactively via
2070 * /proc/sys/vm/compact_memory2069 * 1. /proc/sys/vm/compact_memory
2070 * 2. /sys/devices/system/node/nodex/compact
2071 * 3. /proc/sys/vm/compaction_proactiveness
2071 */2072 */
2072static inline bool is_via_compact_memory(int order)2073static inline bool is_via_compact_memory(int order)
2073{2074{
@@ -2377,6 +2378,30 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, · 303 unmodified lines
2377 return false;2378 return false;
2378}2379}
2379 2380
2381/*
2382 * Should we do compaction for target allocation order.
2383 * Return COMPACT_SUCCESS if allocation for target order can be already
2384 * satisfied
2385 * Return COMPACT_SKIPPED if compaction for target order is likely to fail
2386 * Return COMPACT_CONTINUE if compaction for target order should be ran
2387 */
2388static enum compact_result
2389compaction_suit_allocation_order(struct zone *zone, unsigned int order,
2390 int highest_zoneidx, unsigned int alloc_flags)
2391{
2392 unsigned long watermark;
2393
2394 watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
2395 if (zone_watermark_ok(zone, order, watermark, highest_zoneidx,
2396 alloc_flags))
2397 return COMPACT_SUCCESS;
2398
2399 if (!compaction_suitable(zone, order, highest_zoneidx))
2400 return COMPACT_SKIPPED;
2401
2402 return COMPACT_CONTINUE;
2403}
2404
2380static enum compact_result2405static enum compact_result
2381compact_zone(struct compact_control *cc, struct capture_control *capc)2406compact_zone(struct compact_control *cc, struct capture_control *capc)
2382{2407{
@@ -2402,19 +2427,11 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) · 19 unmodified lines
2402 cc->migratetype = gfp_migratetype(cc->gfp_mask);2427 cc->migratetype = gfp_migratetype(cc->gfp_mask);
2403 2428
2404 if (!is_via_compact_memory(cc->order)) {2429 if (!is_via_compact_memory(cc->order)) {
2405 unsigned long watermark;2430 ret = compaction_suit_allocation_order(cc->zone, cc->order,
2406 2431 cc->highest_zoneidx,
2407 /* Allocation can already succeed, nothing to do */2432 cc->alloc_flags);
2408 watermark = wmark_pages(cc->zone,2433 if (ret != COMPACT_CONTINUE)
2409 cc->alloc_flags & ALLOC_WMARK_MASK);2434 return ret;
2410 if (zone_watermark_ok(cc->zone, cc->order, watermark,
2411 cc->highest_zoneidx, cc->alloc_flags))
2412 return COMPACT_SUCCESS;
2413
2414 /* Compaction is likely to fail */
2415 if (!compaction_suitable(cc->zone, cc->order,
2416 cc->highest_zoneidx))
2417 return COMPACT_SKIPPED;
2418 }2435 }
2419 2436
2420 /*2437 /*
@@ -2913,6 +2930,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 492 unmodified lines
2913 int zoneid;2930 int zoneid;
2914 struct zone *zone;2931 struct zone *zone;
2915 enum zone_type highest_zoneidx = pgdat->kcompactd_highest_zoneidx;2932 enum zone_type highest_zoneidx = pgdat->kcompactd_highest_zoneidx;
2933 enum compact_result ret;
2916 2934
2917 for (zoneid = 0; zoneid <= highest_zoneidx; zoneid++) {2935 for (zoneid = 0; zoneid <= highest_zoneidx; zoneid++) {
2918 zone = &pgdat->node_zones[zoneid];2936 zone = &pgdat->node_zones[zoneid];
@@ -2920,14 +2938,10 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 1 unmodified line
2920 if (!populated_zone(zone))2938 if (!populated_zone(zone))
2921 continue;2939 continue;
2922 2940
2923 /* Allocation can already succeed, check other zones */2941 ret = compaction_suit_allocation_order(zone,
2924 if (zone_watermark_ok(zone, pgdat->kcompactd_max_order,2942 pgdat->kcompactd_max_order,
2925 min_wmark_pages(zone),2943 highest_zoneidx, ALLOC_WMARK_MIN);
2926 highest_zoneidx, 0))2944 if (ret == COMPACT_CONTINUE)
2927 continue;
2928
2929 if (compaction_suitable(zone, pgdat->kcompactd_max_order,
2930 highest_zoneidx))
2931 return true;2945 return true;
2932 }2946 }
2933 2947
@@ -2950,6 +2964,8 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 16 unmodified lines
2950 .ignore_skip_hint = false,2964 .ignore_skip_hint = false,
2951 .gfp_mask = GFP_KERNEL,2965 .gfp_mask = GFP_KERNEL,
2952 };2966 };
2967 enum compact_result ret;
2968
2953 trace_mm_compaction_kcompactd_wake(pgdat->node_id, cc.order,2969 trace_mm_compaction_kcompactd_wake(pgdat->node_id, cc.order,
2954 cc.highest_zoneidx);2970 cc.highest_zoneidx);
2955 count_compact_event(KCOMPACTD_WAKE);2971 count_compact_event(KCOMPACTD_WAKE);
@@ -2964,12 +2980,9 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 8 unmodified lines
2964 if (compaction_deferred(zone, cc.order))2980 if (compaction_deferred(zone, cc.order))
2965 continue;2981 continue;
2966 2982
2967 /* Allocation can already succeed, nothing to do */2983 ret = compaction_suit_allocation_order(zone,
2968 if (zone_watermark_ok(zone, cc.order,2984 cc.order, zoneid, ALLOC_WMARK_MIN);
2969 min_wmark_pages(zone), zoneid, 0))2985 if (ret != COMPACT_CONTINUE)
2970 continue;
2971
2972 if (!compaction_suitable(zone, cc.order, zoneid))
2973 continue;2986 continue;
2974 2987
2975 if (kthread_should_stop())2988 if (kthread_should_stop())
mm/damon/Kconfig+12−0
@@ -59,6 +59,18 @@ config DAMON_SYSFS · 58 unmodified lines
59 This builds the sysfs interface for DAMON. The user space can use59 This builds the sysfs interface for DAMON. The user space can use
60 the interface for arbitrary data access monitoring.60 the interface for arbitrary data access monitoring.
61 61
62config DAMON_SYSFS_KUNIT_TEST
63 bool "Test for damon debugfs interface" if !KUNIT_ALL_TESTS
64 depends on DAMON_SYSFS && KUNIT=y
65 default KUNIT_ALL_TESTS
66 help
67 This builds the DAMON sysfs interface Kunit test suite.
68
69 For more information on KUnit and unit tests in general, please refer
70 to the KUnit documentation.
71
72 If unsure, say N.
73
62config DAMON_DBGFS74config DAMON_DBGFS
63 bool "DAMON debugfs interface (DEPRECATED!)"75 bool "DAMON debugfs interface (DEPRECATED!)"
64 depends on DAMON_VADDR && DAMON_PADDR && DEBUG_FS76 depends on DAMON_VADDR && DAMON_PADDR && DEBUG_FS
mm/damon/core-test.h+28−1
@@ -30,7 +30,7 @@ static void damon_test_regions(struct kunit *test) · 29 unmodified lines
30 damon_add_region(r, t);30 damon_add_region(r, t);
31 KUNIT_EXPECT_EQ(test, 1u, damon_nr_regions(t));31 KUNIT_EXPECT_EQ(test, 1u, damon_nr_regions(t));
32 32
33 damon_del_region(r, t);33 damon_destroy_region(r, t);
34 KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t));34 KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t));
35 35
36 damon_free_target(t);36 damon_free_target(t);
@@ -94,6 +94,7 @@ static void damon_test_aggregate(struct kunit *test) · 57 unmodified lines
94 for (ir = 0; ir < 3; ir++) {94 for (ir = 0; ir < 3; ir++) {
95 r = damon_new_region(saddr[it][ir], eaddr[it][ir]);95 r = damon_new_region(saddr[it][ir], eaddr[it][ir]);
96 r->nr_accesses = accesses[it][ir];96 r->nr_accesses = accesses[it][ir];
97 r->nr_accesses_bp = accesses[it][ir] * 10000;
97 damon_add_region(r, t);98 damon_add_region(r, t);
98 }99 }
99 it++;100 it++;
@@ -147,9 +148,11 @@ static void damon_test_merge_two(struct kunit *test) · 47 unmodified lines
147 t = damon_new_target();148 t = damon_new_target();
148 r = damon_new_region(0, 100);149 r = damon_new_region(0, 100);
149 r->nr_accesses = 10;150 r->nr_accesses = 10;
151 r->nr_accesses_bp = 100000;
150 damon_add_region(r, t);152 damon_add_region(r, t);
151 r2 = damon_new_region(100, 300);153 r2 = damon_new_region(100, 300);
152 r2->nr_accesses = 20;154 r2->nr_accesses = 20;
155 r2->nr_accesses_bp = 200000;
153 damon_add_region(r2, t);156 damon_add_region(r2, t);
154 157
155 damon_merge_two_regions(t, r, r2);158 damon_merge_two_regions(t, r, r2);
@@ -196,6 +199,7 @@ static void damon_test_merge_regions_of(struct kunit *test) · 40 unmodified lines
196 for (i = 0; i < ARRAY_SIZE(sa); i++) {199 for (i = 0; i < ARRAY_SIZE(sa); i++) {
197 r = damon_new_region(sa[i], ea[i]);200 r = damon_new_region(sa[i], ea[i]);
198 r->nr_accesses = nrs[i];201 r->nr_accesses = nrs[i];
202 r->nr_accesses_bp = nrs[i] * 10000;
199 damon_add_region(r, t);203 damon_add_region(r, t);
200 }204 }
201 205
@@ -265,6 +269,8 @@ static void damon_test_ops_registration(struct kunit *test) · 63 unmodified lines
265 269
266 /* Check double-registration failure again */270 /* Check double-registration failure again */
267 KUNIT_EXPECT_EQ(test, damon_register_ops(&ops), -EINVAL);271 KUNIT_EXPECT_EQ(test, damon_register_ops(&ops), -EINVAL);
272
273 damon_destroy_ctx(c);
268}274}
269 275
270static void damon_test_set_regions(struct kunit *test)276static void damon_test_set_regions(struct kunit *test)
@@ -297,6 +303,7 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 26 unmodified lines
297 struct damon_region *r = damon_new_region(3, 7);303 struct damon_region *r = damon_new_region(3, 7);
298 304
299 r->nr_accesses = 15;305 r->nr_accesses = 15;
306 r->nr_accesses_bp = 150000;
300 r->age = 20;307 r->age = 20;
301 308
302 new_attrs = (struct damon_attrs){309 new_attrs = (struct damon_attrs){
@@ -316,6 +323,8 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 13 unmodified lines
316 damon_update_monitoring_result(r, &old_attrs, &new_attrs);323 damon_update_monitoring_result(r, &old_attrs, &new_attrs);
317 KUNIT_EXPECT_EQ(test, r->nr_accesses, 150);324 KUNIT_EXPECT_EQ(test, r->nr_accesses, 150);
318 KUNIT_EXPECT_EQ(test, r->age, 20);325 KUNIT_EXPECT_EQ(test, r->age, 20);
326
327 damon_free_region(r);
319}328}
320 329
321static void damon_test_set_attrs(struct kunit *test)330static void damon_test_set_attrs(struct kunit *test)
@@ -339,6 +348,23 @@ static void damon_test_set_attrs(struct kunit *test) · 17 unmodified lines
339 invalid_attrs = valid_attrs;348 invalid_attrs = valid_attrs;
340 invalid_attrs.aggr_interval = 4999;349 invalid_attrs.aggr_interval = 4999;
341 KUNIT_EXPECT_EQ(test, damon_set_attrs(c, &invalid_attrs), -EINVAL);350 KUNIT_EXPECT_EQ(test, damon_set_attrs(c, &invalid_attrs), -EINVAL);
351
352 damon_destroy_ctx(c);
353}
354
355static void damon_test_moving_sum(struct kunit *test)
356{
357 unsigned int mvsum = 50000, nomvsum = 50000, len_window = 10;
358 unsigned int new_values[] = {10000, 0, 10000, 0, 0, 0, 10000, 0, 0, 0};
359 unsigned int expects[] = {55000, 50000, 55000, 50000, 45000, 40000,
360 45000, 40000, 35000, 30000};
361 int i;
362
363 for (i = 0; i < ARRAY_SIZE(new_values); i++) {
364 mvsum = damon_moving_sum(mvsum, nomvsum, len_window,
365 new_values[i]);
366 KUNIT_EXPECT_EQ(test, mvsum, expects[i]);
367 }
342}368}
343 369
344static void damos_test_new_filter(struct kunit *test)370static void damos_test_new_filter(struct kunit *test)
@@ -425,6 +451,7 @@ static struct kunit_case damon_test_cases[] = { · 80 unmodified lines
425 KUNIT_CASE(damon_test_set_regions),451 KUNIT_CASE(damon_test_set_regions),
426 KUNIT_CASE(damon_test_update_monitoring_result),452 KUNIT_CASE(damon_test_update_monitoring_result),
427 KUNIT_CASE(damon_test_set_attrs),453 KUNIT_CASE(damon_test_set_attrs),
454 KUNIT_CASE(damon_test_moving_sum),
428 KUNIT_CASE(damos_test_new_filter),455 KUNIT_CASE(damos_test_new_filter),
429 KUNIT_CASE(damos_test_filter_out),456 KUNIT_CASE(damos_test_filter_out),
430 {},457 {},
mm/damon/core.c+226−69
@@ -128,6 +128,7 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 127 unmodified lines
128 region->ar.start = start;128 region->ar.start = start;
129 region->ar.end = end;129 region->ar.end = end;
130 region->nr_accesses = 0;130 region->nr_accesses = 0;
131 region->nr_accesses_bp = 0;
131 INIT_LIST_HEAD(&region->list);132 INIT_LIST_HEAD(&region->list);
132 133
133 region->age = 0;134 region->age = 0;
@@ -312,7 +313,9 @@ static struct damos_quota *damos_quota_init_priv(struct damos_quota *quota) · 178 unmodified lines
312}313}
313 314
314struct damos *damon_new_scheme(struct damos_access_pattern *pattern,315struct damos *damon_new_scheme(struct damos_access_pattern *pattern,
315 enum damos_action action, struct damos_quota *quota,316 enum damos_action action,
317 unsigned long apply_interval_us,
318 struct damos_quota *quota,
316 struct damos_watermarks *wmarks)319 struct damos_watermarks *wmarks)
317{320{
318 struct damos *scheme;321 struct damos *scheme;
@@ -322,6 +325,13 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 3 unmodified lines
322 return NULL;325 return NULL;
323 scheme->pattern = *pattern;326 scheme->pattern = *pattern;
324 scheme->action = action;327 scheme->action = action;
328 scheme->apply_interval_us = apply_interval_us;
329 /*
330 * next_apply_sis will be set when kdamond starts. While kdamond is
331 * running, it will also updated when it is added to the DAMON context,
332 * or damon_attrs are updated.
333 */
334 scheme->next_apply_sis = 0;
325 INIT_LIST_HEAD(&scheme->filters);335 INIT_LIST_HEAD(&scheme->filters);
326 scheme->stat = (struct damos_stat){};336 scheme->stat = (struct damos_stat){};
327 INIT_LIST_HEAD(&scheme->list);337 INIT_LIST_HEAD(&scheme->list);
@@ -334,9 +344,21 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 6 unmodified lines
334 return scheme;344 return scheme;
335}345}
336 346
347static void damos_set_next_apply_sis(struct damos *s, struct damon_ctx *ctx)
348{
349 unsigned long sample_interval = ctx->attrs.sample_interval ?
350 ctx->attrs.sample_interval : 1;
351 unsigned long apply_interval = s->apply_interval_us ?
352 s->apply_interval_us : ctx->attrs.aggr_interval;
353
354 s->next_apply_sis = ctx->passed_sample_intervals +
355 apply_interval / sample_interval;
356}
357
337void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)358void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)
338{359{
339 list_add_tail(&s->list, &ctx->schemes);360 list_add_tail(&s->list, &ctx->schemes);
361 damos_set_next_apply_sis(s, ctx);
340}362}
341 363
342static void damon_del_scheme(struct damos *s)364static void damon_del_scheme(struct damos *s)
@@ -427,8 +449,10 @@ struct damon_ctx *damon_new_ctx(void) · 84 unmodified lines
427 ctx->attrs.aggr_interval = 100 * 1000;449 ctx->attrs.aggr_interval = 100 * 1000;
428 ctx->attrs.ops_update_interval = 60 * 1000 * 1000;450 ctx->attrs.ops_update_interval = 60 * 1000 * 1000;
429 451
430 ktime_get_coarse_ts64(&ctx->last_aggregation);452 ctx->passed_sample_intervals = 0;
431 ctx->last_ops_update = ctx->last_aggregation;453 /* These will be set from kdamond_init_intervals_sis() */
454 ctx->next_aggregation_sis = 0;
455 ctx->next_ops_update_sis = 0;
432 456
433 mutex_init(&ctx->kdamond_lock);457 mutex_init(&ctx->kdamond_lock);
434 458
@@ -476,20 +500,14 @@ static unsigned int damon_age_for_new_attrs(unsigned int age, · 41 unmodified lines
476static unsigned int damon_accesses_bp_to_nr_accesses(500static unsigned int damon_accesses_bp_to_nr_accesses(
477 unsigned int accesses_bp, struct damon_attrs *attrs)501 unsigned int accesses_bp, struct damon_attrs *attrs)
478{502{
479 unsigned int max_nr_accesses =503 return accesses_bp * damon_max_nr_accesses(attrs) / 10000;
480 attrs->aggr_interval / attrs->sample_interval;
481
482 return accesses_bp * max_nr_accesses / 10000;
483}504}
484 505
485/* convert nr_accesses to access ratio in bp (per 10,000) */506/* convert nr_accesses to access ratio in bp (per 10,000) */
486static unsigned int damon_nr_accesses_to_accesses_bp(507static unsigned int damon_nr_accesses_to_accesses_bp(
487 unsigned int nr_accesses, struct damon_attrs *attrs)508 unsigned int nr_accesses, struct damon_attrs *attrs)
488{509{
489 unsigned int max_nr_accesses =510 return nr_accesses * 10000 / damon_max_nr_accesses(attrs);
490 attrs->aggr_interval / attrs->sample_interval;
491
492 return nr_accesses * 10000 / max_nr_accesses;
493}511}
494 512
495static unsigned int damon_nr_accesses_for_new_attrs(unsigned int nr_accesses,513static unsigned int damon_nr_accesses_for_new_attrs(unsigned int nr_accesses,
@@ -506,6 +524,7 @@ static void damon_update_monitoring_result(struct damon_region *r, · 10 unmodified lines
506{524{
507 r->nr_accesses = damon_nr_accesses_for_new_attrs(r->nr_accesses,525 r->nr_accesses = damon_nr_accesses_for_new_attrs(r->nr_accesses,
508 old_attrs, new_attrs);526 old_attrs, new_attrs);
527 r->nr_accesses_bp = r->nr_accesses * 10000;
509 r->age = damon_age_for_new_attrs(r->age, old_attrs, new_attrs);528 r->age = damon_age_for_new_attrs(r->age, old_attrs, new_attrs);
510}529}
511 530
@@ -541,13 +560,21 @@ static void damon_update_monitoring_results(struct damon_ctx *ctx, · 29 unmodified lines
541 * @ctx: monitoring context560 * @ctx: monitoring context
542 * @attrs: monitoring attributes561 * @attrs: monitoring attributes
543 *562 *
544 * This function should not be called while the kdamond is running.563 * This function should be called while the kdamond is not running, or an
564 * access check results aggregation is not ongoing (e.g., from
565 * &struct damon_callback->after_aggregation or
566 * &struct damon_callback->after_wmarks_check callbacks).
567 *
545 * Every time interval is in micro-seconds.568 * Every time interval is in micro-seconds.
546 *569 *
547 * Return: 0 on success, negative error code otherwise.570 * Return: 0 on success, negative error code otherwise.
548 */571 */
549int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs)572int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs)
550{573{
574 unsigned long sample_interval = attrs->sample_interval ?
575 attrs->sample_interval : 1;
576 struct damos *s;
577
551 if (attrs->min_nr_regions < 3)578 if (attrs->min_nr_regions < 3)
552 return -EINVAL;579 return -EINVAL;
553 if (attrs->min_nr_regions > attrs->max_nr_regions)580 if (attrs->min_nr_regions > attrs->max_nr_regions)
@@ -555,8 +582,17 @@ int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) · 1 unmodified line
555 if (attrs->sample_interval > attrs->aggr_interval)582 if (attrs->sample_interval > attrs->aggr_interval)
556 return -EINVAL;583 return -EINVAL;
557 584
585 ctx->next_aggregation_sis = ctx->passed_sample_intervals +
586 attrs->aggr_interval / sample_interval;
587 ctx->next_ops_update_sis = ctx->passed_sample_intervals +
588 attrs->ops_update_interval / sample_interval;
589
558 damon_update_monitoring_results(ctx, attrs);590 damon_update_monitoring_results(ctx, attrs);
559 ctx->attrs = *attrs;591 ctx->attrs = *attrs;
592
593 damon_for_each_scheme(s, ctx)
594 damos_set_next_apply_sis(s, ctx);
595
560 return 0;596 return 0;
561}597}
562 598
@@ -728,38 +764,6 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs) · 165 unmodified lines
728 return err;764 return err;
729}765}
730 766
731/*
732 * damon_check_reset_time_interval() - Check if a time interval is elapsed.
733 * @baseline: the time to check whether the interval has elapsed since
734 * @interval: the time interval (microseconds)
735 *
736 * See whether the given time interval has passed since the given baseline
737 * time. If so, it also updates the baseline to current time for next check.
738 *
739 * Return: true if the time interval has passed, or false otherwise.
740 */
741static bool damon_check_reset_time_interval(struct timespec64 *baseline,
742 unsigned long interval)
743{
744 struct timespec64 now;
745
746 ktime_get_coarse_ts64(&now);
747 if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) <
748 interval * 1000)
749 return false;
750 *baseline = now;
751 return true;
752}
753
754/*
755 * Check whether it is time to flush the aggregated information
756 */
757static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx)
758{
759 return damon_check_reset_time_interval(&ctx->last_aggregation,
760 ctx->attrs.aggr_interval);
761}
762
763/*767/*
764 * Reset the aggregated monitoring results ('nr_accesses' of each region).768 * Reset the aggregated monitoring results ('nr_accesses' of each region).
765 */769 */
@@ -772,7 +776,7 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) · 6 unmodified lines
772 struct damon_region *r;776 struct damon_region *r;
773 777
774 damon_for_each_region(r, t) {778 damon_for_each_region(r, t) {
775 trace_damon_aggregated(t, ti, r, damon_nr_regions(t));779 trace_damon_aggregated(ti, r, damon_nr_regions(t));
776 r->last_nr_accesses = r->nr_accesses;780 r->last_nr_accesses = r->nr_accesses;
777 r->nr_accesses = 0;781 r->nr_accesses = 0;
778 }782 }
@@ -786,12 +790,13 @@ static void damon_split_region_at(struct damon_target *t, · 7 unmodified lines
786static bool __damos_valid_target(struct damon_region *r, struct damos *s)790static bool __damos_valid_target(struct damon_region *r, struct damos *s)
787{791{
788 unsigned long sz;792 unsigned long sz;
793 unsigned int nr_accesses = r->nr_accesses_bp / 10000;
789 794
790 sz = damon_sz_region(r);795 sz = damon_sz_region(r);
791 return s->pattern.min_sz_region <= sz &&796 return s->pattern.min_sz_region <= sz &&
792 sz <= s->pattern.max_sz_region &&797 sz <= s->pattern.max_sz_region &&
793 s->pattern.min_nr_accesses <= r->nr_accesses &&798 s->pattern.min_nr_accesses <= nr_accesses &&
794 r->nr_accesses <= s->pattern.max_nr_accesses &&799 nr_accesses <= s->pattern.max_nr_accesses &&
795 s->pattern.min_age_region <= r->age &&800 s->pattern.min_age_region <= r->age &&
796 r->age <= s->pattern.max_age_region;801 r->age <= s->pattern.max_age_region;
797}802}
@@ -946,6 +951,33 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 148 unmodified lines
946 struct timespec64 begin, end;951 struct timespec64 begin, end;
947 unsigned long sz_applied = 0;952 unsigned long sz_applied = 0;
948 int err = 0;953 int err = 0;
954 /*
955 * We plan to support multiple context per kdamond, as DAMON sysfs
956 * implies with 'nr_contexts' file. Nevertheless, only single context
957 * per kdamond is supported for now. So, we can simply use '0' context
958 * index here.
959 */
960 unsigned int cidx = 0;
961 struct damos *siter; /* schemes iterator */
962 unsigned int sidx = 0;
963 struct damon_target *titer; /* targets iterator */
964 unsigned int tidx = 0;
965 bool do_trace = false;
966
967 /* get indices for trace_damos_before_apply() */
968 if (trace_damos_before_apply_enabled()) {
969 damon_for_each_scheme(siter, c) {
970 if (siter == s)
971 break;
972 sidx++;
973 }
974 damon_for_each_target(titer, c) {
975 if (titer == t)
976 break;
977 tidx++;
978 }
979 do_trace = true;
980 }
949 981
950 if (c->ops.apply_scheme) {982 if (c->ops.apply_scheme) {
951 if (quota->esz && quota->charged_sz + sz > quota->esz) {983 if (quota->esz && quota->charged_sz + sz > quota->esz) {
@@ -960,8 +992,11 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 8 unmodified lines
960 ktime_get_coarse_ts64(&begin);992 ktime_get_coarse_ts64(&begin);
961 if (c->callback.before_damos_apply)993 if (c->callback.before_damos_apply)
962 err = c->callback.before_damos_apply(c, t, r, s);994 err = c->callback.before_damos_apply(c, t, r, s);
963 if (!err)995 if (!err) {
996 trace_damos_before_apply(cidx, sidx, tidx, r,
997 damon_nr_regions(t), do_trace);
964 sz_applied = c->ops.apply_scheme(c, t, r, s);998 sz_applied = c->ops.apply_scheme(c, t, r, s);
999 }
965 ktime_get_coarse_ts64(&end);1000 ktime_get_coarse_ts64(&end);
966 quota->total_charged_ns += timespec64_to_ns(&end) -1001 quota->total_charged_ns += timespec64_to_ns(&end) -
967 timespec64_to_ns(&begin);1002 timespec64_to_ns(&begin);
@@ -1079,14 +1114,29 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 111 unmodified lines
1079 struct damon_target *t;1114 struct damon_target *t;
1080 struct damon_region *r, *next_r;1115 struct damon_region *r, *next_r;
1081 struct damos *s;1116 struct damos *s;
1117 unsigned long sample_interval = c->attrs.sample_interval ?
1118 c->attrs.sample_interval : 1;
1119 bool has_schemes_to_apply = false;
1082 1120
1083 damon_for_each_scheme(s, c) {1121 damon_for_each_scheme(s, c) {
1122 if (c->passed_sample_intervals != s->next_apply_sis)
1123 continue;
1124
1125 s->next_apply_sis +=
1126 (s->apply_interval_us ? s->apply_interval_us :
1127 c->attrs.aggr_interval) / sample_interval;
1128
1084 if (!s->wmarks.activated)1129 if (!s->wmarks.activated)
1085 continue;1130 continue;
1086 1131
1132 has_schemes_to_apply = true;
1133
1087 damos_adjust_quota(c, s);1134 damos_adjust_quota(c, s);
1088 }1135 }
1089 1136
1137 if (!has_schemes_to_apply)
1138 return;
1139
1090 damon_for_each_target(t, c) {1140 damon_for_each_target(t, c) {
1091 damon_for_each_region_safe(r, next_r, t)1141 damon_for_each_region_safe(r, next_r, t)
1092 damon_do_apply_schemes(c, t, r);1142 damon_do_apply_schemes(c, t, r);
@@ -1103,6 +1153,7 @@ static void damon_merge_two_regions(struct damon_target *t, · 10 unmodified lines
1103 1153
1104 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /1154 l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
1105 (sz_l + sz_r);1155 (sz_l + sz_r);
1156 l->nr_accesses_bp = l->nr_accesses * 10000;
1106 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);1157 l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
1107 l->ar.end = r->ar.end;1158 l->ar.end = r->ar.end;
1108 damon_destroy_region(r, t);1159 damon_destroy_region(r, t);
@@ -1174,6 +1225,7 @@ static void damon_split_region_at(struct damon_target *t, · 65 unmodified lines
1174 1225
1175 new->age = r->age;1226 new->age = r->age;
1176 new->last_nr_accesses = r->last_nr_accesses;1227 new->last_nr_accesses = r->last_nr_accesses;
1228 new->nr_accesses_bp = r->nr_accesses_bp;
1177 1229
1178 damon_insert_region(new, r, damon_next_region(r), t);1230 damon_insert_region(new, r, damon_next_region(r), t);
1179}1231}
@@ -1240,18 +1292,6 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 60 unmodified lines
1240 last_nr_regions = nr_regions;1292 last_nr_regions = nr_regions;
1241}1293}
1242 1294
1243/*
1244 * Check whether it is time to check and apply the operations-related data
1245 * structures.
1246 *
1247 * Returns true if it is.
1248 */
1249static bool kdamond_need_update_operations(struct damon_ctx *ctx)
1250{
1251 return damon_check_reset_time_interval(&ctx->last_ops_update,
1252 ctx->attrs.ops_update_interval);
1253}
1254
1255/*1295/*
1256 * Check whether current monitoring should be stopped1296 * Check whether current monitoring should be stopped
1257 *1297 *
@@ -1280,12 +1320,10 @@ static bool kdamond_need_stop(struct damon_ctx *ctx) · 22 unmodified lines
1280 1320
1281static unsigned long damos_wmark_metric_value(enum damos_wmark_metric metric)1321static unsigned long damos_wmark_metric_value(enum damos_wmark_metric metric)
1282{1322{
1283 struct sysinfo i;
1284
1285 switch (metric) {1323 switch (metric) {
1286 case DAMOS_WMARK_FREE_MEM_RATE:1324 case DAMOS_WMARK_FREE_MEM_RATE:
1287 si_meminfo(&i);1325 return global_zone_page_state(NR_FREE_PAGES) * 1000 /
1288 return i.freeram * 1000 / i.totalram;1326 totalram_pages();
1289 default:1327 default:
1290 break;1328 break;
1291 }1329 }
@@ -1363,6 +1401,25 @@ static int kdamond_wait_activation(struct damon_ctx *ctx) · 71 unmodified lines
1363 return -EBUSY;1401 return -EBUSY;
1364}1402}
1365 1403
1404static void kdamond_init_intervals_sis(struct damon_ctx *ctx)
1405{
1406 unsigned long sample_interval = ctx->attrs.sample_interval ?
1407 ctx->attrs.sample_interval : 1;
1408 unsigned long apply_interval;
1409 struct damos *scheme;
1410
1411 ctx->passed_sample_intervals = 0;
1412 ctx->next_aggregation_sis = ctx->attrs.aggr_interval / sample_interval;
1413 ctx->next_ops_update_sis = ctx->attrs.ops_update_interval /
1414 sample_interval;
1415
1416 damon_for_each_scheme(scheme, ctx) {
1417 apply_interval = scheme->apply_interval_us ?
1418 scheme->apply_interval_us : ctx->attrs.aggr_interval;
1419 scheme->next_apply_sis = apply_interval / sample_interval;
1420 }
1421}
1422
1366/*1423/*
1367 * The monitoring daemon that runs as a kernel thread1424 * The monitoring daemon that runs as a kernel thread
1368 */1425 */
@@ -1376,6 +1433,8 @@ static int kdamond_fn(void *data) · 7 unmodified lines
1376 1433
1377 pr_debug("kdamond (%d) starts\n", current->pid);1434 pr_debug("kdamond (%d) starts\n", current->pid);
1378 1435
1436 kdamond_init_intervals_sis(ctx);
1437
1379 if (ctx->ops.init)1438 if (ctx->ops.init)
1380 ctx->ops.init(ctx);1439 ctx->ops.init(ctx);
1381 if (ctx->callback.before_start && ctx->callback.before_start(ctx))1440 if (ctx->callback.before_start && ctx->callback.before_start(ctx))
@@ -1384,6 +1443,17 @@ static int kdamond_fn(void *data) · 2 unmodified lines
1384 sz_limit = damon_region_sz_limit(ctx);1443 sz_limit = damon_region_sz_limit(ctx);
1385 1444
1386 while (!kdamond_need_stop(ctx)) {1445 while (!kdamond_need_stop(ctx)) {
1446 /*
1447 * ctx->attrs and ctx->next_{aggregation,ops_update}_sis could
1448 * be changed from after_wmarks_check() or after_aggregation()
1449 * callbacks. Read the values here, and use those for this
1450 * iteration. That is, damon_set_attrs() updated new values
1451 * are respected from next iteration.
1452 */
1453 unsigned long next_aggregation_sis = ctx->next_aggregation_sis;
1454 unsigned long next_ops_update_sis = ctx->next_ops_update_sis;
1455 unsigned long sample_interval = ctx->attrs.sample_interval;
1456
1387 if (kdamond_wait_activation(ctx))1457 if (kdamond_wait_activation(ctx))
1388 break;1458 break;
1389 1459
@@ -1393,27 +1463,44 @@ static int kdamond_fn(void *data) · 3 unmodified lines
1393 ctx->callback.after_sampling(ctx))1463 ctx->callback.after_sampling(ctx))
1394 break;1464 break;
1395 1465
1396 kdamond_usleep(ctx->attrs.sample_interval);1466 kdamond_usleep(sample_interval);
1467 ctx->passed_sample_intervals++;
1397 1468
1398 if (ctx->ops.check_accesses)1469 if (ctx->ops.check_accesses)
1399 max_nr_accesses = ctx->ops.check_accesses(ctx);1470 max_nr_accesses = ctx->ops.check_accesses(ctx);
1400 1471
1401 if (kdamond_aggregate_interval_passed(ctx)) {1472 if (ctx->passed_sample_intervals == next_aggregation_sis) {
1402 kdamond_merge_regions(ctx,1473 kdamond_merge_regions(ctx,
1403 max_nr_accesses / 10,1474 max_nr_accesses / 10,
1404 sz_limit);1475 sz_limit);
1405 if (ctx->callback.after_aggregation &&1476 if (ctx->callback.after_aggregation &&
1406 ctx->callback.after_aggregation(ctx))1477 ctx->callback.after_aggregation(ctx))
1407 break;1478 break;
1408 if (!list_empty(&ctx->schemes))1479 }
1409 kdamond_apply_schemes(ctx);1480
1481 /*
1482 * do kdamond_apply_schemes() after kdamond_merge_regions() if
1483 * possible, to reduce overhead
1484 */
1485 if (!list_empty(&ctx->schemes))
1486 kdamond_apply_schemes(ctx);
1487
1488 sample_interval = ctx->attrs.sample_interval ?
1489 ctx->attrs.sample_interval : 1;
1490 if (ctx->passed_sample_intervals == next_aggregation_sis) {
1491 ctx->next_aggregation_sis = next_aggregation_sis +
1492 ctx->attrs.aggr_interval / sample_interval;
1493
1410 kdamond_reset_aggregated(ctx);1494 kdamond_reset_aggregated(ctx);
1411 kdamond_split_regions(ctx);1495 kdamond_split_regions(ctx);
1412 if (ctx->ops.reset_aggregated)1496 if (ctx->ops.reset_aggregated)
1413 ctx->ops.reset_aggregated(ctx);1497 ctx->ops.reset_aggregated(ctx);
1414 }1498 }
1415 1499
1416 if (kdamond_need_update_operations(ctx)) {1500 if (ctx->passed_sample_intervals == next_ops_update_sis) {
1501 ctx->next_ops_update_sis = next_ops_update_sis +
1502 ctx->attrs.ops_update_interval /
1503 sample_interval;
1417 if (ctx->ops.update)1504 if (ctx->ops.update)
1418 ctx->ops.update(ctx);1505 ctx->ops.update(ctx);
1419 sz_limit = damon_region_sz_limit(ctx);1506 sz_limit = damon_region_sz_limit(ctx);
@@ -1517,6 +1604,76 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 97 unmodified lines
1517 return damon_set_regions(t, &addr_range, 1);1604 return damon_set_regions(t, &addr_range, 1);
1518}1605}
1519 1606
1607/*
1608 * damon_moving_sum() - Calculate an inferred moving sum value.
1609 * @mvsum: Inferred sum of the last @len_window values.
1610 * @nomvsum: Non-moving sum of the last discrete @len_window window values.
1611 * @len_window: The number of last values to take care of.
1612 * @new_value: New value that will be added to the pseudo moving sum.
1613 *
1614 * Moving sum (moving average * window size) is good for handling noise, but
1615 * the cost of keeping past values can be high for arbitrary window size. This
1616 * function implements a lightweight pseudo moving sum function that doesn't
1617 * keep the past window values.
1618 *
1619 * It simply assumes there was no noise in the past, and get the no-noise
1620 * assumed past value to drop from @nomvsum and @len_window. @nomvsum is a
1621 * non-moving sum of the last window. For example, if @len_window is 10 and we
1622 * have 25 values, @nomvsum is the sum of the 11th to 20th values of the 25
1623 * values. Hence, this function simply drops @nomvsum / @len_window from
1624 * given @mvsum and add @new_value.
1625 *
1626 * For example, if @len_window is 10 and @nomvsum is 50, the last 10 values for
1627 * the last window could be vary, e.g., 0, 10, 0, 10, 0, 10, 0, 0, 0, 20. For
1628 * calculating next moving sum with a new value, we should drop 0 from 50 and
1629 * add the new value. However, this function assumes it got value 5 for each
1630 * of the last ten times. Based on the assumption, when the next value is
1631 * measured, it drops the assumed past value, 5 from the current sum, and add
1632 * the new value to get the updated pseduo-moving average.
1633 *
1634 * This means the value could have errors, but the errors will be disappeared
1635 * for every @len_window aligned calls. For example, if @len_window is 10, the
1636 * pseudo moving sum with 11th value to 19th value would have an error. But
1637 * the sum with 20th value will not have the error.
1638 *
1639 * Return: Pseudo-moving average after getting the @new_value.
1640 */
1641static unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum,
1642 unsigned int len_window, unsigned int new_value)
1643{
1644 return mvsum - nomvsum / len_window + new_value;
1645}
1646
1647/**
1648 * damon_update_region_access_rate() - Update the access rate of a region.
1649 * @r: The DAMON region to update for its access check result.
1650 * @accessed: Whether the region has accessed during last sampling interval.
1651 * @attrs: The damon_attrs of the DAMON context.
1652 *
1653 * Update the access rate of a region with the region's last sampling interval
1654 * access check result.
1655 *
1656 * Usually this will be called by &damon_operations->check_accesses callback.
1657 */
1658void damon_update_region_access_rate(struct damon_region *r, bool accessed,
1659 struct damon_attrs *attrs)
1660{
1661 unsigned int len_window = 1;
1662
1663 /*
1664 * sample_interval can be zero, but cannot be larger than
1665 * aggr_interval, owing to validation of damon_set_attrs().
1666 */
1667 if (attrs->sample_interval)
1668 len_window = damon_max_nr_accesses(attrs);
1669 r->nr_accesses_bp = damon_moving_sum(r->nr_accesses_bp,
1670 r->last_nr_accesses * 10000, len_window,
1671 accessed ? 10000 : 0);
1672
1673 if (accessed)
1674 r->nr_accesses++;
1675}
1676
1520static int __init damon_init(void)1677static int __init damon_init(void)
1521{1678{
1522 damon_region_cache = KMEM_CACHE(damon_region, 0);1679 damon_region_cache = KMEM_CACHE(damon_region, 0);
mm/damon/dbgfs.c+2−1
@@ -278,7 +278,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len, · 277 unmodified lines
278 goto fail;278 goto fail;
279 279
280 pos += parsed;280 pos += parsed;
281 scheme = damon_new_scheme(&pattern, action, &quota, &wmarks);281 scheme = damon_new_scheme(&pattern, action, 0, &quota,
282 &wmarks);
282 if (!scheme)283 if (!scheme)
283 goto fail;284 goto fail;
284 285
mm/damon/lru_sort.c+3−3
@@ -158,6 +158,8 @@ static struct damos *damon_lru_sort_new_scheme( · 157 unmodified lines
158 pattern,158 pattern,
159 /* (de)prioritize on LRU-lists */159 /* (de)prioritize on LRU-lists */
160 action,160 action,
161 /* for each aggregation interval */
162 0,
161 /* under the quota. */163 /* under the quota. */
162 &quota,164 &quota,
163 /* (De)activate this according to the watermarks. */165 /* (De)activate this according to the watermarks. */
@@ -193,9 +195,7 @@ static int damon_lru_sort_apply_parameters(void) · 29 unmodified lines
193 if (err)195 if (err)
194 return err;196 return err;
195 197
196 /* aggr_interval / sample_interval is the maximum nr_accesses */198 hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) *
197 hot_thres = damon_lru_sort_mon_attrs.aggr_interval /
198 damon_lru_sort_mon_attrs.sample_interval *
199 hot_thres_access_freq / 1000;199 hot_thres_access_freq / 1000;
200 scheme = damon_lru_sort_new_hot_scheme(hot_thres);200 scheme = damon_lru_sort_new_hot_scheme(hot_thres);
201 if (!scheme)201 if (!scheme)
mm/damon/ops-common.c+2−3
@@ -73,7 +73,6 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr · 72 unmodified lines
73int damon_hot_score(struct damon_ctx *c, struct damon_region *r,73int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
74 struct damos *s)74 struct damos *s)
75{75{
76 unsigned int max_nr_accesses;
77 int freq_subscore;76 int freq_subscore;
78 unsigned int age_in_sec;77 unsigned int age_in_sec;
79 int age_in_log, age_subscore;78 int age_in_log, age_subscore;
@@ -81,8 +80,8 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, · 1 unmodified line
81 unsigned int age_weight = s->quota.weight_age;80 unsigned int age_weight = s->quota.weight_age;
82 int hotness;81 int hotness;
83 82
84 max_nr_accesses = c->attrs.aggr_interval / c->attrs.sample_interval;83 freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE /
85 freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / max_nr_accesses;84 damon_max_nr_accesses(&c->attrs);
86 85
87 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;86 age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
88 for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;87 for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
mm/damon/paddr.c+5−6
@@ -148,7 +148,8 @@ out: · 147 unmodified lines
148 return accessed;148 return accessed;
149}149}
150 150
151static void __damon_pa_check_access(struct damon_region *r)151static void __damon_pa_check_access(struct damon_region *r,
152 struct damon_attrs *attrs)
152{153{
153 static unsigned long last_addr;154 static unsigned long last_addr;
154 static unsigned long last_folio_sz = PAGE_SIZE;155 static unsigned long last_folio_sz = PAGE_SIZE;
@@ -157,14 +158,12 @@ static void __damon_pa_check_access(struct damon_region *r) · 2 unmodified lines
157 /* If the region is in the last checked page, reuse the result */158 /* If the region is in the last checked page, reuse the result */
158 if (ALIGN_DOWN(last_addr, last_folio_sz) ==159 if (ALIGN_DOWN(last_addr, last_folio_sz) ==
159 ALIGN_DOWN(r->sampling_addr, last_folio_sz)) {160 ALIGN_DOWN(r->sampling_addr, last_folio_sz)) {
160 if (last_accessed)161 damon_update_region_access_rate(r, last_accessed, attrs);
161 r->nr_accesses++;
162 return;162 return;
163 }163 }
164 164
165 last_accessed = damon_pa_young(r->sampling_addr, &last_folio_sz);165 last_accessed = damon_pa_young(r->sampling_addr, &last_folio_sz);
166 if (last_accessed)166 damon_update_region_access_rate(r, last_accessed, attrs);
167 r->nr_accesses++;
168 167
169 last_addr = r->sampling_addr;168 last_addr = r->sampling_addr;
170}169}
@@ -177,7 +176,7 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx) · 6 unmodified lines
177 176
178 damon_for_each_target(t, ctx) {177 damon_for_each_target(t, ctx) {
179 damon_for_each_region(r, t) {178 damon_for_each_region(r, t) {
180 __damon_pa_check_access(r);179 __damon_pa_check_access(r, &ctx->attrs);
181 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);180 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
182 }181 }
183 }182 }
mm/damon/reclaim.c+2−0
@@ -142,6 +142,8 @@ static struct damos *damon_reclaim_new_scheme(void) · 141 unmodified lines
142 &pattern,142 &pattern,
143 /* page out those, as soon as found */143 /* page out those, as soon as found */
144 DAMOS_PAGEOUT,144 DAMOS_PAGEOUT,
145 /* for each aggregation interval */
146 0,
145 /* under the quota. */147 /* under the quota. */
146 &damon_reclaim_quota,148 &damon_reclaim_quota,
147 /* (De)activate this according to the watermarks. */149 /* (De)activate this according to the watermarks. */
mm/damon/sysfs-common.h+2−0
@@ -49,6 +49,8 @@ int damon_sysfs_schemes_update_regions_start( · 48 unmodified lines
49 struct damon_sysfs_schemes *sysfs_schemes,49 struct damon_sysfs_schemes *sysfs_schemes,
50 struct damon_ctx *ctx, bool total_bytes_only);50 struct damon_ctx *ctx, bool total_bytes_only);
51 51
52bool damos_sysfs_regions_upd_done(void);
53
52int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx);54int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx);
53 55
54int damon_sysfs_schemes_clear_regions(56int damon_sysfs_schemes_clear_regions(
mm/damon/sysfs-schemes.c+128−5
@@ -31,7 +31,7 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc( · 30 unmodified lines
31 return NULL;31 return NULL;
32 sysfs_region->kobj = (struct kobject){};32 sysfs_region->kobj = (struct kobject){};
33 sysfs_region->ar = region->ar;33 sysfs_region->ar = region->ar;
34 sysfs_region->nr_accesses = region->nr_accesses;34 sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;
35 sysfs_region->age = region->age;35 sysfs_region->age = region->age;
36 INIT_LIST_HEAD(&sysfs_region->list);36 INIT_LIST_HEAD(&sysfs_region->list);
37 return sysfs_region;37 return sysfs_region;
@@ -113,11 +113,47 @@ static const struct kobj_type damon_sysfs_scheme_region_ktype = { · 75 unmodified lines
113 * scheme regions directory113 * scheme regions directory
114 */114 */
115 115
116/*
117 * enum damos_sysfs_regions_upd_status - Represent DAMOS tried regions update
118 * status
119 * @DAMOS_TRIED_REGIONS_UPD_IDLE: Waiting for next request.
120 * @DAMOS_TRIED_REGIONS_UPD_STARTED: Update started.
121 * @DAMOS_TRIED_REGIONS_UPD_FINISHED: Update finished.
122 *
123 * Each DAMON-based operation scheme (&struct damos) has its own apply
124 * interval, and we need to expose the scheme tried regions based on only
125 * single snapshot. For this, we keep the tried regions update status for each
126 * scheme. The status becomes 'idle' at the beginning.
127 *
128 * Once the tried regions update request is received, the request handling
129 * start function (damon_sysfs_scheme_update_regions_start()) sets the status
130 * of all schemes as 'idle' again, and register ->before_damos_apply() and
131 * ->after_sampling() callbacks.
132 *
133 * Then, the first followup ->before_damos_apply() callback
134 * (damon_sysfs_before_damos_apply()) sets the status 'started'. The first
135 * ->after_sampling() callback (damon_sysfs_after_sampling()) after the call
136 * is called only after the scheme is completely applied
137 * to the given snapshot. Hence the callback knows the situation by showing
138 * 'started' status, and sets the status as 'finished'. Then,
139 * damon_sysfs_before_damos_apply() understands the situation by showing the
140 * 'finished' status and do nothing.
141 *
142 * Finally, the tried regions request handling finisher function
143 * (damon_sysfs_schemes_update_regions_stop()) unregisters the callbacks.
144 */
145enum damos_sysfs_regions_upd_status {
146 DAMOS_TRIED_REGIONS_UPD_IDLE,
147 DAMOS_TRIED_REGIONS_UPD_STARTED,
148 DAMOS_TRIED_REGIONS_UPD_FINISHED,
149};
150
116struct damon_sysfs_scheme_regions {151struct damon_sysfs_scheme_regions {
117 struct kobject kobj;152 struct kobject kobj;
118 struct list_head regions_list;153 struct list_head regions_list;
119 int nr_regions;154 int nr_regions;
120 unsigned long total_bytes;155 unsigned long total_bytes;
156 enum damos_sysfs_regions_upd_status upd_status;
121};157};
122 158
123static struct damon_sysfs_scheme_regions *159static struct damon_sysfs_scheme_regions *
@@ -130,6 +166,7 @@ damon_sysfs_scheme_regions_alloc(void) · 6 unmodified lines
130 INIT_LIST_HEAD(&regions->regions_list);166 INIT_LIST_HEAD(&regions->regions_list);
131 regions->nr_regions = 0;167 regions->nr_regions = 0;
132 regions->total_bytes = 0;168 regions->total_bytes = 0;
169 regions->upd_status = DAMOS_TRIED_REGIONS_UPD_IDLE;
133 return regions;170 return regions;
134}171}
135 172
@@ -1121,6 +1158,7 @@ struct damon_sysfs_scheme { · 985 unmodified lines
1121 struct kobject kobj;1158 struct kobject kobj;
1122 enum damos_action action;1159 enum damos_action action;
1123 struct damon_sysfs_access_pattern *access_pattern;1160 struct damon_sysfs_access_pattern *access_pattern;
1161 unsigned long apply_interval_us;
1124 struct damon_sysfs_quotas *quotas;1162 struct damon_sysfs_quotas *quotas;
1125 struct damon_sysfs_watermarks *watermarks;1163 struct damon_sysfs_watermarks *watermarks;
1126 struct damon_sysfs_scheme_filters *filters;1164 struct damon_sysfs_scheme_filters *filters;
@@ -1141,7 +1179,7 @@ static const char * const damon_sysfs_damos_action_strs[] = { · 14 unmodified lines
1141};1179};
1142 1180
1143static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc(1181static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc(
1144 enum damos_action action)1182 enum damos_action action, unsigned long apply_interval_us)
1145{1183{
1146 struct damon_sysfs_scheme *scheme = kmalloc(sizeof(*scheme),1184 struct damon_sysfs_scheme *scheme = kmalloc(sizeof(*scheme),
1147 GFP_KERNEL);1185 GFP_KERNEL);
@@ -1150,6 +1188,7 @@ static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc( · 2 unmodified lines
1150 return NULL;1188 return NULL;
1151 scheme->kobj = (struct kobject){};1189 scheme->kobj = (struct kobject){};
1152 scheme->action = action;1190 scheme->action = action;
1191 scheme->apply_interval_us = apply_interval_us;
1153 return scheme;1192 return scheme;
1154}1193}
1155 1194
@@ -1353,6 +1392,25 @@ static ssize_t action_store(struct kobject *kobj, struct kobj_attribute *attr, · 197 unmodified lines
1353 return -EINVAL;1392 return -EINVAL;
1354}1393}
1355 1394
1395static ssize_t apply_interval_us_show(struct kobject *kobj,
1396 struct kobj_attribute *attr, char *buf)
1397{
1398 struct damon_sysfs_scheme *scheme = container_of(kobj,
1399 struct damon_sysfs_scheme, kobj);
1400
1401 return sysfs_emit(buf, "%lu\n", scheme->apply_interval_us);
1402}
1403
1404static ssize_t apply_interval_us_store(struct kobject *kobj,
1405 struct kobj_attribute *attr, const char *buf, size_t count)
1406{
1407 struct damon_sysfs_scheme *scheme = container_of(kobj,
1408 struct damon_sysfs_scheme, kobj);
1409 int err = kstrtoul(buf, 0, &scheme->apply_interval_us);
1410
1411 return err ? err : count;
1412}
1413
1356static void damon_sysfs_scheme_release(struct kobject *kobj)1414static void damon_sysfs_scheme_release(struct kobject *kobj)
1357{1415{
1358 kfree(container_of(kobj, struct damon_sysfs_scheme, kobj));1416 kfree(container_of(kobj, struct damon_sysfs_scheme, kobj));
@@ -1361,8 +1419,12 @@ static void damon_sysfs_scheme_release(struct kobject *kobj) · 2 unmodified lines
1361static struct kobj_attribute damon_sysfs_scheme_action_attr =1419static struct kobj_attribute damon_sysfs_scheme_action_attr =
1362 __ATTR_RW_MODE(action, 0600);1420 __ATTR_RW_MODE(action, 0600);
1363 1421
1422static struct kobj_attribute damon_sysfs_scheme_apply_interval_us_attr =
1423 __ATTR_RW_MODE(apply_interval_us, 0600);
1424
1364static struct attribute *damon_sysfs_scheme_attrs[] = {1425static struct attribute *damon_sysfs_scheme_attrs[] = {
1365 &damon_sysfs_scheme_action_attr.attr,1426 &damon_sysfs_scheme_action_attr.attr,
1427 &damon_sysfs_scheme_apply_interval_us_attr.attr,
1366 NULL,1428 NULL,
1367};1429};
1368ATTRIBUTE_GROUPS(damon_sysfs_scheme);1430ATTRIBUTE_GROUPS(damon_sysfs_scheme);
@@ -1413,7 +1475,11 @@ static int damon_sysfs_schemes_add_dirs(struct damon_sysfs_schemes *schemes, · 44 unmodified lines
1413 schemes->schemes_arr = schemes_arr;1475 schemes->schemes_arr = schemes_arr;
1414 1476
1415 for (i = 0; i < nr_schemes; i++) {1477 for (i = 0; i < nr_schemes; i++) {
1416 scheme = damon_sysfs_scheme_alloc(DAMOS_STAT);1478 /*
1479 * apply_interval_us as 0 means same to aggregation interval
1480 * (same to before-apply_interval behavior)
1481 */
1482 scheme = damon_sysfs_scheme_alloc(DAMOS_STAT, 0);
1417 if (!scheme) {1483 if (!scheme) {
1418 damon_sysfs_schemes_rm_dirs(schemes);1484 damon_sysfs_schemes_rm_dirs(schemes);
1419 return -ENOMEM;1485 return -ENOMEM;
@@ -1610,8 +1676,8 @@ static struct damos *damon_sysfs_mk_scheme( · 190 unmodified lines
1610 .low = sysfs_wmarks->low,1676 .low = sysfs_wmarks->low,
1611 };1677 };
1612 1678
1613 scheme = damon_new_scheme(&pattern, sysfs_scheme->action, &quota,1679 scheme = damon_new_scheme(&pattern, sysfs_scheme->action,
1614 &wmarks);1680 sysfs_scheme->apply_interval_us, &quota, &wmarks);
1615 if (!scheme)1681 if (!scheme)
1616 return NULL;1682 return NULL;
1617 1683
@@ -1641,6 +1707,7 @@ static void damon_sysfs_update_scheme(struct damos *scheme, · 23 unmodified lines
1641 scheme->pattern.max_age_region = access_pattern->age->max;1707 scheme->pattern.max_age_region = access_pattern->age->max;
1642 1708
1643 scheme->action = sysfs_scheme->action;1709 scheme->action = sysfs_scheme->action;
1710 scheme->apply_interval_us = sysfs_scheme->apply_interval_us;
1644 1711
1645 scheme->quota.ms = sysfs_quotas->ms;1712 scheme->quota.ms = sysfs_quotas->ms;
1646 scheme->quota.sz = sysfs_quotas->sz;1713 scheme->quota.sz = sysfs_quotas->sz;
@@ -1747,6 +1814,10 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 100 unmodified lines
1747 return 0;1814 return 0;
1748 1815
1749 sysfs_regions = sysfs_schemes->schemes_arr[schemes_idx]->tried_regions;1816 sysfs_regions = sysfs_schemes->schemes_arr[schemes_idx]->tried_regions;
1817 if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_FINISHED)
1818 return 0;
1819 if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_IDLE)
1820 sysfs_regions->upd_status = DAMOS_TRIED_REGIONS_UPD_STARTED;
1750 sysfs_regions->total_bytes += r->ar.end - r->ar.start;1821 sysfs_regions->total_bytes += r->ar.end - r->ar.start;
1751 if (damos_regions_upd_total_bytes_only)1822 if (damos_regions_upd_total_bytes_only)
1752 return 0;1823 return 0;
@@ -1763,6 +1834,29 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 10 unmodified lines
1763 return 0;1834 return 0;
1764}1835}
1765 1836
1837/*
1838 * DAMON callback that called after each accesses sampling. While this
1839 * callback is registered, damon_sysfs_lock should be held to ensure the
1840 * regions directories exist.
1841 */
1842static int damon_sysfs_after_sampling(struct damon_ctx *ctx)
1843{
1844 struct damon_sysfs_schemes *sysfs_schemes =
1845 damon_sysfs_schemes_for_damos_callback;
1846 struct damon_sysfs_scheme_regions *sysfs_regions;
1847 int i;
1848
1849 for (i = 0; i < sysfs_schemes->nr; i++) {
1850 sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions;
1851 if (sysfs_regions->upd_status ==
1852 DAMOS_TRIED_REGIONS_UPD_STARTED)
1853 sysfs_regions->upd_status =
1854 DAMOS_TRIED_REGIONS_UPD_FINISHED;
1855 }
1856
1857 return 0;
1858}
1859
1766/* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */1860/* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */
1767int damon_sysfs_schemes_clear_regions(1861int damon_sysfs_schemes_clear_regions(
1768 struct damon_sysfs_schemes *sysfs_schemes,1862 struct damon_sysfs_schemes *sysfs_schemes,
@@ -1786,6 +1880,16 @@ int damon_sysfs_schemes_clear_regions( · 17 unmodified lines
1786 return 0;1880 return 0;
1787}1881}
1788 1882
1883static void damos_tried_regions_init_upd_status(
1884 struct damon_sysfs_schemes *sysfs_schemes)
1885{
1886 int i;
1887
1888 for (i = 0; i < sysfs_schemes->nr; i++)
1889 sysfs_schemes->schemes_arr[i]->tried_regions->upd_status =
1890 DAMOS_TRIED_REGIONS_UPD_IDLE;
1891}
1892
1789/* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */1893/* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */
1790int damon_sysfs_schemes_update_regions_start(1894int damon_sysfs_schemes_update_regions_start(
1791 struct damon_sysfs_schemes *sysfs_schemes,1895 struct damon_sysfs_schemes *sysfs_schemes,
@@ -1793,11 +1897,29 @@ int damon_sysfs_schemes_update_regions_start( · 1 unmodified line
1793{1897{
1794 damon_sysfs_schemes_clear_regions(sysfs_schemes, ctx);1898 damon_sysfs_schemes_clear_regions(sysfs_schemes, ctx);
1795 damon_sysfs_schemes_for_damos_callback = sysfs_schemes;1899 damon_sysfs_schemes_for_damos_callback = sysfs_schemes;
1900 damos_tried_regions_init_upd_status(sysfs_schemes);
1796 damos_regions_upd_total_bytes_only = total_bytes_only;1901 damos_regions_upd_total_bytes_only = total_bytes_only;
1797 ctx->callback.before_damos_apply = damon_sysfs_before_damos_apply;1902 ctx->callback.before_damos_apply = damon_sysfs_before_damos_apply;
1903 ctx->callback.after_sampling = damon_sysfs_after_sampling;
1798 return 0;1904 return 0;
1799}1905}
1800 1906
1907bool damos_sysfs_regions_upd_done(void)
1908{
1909 struct damon_sysfs_schemes *sysfs_schemes =
1910 damon_sysfs_schemes_for_damos_callback;
1911 struct damon_sysfs_scheme_regions *sysfs_regions;
1912 int i;
1913
1914 for (i = 0; i < sysfs_schemes->nr; i++) {
1915 sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions;
1916 if (sysfs_regions->upd_status !=
1917 DAMOS_TRIED_REGIONS_UPD_FINISHED)
1918 return false;
1919 }
1920 return true;
1921}
1922
1801/*1923/*
1802 * Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock. Caller1924 * Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock. Caller
1803 * should unlock damon_sysfs_lock which held before1925 * should unlock damon_sysfs_lock which held before
@@ -1807,6 +1929,7 @@ int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx) · 3 unmodified lines
1807{1929{
1808 damon_sysfs_schemes_for_damos_callback = NULL;1930 damon_sysfs_schemes_for_damos_callback = NULL;
1809 ctx->callback.before_damos_apply = NULL;1931 ctx->callback.before_damos_apply = NULL;
1932 ctx->callback.after_sampling = NULL;
1810 damon_sysfs_schemes_region_idx = 0;1933 damon_sysfs_schemes_region_idx = 0;
1811 return 0;1934 return 0;
1812}1935}
mm/damon/sysfs-test.h+86−0
@@ -0,0 +1,86 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2/*
3 * Data Access Monitor Unit Tests
4 *
5 * Author: SeongJae Park <sj@kernel.org>
6 */
7
8#ifdef CONFIG_DAMON_SYSFS_KUNIT_TEST
9
10#ifndef _DAMON_SYSFS_TEST_H
11#define _DAMON_SYSFS_TEST_H
12
13#include <kunit/test.h>
14
15static unsigned int nr_damon_targets(struct damon_ctx *ctx)
16{
17 struct damon_target *t;
18 unsigned int nr_targets = 0;
19
20 damon_for_each_target(t, ctx)
21 nr_targets++;
22
23 return nr_targets;
24}
25
26static int __damon_sysfs_test_get_any_pid(int min, int max)
27{
28 struct pid *pid;
29 int i;
30
31 for (i = min; i <= max; i++) {
32 pid = find_get_pid(i);
33 if (pid) {
34 put_pid(pid);
35 return i;
36 }
37 }
38 return -1;
39}
40
41static void damon_sysfs_test_set_targets(struct kunit *test)
42{
43 struct damon_sysfs_targets *sysfs_targets;
44 struct damon_sysfs_target *sysfs_target;
45 struct damon_ctx *ctx;
46
47 sysfs_targets = damon_sysfs_targets_alloc();
48 sysfs_targets->nr = 1;
49 sysfs_targets->targets_arr = kmalloc_array(1,
50 sizeof(*sysfs_targets->targets_arr), GFP_KERNEL);
51
52 sysfs_target = damon_sysfs_target_alloc();
53 sysfs_target->pid = __damon_sysfs_test_get_any_pid(12, 100);
54 sysfs_target->regions = damon_sysfs_regions_alloc();
55 sysfs_targets->targets_arr[0] = sysfs_target;
56
57 ctx = damon_new_ctx();
58
59 damon_sysfs_set_targets(ctx, sysfs_targets);
60 KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));
61
62 sysfs_target->pid = __damon_sysfs_test_get_any_pid(
63 sysfs_target->pid + 1, 200);
64 damon_sysfs_set_targets(ctx, sysfs_targets);
65 KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));
66
67 damon_destroy_ctx(ctx);
68 kfree(sysfs_targets->targets_arr);
69 kfree(sysfs_targets);
70 kfree(sysfs_target);
71}
72
73static struct kunit_case damon_sysfs_test_cases[] = {
74 KUNIT_CASE(damon_sysfs_test_set_targets),
75 {},
76};
77
78static struct kunit_suite damon_sysfs_test_suite = {
79 .name = "damon-sysfs",
80 .test_cases = damon_sysfs_test_cases,
81};
82kunit_test_suite(damon_sysfs_test_suite);
83
84#endif /* _DAMON_SYSFS_TEST_H */
85
86#endif /* CONFIG_DAMON_SYSFS_KUNIT_TEST */
mm/damon/sysfs.c+83−40
@@ -1150,58 +1150,73 @@ destroy_targets_out: · 1149 unmodified lines
1150 return err;1150 return err;
1151}1151}
1152 1152
1153/*1153static int damon_sysfs_update_target_pid(struct damon_target *target, int pid)
1154 * Search a target in a context that corresponds to the sysfs target input.
1155 *
1156 * Return: pointer to the target if found, NULL if not found, or negative
1157 * error code if the search failed.
1158 */
1159static struct damon_target *damon_sysfs_existing_target(
1160 struct damon_sysfs_target *sys_target, struct damon_ctx *ctx)
1161{1154{
1162 struct pid *pid;1155 struct pid *pid_new;
1163 struct damon_target *t;
1164 1156
1165 if (!damon_target_has_pid(ctx)) {1157 pid_new = find_get_pid(pid);
1166 /* Up to only one target for paddr could exist */1158 if (!pid_new)
1167 damon_for_each_target(t, ctx)1159 return -EINVAL;
1168 return t;1160
1169 return NULL;1161 if (pid_new == target->pid) {
1162 put_pid(pid_new);
1163 return 0;
1170 }1164 }
1171 1165
1172 /* ops.id should be DAMON_OPS_VADDR or DAMON_OPS_FVADDR */1166 put_pid(target->pid);
1173 pid = find_get_pid(sys_target->pid);1167 target->pid = pid_new;
1174 if (!pid)1168 return 0;
1175 return ERR_PTR(-EINVAL);1169}
1176 damon_for_each_target(t, ctx) {1170
1177 if (t->pid == pid) {1171static int damon_sysfs_update_target(struct damon_target *target,
1178 put_pid(pid);1172 struct damon_ctx *ctx,
1179 return t;1173 struct damon_sysfs_target *sys_target)
1180 }1174{
1175 int err;
1176
1177 if (damon_target_has_pid(ctx)) {
1178 err = damon_sysfs_update_target_pid(target, sys_target->pid);
1179 if (err)
1180 return err;
1181 }1181 }
1182 put_pid(pid);1182
1183 return NULL;1183 /*
1184 * Do monitoring target region boundary update only if one or more
1185 * regions are set by the user. This is for keeping current monitoring
1186 * target results and range easier, especially for dynamic monitoring
1187 * target regions update ops like 'vaddr'.
1188 */
1189 if (sys_target->regions->nr)
1190 err = damon_sysfs_set_regions(target, sys_target->regions);
1191 return err;
1184}1192}
1185 1193
1186static int damon_sysfs_set_targets(struct damon_ctx *ctx,1194static int damon_sysfs_set_targets(struct damon_ctx *ctx,
1187 struct damon_sysfs_targets *sysfs_targets)1195 struct damon_sysfs_targets *sysfs_targets)
1188{1196{
1189 int i, err;1197 struct damon_target *t, *next;
1198 int i = 0, err;
1190 1199
1191 /* Multiple physical address space monitoring targets makes no sense */1200 /* Multiple physical address space monitoring targets makes no sense */
1192 if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1)1201 if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1)
1193 return -EINVAL;1202 return -EINVAL;
1194 1203
1195 for (i = 0; i < sysfs_targets->nr; i++) {1204 damon_for_each_target_safe(t, next, ctx) {
1205 if (i < sysfs_targets->nr) {
1206 damon_sysfs_update_target(t, ctx,
1207 sysfs_targets->targets_arr[i]);
1208 } else {
1209 if (damon_target_has_pid(ctx))
1210 put_pid(t->pid);
1211 damon_destroy_target(t);
1212 }
1213 i++;
1214 }
1215
1216 for (; i < sysfs_targets->nr; i++) {
1196 struct damon_sysfs_target *st = sysfs_targets->targets_arr[i];1217 struct damon_sysfs_target *st = sysfs_targets->targets_arr[i];
1197 struct damon_target *t = damon_sysfs_existing_target(st, ctx);1218
1198 1219 err = damon_sysfs_add_target(st, ctx);
1199 if (IS_ERR(t))
1200 return PTR_ERR(t);
1201 if (!t)
1202 err = damon_sysfs_add_target(st, ctx);
1203 else
1204 err = damon_sysfs_set_regions(t, st->regions);
1205 if (err)1220 if (err)
1206 return err;1221 return err;
1207 }1222 }
@@ -1336,12 +1351,13 @@ static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond) · 128 unmodified lines
1336 1351
1337/*1352/*
1338 * damon_sysfs_cmd_request_callback() - DAMON callback for handling requests.1353 * damon_sysfs_cmd_request_callback() - DAMON callback for handling requests.
1339 * @c: The DAMON context of the callback.1354 * @c: The DAMON context of the callback.
1355 * @active: Whether @c is not deactivated due to watermarks.
1340 *1356 *
1341 * This function is periodically called back from the kdamond thread for @c.1357 * This function is periodically called back from the kdamond thread for @c.
1342 * Then, it checks if there is a waiting DAMON sysfs request and handles it.1358 * Then, it checks if there is a waiting DAMON sysfs request and handles it.
1343 */1359 */
1344static int damon_sysfs_cmd_request_callback(struct damon_ctx *c)1360static int damon_sysfs_cmd_request_callback(struct damon_ctx *c, bool active)
1345{1361{
1346 struct damon_sysfs_kdamond *kdamond;1362 struct damon_sysfs_kdamond *kdamond;
1347 bool total_bytes_only = false;1363 bool total_bytes_only = false;
@@ -1373,6 +1389,13 @@ static int damon_sysfs_cmd_request_callback(struct damon_ctx *c) · 25 unmodified lines
1373 goto keep_lock_out;1389 goto keep_lock_out;
1374 }1390 }
1375 } else {1391 } else {
1392 /*
1393 * Continue regions updating if DAMON is till
1394 * active and the update for all schemes is not
1395 * finished.
1396 */
1397 if (active && !damos_sysfs_regions_upd_done())
1398 goto keep_lock_out;
1376 err = damon_sysfs_upd_schemes_regions_stop(kdamond);1399 err = damon_sysfs_upd_schemes_regions_stop(kdamond);
1377 damon_sysfs_schemes_regions_updating = false;1400 damon_sysfs_schemes_regions_updating = false;
1378 }1401 }
@@ -1392,6 +1415,24 @@ keep_lock_out: · 13 unmodified lines
1392 return err;1415 return err;
1393}1416}
1394 1417
1418static int damon_sysfs_after_wmarks_check(struct damon_ctx *c)
1419{
1420 /*
1421 * after_wmarks_check() is called back while the context is deactivated
1422 * by watermarks.
1423 */
1424 return damon_sysfs_cmd_request_callback(c, false);
1425}
1426
1427static int damon_sysfs_after_aggregation(struct damon_ctx *c)
1428{
1429 /*
1430 * after_aggregation() is called back only while the context is not
1431 * deactivated by watermarks.
1432 */
1433 return damon_sysfs_cmd_request_callback(c, true);
1434}
1435
1395static struct damon_ctx *damon_sysfs_build_ctx(1436static struct damon_ctx *damon_sysfs_build_ctx(
1396 struct damon_sysfs_context *sys_ctx)1437 struct damon_sysfs_context *sys_ctx)
1397{1438{
@@ -1407,8 +1448,8 @@ static struct damon_ctx *damon_sysfs_build_ctx( · 9 unmodified lines
1407 return ERR_PTR(err);1448 return ERR_PTR(err);
1408 }1449 }
1409 1450
1410 ctx->callback.after_wmarks_check = damon_sysfs_cmd_request_callback;1451 ctx->callback.after_wmarks_check = damon_sysfs_after_wmarks_check;
1411 ctx->callback.after_aggregation = damon_sysfs_cmd_request_callback;1452 ctx->callback.after_aggregation = damon_sysfs_after_aggregation;
1412 ctx->callback.before_terminate = damon_sysfs_before_terminate;1453 ctx->callback.before_terminate = damon_sysfs_before_terminate;
1413 return ctx;1454 return ctx;
1414}1455}
@@ -1810,3 +1851,5 @@ out: · 395 unmodified lines
1810 return err;1851 return err;
1811}1852}
1812subsys_initcall(damon_sysfs_init);1853subsys_initcall(damon_sysfs_init);
1854
1855#include "sysfs-test.h"
mm/damon/vaddr.c+13−9
@@ -558,23 +558,27 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, · 557 unmodified lines
558 * r the region to be checked558 * r the region to be checked
559 */559 */
560static void __damon_va_check_access(struct mm_struct *mm,560static void __damon_va_check_access(struct mm_struct *mm,
561 struct damon_region *r, bool same_target)561 struct damon_region *r, bool same_target,
562 struct damon_attrs *attrs)
562{563{
563 static unsigned long last_addr;564 static unsigned long last_addr;
564 static unsigned long last_folio_sz = PAGE_SIZE;565 static unsigned long last_folio_sz = PAGE_SIZE;
565 static bool last_accessed;566 static bool last_accessed;
566 567
568 if (!mm) {
569 damon_update_region_access_rate(r, false, attrs);
570 return;
571 }
572
567 /* If the region is in the last checked page, reuse the result */573 /* If the region is in the last checked page, reuse the result */
568 if (same_target && (ALIGN_DOWN(last_addr, last_folio_sz) ==574 if (same_target && (ALIGN_DOWN(last_addr, last_folio_sz) ==
569 ALIGN_DOWN(r->sampling_addr, last_folio_sz))) {575 ALIGN_DOWN(r->sampling_addr, last_folio_sz))) {
570 if (last_accessed)576 damon_update_region_access_rate(r, last_accessed, attrs);
571 r->nr_accesses++;
572 return;577 return;
573 }578 }
574 579
575 last_accessed = damon_va_young(mm, r->sampling_addr, &last_folio_sz);580 last_accessed = damon_va_young(mm, r->sampling_addr, &last_folio_sz);
576 if (last_accessed)581 damon_update_region_access_rate(r, last_accessed, attrs);
577 r->nr_accesses++;
578 582
579 last_addr = r->sampling_addr;583 last_addr = r->sampling_addr;
580}584}
@@ -589,15 +593,15 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx) · 8 unmodified lines
589 593
590 damon_for_each_target(t, ctx) {594 damon_for_each_target(t, ctx) {
591 mm = damon_get_mm(t);595 mm = damon_get_mm(t);
592 if (!mm)
593 continue;
594 same_target = false;596 same_target = false;
595 damon_for_each_region(r, t) {597 damon_for_each_region(r, t) {
596 __damon_va_check_access(mm, r, same_target);598 __damon_va_check_access(mm, r, same_target,
599 &ctx->attrs);
597 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);600 max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
598 same_target = true;601 same_target = true;
599 }602 }
600 mmput(mm);603 if (mm)
604 mmput(mm);
601 }605 }
602 606
603 return max_nr_accesses;607 return max_nr_accesses;
mm/debug_vm_pgtable.c+2−2
@@ -1322,8 +1322,8 @@ static int __init debug_vm_pgtable(void) · 1321 unmodified lines
1322 * true irrespective of the starting protection value for a1322 * true irrespective of the starting protection value for a
1323 * given page table entry.1323 * given page table entry.
1324 *1324 *
1325 * Protection based vm_flags combinatins are always linear1325 * Protection based vm_flags combinations are always linear
1326 * and increasing i.e starting from VM_NONE and going upto1326 * and increasing i.e starting from VM_NONE and going up to
1327 * (VM_SHARED | READ | WRITE | EXEC).1327 * (VM_SHARED | READ | WRITE | EXEC).
1328 */1328 */
1329#define VM_FLAGS_START (VM_NONE)1329#define VM_FLAGS_START (VM_NONE)
mm/filemap.c+82−133
@@ -131,11 +131,8 @@ static void page_cache_delete(struct address_space *mapping, · 130 unmodified lines
131 131
132 mapping_set_update(&xas, mapping);132 mapping_set_update(&xas, mapping);
133 133
134 /* hugetlb pages are represented by a single entry in the xarray */134 xas_set_order(&xas, folio->index, folio_order(folio));
135 if (!folio_test_hugetlb(folio)) {135 nr = folio_nr_pages(folio);
136 xas_set_order(&xas, folio->index, folio_order(folio));
137 nr = folio_nr_pages(folio);
138 }
139 136
140 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);137 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
141 138
@@ -234,7 +231,7 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio) · 92 unmodified lines
234 if (free_folio)231 if (free_folio)
235 free_folio(folio);232 free_folio(folio);
236 233
237 if (folio_test_large(folio) && !folio_test_hugetlb(folio))234 if (folio_test_large(folio))
238 refs = folio_nr_pages(folio);235 refs = folio_nr_pages(folio);
239 folio_put_refs(folio, refs);236 folio_put_refs(folio, refs);
240}237}
@@ -819,7 +816,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) · 578 unmodified lines
819 new->mapping = mapping;816 new->mapping = mapping;
820 new->index = offset;817 new->index = offset;
821 818
822 mem_cgroup_migrate(old, new);819 mem_cgroup_replace_folio(old, new);
823 820
824 xas_lock_irq(&xas);821 xas_lock_irq(&xas);
825 xas_store(&xas, new);822 xas_store(&xas, new);
@@ -855,14 +852,15 @@ noinline int __filemap_add_folio(struct address_space *mapping, · 29 unmodified lines
855 852
856 if (!huge) {853 if (!huge) {
857 int error = mem_cgroup_charge(folio, NULL, gfp);854 int error = mem_cgroup_charge(folio, NULL, gfp);
858 VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio);
859 if (error)855 if (error)
860 return error;856 return error;
861 charged = true;857 charged = true;
862 xas_set_order(&xas, index, folio_order(folio));
863 nr = folio_nr_pages(folio);
864 }858 }
865 859
860 VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio);
861 xas_set_order(&xas, index, folio_order(folio));
862 nr = folio_nr_pages(folio);
863
866 gfp &= GFP_RECLAIM_MASK;864 gfp &= GFP_RECLAIM_MASK;
867 folio_ref_add(folio, nr);865 folio_ref_add(folio, nr);
868 folio->mapping = mapping;866 folio->mapping = mapping;
@@ -1135,32 +1133,13 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 266 unmodified lines
1135 wait_queue_head_t *q = folio_waitqueue(folio);1133 wait_queue_head_t *q = folio_waitqueue(folio);
1136 struct wait_page_key key;1134 struct wait_page_key key;
1137 unsigned long flags;1135 unsigned long flags;
1138 wait_queue_entry_t bookmark;
1139 1136
1140 key.folio = folio;1137 key.folio = folio;
1141 key.bit_nr = bit_nr;1138 key.bit_nr = bit_nr;
1142 key.page_match = 0;1139 key.page_match = 0;
1143 1140
1144 bookmark.flags = 0;
1145 bookmark.private = NULL;
1146 bookmark.func = NULL;
1147 INIT_LIST_HEAD(&bookmark.entry);
1148
1149 spin_lock_irqsave(&q->lock, flags);1141 spin_lock_irqsave(&q->lock, flags);
1150 __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark);1142 __wake_up_locked_key(q, TASK_NORMAL, &key);
1151
1152 while (bookmark.flags & WQ_FLAG_BOOKMARK) {
1153 /*
1154 * Take a breather from holding the lock,
1155 * allow pages that finish wake up asynchronously
1156 * to acquire the lock and remove themselves
1157 * from wait queue
1158 */
1159 spin_unlock_irqrestore(&q->lock, flags);
1160 cpu_relax();
1161 spin_lock_irqsave(&q->lock, flags);
1162 __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark);
1163 }
1164 1143
1165 /*1144 /*
1166 * It's possible to miss clearing waiters here, when we woke our page1145 * It's possible to miss clearing waiters here, when we woke our page
@@ -1177,13 +1156,6 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 10 unmodified lines
1177 spin_unlock_irqrestore(&q->lock, flags);1156 spin_unlock_irqrestore(&q->lock, flags);
1178}1157}
1179 1158
1180static void folio_wake(struct folio *folio, int bit)
1181{
1182 if (!folio_test_waiters(folio))
1183 return;
1184 folio_wake_bit(folio, bit);
1185}
1186
1187/*1159/*
1188 * A choice of three behaviors for folio_wait_bit_common():1160 * A choice of three behaviors for folio_wait_bit_common():
1189 */1161 */
@@ -1484,29 +1456,6 @@ void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) · 294 unmodified lines
1484}1456}
1485EXPORT_SYMBOL_GPL(folio_add_wait_queue);1457EXPORT_SYMBOL_GPL(folio_add_wait_queue);
1486 1458
1487#ifndef clear_bit_unlock_is_negative_byte
1488
1489/*
1490 * PG_waiters is the high bit in the same byte as PG_lock.
1491 *
1492 * On x86 (and on many other architectures), we can clear PG_lock and
1493 * test the sign bit at the same time. But if the architecture does
1494 * not support that special operation, we just do this all by hand
1495 * instead.
1496 *
1497 * The read of PG_waiters has to be after (or concurrently with) PG_locked
1498 * being cleared, but a memory barrier should be unnecessary since it is
1499 * in the same byte as PG_locked.
1500 */
1501static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem)
1502{
1503 clear_bit_unlock(nr, mem);
1504 /* smp_mb__after_atomic(); */
1505 return test_bit(PG_waiters, mem);
1506}
1507
1508#endif
1509
1510/**1459/**
1511 * folio_unlock - Unlock a locked folio.1460 * folio_unlock - Unlock a locked folio.
1512 * @folio: The folio.1461 * @folio: The folio.
@@ -1522,11 +1471,41 @@ void folio_unlock(struct folio *folio) · 9 unmodified lines
1522 BUILD_BUG_ON(PG_waiters != 7);1471 BUILD_BUG_ON(PG_waiters != 7);
1523 BUILD_BUG_ON(PG_locked > 7);1472 BUILD_BUG_ON(PG_locked > 7);
1524 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);1473 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
1525 if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0)))1474 if (folio_xor_flags_has_waiters(folio, 1 << PG_locked))
1526 folio_wake_bit(folio, PG_locked);1475 folio_wake_bit(folio, PG_locked);
1527}1476}
1528EXPORT_SYMBOL(folio_unlock);1477EXPORT_SYMBOL(folio_unlock);
1529 1478
1479/**
1480 * folio_end_read - End read on a folio.
1481 * @folio: The folio.
1482 * @success: True if all reads completed successfully.
1483 *
1484 * When all reads against a folio have completed, filesystems should
1485 * call this function to let the pagecache know that no more reads
1486 * are outstanding. This will unlock the folio and wake up any thread
1487 * sleeping on the lock. The folio will also be marked uptodate if all
1488 * reads succeeded.
1489 *
1490 * Context: May be called from interrupt or process context. May not be
1491 * called from NMI context.
1492 */
1493void folio_end_read(struct folio *folio, bool success)
1494{
1495 unsigned long mask = 1 << PG_locked;
1496
1497 /* Must be in bottom byte for x86 to work */
1498 BUILD_BUG_ON(PG_uptodate > 7);
1499 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
1500 VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio);
1501
1502 if (likely(success))
1503 mask |= 1 << PG_uptodate;
1504 if (folio_xor_flags_has_waiters(folio, mask))
1505 folio_wake_bit(folio, PG_locked);
1506}
1507EXPORT_SYMBOL(folio_end_read);
1508
1530/**1509/**
1531 * folio_end_private_2 - Clear PG_private_2 and wake any waiters.1510 * folio_end_private_2 - Clear PG_private_2 and wake any waiters.
1532 * @folio: The folio.1511 * @folio: The folio.
@@ -1588,9 +1567,15 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); · 55 unmodified lines
1588/**1567/**
1589 * folio_end_writeback - End writeback against a folio.1568 * folio_end_writeback - End writeback against a folio.
1590 * @folio: The folio.1569 * @folio: The folio.
1570 *
1571 * The folio must actually be under writeback.
1572 *
1573 * Context: May be called from process or interrupt context.
1591 */1574 */
1592void folio_end_writeback(struct folio *folio)1575void folio_end_writeback(struct folio *folio)
1593{1576{
1577 VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio);
1578
1594 /*1579 /*
1595 * folio_test_clear_reclaim() could be used here but it is an1580 * folio_test_clear_reclaim() could be used here but it is an
1596 * atomic operation and overkill in this particular case. Failing1581 * atomic operation and overkill in this particular case. Failing
@@ -1607,14 +1592,11 @@ void folio_end_writeback(struct folio *folio) · 10 unmodified lines
1607 * Writeback does not hold a folio reference of its own, relying1592 * Writeback does not hold a folio reference of its own, relying
1608 * on truncation to wait for the clearing of PG_writeback.1593 * on truncation to wait for the clearing of PG_writeback.
1609 * But here we must make sure that the folio is not freed and1594 * But here we must make sure that the folio is not freed and
1610 * reused before the folio_wake().1595 * reused before the folio_wake_bit().
1611 */1596 */
1612 folio_get(folio);1597 folio_get(folio);
1613 if (!__folio_end_writeback(folio))1598 if (__folio_end_writeback(folio))
1614 BUG();1599 folio_wake_bit(folio, PG_writeback);
1615
1616 smp_mb__after_atomic();
1617 folio_wake(folio, PG_writeback);
1618 acct_reclaim_writeback(folio);1600 acct_reclaim_writeback(folio);
1619 folio_put(folio);1601 folio_put(folio);
1620}1602}
@@ -2040,7 +2022,7 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, · 419 unmodified lines
2040 int idx = folio_batch_count(fbatch) - 1;2022 int idx = folio_batch_count(fbatch) - 1;
2041 2023
2042 folio = fbatch->folios[idx];2024 folio = fbatch->folios[idx];
2043 if (!xa_is_value(folio) && !folio_test_hugetlb(folio))2025 if (!xa_is_value(folio))
2044 nr = folio_nr_pages(folio);2026 nr = folio_nr_pages(folio);
2045 *start = indices[idx] + nr;2027 *start = indices[idx] + nr;
2046 }2028 }
@@ -2104,7 +2086,7 @@ put: · 57 unmodified lines
2104 int idx = folio_batch_count(fbatch) - 1;2086 int idx = folio_batch_count(fbatch) - 1;
2105 2087
2106 folio = fbatch->folios[idx];2088 folio = fbatch->folios[idx];
2107 if (!xa_is_value(folio) && !folio_test_hugetlb(folio))2089 if (!xa_is_value(folio))
2108 nr = folio_nr_pages(folio);2090 nr = folio_nr_pages(folio);
2109 *start = indices[idx] + nr;2091 *start = indices[idx] + nr;
2110 }2092 }
@@ -2122,51 +2104,13 @@ put: · 11 unmodified lines
2122 * index @start and up to index @end (inclusive). The folios are returned2104 * index @start and up to index @end (inclusive). The folios are returned
2123 * in @fbatch with an elevated reference count.2105 * in @fbatch with an elevated reference count.
2124 *2106 *
2125 * The first folio may start before @start; if it does, it will contain
2126 * @start. The final folio may extend beyond @end; if it does, it will
2127 * contain @end. The folios have ascending indices. There may be gaps
2128 * between the folios if there are indices which have no folio in the
2129 * page cache. If folios are added to or removed from the page cache
2130 * while this is running, they may or may not be found by this call.
2131 *
2132 * Return: The number of folios which were found.2107 * Return: The number of folios which were found.
2133 * We also update @start to index the next folio for the traversal.2108 * We also update @start to index the next folio for the traversal.
2134 */2109 */
2135unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start,2110unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start,
2136 pgoff_t end, struct folio_batch *fbatch)2111 pgoff_t end, struct folio_batch *fbatch)
2137{2112{
2138 XA_STATE(xas, &mapping->i_pages, *start);2113 return filemap_get_folios_tag(mapping, start, end, XA_PRESENT, fbatch);
2139 struct folio *folio;
2140
2141 rcu_read_lock();
2142 while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) {
2143 /* Skip over shadow, swap and DAX entries */
2144 if (xa_is_value(folio))
2145 continue;
2146 if (!folio_batch_add(fbatch, folio)) {
2147 unsigned long nr = folio_nr_pages(folio);
2148
2149 if (folio_test_hugetlb(folio))
2150 nr = 1;
2151 *start = folio->index + nr;
2152 goto out;
2153 }
2154 }
2155
2156 /*
2157 * We come here when there is no page beyond @end. We take care to not
2158 * overflow the index @start as it confuses some of the callers. This
2159 * breaks the iteration when there is a page at index -1 but that is
2160 * already broken anyway.
2161 */
2162 if (end == (pgoff_t)-1)
2163 *start = (pgoff_t)-1;
2164 else
2165 *start = end + 1;
2166out:
2167 rcu_read_unlock();
2168
2169 return folio_batch_count(fbatch);
2170}2114}
2171EXPORT_SYMBOL(filemap_get_folios);2115EXPORT_SYMBOL(filemap_get_folios);
2172 2116
@@ -2213,9 +2157,6 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, · 40 unmodified lines
2213 2157
2214 if (!folio_batch_add(fbatch, folio)) {2158 if (!folio_batch_add(fbatch, folio)) {
2215 nr = folio_nr_pages(folio);2159 nr = folio_nr_pages(folio);
2216
2217 if (folio_test_hugetlb(folio))
2218 nr = 1;
2219 *start = folio->index + nr;2160 *start = folio->index + nr;
2220 goto out;2161 goto out;
2221 }2162 }
@@ -2232,10 +2173,7 @@ update_start: · 10 unmodified lines
2232 2173
2233 if (nr) {2174 if (nr) {
2234 folio = fbatch->folios[nr - 1];2175 folio = fbatch->folios[nr - 1];
2235 if (folio_test_hugetlb(folio))2176 *start = folio->index + folio_nr_pages(folio);
2236 *start = folio->index + 1;
2237 else
2238 *start = folio_next_index(folio);
2239 }2177 }
2240out:2178out:
2241 rcu_read_unlock();2179 rcu_read_unlock();
@@ -2251,7 +2189,13 @@ EXPORT_SYMBOL(filemap_get_folios_contig); · 9 unmodified lines
2251 * @tag: The tag index2189 * @tag: The tag index
2252 * @fbatch: The batch to fill2190 * @fbatch: The batch to fill
2253 *2191 *
2254 * Same as filemap_get_folios(), but only returning folios tagged with @tag.2192 * The first folio may start before @start; if it does, it will contain
2193 * @start. The final folio may extend beyond @end; if it does, it will
2194 * contain @end. The folios have ascending indices. There may be gaps
2195 * between the folios if there are indices which have no folio in the
2196 * page cache. If folios are added to or removed from the page cache
2197 * while this is running, they may or may not be found by this call.
2198 * Only returns folios that are tagged with @tag.
2255 *2199 *
2256 * Return: The number of folios found.2200 * Return: The number of folios found.
2257 * Also update @start to index the next folio for traversal.2201 * Also update @start to index the next folio for traversal.
@@ -2273,9 +2217,6 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, · 15 unmodified lines
2273 continue;2217 continue;
2274 if (!folio_batch_add(fbatch, folio)) {2218 if (!folio_batch_add(fbatch, folio)) {
2275 unsigned long nr = folio_nr_pages(folio);2219 unsigned long nr = folio_nr_pages(folio);
2276
2277 if (folio_test_hugetlb(folio))
2278 nr = 1;
2279 *start = folio->index + nr;2220 *start = folio->index + nr;
2280 goto out;2221 goto out;
2281 }2222 }
@@ -3104,7 +3045,7 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 822 unmodified lines
3104 3045
3105 /*3046 /*
3106 * NOTE! This will make us return with VM_FAULT_RETRY, but with3047 * NOTE! This will make us return with VM_FAULT_RETRY, but with
3107 * the mmap_lock still held. That's how FAULT_FLAG_RETRY_NOWAIT3048 * the fault lock still held. That's how FAULT_FLAG_RETRY_NOWAIT
3108 * is supposed to work. We have way too many special cases..3049 * is supposed to work. We have way too many special cases..
3109 */3050 */
3110 if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)3051 if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
@@ -3114,13 +3055,14 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 3 unmodified lines
3114 if (vmf->flags & FAULT_FLAG_KILLABLE) {3055 if (vmf->flags & FAULT_FLAG_KILLABLE) {
3115 if (__folio_lock_killable(folio)) {3056 if (__folio_lock_killable(folio)) {
3116 /*3057 /*
3117 * We didn't have the right flags to drop the mmap_lock,3058 * We didn't have the right flags to drop the
3118 * but all fault_handlers only check for fatal signals3059 * fault lock, but all fault_handlers only check
3119 * if we return VM_FAULT_RETRY, so we need to drop the3060 * for fatal signals if we return VM_FAULT_RETRY,
3120 * mmap_lock here and return 0 if we don't have a fpin.3061 * so we need to drop the fault lock here and
3062 * return 0 if we don't have a fpin.
3121 */3063 */
3122 if (*fpin == NULL)3064 if (*fpin == NULL)
3123 mmap_read_unlock(vmf->vma->vm_mm);3065 release_fault_lock(vmf);
3124 return 0;3066 return 0;
3125 }3067 }
3126 } else3068 } else
@@ -3321,21 +3263,28 @@ retry_find: · 194 unmodified lines
3321 VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);3263 VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio);
3322 3264
3323 /*3265 /*
3324 * We have a locked page in the page cache, now we need to check3266 * We have a locked folio in the page cache, now we need to check
3325 * that it's up-to-date. If not, it is going to be due to an error.3267 * that it's up-to-date. If not, it is going to be due to an error,
3268 * or because readahead was otherwise unable to retrieve it.
3326 */3269 */
3327 if (unlikely(!folio_test_uptodate(folio))) {3270 if (unlikely(!folio_test_uptodate(folio))) {
3328 /*3271 /*
3329 * The page was in cache and uptodate and now it is not.3272 * If the invalidate lock is not held, the folio was in cache
3330 * Strange but possible since we didn't hold the page lock all3273 * and uptodate and now it is not. Strange but possible since we
3331 * the time. Let's drop everything get the invalidate lock and3274 * didn't hold the page lock all the time. Let's drop
3332 * try again.3275 * everything, get the invalidate lock and try again.
3333 */3276 */
3334 if (!mapping_locked) {3277 if (!mapping_locked) {
3335 folio_unlock(folio);3278 folio_unlock(folio);
3336 folio_put(folio);3279 folio_put(folio);
3337 goto retry_find;3280 goto retry_find;
3338 }3281 }
3282
3283 /*
3284 * OK, the folio is really not uptodate. This can be because the
3285 * VMA has the VM_RAND_READ flag set, or because an error
3286 * arose. Let's read it in directly.
3287 */
3339 goto page_not_uptodate;3288 goto page_not_uptodate;
3340 }3289 }
3341 3290
@@ -3591,7 +3540,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 249 unmodified lines
3591 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;3540 addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
3592 vmf->pte += xas.xa_index - last_pgoff;3541 vmf->pte += xas.xa_index - last_pgoff;
3593 last_pgoff = xas.xa_index;3542 last_pgoff = xas.xa_index;
3594 end = folio->index + folio_nr_pages(folio) - 1;3543 end = folio_next_index(folio) - 1;
3595 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;3544 nr_pages = min(end, end_pgoff) - xas.xa_index + 1;
3596 3545
3597 if (!folio_test_large(folio))3546 if (!folio_test_large(folio))
@@ -3669,7 +3618,7 @@ int generic_file_mmap(struct file *file, struct vm_area_struct *vma) · 71 unmodified lines
3669 */3618 */
3670int generic_file_readonly_mmap(struct file *file, struct vm_area_struct *vma)3619int generic_file_readonly_mmap(struct file *file, struct vm_area_struct *vma)
3671{3620{
3672 if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_MAYWRITE))3621 if (vma_is_shared_maywrite(vma))
3673 return -EINVAL;3622 return -EINVAL;
3674 return generic_file_mmap(file, vma);3623 return generic_file_mmap(file, vma);
3675}3624}
mm/gup.c+13−3
@@ -1471,6 +1471,9 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, · 1470 unmodified lines
1471 long ret, pages_done;1471 long ret, pages_done;
1472 bool must_unlock = false;1472 bool must_unlock = false;
1473 1473
1474 if (!nr_pages)
1475 return 0;
1476
1474 /*1477 /*
1475 * The internal caller expects GUP to manage the lock internally and the1478 * The internal caller expects GUP to manage the lock internally and the
1476 * lock must be released when this returns.1479 * lock must be released when this returns.
@@ -1595,6 +1598,14 @@ retry: · 118 unmodified lines
1595 mmap_read_unlock(mm);1598 mmap_read_unlock(mm);
1596 *locked = 0;1599 *locked = 0;
1597 }1600 }
1601
1602 /*
1603 * Failing to pin anything implies something has gone wrong (except when
1604 * FOLL_NOWAIT is specified).
1605 */
1606 if (WARN_ON_ONCE(pages_done == 0 && !(flags & FOLL_NOWAIT)))
1607 return -EFAULT;
1608
1598 return pages_done;1609 return pages_done;
1599}1610}
1600 1611
@@ -2227,12 +2238,11 @@ static bool is_valid_gup_args(struct page **pages, int *locked, · 626 unmodified lines
2227 /*2238 /*
2228 * These flags not allowed to be specified externally to the gup2239 * These flags not allowed to be specified externally to the gup
2229 * interfaces:2240 * interfaces:
2230 * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only2241 * - FOLL_TOUCH/FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only
2231 * - FOLL_REMOTE is internal only and used on follow_page()2242 * - FOLL_REMOTE is internal only and used on follow_page()
2232 * - FOLL_UNLOCKABLE is internal only and used if locked is !NULL2243 * - FOLL_UNLOCKABLE is internal only and used if locked is !NULL
2233 */2244 */
2234 if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCKABLE |2245 if (WARN_ON_ONCE(gup_flags & INTERNAL_GUP_FLAGS))
2235 FOLL_REMOTE | FOLL_FAST_ONLY)))
2236 return false;2246 return false;
2237 2247
2238 gup_flags |= to_set;2248 gup_flags |= to_set;
mm/huge_memory.c+79−55
@@ -65,7 +65,11 @@ unsigned long transparent_hugepage_flags __read_mostly = · 64 unmodified lines
65 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|65 (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
66 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);66 (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
67 67
68static struct shrinker deferred_split_shrinker;68static struct shrinker *deferred_split_shrinker;
69static unsigned long deferred_split_count(struct shrinker *shrink,
70 struct shrink_control *sc);
71static unsigned long deferred_split_scan(struct shrinker *shrink,
72 struct shrink_control *sc);
69 73
70static atomic_t huge_zero_refcount;74static atomic_t huge_zero_refcount;
71struct page *huge_zero_page __read_mostly;75struct page *huge_zero_page __read_mostly;
@@ -96,11 +100,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 24 unmodified lines
96 return in_pf;100 return in_pf;
97 101
98 /*102 /*
99 * Special VMA and hugetlb VMA.103 * khugepaged special VMA and hugetlb VMA.
100 * Must be checked after dax since some dax mappings may have104 * Must be checked after dax since some dax mappings may have
101 * VM_MIXEDMAP set.105 * VM_MIXEDMAP set.
102 */106 */
103 if (vm_flags & VM_NO_KHUGEPAGED)107 if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED))
104 return false;108 return false;
105 109
106 /*110 /*
@@ -128,12 +132,18 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 21 unmodified lines
128 !hugepage_flags_always())))132 !hugepage_flags_always())))
129 return false;133 return false;
130 134
131 /* Only regular file is valid */135 if (!vma_is_anonymous(vma)) {
132 if (!in_pf && file_thp_enabled(vma))136 /*
133 return true;137 * Trust that ->huge_fault() handlers know what they are doing
134 138 * in fault path.
135 if (!vma_is_anonymous(vma))139 */
140 if (((in_pf || smaps)) && vma->vm_ops->huge_fault)
141 return true;
142 /* Only regular file is valid in collapse path */
143 if (((!in_pf || smaps)) && file_thp_enabled(vma))
144 return true;
136 return false;145 return false;
146 }
137 147
138 if (vma_is_temporary_stack(vma))148 if (vma_is_temporary_stack(vma))
139 return false;149 return false;
@@ -229,11 +239,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, · 89 unmodified lines
229 return 0;239 return 0;
230}240}
231 241
232static struct shrinker huge_zero_page_shrinker = {242static struct shrinker *huge_zero_page_shrinker;
233 .count_objects = shrink_huge_zero_page_count,
234 .scan_objects = shrink_huge_zero_page_scan,
235 .seeks = DEFAULT_SEEKS,
236};
237 243
238#ifdef CONFIG_SYSFS244#ifdef CONFIG_SYSFS
239static ssize_t enabled_show(struct kobject *kobj,245static ssize_t enabled_show(struct kobject *kobj,
@@ -454,6 +460,38 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · 214 unmodified lines
454}460}
455#endif /* CONFIG_SYSFS */461#endif /* CONFIG_SYSFS */
456 462
463static int __init thp_shrinker_init(void)
464{
465 huge_zero_page_shrinker = shrinker_alloc(0, "thp-zero");
466 if (!huge_zero_page_shrinker)
467 return -ENOMEM;
468
469 deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
470 SHRINKER_MEMCG_AWARE |
471 SHRINKER_NONSLAB,
472 "thp-deferred_split");
473 if (!deferred_split_shrinker) {
474 shrinker_free(huge_zero_page_shrinker);
475 return -ENOMEM;
476 }
477
478 huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count;
479 huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan;
480 shrinker_register(huge_zero_page_shrinker);
481
482 deferred_split_shrinker->count_objects = deferred_split_count;
483 deferred_split_shrinker->scan_objects = deferred_split_scan;
484 shrinker_register(deferred_split_shrinker);
485
486 return 0;
487}
488
489static void __init thp_shrinker_exit(void)
490{
491 shrinker_free(huge_zero_page_shrinker);
492 shrinker_free(deferred_split_shrinker);
493}
494
457static int __init hugepage_init(void)495static int __init hugepage_init(void)
458{496{
459 int err;497 int err;
@@ -482,12 +520,9 @@ static int __init hugepage_init(void) · 22 unmodified lines
482 if (err)520 if (err)
483 goto err_slab;521 goto err_slab;
484 522
485 err = register_shrinker(&huge_zero_page_shrinker, "thp-zero");523 err = thp_shrinker_init();
486 if (err)524 if (err)
487 goto err_hzp_shrinker;525 goto err_shrinker;
488 err = register_shrinker(&deferred_split_shrinker, "thp-deferred_split");
489 if (err)
490 goto err_split_shrinker;
491 526
492 /*527 /*
493 * By default disable transparent hugepages on smaller systems,528 * By default disable transparent hugepages on smaller systems,
@@ -505,10 +540,8 @@ static int __init hugepage_init(void) · 11 unmodified lines
505 540
506 return 0;541 return 0;
507err_khugepaged:542err_khugepaged:
508 unregister_shrinker(&deferred_split_shrinker);543 thp_shrinker_exit();
509err_split_shrinker:544err_shrinker:
510 unregister_shrinker(&huge_zero_page_shrinker);
511err_hzp_shrinker:
512 khugepaged_destroy();545 khugepaged_destroy();
513err_slab:546err_slab:
514 hugepage_exit_sysfs(hugepage_kobj);547 hugepage_exit_sysfs(hugepage_kobj);
@@ -1349,7 +1382,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) · 834 unmodified lines
1349 if (folio_ref_count(folio) == 1) {1382 if (folio_ref_count(folio) == 1) {
1350 pmd_t entry;1383 pmd_t entry;
1351 1384
1352 page_move_anon_rmap(page, vma);1385 folio_move_anon_rmap(folio, vma);
1386 SetPageAnonExclusive(page);
1353 folio_unlock(folio);1387 folio_unlock(folio);
1354reuse:1388reuse:
1355 if (unlikely(unshare)) {1389 if (unlikely(unshare)) {
@@ -1490,9 +1524,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 134 unmodified lines
1490 struct vm_area_struct *vma = vmf->vma;1524 struct vm_area_struct *vma = vmf->vma;
1491 pmd_t oldpmd = vmf->orig_pmd;1525 pmd_t oldpmd = vmf->orig_pmd;
1492 pmd_t pmd;1526 pmd_t pmd;
1493 struct page *page;1527 struct folio *folio;
1494 unsigned long haddr = vmf->address & HPAGE_PMD_MASK;1528 unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
1495 int page_nid = NUMA_NO_NODE;1529 int nid = NUMA_NO_NODE;
1496 int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);1530 int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
1497 bool migrated = false, writable = false;1531 bool migrated = false, writable = false;
1498 int flags = 0;1532 int flags = 0;
@@ -1514,36 +1548,34 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 15 unmodified lines
1514 can_change_pmd_writable(vma, vmf->address, pmd))1548 can_change_pmd_writable(vma, vmf->address, pmd))
1515 writable = true;1549 writable = true;
1516 1550
1517 page = vm_normal_page_pmd(vma, haddr, pmd);1551 folio = vm_normal_folio_pmd(vma, haddr, pmd);
1518 if (!page)1552 if (!folio)
1519 goto out_map;1553 goto out_map;
1520 1554
1521 /* See similar comment in do_numa_page for explanation */1555 /* See similar comment in do_numa_page for explanation */
1522 if (!writable)1556 if (!writable)
1523 flags |= TNF_NO_GROUP;1557 flags |= TNF_NO_GROUP;
1524 1558
1525 page_nid = page_to_nid(page);1559 nid = folio_nid(folio);
1526 /*1560 /*
1527 * For memory tiering mode, cpupid of slow memory page is used1561 * For memory tiering mode, cpupid of slow memory page is used
1528 * to record page access time. So use default value.1562 * to record page access time. So use default value.
1529 */1563 */
1530 if (node_is_toptier(page_nid))1564 if (node_is_toptier(nid))
1531 last_cpupid = page_cpupid_last(page);1565 last_cpupid = folio_last_cpupid(folio);
1532 target_nid = numa_migrate_prep(page, vma, haddr, page_nid,1566 target_nid = numa_migrate_prep(folio, vma, haddr, nid, &flags);
1533 &flags);
1534
1535 if (target_nid == NUMA_NO_NODE) {1567 if (target_nid == NUMA_NO_NODE) {
1536 put_page(page);1568 folio_put(folio);
1537 goto out_map;1569 goto out_map;
1538 }1570 }
1539 1571
1540 spin_unlock(vmf->ptl);1572 spin_unlock(vmf->ptl);
1541 writable = false;1573 writable = false;
1542 1574
1543 migrated = migrate_misplaced_page(page, vma, target_nid);1575 migrated = migrate_misplaced_folio(folio, vma, target_nid);
1544 if (migrated) {1576 if (migrated) {
1545 flags |= TNF_MIGRATED;1577 flags |= TNF_MIGRATED;
1546 page_nid = target_nid;1578 nid = target_nid;
1547 } else {1579 } else {
1548 flags |= TNF_MIGRATE_FAIL;1580 flags |= TNF_MIGRATE_FAIL;
1549 vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);1581 vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -1555,9 +1587,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 5 unmodified lines
1555 }1587 }
1556 1588
1557out:1589out:
1558 if (page_nid != NUMA_NO_NODE)1590 if (nid != NUMA_NO_NODE)
1559 task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,1591 task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
1560 flags);
1561 1592
1562 return 0;1593 return 0;
1563 1594
@@ -1825,7 +1856,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 261 unmodified lines
1825#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION1856#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
1826 if (is_swap_pmd(*pmd)) {1857 if (is_swap_pmd(*pmd)) {
1827 swp_entry_t entry = pmd_to_swp_entry(*pmd);1858 swp_entry_t entry = pmd_to_swp_entry(*pmd);
1828 struct page *page = pfn_swap_entry_to_page(entry);1859 struct folio *folio = page_folio(pfn_swap_entry_to_page(entry));
1829 pmd_t newpmd;1860 pmd_t newpmd;
1830 1861
1831 VM_BUG_ON(!is_pmd_migration_entry(*pmd));1862 VM_BUG_ON(!is_pmd_migration_entry(*pmd));
@@ -1834,7 +1865,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 2 unmodified lines
1834 * A protection check is difficult so1865 * A protection check is difficult so
1835 * just be safe and disable write1866 * just be safe and disable write
1836 */1867 */
1837 if (PageAnon(page))1868 if (folio_test_anon(folio))
1838 entry = make_readable_exclusive_migration_entry(swp_offset(entry));1869 entry = make_readable_exclusive_migration_entry(swp_offset(entry));
1839 else1870 else
1840 entry = make_readable_migration_entry(swp_offset(entry));1871 entry = make_readable_migration_entry(swp_offset(entry));
@@ -1856,7 +1887,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 15 unmodified lines
1856#endif1887#endif
1857 1888
1858 if (prot_numa) {1889 if (prot_numa) {
1859 struct page *page;1890 struct folio *folio;
1860 bool toptier;1891 bool toptier;
1861 /*1892 /*
1862 * Avoid trapping faults against the zero page. The read-only1893 * Avoid trapping faults against the zero page. The read-only
@@ -1869,8 +1900,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 6 unmodified lines
1869 if (pmd_protnone(*pmd))1900 if (pmd_protnone(*pmd))
1870 goto unlock;1901 goto unlock;
1871 1902
1872 page = pmd_page(*pmd);1903 folio = page_folio(pmd_page(*pmd));
1873 toptier = node_is_toptier(page_to_nid(page));1904 toptier = node_is_toptier(folio_nid(folio));
1874 /*1905 /*
1875 * Skip scanning top tier node if normal numa1906 * Skip scanning top tier node if normal numa
1876 * balancing is disabled1907 * balancing is disabled
@@ -1881,7 +1912,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 4 unmodified lines
1881 1912
1882 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&1913 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
1883 !toptier)1914 !toptier)
1884 xchg_page_access_time(page, jiffies_to_msecs(jiffies));1915 folio_xchg_access_time(folio,
1916 jiffies_to_msecs(jiffies));
1885 }1917 }
1886 /*1918 /*
1887 * In case prot_numa, we are under mmap_read_lock(mm). It's critical1919 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
@@ -2483,7 +2515,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, · 595 unmodified lines
2483 if (page_is_idle(head))2515 if (page_is_idle(head))
2484 set_page_idle(page_tail);2516 set_page_idle(page_tail);
2485 2517
2486 page_cpupid_xchg_last(page_tail, page_cpupid_last(head));2518 folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio));
2487 2519
2488 /*2520 /*
2489 * always add to the tail because some iterators expect new2521 * always add to the tail because some iterators expect new
@@ -2828,7 +2860,7 @@ void deferred_split_folio(struct folio *folio) · 338 unmodified lines
2828#ifdef CONFIG_MEMCG2860#ifdef CONFIG_MEMCG
2829 if (memcg)2861 if (memcg)
2830 set_shrinker_bit(memcg, folio_nid(folio),2862 set_shrinker_bit(memcg, folio_nid(folio),
2831 deferred_split_shrinker.id);2863 deferred_split_shrinker->id);
2832#endif2864#endif
2833 }2865 }
2834 spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);2866 spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
@@ -2902,14 +2934,6 @@ next: · 67 unmodified lines
2902 return split;2934 return split;
2903}2935}
2904 2936
2905static struct shrinker deferred_split_shrinker = {
2906 .count_objects = deferred_split_count,
2907 .scan_objects = deferred_split_scan,
2908 .seeks = DEFAULT_SEEKS,
2909 .flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE |
2910 SHRINKER_NONSLAB,
2911};
2912
2913#ifdef CONFIG_DEBUG_FS2937#ifdef CONFIG_DEBUG_FS
2914static void split_huge_pages_all(void)2938static void split_huge_pages_all(void)
2915{2939{
mm/hugetlb.c+458−167
@@ -984,7 +984,7 @@ static long region_count(struct resv_map *resv, long f, long t) · 983 unmodified lines
984 984
985/*985/*
986 * Convert the address within this vma to the page offset within986 * Convert the address within this vma to the page offset within
987 * the mapping, in pagecache page units; huge pages here.987 * the mapping, huge page units here.
988 */988 */
989static pgoff_t vma_hugecache_offset(struct hstate *h,989static pgoff_t vma_hugecache_offset(struct hstate *h,
990 struct vm_area_struct *vma, unsigned long address)990 struct vm_area_struct *vma, unsigned long address)
@@ -993,13 +993,6 @@ static pgoff_t vma_hugecache_offset(struct hstate *h, · 2 unmodified lines
993 (vma->vm_pgoff >> huge_page_order(h));993 (vma->vm_pgoff >> huge_page_order(h));
994}994}
995 995
996pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
997 unsigned long address)
998{
999 return vma_hugecache_offset(hstate_vma(vma), vma, address);
1000}
1001EXPORT_SYMBOL_GPL(linear_hugepage_index);
1002
1003/**996/**
1004 * vma_kernel_pagesize - Page size granularity for this VMA.997 * vma_kernel_pagesize - Page size granularity for this VMA.
1005 * @vma: The user mapping.998 * @vma: The user mapping.
@@ -1478,7 +1471,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, · 472 unmodified lines
1478}1471}
1479 1472
1480/*1473/*
1481 * helper for remove_pool_huge_page() - return the previously saved1474 * helper for remove_pool_hugetlb_folio() - return the previously saved
1482 * node ["this node"] from which to free a huge page. Advance the1475 * node ["this node"] from which to free a huge page. Advance the
1483 * next node id whether or not we find a free huge page to free so1476 * next node id whether or not we find a free huge page to free so
1484 * that the next attempt to free addresses the next node.1477 * that the next attempt to free addresses the next node.
@@ -1752,7 +1745,12 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, · 267 unmodified lines
1752 if (folio_test_hugetlb_raw_hwp_unreliable(folio))1745 if (folio_test_hugetlb_raw_hwp_unreliable(folio))
1753 return;1746 return;
1754 1747
1755 if (hugetlb_vmemmap_restore(h, &folio->page)) {1748 /*
1749 * If folio is not vmemmap optimized (!clear_dtor), then the folio
1750 * is no longer identified as a hugetlb page. hugetlb_vmemmap_restore_folio
1751 * can only be passed hugetlb pages and will BUG otherwise.
1752 */
1753 if (clear_dtor && hugetlb_vmemmap_restore_folio(h, folio)) {
1756 spin_lock_irq(&hugetlb_lock);1754 spin_lock_irq(&hugetlb_lock);
1757 /*1755 /*
1758 * If we cannot allocate vmemmap pages, just refuse to free the1756 * If we cannot allocate vmemmap pages, just refuse to free the
@@ -1814,22 +1812,22 @@ static void free_hpage_workfn(struct work_struct *work) · 55 unmodified lines
1814 node = llist_del_all(&hpage_freelist);1812 node = llist_del_all(&hpage_freelist);
1815 1813
1816 while (node) {1814 while (node) {
1817 struct page *page;1815 struct folio *folio;
1818 struct hstate *h;1816 struct hstate *h;
1819 1817
1820 page = container_of((struct address_space **)node,1818 folio = container_of((struct address_space **)node,
1821 struct page, mapping);1819 struct folio, mapping);
1822 node = node->next;1820 node = node->next;
1823 page->mapping = NULL;1821 folio->mapping = NULL;
1824 /*1822 /*
1825 * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in1823 * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in
1826 * folio_hstate() is going to trigger because a previous call to1824 * folio_hstate() is going to trigger because a previous call to
1827 * remove_hugetlb_folio() will clear the hugetlb bit, so do1825 * remove_hugetlb_folio() will clear the hugetlb bit, so do
1828 * not use folio_hstate() directly.1826 * not use folio_hstate() directly.
1829 */1827 */
1830 h = size_to_hstate(page_size(page));1828 h = size_to_hstate(folio_size(folio));
1831 1829
1832 __update_and_free_hugetlb_folio(h, page_folio(page));1830 __update_and_free_hugetlb_folio(h, folio);
1833 1831
1834 cond_resched();1832 cond_resched();
1835 }1833 }
@@ -1861,13 +1859,93 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, · 25 unmodified lines
1861 schedule_work(&free_hpage_work);1859 schedule_work(&free_hpage_work);
1862}1860}
1863 1861
1864static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list)1862static void bulk_vmemmap_restore_error(struct hstate *h,
1863 struct list_head *folio_list,
1864 struct list_head *non_hvo_folios)
1865{1865{
1866 struct page *page, *t_page;1866 struct folio *folio, *t_folio;
1867 struct folio *folio;1867
1868 if (!list_empty(non_hvo_folios)) {
1869 /*
1870 * Free any restored hugetlb pages so that restore of the
1871 * entire list can be retried.
1872 * The idea is that in the common case of ENOMEM errors freeing
1873 * hugetlb pages with vmemmap we will free up memory so that we
1874 * can allocate vmemmap for more hugetlb pages.
1875 */
1876 list_for_each_entry_safe(folio, t_folio, non_hvo_folios, lru) {
1877 list_del(&folio->lru);
1878 spin_lock_irq(&hugetlb_lock);
1879 __clear_hugetlb_destructor(h, folio);
1880 spin_unlock_irq(&hugetlb_lock);
1881 update_and_free_hugetlb_folio(h, folio, false);
1882 cond_resched();
1883 }
1884 } else {
1885 /*
1886 * In the case where there are no folios which can be
1887 * immediately freed, we loop through the list trying to restore
1888 * vmemmap individually in the hope that someone elsewhere may
1889 * have done something to cause success (such as freeing some
1890 * memory). If unable to restore a hugetlb page, the hugetlb
1891 * page is made a surplus page and removed from the list.
1892 * If are able to restore vmemmap and free one hugetlb page, we
1893 * quit processing the list to retry the bulk operation.
1894 */
1895 list_for_each_entry_safe(folio, t_folio, folio_list, lru)
1896 if (hugetlb_vmemmap_restore_folio(h, folio)) {
1897 list_del(&folio->lru);
1898 spin_lock_irq(&hugetlb_lock);
1899 add_hugetlb_folio(h, folio, true);
1900 spin_unlock_irq(&hugetlb_lock);
1901 } else {
1902 list_del(&folio->lru);
1903 spin_lock_irq(&hugetlb_lock);
1904 __clear_hugetlb_destructor(h, folio);
1905 spin_unlock_irq(&hugetlb_lock);
1906 update_and_free_hugetlb_folio(h, folio, false);
1907 cond_resched();
1908 break;
1909 }
1910 }
1911}
1912
1913static void update_and_free_pages_bulk(struct hstate *h,
1914 struct list_head *folio_list)
1915{
1916 long ret;
1917 struct folio *folio, *t_folio;
1918 LIST_HEAD(non_hvo_folios);
1919
1920 /*
1921 * First allocate required vmemmmap (if necessary) for all folios.
1922 * Carefully handle errors and free up any available hugetlb pages
1923 * in an effort to make forward progress.
1924 */
1925retry:
1926 ret = hugetlb_vmemmap_restore_folios(h, folio_list, &non_hvo_folios);
1927 if (ret < 0) {
1928 bulk_vmemmap_restore_error(h, folio_list, &non_hvo_folios);
1929 goto retry;
1930 }
1931
1932 /*
1933 * At this point, list should be empty, ret should be >= 0 and there
1934 * should only be pages on the non_hvo_folios list.
1935 * Do note that the non_hvo_folios list could be empty.
1936 * Without HVO enabled, ret will be 0 and there is no need to call
1937 * __clear_hugetlb_destructor as this was done previously.
1938 */
1939 VM_WARN_ON(!list_empty(folio_list));
1940 VM_WARN_ON(ret < 0);
1941 if (!list_empty(&non_hvo_folios) && ret) {
1942 spin_lock_irq(&hugetlb_lock);
1943 list_for_each_entry(folio, &non_hvo_folios, lru)
1944 __clear_hugetlb_destructor(h, folio);
1945 spin_unlock_irq(&hugetlb_lock);
1946 }
1868 1947
1869 list_for_each_entry_safe(page, t_page, list, lru) {1948 list_for_each_entry_safe(folio, t_folio, &non_hvo_folios, lru) {
1870 folio = page_folio(page);
1871 update_and_free_hugetlb_folio(h, folio, false);1949 update_and_free_hugetlb_folio(h, folio, false);
1872 cond_resched();1950 cond_resched();
1873 }1951 }
@@ -1931,6 +2009,7 @@ void free_huge_folio(struct folio *folio) · 57 unmodified lines
1931 pages_per_huge_page(h), folio);2009 pages_per_huge_page(h), folio);
1932 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),2010 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
1933 pages_per_huge_page(h), folio);2011 pages_per_huge_page(h), folio);
2012 mem_cgroup_uncharge(folio);
1934 if (restore_reserve)2013 if (restore_reserve)
1935 h->resv_huge_pages++;2014 h->resv_huge_pages++;
1936 2015
@@ -1960,16 +2039,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) · 23 unmodified lines
1960 h->nr_huge_pages_node[nid]++;2039 h->nr_huge_pages_node[nid]++;
1961}2040}
1962 2041
1963static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio)2042static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio)
1964{2043{
1965 hugetlb_vmemmap_optimize(h, &folio->page);
1966 INIT_LIST_HEAD(&folio->lru);
1967 folio_set_hugetlb(folio);2044 folio_set_hugetlb(folio);
2045 INIT_LIST_HEAD(&folio->lru);
1968 hugetlb_set_folio_subpool(folio, NULL);2046 hugetlb_set_folio_subpool(folio, NULL);
1969 set_hugetlb_cgroup(folio, NULL);2047 set_hugetlb_cgroup(folio, NULL);
1970 set_hugetlb_cgroup_rsvd(folio, NULL);2048 set_hugetlb_cgroup_rsvd(folio, NULL);
1971}2049}
1972 2050
2051static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio)
2052{
2053 init_new_hugetlb_folio(h, folio);
2054 hugetlb_vmemmap_optimize_folio(h, folio);
2055}
2056
1973static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid)2057static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid)
1974{2058{
1975 __prep_new_hugetlb_folio(h, folio);2059 __prep_new_hugetlb_folio(h, folio);
@@ -2103,20 +2187,6 @@ struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage) · 127 unmodified lines
2103 return NULL;2187 return NULL;
2104}2188}
2105 2189
2106pgoff_t hugetlb_basepage_index(struct page *page)
2107{
2108 struct page *page_head = compound_head(page);
2109 pgoff_t index = page_index(page_head);
2110 unsigned long compound_idx;
2111
2112 if (compound_order(page_head) > MAX_ORDER)
2113 compound_idx = page_to_pfn(page) - page_to_pfn(page_head);
2114 else
2115 compound_idx = page - page_head;
2116
2117 return (index << compound_order(page_head)) + compound_idx;
2118}
2119
2120static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h,2190static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h,
2121 gfp_t gfp_mask, int nid, nodemask_t *nmask,2191 gfp_t gfp_mask, int nid, nodemask_t *nmask,
2122 nodemask_t *node_alloc_noretry)2192 nodemask_t *node_alloc_noretry)
@@ -2180,16 +2250,9 @@ retry: · 57 unmodified lines
2180 return page_folio(page);2250 return page_folio(page);
2181}2251}
2182 2252
2183/*2253static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h,
2184 * Common helper to allocate a fresh hugetlb page. All specific allocators2254 gfp_t gfp_mask, int nid, nodemask_t *nmask,
2185 * should use this function to get new hugetlb pages2255 nodemask_t *node_alloc_noretry)
2186 *
2187 * Note that returned page is 'frozen': ref count of head page and all tail
2188 * pages is zero.
2189 */
2190static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
2191 gfp_t gfp_mask, int nid, nodemask_t *nmask,
2192 nodemask_t *node_alloc_noretry)
2193{2256{
2194 struct folio *folio;2257 struct folio *folio;
2195 bool retry = false;2258 bool retry = false;
@@ -2202,6 +2265,7 @@ retry: · 6 unmodified lines
2202 nid, nmask, node_alloc_noretry);2265 nid, nmask, node_alloc_noretry);
2203 if (!folio)2266 if (!folio)
2204 return NULL;2267 return NULL;
2268
2205 if (hstate_is_gigantic(h)) {2269 if (hstate_is_gigantic(h)) {
2206 if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) {2270 if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) {
2207 /*2271 /*
@@ -2216,32 +2280,84 @@ retry: · 8 unmodified lines
2216 return NULL;2280 return NULL;
2217 }2281 }
2218 }2282 }
2219 prep_new_hugetlb_folio(h, folio, folio_nid(folio));
2220 2283
2221 return folio;2284 return folio;
2222}2285}
2223 2286
2287static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
2288 gfp_t gfp_mask, int nid, nodemask_t *nmask,
2289 nodemask_t *node_alloc_noretry)
2290{
2291 struct folio *folio;
2292
2293 folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask,
2294 node_alloc_noretry);
2295 if (folio)
2296 init_new_hugetlb_folio(h, folio);
2297 return folio;
2298}
2299
2224/*2300/*
2225 * Allocates a fresh page to the hugetlb allocator pool in the node interleaved2301 * Common helper to allocate a fresh hugetlb page. All specific allocators
2226 * manner.2302 * should use this function to get new hugetlb pages
2303 *
2304 * Note that returned page is 'frozen': ref count of head page and all tail
2305 * pages is zero.
2227 */2306 */
2228static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,2307static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
2229 nodemask_t *node_alloc_noretry)2308 gfp_t gfp_mask, int nid, nodemask_t *nmask,
2309 nodemask_t *node_alloc_noretry)
2230{2310{
2231 struct folio *folio;2311 struct folio *folio;
2232 int nr_nodes, node;2312
2313 folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask,
2314 node_alloc_noretry);
2315 if (!folio)
2316 return NULL;
2317
2318 prep_new_hugetlb_folio(h, folio, folio_nid(folio));
2319 return folio;
2320}
2321
2322static void prep_and_add_allocated_folios(struct hstate *h,
2323 struct list_head *folio_list)
2324{
2325 unsigned long flags;
2326 struct folio *folio, *tmp_f;
2327
2328 /* Send list for bulk vmemmap optimization processing */
2329 hugetlb_vmemmap_optimize_folios(h, folio_list);
2330
2331 /* Add all new pool pages to free lists in one lock cycle */
2332 spin_lock_irqsave(&hugetlb_lock, flags);
2333 list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
2334 __prep_account_new_huge_page(h, folio_nid(folio));
2335 enqueue_hugetlb_folio(h, folio);
2336 }
2337 spin_unlock_irqrestore(&hugetlb_lock, flags);
2338}
2339
2340/*
2341 * Allocates a fresh hugetlb page in a node interleaved manner. The page
2342 * will later be added to the appropriate hugetlb pool.
2343 */
2344static struct folio *alloc_pool_huge_folio(struct hstate *h,
2345 nodemask_t *nodes_allowed,
2346 nodemask_t *node_alloc_noretry)
2347{
2233 gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;2348 gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
2349 int nr_nodes, node;
2234 2350
2235 for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {2351 for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) {
2236 folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node,2352 struct folio *folio;
2353
2354 folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node,
2237 nodes_allowed, node_alloc_noretry);2355 nodes_allowed, node_alloc_noretry);
2238 if (folio) {2356 if (folio)
2239 free_huge_folio(folio); /* free it into the hugepage allocator */2357 return folio;
2240 return 1;
2241 }
2242 }2358 }
2243 2359
2244 return 0;2360 return NULL;
2245}2361}
2246 2362
2247/*2363/*
@@ -2251,13 +2367,11 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, · 3 unmodified lines
2251 * an additional call to free the page to low level allocators.2367 * an additional call to free the page to low level allocators.
2252 * Called with hugetlb_lock locked.2368 * Called with hugetlb_lock locked.
2253 */2369 */
2254static struct page *remove_pool_huge_page(struct hstate *h,2370static struct folio *remove_pool_hugetlb_folio(struct hstate *h,
2255 nodemask_t *nodes_allowed,2371 nodemask_t *nodes_allowed, bool acct_surplus)
2256 bool acct_surplus)
2257{2372{
2258 int nr_nodes, node;2373 int nr_nodes, node;
2259 struct page *page = NULL;2374 struct folio *folio = NULL;
2260 struct folio *folio;
2261 2375
2262 lockdep_assert_held(&hugetlb_lock);2376 lockdep_assert_held(&hugetlb_lock);
2263 for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {2377 for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
@@ -2267,15 +2381,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, · 3 unmodified lines
2267 */2381 */
2268 if ((!acct_surplus || h->surplus_huge_pages_node[node]) &&2382 if ((!acct_surplus || h->surplus_huge_pages_node[node]) &&
2269 !list_empty(&h->hugepage_freelists[node])) {2383 !list_empty(&h->hugepage_freelists[node])) {
2270 page = list_entry(h->hugepage_freelists[node].next,2384 folio = list_entry(h->hugepage_freelists[node].next,
2271 struct page, lru);2385 struct folio, lru);
2272 folio = page_folio(page);
2273 remove_hugetlb_folio(h, folio, acct_surplus);2386 remove_hugetlb_folio(h, folio, acct_surplus);
2274 break;2387 break;
2275 }2388 }
2276 }2389 }
2277 2390
2278 return page;2391 return folio;
2279}2392}
2280 2393
2281/*2394/*
@@ -2343,17 +2456,23 @@ retry: · 61 unmodified lines
2343 * need to adjust max_huge_pages if the page is not freed.2456 * need to adjust max_huge_pages if the page is not freed.
2344 * Attempt to allocate vmemmmap here so that we can take2457 * Attempt to allocate vmemmmap here so that we can take
2345 * appropriate action on failure.2458 * appropriate action on failure.
2459 *
2460 * The folio_test_hugetlb check here is because
2461 * remove_hugetlb_folio will clear hugetlb folio flag for
2462 * non-vmemmap optimized hugetlb folios.
2346 */2463 */
2347 rc = hugetlb_vmemmap_restore(h, &folio->page);2464 if (folio_test_hugetlb(folio)) {
2348 if (!rc) {2465 rc = hugetlb_vmemmap_restore_folio(h, folio);
2349 update_and_free_hugetlb_folio(h, folio, false);2466 if (rc) {
2350 } else {2467 spin_lock_irq(&hugetlb_lock);
2351 spin_lock_irq(&hugetlb_lock);2468 add_hugetlb_folio(h, folio, false);
2352 add_hugetlb_folio(h, folio, false);2469 h->max_huge_pages++;
2353 h->max_huge_pages++;2470 goto out;
2354 spin_unlock_irq(&hugetlb_lock);2471 }
2355 }2472 } else
2473 rc = 0;
2356 2474
2475 update_and_free_hugetlb_folio(h, folio, false);
2357 return rc;2476 return rc;
2358 }2477 }
2359out:2478out:
@@ -2511,24 +2630,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 151 unmodified lines
2511 return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask);2630 return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask);
2512}2631}
2513 2632
2514/* mempolicy aware migration callback */
2515struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma,
2516 unsigned long address)
2517{
2518 struct mempolicy *mpol;
2519 nodemask_t *nodemask;
2520 struct folio *folio;
2521 gfp_t gfp_mask;
2522 int node;
2523
2524 gfp_mask = htlb_alloc_mask(h);
2525 node = huge_node(vma, address, gfp_mask, &mpol, &nodemask);
2526 folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask);
2527 mpol_cond_put(mpol);
2528
2529 return folio;
2530}
2531
2532/*2633/*
2533 * Increase the hugetlb pool such that it can accommodate a reservation2634 * Increase the hugetlb pool such that it can accommodate a reservation
2534 * of size 'delta'.2635 * of size 'delta'.
@@ -2629,7 +2730,6 @@ static void return_unused_surplus_pages(struct hstate *h, · 94 unmodified lines
2629 unsigned long unused_resv_pages)2730 unsigned long unused_resv_pages)
2630{2731{
2631 unsigned long nr_pages;2732 unsigned long nr_pages;
2632 struct page *page;
2633 LIST_HEAD(page_list);2733 LIST_HEAD(page_list);
2634 2734
2635 lockdep_assert_held(&hugetlb_lock);2735 lockdep_assert_held(&hugetlb_lock);
@@ -2650,15 +2750,17 @@ static void return_unused_surplus_pages(struct hstate *h, · 14 unmodified lines
2650 * evenly across all nodes with memory. Iterate across these nodes2750 * evenly across all nodes with memory. Iterate across these nodes
2651 * until we can no longer free unreserved surplus pages. This occurs2751 * until we can no longer free unreserved surplus pages. This occurs
2652 * when the nodes with surplus pages have no free pages.2752 * when the nodes with surplus pages have no free pages.
2653 * remove_pool_huge_page() will balance the freed pages across the2753 * remove_pool_hugetlb_folio() will balance the freed pages across the
2654 * on-line nodes with memory and will handle the hstate accounting.2754 * on-line nodes with memory and will handle the hstate accounting.
2655 */2755 */
2656 while (nr_pages--) {2756 while (nr_pages--) {
2657 page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1);2757 struct folio *folio;
2658 if (!page)2758
2759 folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1);
2760 if (!folio)
2659 goto out;2761 goto out;
2660 2762
2661 list_add(&page->lru, &page_list);2763 list_add(&folio->lru, &page_list);
2662 }2764 }
2663 2765
2664out:2766out:
@@ -3040,11 +3142,20 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 375 unmodified lines
3040 struct hugepage_subpool *spool = subpool_vma(vma);3142 struct hugepage_subpool *spool = subpool_vma(vma);
3041 struct hstate *h = hstate_vma(vma);3143 struct hstate *h = hstate_vma(vma);
3042 struct folio *folio;3144 struct folio *folio;
3043 long map_chg, map_commit;3145 long map_chg, map_commit, nr_pages = pages_per_huge_page(h);
3044 long gbl_chg;3146 long gbl_chg;
3045 int ret, idx;3147 int memcg_charge_ret, ret, idx;
3046 struct hugetlb_cgroup *h_cg = NULL;3148 struct hugetlb_cgroup *h_cg = NULL;
3149 struct mem_cgroup *memcg;
3047 bool deferred_reserve;3150 bool deferred_reserve;
3151 gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
3152
3153 memcg = get_mem_cgroup_from_current();
3154 memcg_charge_ret = mem_cgroup_hugetlb_try_charge(memcg, gfp, nr_pages);
3155 if (memcg_charge_ret == -ENOMEM) {
3156 mem_cgroup_put(memcg);
3157 return ERR_PTR(-ENOMEM);
3158 }
3048 3159
3049 idx = hstate_index(h);3160 idx = hstate_index(h);
3050 /*3161 /*
@@ -3053,8 +3164,12 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 2 unmodified lines
3053 * code of zero indicates a reservation exists (no change).3164 * code of zero indicates a reservation exists (no change).
3054 */3165 */
3055 map_chg = gbl_chg = vma_needs_reservation(h, vma, addr);3166 map_chg = gbl_chg = vma_needs_reservation(h, vma, addr);
3056 if (map_chg < 0)3167 if (map_chg < 0) {
3168 if (!memcg_charge_ret)
3169 mem_cgroup_cancel_charge(memcg, nr_pages);
3170 mem_cgroup_put(memcg);
3057 return ERR_PTR(-ENOMEM);3171 return ERR_PTR(-ENOMEM);
3172 }
3058 3173
3059 /*3174 /*
3060 * Processes that did not create the mapping will have no3175 * Processes that did not create the mapping will have no
@@ -3065,10 +3180,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 4 unmodified lines
3065 */3180 */
3066 if (map_chg || avoid_reserve) {3181 if (map_chg || avoid_reserve) {
3067 gbl_chg = hugepage_subpool_get_pages(spool, 1);3182 gbl_chg = hugepage_subpool_get_pages(spool, 1);
3068 if (gbl_chg < 0) {3183 if (gbl_chg < 0)
3069 vma_end_reservation(h, vma, addr);3184 goto out_end_reservation;
3070 return ERR_PTR(-ENOSPC);
3071 }
3072 3185
3073 /*3186 /*
3074 * Even though there was no reservation in the region/reserve3187 * Even though there was no reservation in the region/reserve
@@ -3150,6 +3263,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 75 unmodified lines
3150 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),3263 hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
3151 pages_per_huge_page(h), folio);3264 pages_per_huge_page(h), folio);
3152 }3265 }
3266
3267 if (!memcg_charge_ret)
3268 mem_cgroup_commit_charge(folio, memcg);
3269 mem_cgroup_put(memcg);
3270
3153 return folio;3271 return folio;
3154 3272
3155out_uncharge_cgroup:3273out_uncharge_cgroup:
@@ -3161,7 +3279,11 @@ out_uncharge_cgroup_reservation: · 5 unmodified lines
3161out_subpool_put:3279out_subpool_put:
3162 if (map_chg || avoid_reserve)3280 if (map_chg || avoid_reserve)
3163 hugepage_subpool_put_pages(spool, 1);3281 hugepage_subpool_put_pages(spool, 1);
3282out_end_reservation:
3164 vma_end_reservation(h, vma, addr);3283 vma_end_reservation(h, vma, addr);
3284 if (!memcg_charge_ret)
3285 mem_cgroup_cancel_charge(memcg, nr_pages);
3286 mem_cgroup_put(memcg);
3165 return ERR_PTR(-ENOSPC);3287 return ERR_PTR(-ENOSPC);
3166}3288}
3167 3289
@@ -3196,6 +3318,16 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) · 28 unmodified lines
3196 }3318 }
3197 3319
3198found:3320found:
3321
3322 /*
3323 * Only initialize the head struct page in memmap_init_reserved_pages,
3324 * rest of the struct pages will be initialized by the HugeTLB
3325 * subsystem itself.
3326 * The head struct page is used to get folio information by the HugeTLB
3327 * subsystem like zone id and node id.
3328 */
3329 memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE),
3330 huge_page_size(h) - PAGE_SIZE);
3199 /* Put them into a private list first because mem_map is not up yet */3331 /* Put them into a private list first because mem_map is not up yet */
3200 INIT_LIST_HEAD(&m->list);3332 INIT_LIST_HEAD(&m->list);
3201 list_add(&m->list, &huge_boot_pages);3333 list_add(&m->list, &huge_boot_pages);
@@ -3203,29 +3335,102 @@ found: · 1 unmodified line
3203 return 1;3335 return 1;
3204}3336}
3205 3337
3338/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */
3339static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
3340 unsigned long start_page_number,
3341 unsigned long end_page_number)
3342{
3343 enum zone_type zone = zone_idx(folio_zone(folio));
3344 int nid = folio_nid(folio);
3345 unsigned long head_pfn = folio_pfn(folio);
3346 unsigned long pfn, end_pfn = head_pfn + end_page_number;
3347 int ret;
3348
3349 for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) {
3350 struct page *page = pfn_to_page(pfn);
3351
3352 __init_single_page(page, pfn, zone, nid);
3353 prep_compound_tail((struct page *)folio, pfn - head_pfn);
3354 ret = page_ref_freeze(page, 1);
3355 VM_BUG_ON(!ret);
3356 }
3357}
3358
3359static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
3360 struct hstate *h,
3361 unsigned long nr_pages)
3362{
3363 int ret;
3364
3365 /* Prepare folio head */
3366 __folio_clear_reserved(folio);
3367 __folio_set_head(folio);
3368 ret = folio_ref_freeze(folio, 1);
3369 VM_BUG_ON(!ret);
3370 /* Initialize the necessary tail struct pages */
3371 hugetlb_folio_init_tail_vmemmap(folio, 1, nr_pages);
3372 prep_compound_head((struct page *)folio, huge_page_order(h));
3373}
3374
3375static void __init prep_and_add_bootmem_folios(struct hstate *h,
3376 struct list_head *folio_list)
3377{
3378 unsigned long flags;
3379 struct folio *folio, *tmp_f;
3380
3381 /* Send list for bulk vmemmap optimization processing */
3382 hugetlb_vmemmap_optimize_folios(h, folio_list);
3383
3384 /* Add all new pool pages to free lists in one lock cycle */
3385 spin_lock_irqsave(&hugetlb_lock, flags);
3386 list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
3387 if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
3388 /*
3389 * If HVO fails, initialize all tail struct pages
3390 * We do not worry about potential long lock hold
3391 * time as this is early in boot and there should
3392 * be no contention.
3393 */
3394 hugetlb_folio_init_tail_vmemmap(folio,
3395 HUGETLB_VMEMMAP_RESERVE_PAGES,
3396 pages_per_huge_page(h));
3397 }
3398 __prep_account_new_huge_page(h, folio_nid(folio));
3399 enqueue_hugetlb_folio(h, folio);
3400 }
3401 spin_unlock_irqrestore(&hugetlb_lock, flags);
3402}
3403
3206/*3404/*
3207 * Put bootmem huge pages into the standard lists after mem_map is up.3405 * Put bootmem huge pages into the standard lists after mem_map is up.
3208 * Note: This only applies to gigantic (order > MAX_ORDER) pages.3406 * Note: This only applies to gigantic (order > MAX_ORDER) pages.
3209 */3407 */
3210static void __init gather_bootmem_prealloc(void)3408static void __init gather_bootmem_prealloc(void)
3211{3409{
3410 LIST_HEAD(folio_list);
3212 struct huge_bootmem_page *m;3411 struct huge_bootmem_page *m;
3412 struct hstate *h = NULL, *prev_h = NULL;
3213 3413
3214 list_for_each_entry(m, &huge_boot_pages, list) {3414 list_for_each_entry(m, &huge_boot_pages, list) {
3215 struct page *page = virt_to_page(m);3415 struct page *page = virt_to_page(m);
3216 struct folio *folio = page_folio(page);3416 struct folio *folio = (void *)page;
3217 struct hstate *h = m->hstate;3417
3418 h = m->hstate;
3419 /*
3420 * It is possible to have multiple huge page sizes (hstates)
3421 * in this list. If so, process each size separately.
3422 */
3423 if (h != prev_h && prev_h != NULL)
3424 prep_and_add_bootmem_folios(prev_h, &folio_list);
3425 prev_h = h;
3218 3426
3219 VM_BUG_ON(!hstate_is_gigantic(h));3427 VM_BUG_ON(!hstate_is_gigantic(h));
3220 WARN_ON(folio_ref_count(folio) != 1);3428 WARN_ON(folio_ref_count(folio) != 1);
3221 if (prep_compound_gigantic_folio(folio, huge_page_order(h))) {3429
3222 WARN_ON(folio_test_reserved(folio));3430 hugetlb_folio_init_vmemmap(folio, h,
3223 prep_new_hugetlb_folio(h, folio, folio_nid(folio));3431 HUGETLB_VMEMMAP_RESERVE_PAGES);
3224 free_huge_folio(folio); /* add to the hugepage allocator */3432 init_new_hugetlb_folio(h, folio);
3225 } else {3433 list_add(&folio->lru, &folio_list);
3226 /* VERY unlikely inflated ref count on a tail page */
3227 free_gigantic_folio(folio, huge_page_order(h));
3228 }
3229 3434
3230 /*3435 /*
3231 * We need to restore the 'stolen' pages to totalram_pages3436 * We need to restore the 'stolen' pages to totalram_pages
@@ -3235,7 +3440,10 @@ static void __init gather_bootmem_prealloc(void) · 3 unmodified lines
3235 adjust_managed_page_count(page, pages_per_huge_page(h));3440 adjust_managed_page_count(page, pages_per_huge_page(h));
3236 cond_resched();3441 cond_resched();
3237 }3442 }
3443
3444 prep_and_add_bootmem_folios(h, &folio_list);
3238}3445}
3446
3239static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)3447static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
3240{3448{
3241 unsigned long i;3449 unsigned long i;
@@ -3267,9 +3475,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 25 unmodified lines
3267 h->max_huge_pages_node[nid] = i;3475 h->max_huge_pages_node[nid] = i;
3268}3476}
3269 3477
3478/*
3479 * NOTE: this routine is called in different contexts for gigantic and
3480 * non-gigantic pages.
3481 * - For gigantic pages, this is called early in the boot process and
3482 * pages are allocated from memblock allocated or something similar.
3483 * Gigantic pages are actually added to pools later with the routine
3484 * gather_bootmem_prealloc.
3485 * - For non-gigantic pages, this is called later in the boot process after
3486 * all of mm is up and functional. Pages are allocated from buddy and
3487 * then added to hugetlb pools.
3488 */
3270static void __init hugetlb_hstate_alloc_pages(struct hstate *h)3489static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
3271{3490{
3272 unsigned long i;3491 unsigned long i;
3492 struct folio *folio;
3493 LIST_HEAD(folio_list);
3273 nodemask_t *node_alloc_noretry;3494 nodemask_t *node_alloc_noretry;
3274 bool node_specific_alloc = false;3495 bool node_specific_alloc = false;
3275 3496
@@ -3311,14 +3532,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · 35 unmodified lines
3311 3532
3312 for (i = 0; i < h->max_huge_pages; ++i) {3533 for (i = 0; i < h->max_huge_pages; ++i) {
3313 if (hstate_is_gigantic(h)) {3534 if (hstate_is_gigantic(h)) {
3535 /*
3536 * gigantic pages not added to list as they are not
3537 * added to pools now.
3538 */
3314 if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))3539 if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
3315 break;3540 break;
3316 } else if (!alloc_pool_huge_page(h,3541 } else {
3317 &node_states[N_MEMORY],3542 folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY],
3318 node_alloc_noretry))3543 node_alloc_noretry);
3319 break;3544 if (!folio)
3545 break;
3546 list_add(&folio->lru, &folio_list);
3547 }
3320 cond_resched();3548 cond_resched();
3321 }3549 }
3550
3551 /* list will be empty if hstate_is_gigantic */
3552 prep_and_add_allocated_folios(h, &folio_list);
3553
3322 if (i < h->max_huge_pages) {3554 if (i < h->max_huge_pages) {
3323 char buf[32];3555 char buf[32];
3324 3556
@@ -3391,15 +3623,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, · 66 unmodified lines
3391 * Collect pages to be freed on a list, and free after dropping lock3623 * Collect pages to be freed on a list, and free after dropping lock
3392 */3624 */
3393 for_each_node_mask(i, *nodes_allowed) {3625 for_each_node_mask(i, *nodes_allowed) {
3394 struct page *page, *next;3626 struct folio *folio, *next;
3395 struct list_head *freel = &h->hugepage_freelists[i];3627 struct list_head *freel = &h->hugepage_freelists[i];
3396 list_for_each_entry_safe(page, next, freel, lru) {3628 list_for_each_entry_safe(folio, next, freel, lru) {
3397 if (count >= h->nr_huge_pages)3629 if (count >= h->nr_huge_pages)
3398 goto out;3630 goto out;
3399 if (PageHighMem(page))3631 if (folio_test_highmem(folio))
3400 continue;3632 continue;
3401 remove_hugetlb_folio(h, page_folio(page), false);3633 remove_hugetlb_folio(h, folio, false);
3402 list_add(&page->lru, &page_list);3634 list_add(&folio->lru, &page_list);
3403 }3635 }
3404 }3636 }
3405 3637
@@ -3452,8 +3684,9 @@ found: · 46 unmodified lines
3452static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,3684static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
3453 nodemask_t *nodes_allowed)3685 nodemask_t *nodes_allowed)
3454{3686{
3455 unsigned long min_count, ret;3687 unsigned long min_count;
3456 struct page *page;3688 unsigned long allocated;
3689 struct folio *folio;
3457 LIST_HEAD(page_list);3690 LIST_HEAD(page_list);
3458 NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL);3691 NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL);
3459 3692
@@ -3484,7 +3717,9 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 24 unmodified lines
3484 if (nid != NUMA_NO_NODE) {3717 if (nid != NUMA_NO_NODE) {
3485 unsigned long old_count = count;3718 unsigned long old_count = count;
3486 3719
3487 count += h->nr_huge_pages - h->nr_huge_pages_node[nid];3720 count += persistent_huge_pages(h) -
3721 (h->nr_huge_pages_node[nid] -
3722 h->surplus_huge_pages_node[nid]);
3488 /*3723 /*
3489 * User may have specified a large count value which caused the3724 * User may have specified a large count value which caused the
3490 * above calculation to overflow. In this case, they wanted3725 * above calculation to overflow. In this case, they wanted
@@ -3528,7 +3763,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 37 unmodified lines
3528 break;3763 break;
3529 }3764 }
3530 3765
3531 while (count > persistent_huge_pages(h)) {3766 allocated = 0;
3767 while (count > (persistent_huge_pages(h) + allocated)) {
3532 /*3768 /*
3533 * If this allocation races such that we no longer need the3769 * If this allocation races such that we no longer need the
3534 * page, free_huge_folio will handle it by freeing the page3770 * page, free_huge_folio will handle it by freeing the page
@@ -3539,15 +3775,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 4 unmodified lines
3539 /* yield cpu to avoid soft lockup */3775 /* yield cpu to avoid soft lockup */
3540 cond_resched();3776 cond_resched();
3541 3777
3542 ret = alloc_pool_huge_page(h, nodes_allowed,3778 folio = alloc_pool_huge_folio(h, nodes_allowed,
3543 node_alloc_noretry);3779 node_alloc_noretry);
3544 spin_lock_irq(&hugetlb_lock);3780 if (!folio) {
3545 if (!ret)3781 prep_and_add_allocated_folios(h, &page_list);
3782 spin_lock_irq(&hugetlb_lock);
3546 goto out;3783 goto out;
3784 }
3785
3786 list_add(&folio->lru, &page_list);
3787 allocated++;
3547 3788
3548 /* Bail for signals. Probably ctrl-c from user */3789 /* Bail for signals. Probably ctrl-c from user */
3549 if (signal_pending(current))3790 if (signal_pending(current)) {
3791 prep_and_add_allocated_folios(h, &page_list);
3792 spin_lock_irq(&hugetlb_lock);
3550 goto out;3793 goto out;
3794 }
3795
3796 spin_lock_irq(&hugetlb_lock);
3797 }
3798
3799 /* Add allocated pages to the pool */
3800 if (!list_empty(&page_list)) {
3801 spin_unlock_irq(&hugetlb_lock);
3802 prep_and_add_allocated_folios(h, &page_list);
3803 spin_lock_irq(&hugetlb_lock);
3551 }3804 }
3552 3805
3553 /*3806 /*
@@ -3573,11 +3826,11 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 19 unmodified lines
3573 * Collect pages to be removed on list without dropping lock3826 * Collect pages to be removed on list without dropping lock
3574 */3827 */
3575 while (min_count < persistent_huge_pages(h)) {3828 while (min_count < persistent_huge_pages(h)) {
3576 page = remove_pool_huge_page(h, nodes_allowed, 0);3829 folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0);
3577 if (!page)3830 if (!folio)
3578 break;3831 break;
3579 3832
3580 list_add(&page->lru, &page_list);3833 list_add(&folio->lru, &page_list);
3581 }3834 }
3582 /* free the pages after dropping lock */3835 /* free the pages after dropping lock */
3583 spin_unlock_irq(&hugetlb_lock);3836 spin_unlock_irq(&hugetlb_lock);
@@ -3612,13 +3865,21 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) · 28 unmodified lines
3612 remove_hugetlb_folio_for_demote(h, folio, false);3865 remove_hugetlb_folio_for_demote(h, folio, false);
3613 spin_unlock_irq(&hugetlb_lock);3866 spin_unlock_irq(&hugetlb_lock);
3614 3867
3615 rc = hugetlb_vmemmap_restore(h, &folio->page);3868 /*
3616 if (rc) {3869 * If vmemmap already existed for folio, the remove routine above would
3617 /* Allocation of vmemmmap failed, we can not demote folio */3870 * have cleared the hugetlb folio flag. Hence the folio is technically
3618 spin_lock_irq(&hugetlb_lock);3871 * no longer a hugetlb folio. hugetlb_vmemmap_restore_folio can only be
3619 folio_ref_unfreeze(folio, 1);3872 * passed hugetlb folios and will BUG otherwise.
3620 add_hugetlb_folio(h, folio, false);3873 */
3621 return rc;3874 if (folio_test_hugetlb(folio)) {
3875 rc = hugetlb_vmemmap_restore_folio(h, folio);
3876 if (rc) {
3877 /* Allocation of vmemmmap failed, we can not demote folio */
3878 spin_lock_irq(&hugetlb_lock);
3879 folio_ref_unfreeze(folio, 1);
3880 add_hugetlb_folio(h, folio, false);
3881 return rc;
3882 }
3622 }3883 }
3623 3884
3624 /*3885 /*
@@ -4314,7 +4575,7 @@ void __init hugetlb_add_hstate(unsigned int order) · 689 unmodified lines
4314 return;4575 return;
4315 }4576 }
4316 BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE);4577 BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE);
4317 BUG_ON(order == 0);4578 BUG_ON(order < order_base_2(__NR_USED_SUBPAGE));
4318 h = &hstates[hugetlb_max_hstate++];4579 h = &hstates[hugetlb_max_hstate++];
4319 mutex_init(&h->resize_lock);4580 mutex_init(&h->resize_lock);
4320 h->order = order;4581 h->order = order;
@@ -4997,7 +5258,7 @@ bool is_hugetlb_entry_migration(pte_t pte) · 676 unmodified lines
4997 return false;5258 return false;
4998}5259}
4999 5260
5000static bool is_hugetlb_entry_hwpoisoned(pte_t pte)5261bool is_hugetlb_entry_hwpoisoned(pte_t pte)
5001{5262{
5002 swp_entry_t swp;5263 swp_entry_t swp;
5003 5264
@@ -5605,8 +5866,10 @@ retry_avoidcopy: · 601 unmodified lines
5605 * owner and can reuse this page.5866 * owner and can reuse this page.
5606 */5867 */
5607 if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) {5868 if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) {
5608 if (!PageAnonExclusive(&old_folio->page))5869 if (!PageAnonExclusive(&old_folio->page)) {
5609 page_move_anon_rmap(&old_folio->page, vma);5870 folio_move_anon_rmap(old_folio, vma);
5871 SetPageAnonExclusive(&old_folio->page);
5872 }
5610 if (likely(!unshare))5873 if (likely(!unshare))
5611 set_huge_ptep_writable(vma, haddr, ptep);5874 set_huge_ptep_writable(vma, haddr, ptep);
5612 5875
@@ -5752,7 +6015,7 @@ static bool hugetlbfs_pagecache_present(struct hstate *h, · 139 unmodified lines
5752 struct vm_area_struct *vma, unsigned long address)6015 struct vm_area_struct *vma, unsigned long address)
5753{6016{
5754 struct address_space *mapping = vma->vm_file->f_mapping;6017 struct address_space *mapping = vma->vm_file->f_mapping;
5755 pgoff_t idx = vma_hugecache_offset(h, vma, address);6018 pgoff_t idx = linear_page_index(vma, address);
5756 struct folio *folio;6019 struct folio *folio;
5757 6020
5758 folio = filemap_get_folio(mapping, idx);6021 folio = filemap_get_folio(mapping, idx);
@@ -5769,6 +6032,7 @@ int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping · 10 unmodified lines
5769 struct hstate *h = hstate_inode(inode);6032 struct hstate *h = hstate_inode(inode);
5770 int err;6033 int err;
5771 6034
6035 idx <<= huge_page_order(h);
5772 __folio_set_locked(folio);6036 __folio_set_locked(folio);
5773 err = __filemap_add_folio(mapping, folio, idx, GFP_KERNEL, NULL);6037 err = __filemap_add_folio(mapping, folio, idx, GFP_KERNEL, NULL);
5774 6038
@@ -5876,7 +6140,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, · 101 unmodified lines
5876 * before we get page_table_lock.6140 * before we get page_table_lock.
5877 */6141 */
5878 new_folio = false;6142 new_folio = false;
5879 folio = filemap_lock_folio(mapping, idx);6143 folio = filemap_lock_hugetlb_folio(h, mapping, idx);
5880 if (IS_ERR(folio)) {6144 if (IS_ERR(folio)) {
5881 size = i_size_read(mapping->host) >> huge_page_shift(h);6145 size = i_size_read(mapping->host) >> huge_page_shift(h);
5882 if (idx >= size)6146 if (idx >= size)
@@ -6185,7 +6449,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 302 unmodified lines
6185 /* Just decrements count, does not deallocate */6449 /* Just decrements count, does not deallocate */
6186 vma_end_reservation(h, vma, haddr);6450 vma_end_reservation(h, vma, haddr);
6187 6451
6188 pagecache_folio = filemap_lock_folio(mapping, idx);6452 pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, idx);
6189 if (IS_ERR(pagecache_folio))6453 if (IS_ERR(pagecache_folio))
6190 pagecache_folio = NULL;6454 pagecache_folio = NULL;
6191 }6455 }
@@ -6199,21 +6463,28 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 7 unmodified lines
6199 /* Handle userfault-wp first, before trying to lock more pages */6463 /* Handle userfault-wp first, before trying to lock more pages */
6200 if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&6464 if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
6201 (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {6465 (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
6202 struct vm_fault vmf = {6466 if (!userfaultfd_wp_async(vma)) {
6203 .vma = vma,6467 struct vm_fault vmf = {
6204 .address = haddr,6468 .vma = vma,
6205 .real_address = address,6469 .address = haddr,
6206 .flags = flags,6470 .real_address = address,
6207 };6471 .flags = flags,
6472 };
6208 6473
6209 spin_unlock(ptl);6474 spin_unlock(ptl);
6210 if (pagecache_folio) {6475 if (pagecache_folio) {
6211 folio_unlock(pagecache_folio);6476 folio_unlock(pagecache_folio);
6212 folio_put(pagecache_folio);6477 folio_put(pagecache_folio);
6478 }
6479 hugetlb_vma_unlock_read(vma);
6480 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6481 return handle_userfault(&vmf, VM_UFFD_WP);
6213 }6482 }
6214 hugetlb_vma_unlock_read(vma);6483
6215 mutex_unlock(&hugetlb_fault_mutex_table[hash]);6484 entry = huge_pte_clear_uffd_wp(entry);
6216 return handle_userfault(&vmf, VM_UFFD_WP);6485 set_huge_pte_at(mm, haddr, ptep, entry,
6486 huge_page_size(hstate_vma(vma)));
6487 /* Fallthrough to CoW */
6217 }6488 }
6218 6489
6219 /*6490 /*
@@ -6270,6 +6541,26 @@ out_mutex: · 50 unmodified lines
6270}6541}
6271 6542
6272#ifdef CONFIG_USERFAULTFD6543#ifdef CONFIG_USERFAULTFD
6544/*
6545 * Can probably be eliminated, but still used by hugetlb_mfill_atomic_pte().
6546 */
6547static struct folio *alloc_hugetlb_folio_vma(struct hstate *h,
6548 struct vm_area_struct *vma, unsigned long address)
6549{
6550 struct mempolicy *mpol;
6551 nodemask_t *nodemask;
6552 struct folio *folio;
6553 gfp_t gfp_mask;
6554 int node;
6555
6556 gfp_mask = htlb_alloc_mask(h);
6557 node = huge_node(vma, address, gfp_mask, &mpol, &nodemask);
6558 folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask);
6559 mpol_cond_put(mpol);
6560
6561 return folio;
6562}
6563
6273/*6564/*
6274 * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte6565 * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte
6275 * with modifications for hugetlb pages.6566 * with modifications for hugetlb pages.
@@ -6318,7 +6609,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 42 unmodified lines
6318 6609
6319 if (is_continue) {6610 if (is_continue) {
6320 ret = -EFAULT;6611 ret = -EFAULT;
6321 folio = filemap_lock_folio(mapping, idx);6612 folio = filemap_lock_hugetlb_folio(h, mapping, idx);
6322 if (IS_ERR(folio))6613 if (IS_ERR(folio))
6323 goto out;6614 goto out;
6324 folio_in_pagecache = true;6615 folio_in_pagecache = true;
@@ -6520,7 +6811,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, · 195 unmodified lines
6520 }6811 }
6521 }6812 }
6522 6813
6523 page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT);6814 page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT));
6524 6815
6525 /*6816 /*
6526 * Note that page may be a sub-page, and with vmemmap6817 * Note that page may be a sub-page, and with vmemmap
mm/hugetlb_cgroup.c+2−18
@@ -262,12 +262,6 @@ static int __hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages, · 261 unmodified lines
262 262
263 if (hugetlb_cgroup_disabled())263 if (hugetlb_cgroup_disabled())
264 goto done;264 goto done;
265 /*
266 * We don't charge any cgroup if the compound page have less
267 * than 3 pages.
268 */
269 if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER)
270 goto done;
271again:265again:
272 rcu_read_lock();266 rcu_read_lock();
273 h_cg = hugetlb_cgroup_from_task(current);267 h_cg = hugetlb_cgroup_from_task(current);
@@ -397,9 +391,6 @@ static void __hugetlb_cgroup_uncharge_cgroup(int idx, unsigned long nr_pages, · 123 unmodified lines
397 if (hugetlb_cgroup_disabled() || !h_cg)391 if (hugetlb_cgroup_disabled() || !h_cg)
398 return;392 return;
399 393
400 if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER)
401 return;
402
403 page_counter_uncharge(__hugetlb_cgroup_counter_from_cgroup(h_cg, idx,394 page_counter_uncharge(__hugetlb_cgroup_counter_from_cgroup(h_cg, idx,
404 rsvd),395 rsvd),
405 nr_pages);396 nr_pages);
@@ -869,15 +860,8 @@ void __init hugetlb_cgroup_file_init(void) · 463 unmodified lines
869{860{
870 struct hstate *h;861 struct hstate *h;
871 862
872 for_each_hstate(h) {863 for_each_hstate(h)
873 /*864 __hugetlb_cgroup_file_init(hstate_index(h));
874 * Add cgroup control files only if the huge page consists
875 * of more than two normal pages. This is because we use
876 * page[2].private for storing cgroup details.
877 */
878 if (huge_page_order(h) >= HUGETLB_CGROUP_MIN_ORDER)
879 __hugetlb_cgroup_file_init(hstate_index(h));
880 }
881}865}
882 866
883/*867/*
mm/hugetlb_vmemmap.c+252−49
@@ -13,6 +13,7 @@ · 12 unmodified lines
13#include <linux/pgtable.h>13#include <linux/pgtable.h>
14#include <linux/moduleparam.h>14#include <linux/moduleparam.h>
15#include <linux/bootmem_info.h>15#include <linux/bootmem_info.h>
16#include <linux/mmdebug.h>
16#include <asm/pgalloc.h>17#include <asm/pgalloc.h>
17#include <asm/tlbflush.h>18#include <asm/tlbflush.h>
18#include "hugetlb_vmemmap.h"19#include "hugetlb_vmemmap.h"
@@ -26,6 +27,8 @@ · 7 unmodified lines
26 * @reuse_addr: the virtual address of the @reuse_page page.27 * @reuse_addr: the virtual address of the @reuse_page page.
27 * @vmemmap_pages: the list head of the vmemmap pages that can be freed28 * @vmemmap_pages: the list head of the vmemmap pages that can be freed
28 * or is mapped from.29 * or is mapped from.
30 * @flags: used to modify behavior in vmemmap page table walking
31 * operations.
29 */32 */
30struct vmemmap_remap_walk {33struct vmemmap_remap_walk {
31 void (*remap_pte)(pte_t *pte, unsigned long addr,34 void (*remap_pte)(pte_t *pte, unsigned long addr,
@@ -34,9 +37,15 @@ struct vmemmap_remap_walk { · 2 unmodified lines
34 struct page *reuse_page;37 struct page *reuse_page;
35 unsigned long reuse_addr;38 unsigned long reuse_addr;
36 struct list_head *vmemmap_pages;39 struct list_head *vmemmap_pages;
40
41/* Skip the TLB flush when we split the PMD */
42#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0)
43/* Skip the TLB flush when we remap the PTE */
44#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1)
45 unsigned long flags;
37};46};
38 47
39static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start)48static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush)
40{49{
41 pmd_t __pmd;50 pmd_t __pmd;
42 int i;51 int i;
@@ -79,7 +88,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) · 36 unmodified lines
79 /* Make pte visible before pmd. See comment in pmd_install(). */88 /* Make pte visible before pmd. See comment in pmd_install(). */
80 smp_wmb();89 smp_wmb();
81 pmd_populate_kernel(&init_mm, pmd, pgtable);90 pmd_populate_kernel(&init_mm, pmd, pgtable);
82 flush_tlb_kernel_range(start, start + PMD_SIZE);91 if (flush)
92 flush_tlb_kernel_range(start, start + PMD_SIZE);
83 } else {93 } else {
84 pte_free_kernel(&init_mm, pgtable);94 pte_free_kernel(&init_mm, pgtable);
85 }95 }
@@ -126,11 +136,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, · 40 unmodified lines
126 do {136 do {
127 int ret;137 int ret;
128 138
129 ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK);139 ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK,
140 !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH));
130 if (ret)141 if (ret)
131 return ret;142 return ret;
132 143
133 next = pmd_addr_end(addr, end);144 next = pmd_addr_end(addr, end);
145
146 /*
147 * We are only splitting, not remapping the hugetlb vmemmap
148 * pages.
149 */
150 if (!walk->remap_pte)
151 continue;
152
134 vmemmap_pte_range(pmd, addr, next, walk);153 vmemmap_pte_range(pmd, addr, next, walk);
135 } while (pmd++, addr = next, addr != end);154 } while (pmd++, addr = next, addr != end);
136 155
@@ -197,7 +216,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, · 60 unmodified lines
197 return ret;216 return ret;
198 } while (pgd++, addr = next, addr != end);217 } while (pgd++, addr = next, addr != end);
199 218
200 flush_tlb_kernel_range(start, end);219 if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH))
220 flush_tlb_kernel_range(start, end);
201 221
202 return 0;222 return 0;
203}223}
@@ -250,7 +270,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, · 46 unmodified lines
250 }270 }
251 271
252 entry = mk_pte(walk->reuse_page, pgprot);272 entry = mk_pte(walk->reuse_page, pgprot);
253 list_add_tail(&page->lru, walk->vmemmap_pages);273 list_add(&page->lru, walk->vmemmap_pages);
254 set_pte_at(&init_mm, addr, pte, entry);274 set_pte_at(&init_mm, addr, pte, entry);
255}275}
256 276
@@ -296,6 +316,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 39 unmodified lines
296 set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));316 set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));
297}317}
298 318
319/**
320 * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end)
321 * backing PMDs of the directmap into PTEs
322 * @start: start address of the vmemmap virtual address range that we want
323 * to remap.
324 * @end: end address of the vmemmap virtual address range that we want to
325 * remap.
326 * @reuse: reuse address.
327 *
328 * Return: %0 on success, negative error code otherwise.
329 */
330static int vmemmap_remap_split(unsigned long start, unsigned long end,
331 unsigned long reuse)
332{
333 int ret;
334 struct vmemmap_remap_walk walk = {
335 .remap_pte = NULL,
336 .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH,
337 };
338
339 /* See the comment in the vmemmap_remap_free(). */
340 BUG_ON(start - reuse != PAGE_SIZE);
341
342 mmap_read_lock(&init_mm);
343 ret = vmemmap_remap_range(reuse, end, &walk);
344 mmap_read_unlock(&init_mm);
345
346 return ret;
347}
348
299/**349/**
300 * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end)350 * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end)
301 * to the page which @reuse is mapped to, then free vmemmap351 * to the page which @reuse is mapped to, then free vmemmap
@@ -305,22 +355,26 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 3 unmodified lines
305 * @end: end address of the vmemmap virtual address range that we want to355 * @end: end address of the vmemmap virtual address range that we want to
306 * remap.356 * remap.
307 * @reuse: reuse address.357 * @reuse: reuse address.
358 * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers
359 * responsibility to free pages.
360 * @flags: modifications to vmemmap_remap_walk flags
308 *361 *
309 * Return: %0 on success, negative error code otherwise.362 * Return: %0 on success, negative error code otherwise.
310 */363 */
311static int vmemmap_remap_free(unsigned long start, unsigned long end,364static int vmemmap_remap_free(unsigned long start, unsigned long end,
312 unsigned long reuse)365 unsigned long reuse,
366 struct list_head *vmemmap_pages,
367 unsigned long flags)
313{368{
314 int ret;369 int ret;
315 LIST_HEAD(vmemmap_pages);
316 struct vmemmap_remap_walk walk = {370 struct vmemmap_remap_walk walk = {
317 .remap_pte = vmemmap_remap_pte,371 .remap_pte = vmemmap_remap_pte,
318 .reuse_addr = reuse,372 .reuse_addr = reuse,
319 .vmemmap_pages = &vmemmap_pages,373 .vmemmap_pages = vmemmap_pages,
374 .flags = flags,
320 };375 };
321 int nid = page_to_nid((struct page *)start);376 int nid = page_to_nid((struct page *)reuse);
322 gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY |377 gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN;
323 __GFP_NOWARN;
324 378
325 /*379 /*
326 * Allocate a new head vmemmap page to avoid breaking a contiguous380 * Allocate a new head vmemmap page to avoid breaking a contiguous
@@ -334,7 +388,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 7 unmodified lines
334 if (walk.reuse_page) {388 if (walk.reuse_page) {
335 copy_page(page_to_virt(walk.reuse_page),389 copy_page(page_to_virt(walk.reuse_page),
336 (void *)walk.reuse_addr);390 (void *)walk.reuse_addr);
337 list_add(&walk.reuse_page->lru, &vmemmap_pages);391 list_add(&walk.reuse_page->lru, vmemmap_pages);
338 }392 }
339 393
340 /*394 /*
@@ -365,22 +419,21 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 24 unmodified lines
365 walk = (struct vmemmap_remap_walk) {419 walk = (struct vmemmap_remap_walk) {
366 .remap_pte = vmemmap_restore_pte,420 .remap_pte = vmemmap_restore_pte,
367 .reuse_addr = reuse,421 .reuse_addr = reuse,
368 .vmemmap_pages = &vmemmap_pages,422 .vmemmap_pages = vmemmap_pages,
423 .flags = 0,
369 };424 };
370 425
371 vmemmap_remap_range(reuse, end, &walk);426 vmemmap_remap_range(reuse, end, &walk);
372 }427 }
373 mmap_read_unlock(&init_mm);428 mmap_read_unlock(&init_mm);
374 429
375 free_vmemmap_page_list(&vmemmap_pages);
376
377 return ret;430 return ret;
378}431}
379 432
380static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,433static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
381 struct list_head *list)434 struct list_head *list)
382{435{
383 gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_THISNODE;436 gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL;
384 unsigned long nr_pages = (end - start) >> PAGE_SHIFT;437 unsigned long nr_pages = (end - start) >> PAGE_SHIFT;
385 int nid = page_to_nid((struct page *)start);438 int nid = page_to_nid((struct page *)start);
386 struct page *page, *next;439 struct page *page, *next;
@@ -389,7 +442,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, · 2 unmodified lines
389 page = alloc_pages_node(nid, gfp_mask, 0);442 page = alloc_pages_node(nid, gfp_mask, 0);
390 if (!page)443 if (!page)
391 goto out;444 goto out;
392 list_add_tail(&page->lru, list);445 list_add(&page->lru, list);
393 }446 }
394 447
395 return 0;448 return 0;
@@ -408,17 +461,19 @@ out: · 12 unmodified lines
408 * @end: end address of the vmemmap virtual address range that we want to461 * @end: end address of the vmemmap virtual address range that we want to
409 * remap.462 * remap.
410 * @reuse: reuse address.463 * @reuse: reuse address.
464 * @flags: modifications to vmemmap_remap_walk flags
411 *465 *
412 * Return: %0 on success, negative error code otherwise.466 * Return: %0 on success, negative error code otherwise.
413 */467 */
414static int vmemmap_remap_alloc(unsigned long start, unsigned long end,468static int vmemmap_remap_alloc(unsigned long start, unsigned long end,
415 unsigned long reuse)469 unsigned long reuse, unsigned long flags)
416{470{
417 LIST_HEAD(vmemmap_pages);471 LIST_HEAD(vmemmap_pages);
418 struct vmemmap_remap_walk walk = {472 struct vmemmap_remap_walk walk = {
419 .remap_pte = vmemmap_restore_pte,473 .remap_pte = vmemmap_restore_pte,
420 .reuse_addr = reuse,474 .reuse_addr = reuse,
421 .vmemmap_pages = &vmemmap_pages,475 .vmemmap_pages = &vmemmap_pages,
476 .flags = flags,
422 };477 };
423 478
424 /* See the comment in the vmemmap_remap_free(). */479 /* See the comment in the vmemmap_remap_free(). */
@@ -440,23 +495,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); · 15 unmodified lines
440static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);495static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
441core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0);496core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0);
442 497
443/**498static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio, unsigned long flags)
444 * hugetlb_vmemmap_restore - restore previously optimized (by
445 * hugetlb_vmemmap_optimize()) vmemmap pages which
446 * will be reallocated and remapped.
447 * @h: struct hstate.
448 * @head: the head page whose vmemmap pages will be restored.
449 *
450 * Return: %0 if @head's vmemmap pages have been reallocated and remapped,
451 * negative error code otherwise.
452 */
453int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
454{499{
455 int ret;500 int ret;
501 struct page *head = &folio->page;
456 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;502 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
457 unsigned long vmemmap_reuse;503 unsigned long vmemmap_reuse;
458 504
459 if (!HPageVmemmapOptimized(head))505 VM_WARN_ON_ONCE(!PageHuge(head));
506 if (!folio_test_hugetlb_vmemmap_optimized(folio))
460 return 0;507 return 0;
461 508
462 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);509 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
@@ -470,18 +517,77 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) · 7 unmodified lines
470 * When a HugeTLB page is freed to the buddy allocator, previously517 * When a HugeTLB page is freed to the buddy allocator, previously
471 * discarded vmemmap pages must be allocated and remapping.518 * discarded vmemmap pages must be allocated and remapping.
472 */519 */
473 ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse);520 ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags);
474 if (!ret) {521 if (!ret) {
475 ClearHPageVmemmapOptimized(head);522 folio_clear_hugetlb_vmemmap_optimized(folio);
476 static_branch_dec(&hugetlb_optimize_vmemmap_key);523 static_branch_dec(&hugetlb_optimize_vmemmap_key);
477 }524 }
478 525
479 return ret;526 return ret;
480}527}
481 528
529/**
530 * hugetlb_vmemmap_restore_folio - restore previously optimized (by
531 * hugetlb_vmemmap_optimize_folio()) vmemmap pages which
532 * will be reallocated and remapped.
533 * @h: struct hstate.
534 * @folio: the folio whose vmemmap pages will be restored.
535 *
536 * Return: %0 if @folio's vmemmap pages have been reallocated and remapped,
537 * negative error code otherwise.
538 */
539int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio)
540{
541 return __hugetlb_vmemmap_restore_folio(h, folio, 0);
542}
543
544/**
545 * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list.
546 * @h: hstate.
547 * @folio_list: list of folios.
548 * @non_hvo_folios: Output list of folios for which vmemmap exists.
549 *
550 * Return: number of folios for which vmemmap was restored, or an error code
551 * if an error was encountered restoring vmemmap for a folio.
552 * Folios that have vmemmap are moved to the non_hvo_folios
553 * list. Processing of entries stops when the first error is
554 * encountered. The folio that experienced the error and all
555 * non-processed folios will remain on folio_list.
556 */
557long hugetlb_vmemmap_restore_folios(const struct hstate *h,
558 struct list_head *folio_list,
559 struct list_head *non_hvo_folios)
560{
561 struct folio *folio, *t_folio;
562 long restored = 0;
563 long ret = 0;
564
565 list_for_each_entry_safe(folio, t_folio, folio_list, lru) {
566 if (folio_test_hugetlb_vmemmap_optimized(folio)) {
567 ret = __hugetlb_vmemmap_restore_folio(h, folio,
568 VMEMMAP_REMAP_NO_TLB_FLUSH);
569 if (ret)
570 break;
571 restored++;
572 }
573
574 /* Add non-optimized folios to output list */
575 list_move(&folio->lru, non_hvo_folios);
576 }
577
578 if (restored)
579 flush_tlb_all();
580 if (!ret)
581 ret = restored;
582 return ret;
583}
584
482/* Return true iff a HugeTLB whose vmemmap should and can be optimized. */585/* Return true iff a HugeTLB whose vmemmap should and can be optimized. */
483static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head)586static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head)
484{587{
588 if (HPageVmemmapOptimized((struct page *)head))
589 return false;
590
485 if (!READ_ONCE(vmemmap_optimize_enabled))591 if (!READ_ONCE(vmemmap_optimize_enabled))
486 return false;592 return false;
487 593
@@ -535,39 +641,136 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h · 47 unmodified lines
535 return true;641 return true;
536}642}
537 643
644static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
645 struct folio *folio,
646 struct list_head *vmemmap_pages,
647 unsigned long flags)
648{
649 int ret = 0;
650 struct page *head = &folio->page;
651 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
652 unsigned long vmemmap_reuse;
653
654 VM_WARN_ON_ONCE(!PageHuge(head));
655 if (!vmemmap_should_optimize(h, head))
656 return ret;
657
658 static_branch_inc(&hugetlb_optimize_vmemmap_key);
659 /*
660 * Very Subtle
661 * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed
662 * immediately after remapping. As a result, subsequent accesses
663 * and modifications to struct pages associated with the hugetlb
664 * page could be to the OLD struct pages. Set the vmemmap optimized
665 * flag here so that it is copied to the new head page. This keeps
666 * the old and new struct pages in sync.
667 * If there is an error during optimization, we will immediately FLUSH
668 * the TLB and clear the flag below.
669 */
670 folio_set_hugetlb_vmemmap_optimized(folio);
671
672 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
673 vmemmap_reuse = vmemmap_start;
674 vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
675
676 /*
677 * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
678 * to the page which @vmemmap_reuse is mapped to. Add pages previously
679 * mapping the range to vmemmap_pages list so that they can be freed by
680 * the caller.
681 */
682 ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse,
683 vmemmap_pages, flags);
684 if (ret) {
685 static_branch_dec(&hugetlb_optimize_vmemmap_key);
686 folio_clear_hugetlb_vmemmap_optimized(folio);
687 }
688
689 return ret;
690}
691
538/**692/**
539 * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages.693 * hugetlb_vmemmap_optimize_folio - optimize @folio's vmemmap pages.
540 * @h: struct hstate.694 * @h: struct hstate.
541 * @head: the head page whose vmemmap pages will be optimized.695 * @folio: the folio whose vmemmap pages will be optimized.
542 *696 *
543 * This function only tries to optimize @head's vmemmap pages and does not697 * This function only tries to optimize @folio's vmemmap pages and does not
544 * guarantee that the optimization will succeed after it returns. The caller698 * guarantee that the optimization will succeed after it returns. The caller
545 * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages699 * can use folio_test_hugetlb_vmemmap_optimized(@folio) to detect if @folio's
546 * have been optimized.700 * vmemmap pages have been optimized.
547 */701 */
548void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)702void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
703{
704 LIST_HEAD(vmemmap_pages);
705
706 __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0);
707 free_vmemmap_page_list(&vmemmap_pages);
708}
709
710static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head)
549{711{
550 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;712 unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
551 unsigned long vmemmap_reuse;713 unsigned long vmemmap_reuse;
552 714
553 if (!vmemmap_should_optimize(h, head))715 if (!vmemmap_should_optimize(h, head))
554 return;716 return 0;
555
556 static_branch_inc(&hugetlb_optimize_vmemmap_key);
557 717
558 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);718 vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
559 vmemmap_reuse = vmemmap_start;719 vmemmap_reuse = vmemmap_start;
560 vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;720 vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
561 721
562 /*722 /*
563 * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)723 * Split PMDs on the vmemmap virtual address range [@vmemmap_start,
564 * to the page which @vmemmap_reuse is mapped to, then free the pages724 * @vmemmap_end]
565 * which the range [@vmemmap_start, @vmemmap_end] is mapped to.
566 */725 */
567 if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse))726 return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse);
568 static_branch_dec(&hugetlb_optimize_vmemmap_key);727}
569 else728
570 SetHPageVmemmapOptimized(head);729void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
730{
731 struct folio *folio;
732 LIST_HEAD(vmemmap_pages);
733
734 list_for_each_entry(folio, folio_list, lru) {
735 int ret = hugetlb_vmemmap_split(h, &folio->page);
736
737 /*
738 * Spliting the PMD requires allocating a page, thus lets fail
739 * early once we encounter the first OOM. No point in retrying
740 * as it can be dynamically done on remap with the memory
741 * we get back from the vmemmap deduplication.
742 */
743 if (ret == -ENOMEM)
744 break;
745 }
746
747 flush_tlb_all();
748
749 list_for_each_entry(folio, folio_list, lru) {
750 int ret = __hugetlb_vmemmap_optimize_folio(h, folio,
751 &vmemmap_pages,
752 VMEMMAP_REMAP_NO_TLB_FLUSH);
753
754 /*
755 * Pages to be freed may have been accumulated. If we
756 * encounter an ENOMEM, free what we have and try again.
757 * This can occur in the case that both spliting fails
758 * halfway and head page allocation also failed. In this
759 * case __hugetlb_vmemmap_optimize_folio() would free memory
760 * allowing more vmemmap remaps to occur.
761 */
762 if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) {
763 flush_tlb_all();
764 free_vmemmap_page_list(&vmemmap_pages);
765 INIT_LIST_HEAD(&vmemmap_pages);
766 __hugetlb_vmemmap_optimize_folio(h, folio,
767 &vmemmap_pages,
768 VMEMMAP_REMAP_NO_TLB_FLUSH);
769 }
770 }
771
772 flush_tlb_all();
773 free_vmemmap_page_list(&vmemmap_pages);
571}774}
572 775
573static struct ctl_table hugetlb_vmemmap_sysctls[] = {776static struct ctl_table hugetlb_vmemmap_sysctls[] = {
@@ -586,7 +789,7 @@ static int __init hugetlb_vmemmap_init(void) · 12 unmodified lines
586 const struct hstate *h;789 const struct hstate *h;
587 790
588 /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */791 /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */
589 BUILD_BUG_ON(__NR_USED_SUBPAGE * sizeof(struct page) > HUGETLB_VMEMMAP_RESERVE_SIZE);792 BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES);
590 793
591 for_each_hstate(h) {794 for_each_hstate(h) {
592 if (hugetlb_vmemmap_optimizable(h)) {795 if (hugetlb_vmemmap_optimizable(h)) {
mm/hugetlb_vmemmap.h+24−7
@@ -10,15 +10,20 @@ · 9 unmodified lines
10#define _LINUX_HUGETLB_VMEMMAP_H10#define _LINUX_HUGETLB_VMEMMAP_H
11#include <linux/hugetlb.h>11#include <linux/hugetlb.h>
12 12
13#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
14int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head);
15void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head);
16
17/*13/*
18 * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See14 * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See
19 * Documentation/vm/vmemmap_dedup.rst.15 * Documentation/mm/vmemmap_dedup.rst.
20 */16 */
21#define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE17#define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE
18#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page))
19
20#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
21int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio);
22long hugetlb_vmemmap_restore_folios(const struct hstate *h,
23 struct list_head *folio_list,
24 struct list_head *non_hvo_folios);
25void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio);
26void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
22 27
23static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)28static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)
24{29{
@@ -38,12 +43,24 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate · 13 unmodified lines
38 return size > 0 ? size : 0;43 return size > 0 ? size : 0;
39}44}
40#else45#else
41static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)46static inline int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio)
42{47{
43 return 0;48 return 0;
44}49}
45 50
46static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)51static long hugetlb_vmemmap_restore_folios(const struct hstate *h,
52 struct list_head *folio_list,
53 struct list_head *non_hvo_folios)
54{
55 list_splice_init(folio_list, non_hvo_folios);
56 return 0;
57}
58
59static inline void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
60{
61}
62
63static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
47{64{
48}65}
49 66
mm/internal.h+136−11
@@ -415,6 +415,15 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) · 414 unmodified lines
415 415
416void folio_undo_large_rmappable(struct folio *folio);416void folio_undo_large_rmappable(struct folio *folio);
417 417
418static inline struct folio *page_rmappable_folio(struct page *page)
419{
420 struct folio *folio = (struct folio *)page;
421
422 if (folio && folio_order(folio) > 1)
423 folio_prep_large_rmappable(folio);
424 return folio;
425}
426
418static inline void prep_compound_head(struct page *page, unsigned int order)427static inline void prep_compound_head(struct page *page, unsigned int order)
419{428{
420 struct folio *folio = (struct folio *)page;429 struct folio *folio = (struct folio *)page;
@@ -586,6 +595,56 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, · 165 unmodified lines
586 bool write, int *locked);595 bool write, int *locked);
587extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags,596extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags,
588 unsigned long bytes);597 unsigned long bytes);
598
599/*
600 * NOTE: This function can't tell whether the folio is "fully mapped" in the
601 * range.
602 * "fully mapped" means all the pages of folio is associated with the page
603 * table of range while this function just check whether the folio range is
604 * within the range [start, end). Function caller needs to do page table
605 * check if it cares about the page table association.
606 *
607 * Typical usage (like mlock or madvise) is:
608 * Caller knows at least 1 page of folio is associated with page table of VMA
609 * and the range [start, end) is intersect with the VMA range. Caller wants
610 * to know whether the folio is fully associated with the range. It calls
611 * this function to check whether the folio is in the range first. Then checks
612 * the page table to know whether the folio is fully mapped to the range.
613 */
614static inline bool
615folio_within_range(struct folio *folio, struct vm_area_struct *vma,
616 unsigned long start, unsigned long end)
617{
618 pgoff_t pgoff, addr;
619 unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
620
621 VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio);
622 if (start > end)
623 return false;
624
625 if (start < vma->vm_start)
626 start = vma->vm_start;
627
628 if (end > vma->vm_end)
629 end = vma->vm_end;
630
631 pgoff = folio_pgoff(folio);
632
633 /* if folio start address is not in vma range */
634 if (!in_range(pgoff, vma->vm_pgoff, vma_pglen))
635 return false;
636
637 addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
638
639 return !(addr < start || end - addr < folio_size(folio));
640}
641
642static inline bool
643folio_within_vma(struct folio *folio, struct vm_area_struct *vma)
644{
645 return folio_within_range(folio, vma, vma->vm_start, vma->vm_end);
646}
647
589/*648/*
590 * mlock_vma_folio() and munlock_vma_folio():649 * mlock_vma_folio() and munlock_vma_folio():
591 * should be called with vma's mmap_lock held for read or write,650 * should be called with vma's mmap_lock held for read or write,
@@ -594,14 +653,10 @@ extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, · 2 unmodified lines
594 * mlock is usually called at the end of page_add_*_rmap(), munlock at653 * mlock is usually called at the end of page_add_*_rmap(), munlock at
595 * the end of page_remove_rmap(); but new anon folios are managed by654 * the end of page_remove_rmap(); but new anon folios are managed by
596 * folio_add_lru_vma() calling mlock_new_folio().655 * folio_add_lru_vma() calling mlock_new_folio().
597 *
598 * @compound is used to include pmd mappings of THPs, but filter out
599 * pte mappings of THPs, which cannot be consistently counted: a pte
600 * mapping of the THP head cannot be distinguished by the page alone.
601 */656 */
602void mlock_folio(struct folio *folio);657void mlock_folio(struct folio *folio);
603static inline void mlock_vma_folio(struct folio *folio,658static inline void mlock_vma_folio(struct folio *folio,
604 struct vm_area_struct *vma, bool compound)659 struct vm_area_struct *vma)
605{660{
606 /*661 /*
607 * The VM_SPECIAL check here serves two purposes.662 * The VM_SPECIAL check here serves two purposes.
@@ -611,17 +666,24 @@ static inline void mlock_vma_folio(struct folio *folio, · 3 unmodified lines
611 * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may666 * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may
612 * still be set while VM_SPECIAL bits are added: so ignore it then.667 * still be set while VM_SPECIAL bits are added: so ignore it then.
613 */668 */
614 if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) &&669 if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED))
615 (compound || !folio_test_large(folio)))
616 mlock_folio(folio);670 mlock_folio(folio);
617}671}
618 672
619void munlock_folio(struct folio *folio);673void munlock_folio(struct folio *folio);
620static inline void munlock_vma_folio(struct folio *folio,674static inline void munlock_vma_folio(struct folio *folio,
621 struct vm_area_struct *vma, bool compound)675 struct vm_area_struct *vma)
622{676{
623 if (unlikely(vma->vm_flags & VM_LOCKED) &&677 /*
624 (compound || !folio_test_large(folio)))678 * munlock if the function is called. Ideally, we should only
679 * do munlock if any page of folio is unmapped from VMA and
680 * cause folio not fully mapped to VMA.
681 *
682 * But it's not easy to confirm that's the situation. So we
683 * always munlock the folio and page reclaim will correct it
684 * if it's wrong.
685 */
686 if (unlikely(vma->vm_flags & VM_LOCKED))
625 munlock_folio(folio);687 munlock_folio(folio);
626}688}
627 689
@@ -930,7 +992,7 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); · 302 unmodified lines
930 992
931void __vunmap_range_noflush(unsigned long start, unsigned long end);993void __vunmap_range_noflush(unsigned long start, unsigned long end);
932 994
933int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,995int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma,
934 unsigned long addr, int page_nid, int *flags);996 unsigned long addr, int page_nid, int *flags);
935 997
936void free_zone_device_page(struct page *page);998void free_zone_device_page(struct page *page);
@@ -949,6 +1011,13 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · 12 unmodified lines
949 unsigned long addr, pmd_t *pmd,1011 unsigned long addr, pmd_t *pmd,
950 unsigned int flags);1012 unsigned int flags);
951 1013
1014/*
1015 * mm/mmap.c
1016 */
1017struct vm_area_struct *vma_merge_extend(struct vma_iterator *vmi,
1018 struct vm_area_struct *vma,
1019 unsigned long delta);
1020
952enum {1021enum {
953 /* mark page accessed */1022 /* mark page accessed */
954 FOLL_TOUCH = 1 << 16,1023 FOLL_TOUCH = 1 << 16,
@@ -964,6 +1033,9 @@ enum { · 9 unmodified lines
964 FOLL_UNLOCKABLE = 1 << 21,1033 FOLL_UNLOCKABLE = 1 << 21,
965};1034};
966 1035
1036#define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
1037 FOLL_FAST_ONLY | FOLL_UNLOCKABLE)
1038
967/*1039/*
968 * Indicates for which pages that are write-protected in the page table,1040 * Indicates for which pages that are write-protected in the page table,
969 * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the1041 * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the
@@ -1154,4 +1226,57 @@ struct vma_prepare { · 184 unmodified lines
1154 struct vm_area_struct *remove;1226 struct vm_area_struct *remove;
1155 struct vm_area_struct *remove2;1227 struct vm_area_struct *remove2;
1156};1228};
1229
1230void __meminit __init_single_page(struct page *page, unsigned long pfn,
1231 unsigned long zone, int nid);
1232
1233/* shrinker related functions */
1234unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg,
1235 int priority);
1236
1237#ifdef CONFIG_SHRINKER_DEBUG
1238static inline __printf(2, 0) int shrinker_debugfs_name_alloc(
1239 struct shrinker *shrinker, const char *fmt, va_list ap)
1240{
1241 shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap);
1242
1243 return shrinker->name ? 0 : -ENOMEM;
1244}
1245
1246static inline void shrinker_debugfs_name_free(struct shrinker *shrinker)
1247{
1248 kfree_const(shrinker->name);
1249 shrinker->name = NULL;
1250}
1251
1252extern int shrinker_debugfs_add(struct shrinker *shrinker);
1253extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
1254 int *debugfs_id);
1255extern void shrinker_debugfs_remove(struct dentry *debugfs_entry,
1256 int debugfs_id);
1257#else /* CONFIG_SHRINKER_DEBUG */
1258static inline int shrinker_debugfs_add(struct shrinker *shrinker)
1259{
1260 return 0;
1261}
1262static inline int shrinker_debugfs_name_alloc(struct shrinker *shrinker,
1263 const char *fmt, va_list ap)
1264{
1265 return 0;
1266}
1267static inline void shrinker_debugfs_name_free(struct shrinker *shrinker)
1268{
1269}
1270static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
1271 int *debugfs_id)
1272{
1273 *debugfs_id = -1;
1274 return NULL;
1275}
1276static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,
1277 int debugfs_id)
1278{
1279}
1280#endif /* CONFIG_SHRINKER_DEBUG */
1281
1157#endif /* __MM_INTERNAL_H */1282#endif /* __MM_INTERNAL_H */
mm/kasan/kasan_test.c+9−10
@@ -5,7 +5,7 @@ · 4 unmodified lines
5 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>5 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>
6 */6 */
7 7
8#define pr_fmt(fmt) "kasan_test: " fmt8#define pr_fmt(fmt) "kasan: test: " fmt
9 9
10#include <kunit/test.h>10#include <kunit/test.h>
11#include <linux/bitops.h>11#include <linux/bitops.h>
@@ -91,10 +91,11 @@ static void kasan_test_exit(struct kunit *test) · 79 unmodified lines
91}91}
92 92
93/**93/**
94 * KUNIT_EXPECT_KASAN_FAIL() - check that the executed expression produces a94 * KUNIT_EXPECT_KASAN_FAIL - check that the executed expression produces a
95 * KASAN report; causes a test failure otherwise. This relies on a KUnit95 * KASAN report; causes a KUnit test failure otherwise.
96 * resource named "kasan_status". Do not use this name for KUnit resources96 *
97 * outside of KASAN tests.97 * @test: Currently executing KUnit test.
98 * @expression: Expression that must produce a KASAN report.
98 *99 *
99 * For hardware tag-based KASAN, when a synchronous tag fault happens, tag100 * For hardware tag-based KASAN, when a synchronous tag fault happens, tag
100 * checking is auto-disabled. When this happens, this test handler reenables101 * checking is auto-disabled. When this happens, this test handler reenables
@@ -1097,11 +1098,9 @@ static void kasan_bitops_test_and_modify(struct kunit *test, int nr, void *addr) · 996 unmodified lines
1097 KUNIT_EXPECT_KASAN_FAIL(test, test_and_change_bit(nr, addr));1098 KUNIT_EXPECT_KASAN_FAIL(test, test_and_change_bit(nr, addr));
1098 KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr));1099 KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr));
1099 KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr));1100 KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr));
1100 1101 if (nr < 7)
1101#if defined(clear_bit_unlock_is_negative_byte)1102 KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result =
1102 KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result =1103 xor_unlock_is_negative_byte(1 << nr, addr));
1103 clear_bit_unlock_is_negative_byte(nr, addr));
1104#endif
1105}1104}
1106 1105
1107static void kasan_bitops_generic(struct kunit *test)1106static void kasan_bitops_generic(struct kunit *test)
mm/kasan/kasan_test_module.c+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
5 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>5 * Author: Andrey Ryabinin <a.ryabinin@samsung.com>
6 */6 */
7 7
8#define pr_fmt(fmt) "kasan test: %s " fmt, __func__8#define pr_fmt(fmt) "kasan: test: " fmt
9 9
10#include <linux/mman.h>10#include <linux/mman.h>
11#include <linux/module.h>11#include <linux/module.h>
mm/kasan/quarantine.c+3−1
@@ -8,6 +8,8 @@ · 7 unmodified lines
8 * Based on code by Dmitry Chernenkov.8 * Based on code by Dmitry Chernenkov.
9 */9 */
10 10
11#define pr_fmt(fmt) "kasan: " fmt
12
11#include <linux/gfp.h>13#include <linux/gfp.h>
12#include <linux/hash.h>14#include <linux/hash.h>
13#include <linux/kernel.h>15#include <linux/kernel.h>
@@ -414,7 +416,7 @@ static int __init kasan_cpu_quarantine_init(void) · 400 unmodified lines
414 ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "mm/kasan:online",416 ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "mm/kasan:online",
415 kasan_cpu_online, kasan_cpu_offline);417 kasan_cpu_online, kasan_cpu_offline);
416 if (ret < 0)418 if (ret < 0)
417 pr_err("kasan cpu quarantine register failed [%d]\n", ret);419 pr_err("cpu quarantine register failed [%d]\n", ret);
418 return ret;420 return ret;
419}421}
420late_initcall(kasan_cpu_quarantine_init);422late_initcall(kasan_cpu_quarantine_init);
mm/kasan/report.c+2−2
@@ -538,7 +538,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty · 537 unmodified lines
538 538
539 start_report(&flags, true);539 start_report(&flags, true);
540 540
541 memset(&info, 0, sizeof(info));541 __memset(&info, 0, sizeof(info));
542 info.type = type;542 info.type = type;
543 info.access_addr = ptr;543 info.access_addr = ptr;
544 info.access_size = 0;544 info.access_size = 0;
@@ -576,7 +576,7 @@ bool kasan_report(const void *addr, size_t size, bool is_write, · 31 unmodified lines
576 576
577 start_report(&irq_flags, true);577 start_report(&irq_flags, true);
578 578
579 memset(&info, 0, sizeof(info));579 __memset(&info, 0, sizeof(info));
580 info.type = KASAN_REPORT_ACCESS;580 info.type = KASAN_REPORT_ACCESS;
581 info.access_addr = addr;581 info.access_addr = addr;
582 info.access_size = size;582 info.access_size = size;
mm/kasan/report_generic.c+3−3
@@ -220,7 +220,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 219 unmodified lines
220 const size_t tok_len = sep - *frame_descr;220 const size_t tok_len = sep - *frame_descr;
221 221
222 if (tok_len + 1 > max_tok_len) {222 if (tok_len + 1 > max_tok_len) {
223 pr_err("KASAN internal error: frame description too long: %s\n",223 pr_err("internal error: frame description too long: %s\n",
224 *frame_descr);224 *frame_descr);
225 return false;225 return false;
226 }226 }
@@ -233,7 +233,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 6 unmodified lines
233 *frame_descr = sep + 1;233 *frame_descr = sep + 1;
234 234
235 if (value != NULL && kstrtoul(token, 10, value)) {235 if (value != NULL && kstrtoul(token, 10, value)) {
236 pr_err("KASAN internal error: not a valid number: %s\n", token);236 pr_err("internal error: not a valid number: %s\n", token);
237 return false;237 return false;
238 }238 }
239 239
@@ -323,7 +323,7 @@ static bool __must_check get_address_stack_frame_info(const void *addr, · 83 unmodified lines
323 323
324 frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);324 frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE);
325 if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {325 if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) {
326 pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n",326 pr_err("internal error: frame has invalid marker: %lu\n",
327 frame[0]);327 frame[0]);
328 return false;328 return false;
329 }329 }
mm/kasan/shadow.c+1−1
@@ -324,7 +324,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 323 unmodified lines
324 if (!page)324 if (!page)
325 return -ENOMEM;325 return -ENOMEM;
326 326
327 memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);327 __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
328 pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);328 pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
329 329
330 spin_lock(&init_mm.page_table_lock);330 spin_lock(&init_mm.page_table_lock);
mm/khugepaged.c+68−69
@@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, · 523 unmodified lines
524 }524 }
525}525}
526 526
527static bool is_refcount_suitable(struct page *page)527static bool is_refcount_suitable(struct folio *folio)
528{528{
529 int expected_refcount;529 int expected_refcount;
530 530
531 expected_refcount = total_mapcount(page);531 expected_refcount = folio_mapcount(folio);
532 if (PageSwapCache(page))532 if (folio_test_swapcache(folio))
533 expected_refcount += compound_nr(page);533 expected_refcount += folio_nr_pages(folio);
534 534
535 return page_count(page) == expected_refcount;535 return folio_ref_count(folio) == expected_refcount;
536}536}
537 537
538static int __collapse_huge_page_isolate(struct vm_area_struct *vma,538static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
@@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 3 unmodified lines
542 struct list_head *compound_pagelist)542 struct list_head *compound_pagelist)
543{543{
544 struct page *page = NULL;544 struct page *page = NULL;
545 struct folio *folio = NULL;
545 pte_t *_pte;546 pte_t *_pte;
546 int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;547 int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
547 bool writable = false;548 bool writable = false;
@@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 28 unmodified lines
576 goto out;577 goto out;
577 }578 }
578 579
579 VM_BUG_ON_PAGE(!PageAnon(page), page);580 folio = page_folio(page);
581 VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio);
580 582
581 if (page_mapcount(page) > 1) {583 if (page_mapcount(page) > 1) {
582 ++shared;584 ++shared;
@@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 5 unmodified lines
588 }590 }
589 }591 }
590 592
591 if (PageCompound(page)) {593 if (folio_test_large(folio)) {
592 struct page *p;594 struct folio *f;
593 page = compound_head(page);
594 595
595 /*596 /*
596 * Check if we have dealt with the compound page597 * Check if we have dealt with the compound page
597 * already598 * already
598 */599 */
599 list_for_each_entry(p, compound_pagelist, lru) {600 list_for_each_entry(f, compound_pagelist, lru) {
600 if (page == p)601 if (folio == f)
601 goto next;602 goto next;
602 }603 }
603 }604 }
@@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 4 unmodified lines
608 * is needed to serialize against split_huge_page609 * is needed to serialize against split_huge_page
609 * when invoked from the VM.610 * when invoked from the VM.
610 */611 */
611 if (!trylock_page(page)) {612 if (!folio_trylock(folio)) {
612 result = SCAN_PAGE_LOCK;613 result = SCAN_PAGE_LOCK;
613 goto out;614 goto out;
614 }615 }
@@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 9 unmodified lines
624 * but not from this process. The other process cannot write to625 * but not from this process. The other process cannot write to
625 * the page, only trigger CoW.626 * the page, only trigger CoW.
626 */627 */
627 if (!is_refcount_suitable(page)) {628 if (!is_refcount_suitable(folio)) {
628 unlock_page(page);629 folio_unlock(folio);
629 result = SCAN_PAGE_COUNT;630 result = SCAN_PAGE_COUNT;
630 goto out;631 goto out;
631 }632 }
@@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 2 unmodified lines
634 * Isolate the page to avoid collapsing an hugepage635 * Isolate the page to avoid collapsing an hugepage
635 * currently in use by the VM.636 * currently in use by the VM.
636 */637 */
637 if (!isolate_lru_page(page)) {638 if (!folio_isolate_lru(folio)) {
638 unlock_page(page);639 folio_unlock(folio);
639 result = SCAN_DEL_PAGE_LRU;640 result = SCAN_DEL_PAGE_LRU;
640 goto out;641 goto out;
641 }642 }
642 mod_node_page_state(page_pgdat(page),643 node_stat_mod_folio(folio,
643 NR_ISOLATED_ANON + page_is_file_lru(page),644 NR_ISOLATED_ANON + folio_is_file_lru(folio),
644 compound_nr(page));645 folio_nr_pages(folio));
645 VM_BUG_ON_PAGE(!PageLocked(page), page);646 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
646 VM_BUG_ON_PAGE(PageLRU(page), page);647 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
647 648
648 if (PageCompound(page))649 if (folio_test_large(folio))
649 list_add_tail(&page->lru, compound_pagelist);650 list_add_tail(&folio->lru, compound_pagelist);
650next:651next:
651 /*652 /*
652 * If collapse was initiated by khugepaged, check that there is653 * If collapse was initiated by khugepaged, check that there is
653 * enough young pte to justify collapsing the page654 * enough young pte to justify collapsing the page
654 */655 */
655 if (cc->is_khugepaged &&656 if (cc->is_khugepaged &&
656 (pte_young(pteval) || page_is_young(page) ||657 (pte_young(pteval) || folio_test_young(folio) ||
657 PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm,658 folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm,
658 address)))659 address)))
659 referenced++;660 referenced++;
660 661
@@ -668,13 +669,13 @@ next: · 7 unmodified lines
668 result = SCAN_LACK_REFERENCED_PAGE;669 result = SCAN_LACK_REFERENCED_PAGE;
669 } else {670 } else {
670 result = SCAN_SUCCEED;671 result = SCAN_SUCCEED;
671 trace_mm_collapse_huge_page_isolate(page, none_or_zero,672 trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero,
672 referenced, writable, result);673 referenced, writable, result);
673 return result;674 return result;
674 }675 }
675out:676out:
676 release_pte_pages(pte, _pte, compound_pagelist);677 release_pte_pages(pte, _pte, compound_pagelist);
677 trace_mm_collapse_huge_page_isolate(page, none_or_zero,678 trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero,
678 referenced, writable, result);679 referenced, writable, result);
679 return result;680 return result;
680}681}
@@ -887,16 +888,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) · 206 unmodified lines
887}888}
888#endif889#endif
889 890
890static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node,891static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node,
891 nodemask_t *nmask)892 nodemask_t *nmask)
892{893{
893 *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask);894 *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask);
894 if (unlikely(!*hpage)) {895
896 if (unlikely(!*folio)) {
895 count_vm_event(THP_COLLAPSE_ALLOC_FAILED);897 count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
896 return false;898 return false;
897 }899 }
898 900
899 folio_prep_large_rmappable((struct folio *)*hpage);
900 count_vm_event(THP_COLLAPSE_ALLOC);901 count_vm_event(THP_COLLAPSE_ALLOC);
901 return true;902 return true;
902}903}
@@ -1063,17 +1064,20 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, · 160 unmodified lines
1063 int node = hpage_collapse_find_target_node(cc);1064 int node = hpage_collapse_find_target_node(cc);
1064 struct folio *folio;1065 struct folio *folio;
1065 1066
1066 if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask))1067 if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) {
1068 *hpage = NULL;
1067 return SCAN_ALLOC_HUGE_PAGE_FAIL;1069 return SCAN_ALLOC_HUGE_PAGE_FAIL;
1070 }
1068 1071
1069 folio = page_folio(*hpage);
1070 if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {1072 if (unlikely(mem_cgroup_charge(folio, mm, gfp))) {
1071 folio_put(folio);1073 folio_put(folio);
1072 *hpage = NULL;1074 *hpage = NULL;
1073 return SCAN_CGROUP_CHARGE_FAIL;1075 return SCAN_CGROUP_CHARGE_FAIL;
1074 }1076 }
1075 count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC);
1076 1077
1078 count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1);
1079
1080 *hpage = folio_page(folio, 0);
1077 return SCAN_SUCCEED;1081 return SCAN_SUCCEED;
1078}1082}
1079 1083
@@ -1247,6 +1251,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 167 unmodified lines
1247 int result = SCAN_FAIL, referenced = 0;1251 int result = SCAN_FAIL, referenced = 0;
1248 int none_or_zero = 0, shared = 0;1252 int none_or_zero = 0, shared = 0;
1249 struct page *page = NULL;1253 struct page *page = NULL;
1254 struct folio *folio = NULL;
1250 unsigned long _address;1255 unsigned long _address;
1251 spinlock_t *ptl;1256 spinlock_t *ptl;
1252 int node = NUMA_NO_NODE, unmapped = 0;1257 int node = NUMA_NO_NODE, unmapped = 0;
@@ -1333,29 +1338,28 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 80 unmodified lines
1333 }1338 }
1334 }1339 }
1335 1340
1336 page = compound_head(page);1341 folio = page_folio(page);
1337
1338 /*1342 /*
1339 * Record which node the original page is from and save this1343 * Record which node the original page is from and save this
1340 * information to cc->node_load[].1344 * information to cc->node_load[].
1341 * Khugepaged will allocate hugepage from the node has the max1345 * Khugepaged will allocate hugepage from the node has the max
1342 * hit record.1346 * hit record.
1343 */1347 */
1344 node = page_to_nid(page);1348 node = folio_nid(folio);
1345 if (hpage_collapse_scan_abort(node, cc)) {1349 if (hpage_collapse_scan_abort(node, cc)) {
1346 result = SCAN_SCAN_ABORT;1350 result = SCAN_SCAN_ABORT;
1347 goto out_unmap;1351 goto out_unmap;
1348 }1352 }
1349 cc->node_load[node]++;1353 cc->node_load[node]++;
1350 if (!PageLRU(page)) {1354 if (!folio_test_lru(folio)) {
1351 result = SCAN_PAGE_LRU;1355 result = SCAN_PAGE_LRU;
1352 goto out_unmap;1356 goto out_unmap;
1353 }1357 }
1354 if (PageLocked(page)) {1358 if (folio_test_locked(folio)) {
1355 result = SCAN_PAGE_LOCK;1359 result = SCAN_PAGE_LOCK;
1356 goto out_unmap;1360 goto out_unmap;
1357 }1361 }
1358 if (!PageAnon(page)) {1362 if (!folio_test_anon(folio)) {
1359 result = SCAN_PAGE_ANON;1363 result = SCAN_PAGE_ANON;
1360 goto out_unmap;1364 goto out_unmap;
1361 }1365 }
@@ -1370,7 +1374,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 8 unmodified lines
1370 * has excessive GUP pins (i.e. 512). Anyway the same check1374 * has excessive GUP pins (i.e. 512). Anyway the same check
1371 * will be done again later the risk seems low.1375 * will be done again later the risk seems low.
1372 */1376 */
1373 if (!is_refcount_suitable(page)) {1377 if (!is_refcount_suitable(folio)) {
1374 result = SCAN_PAGE_COUNT;1378 result = SCAN_PAGE_COUNT;
1375 goto out_unmap;1379 goto out_unmap;
1376 }1380 }
@@ -1380,8 +1384,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 3 unmodified lines
1380 * enough young pte to justify collapsing the page1384 * enough young pte to justify collapsing the page
1381 */1385 */
1382 if (cc->is_khugepaged &&1386 if (cc->is_khugepaged &&
1383 (pte_young(pteval) || page_is_young(page) ||1387 (pte_young(pteval) || folio_test_young(folio) ||
1384 PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm,1388 folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm,
1385 address)))1389 address)))
1386 referenced++;1390 referenced++;
1387 }1391 }
@@ -1403,7 +1407,7 @@ out_unmap: · 15 unmodified lines
1403 *mmap_locked = false;1407 *mmap_locked = false;
1404 }1408 }
1405out:1409out:
1406 trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,1410 trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced,
1407 none_or_zero, result, unmapped);1411 none_or_zero, result, unmapped);
1408 return result;1412 return result;
1409}1413}
@@ -1473,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 63 unmodified lines
1473 bool notified = false;1477 bool notified = false;
1474 unsigned long haddr = addr & HPAGE_PMD_MASK;1478 unsigned long haddr = addr & HPAGE_PMD_MASK;
1475 struct vm_area_struct *vma = vma_lookup(mm, haddr);1479 struct vm_area_struct *vma = vma_lookup(mm, haddr);
1476 struct page *hpage;1480 struct folio *folio;
1477 pte_t *start_pte, *pte;1481 pte_t *start_pte, *pte;
1478 pmd_t *pmd, pgt_pmd;1482 pmd_t *pmd, pgt_pmd;
1479 spinlock_t *pml = NULL, *ptl;1483 spinlock_t *pml = NULL, *ptl;
@@ -1506,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 26 unmodified lines
1506 if (userfaultfd_wp(vma))1510 if (userfaultfd_wp(vma))
1507 return SCAN_PTE_UFFD_WP;1511 return SCAN_PTE_UFFD_WP;
1508 1512
1509 hpage = find_lock_page(vma->vm_file->f_mapping,1513 folio = filemap_lock_folio(vma->vm_file->f_mapping,
1510 linear_page_index(vma, haddr));1514 linear_page_index(vma, haddr));
1511 if (!hpage)1515 if (IS_ERR(folio))
1512 return SCAN_PAGE_NULL;1516 return SCAN_PAGE_NULL;
1513 1517
1514 if (!PageHead(hpage)) {1518 if (folio_order(folio) != HPAGE_PMD_ORDER) {
1515 result = SCAN_FAIL;
1516 goto drop_hpage;
1517 }
1518
1519 if (compound_order(hpage) != HPAGE_PMD_ORDER) {
1520 result = SCAN_PAGE_COMPOUND;1519 result = SCAN_PAGE_COMPOUND;
1521 goto drop_hpage;1520 goto drop_folio;
1522 }1521 }
1523 1522
1524 result = find_pmd_or_thp_or_none(mm, haddr, &pmd);1523 result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
@@ -1532,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 7 unmodified lines
1532 */1531 */
1533 goto maybe_install_pmd;1532 goto maybe_install_pmd;
1534 default:1533 default:
1535 goto drop_hpage;1534 goto drop_folio;
1536 }1535 }
1537 1536
1538 result = SCAN_FAIL;1537 result = SCAN_FAIL;
1539 start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);1538 start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);
1540 if (!start_pte) /* mmap_lock + page lock should prevent this */1539 if (!start_pte) /* mmap_lock + page lock should prevent this */
1541 goto drop_hpage;1540 goto drop_folio;
1542 1541
1543 /* step 1: check all mapped PTEs are to the right huge page */1542 /* step 1: check all mapped PTEs are to the right huge page */
1544 for (i = 0, addr = haddr, pte = start_pte;1543 for (i = 0, addr = haddr, pte = start_pte;
@@ -1563,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 18 unmodified lines
1563 * Note that uprobe, debugger, or MAP_PRIVATE may change the1562 * Note that uprobe, debugger, or MAP_PRIVATE may change the
1564 * page table, but the new page will not be a subpage of hpage.1563 * page table, but the new page will not be a subpage of hpage.
1565 */1564 */
1566 if (hpage + i != page)1565 if (folio_page(folio, i) != page)
1567 goto abort;1566 goto abort;
1568 }1567 }
1569 1568
@@ -1578,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 8 unmodified lines
1578 * page_table_lock) ptl nests inside pml. The less time we hold pml,1577 * page_table_lock) ptl nests inside pml. The less time we hold pml,
1579 * the better; but userfaultfd's mfill_atomic_pte() on a private VMA1578 * the better; but userfaultfd's mfill_atomic_pte() on a private VMA
1580 * inserts a valid as-if-COWed PTE without even looking up page cache.1579 * inserts a valid as-if-COWed PTE without even looking up page cache.
1581 * So page lock of hpage does not protect from it, so we must not drop1580 * So page lock of folio does not protect from it, so we must not drop
1582 * ptl before pgt_pmd is removed, so uffd private needs pml taken now.1581 * ptl before pgt_pmd is removed, so uffd private needs pml taken now.
1583 */1582 */
1584 if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED))1583 if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED))
@@ -1602,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines
1602 continue;1601 continue;
1603 /*1602 /*
1604 * We dropped ptl after the first scan, to do the mmu_notifier:1603 * We dropped ptl after the first scan, to do the mmu_notifier:
1605 * page lock stops more PTEs of the hpage being faulted in, but1604 * page lock stops more PTEs of the folio being faulted in, but
1606 * does not stop write faults COWing anon copies from existing1605 * does not stop write faults COWing anon copies from existing
1607 * PTEs; and does not stop those being swapped out or migrated.1606 * PTEs; and does not stop those being swapped out or migrated.
1608 */1607 */
@@ -1611,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines
1611 goto abort;1610 goto abort;
1612 }1611 }
1613 page = vm_normal_page(vma, addr, ptent);1612 page = vm_normal_page(vma, addr, ptent);
1614 if (hpage + i != page)1613 if (folio_page(folio, i) != page)
1615 goto abort;1614 goto abort;
1616 1615
1617 /*1616 /*
@@ -1630,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 12 unmodified lines
1630 1629
1631 /* step 3: set proper refcount and mm_counters. */1630 /* step 3: set proper refcount and mm_counters. */
1632 if (nr_ptes) {1631 if (nr_ptes) {
1633 page_ref_sub(hpage, nr_ptes);1632 folio_ref_sub(folio, nr_ptes);
1634 add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes);1633 add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes);
1635 }1634 }
1636 1635
1637 /* step 4: remove empty page table */1636 /* step 4: remove empty page table */
@@ -1655,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines
1655maybe_install_pmd:1654maybe_install_pmd:
1656 /* step 5: install pmd entry */1655 /* step 5: install pmd entry */
1657 result = install_pmd1656 result = install_pmd
1658 ? set_huge_pmd(vma, haddr, pmd, hpage)1657 ? set_huge_pmd(vma, haddr, pmd, &folio->page)
1659 : SCAN_SUCCEED;1658 : SCAN_SUCCEED;
1660 goto drop_hpage;1659 goto drop_folio;
1661abort:1660abort:
1662 if (nr_ptes) {1661 if (nr_ptes) {
1663 flush_tlb_mm(mm);1662 flush_tlb_mm(mm);
1664 page_ref_sub(hpage, nr_ptes);1663 folio_ref_sub(folio, nr_ptes);
1665 add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes);1664 add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes);
1666 }1665 }
1667 if (start_pte)1666 if (start_pte)
1668 pte_unmap_unlock(start_pte, ptl);1667 pte_unmap_unlock(start_pte, ptl);
@@ -1670,9 +1669,9 @@ abort: · 1 unmodified line
1670 spin_unlock(pml);1669 spin_unlock(pml);
1671 if (notified)1670 if (notified)
1672 mmu_notifier_invalidate_range_end(&range);1671 mmu_notifier_invalidate_range_end(&range);
1673drop_hpage:1672drop_folio:
1674 unlock_page(hpage);1673 folio_unlock(folio);
1675 put_page(hpage);1674 folio_put(folio);
1676 return result;1675 return result;
1677}1676}
1678 1677
mm/kmemleak.c+98−46
@@ -583,6 +583,19 @@ static void __remove_object(struct kmemleak_object *object) · 582 unmodified lines
583 object->del_state |= DELSTATE_REMOVED;583 object->del_state |= DELSTATE_REMOVED;
584}584}
585 585
586static struct kmemleak_object *__find_and_remove_object(unsigned long ptr,
587 int alias,
588 bool is_phys)
589{
590 struct kmemleak_object *object;
591
592 object = __lookup_object(ptr, alias, is_phys);
593 if (object)
594 __remove_object(object);
595
596 return object;
597}
598
586/*599/*
587 * Look up an object in the object search tree and remove it from both600 * Look up an object in the object search tree and remove it from both
588 * object_tree_root (or object_phys_tree_root) and object_list. The601 * object_tree_root (or object_phys_tree_root) and object_list. The
@@ -596,9 +609,7 @@ static struct kmemleak_object *find_and_remove_object(unsigned long ptr, int ali · 7 unmodified lines
596 struct kmemleak_object *object;609 struct kmemleak_object *object;
597 610
598 raw_spin_lock_irqsave(&kmemleak_lock, flags);611 raw_spin_lock_irqsave(&kmemleak_lock, flags);
599 object = __lookup_object(ptr, alias, is_phys);612 object = __find_and_remove_object(ptr, alias, is_phys);
600 if (object)
601 __remove_object(object);
602 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);613 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
603 614
604 return object;615 return object;
@@ -623,27 +634,28 @@ static noinline depot_stack_handle_t set_track_prepare(void) · 18 unmodified lines
623 return trace_handle;634 return trace_handle;
624}635}
625 636
626/*637static struct kmemleak_object *__alloc_object(gfp_t gfp)
627 * Create the metadata (struct kmemleak_object) corresponding to an allocated
628 * memory block and add it to the object_list and object_tree_root (or
629 * object_phys_tree_root).
630 */
631static void __create_object(unsigned long ptr, size_t size,
632 int min_count, gfp_t gfp, bool is_phys)
633{638{
634 unsigned long flags;639 struct kmemleak_object *object;
635 struct kmemleak_object *object, *parent;
636 struct rb_node **link, *rb_parent;
637 unsigned long untagged_ptr;
638 unsigned long untagged_objp;
639 640
640 object = mem_pool_alloc(gfp);641 object = mem_pool_alloc(gfp);
641 if (!object) {642 if (!object) {
642 pr_warn("Cannot allocate a kmemleak_object structure\n");643 pr_warn("Cannot allocate a kmemleak_object structure\n");
643 kmemleak_disable();644 kmemleak_disable();
644 return;
645 }645 }
646 646
647 return object;
648}
649
650static int __link_object(struct kmemleak_object *object, unsigned long ptr,
651 size_t size, int min_count, bool is_phys)
652{
653
654 struct kmemleak_object *parent;
655 struct rb_node **link, *rb_parent;
656 unsigned long untagged_ptr;
657 unsigned long untagged_objp;
658
647 INIT_LIST_HEAD(&object->object_list);659 INIT_LIST_HEAD(&object->object_list);
648 INIT_LIST_HEAD(&object->gray_list);660 INIT_LIST_HEAD(&object->gray_list);
649 INIT_HLIST_HEAD(&object->area_list);661 INIT_HLIST_HEAD(&object->area_list);
@@ -680,8 +692,6 @@ static void __create_object(unsigned long ptr, size_t size, · 30 unmodified lines
680 /* kernel backtrace */692 /* kernel backtrace */
681 object->trace_handle = set_track_prepare();693 object->trace_handle = set_track_prepare();
682 694
683 raw_spin_lock_irqsave(&kmemleak_lock, flags);
684
685 untagged_ptr = (unsigned long)kasan_reset_tag((void *)ptr);695 untagged_ptr = (unsigned long)kasan_reset_tag((void *)ptr);
686 /*696 /*
687 * Only update min_addr and max_addr with object697 * Only update min_addr and max_addr with object
@@ -710,16 +720,38 @@ static void __create_object(unsigned long ptr, size_t size, · 22 unmodified lines
710 * be freed while the kmemleak_lock is held.720 * be freed while the kmemleak_lock is held.
711 */721 */
712 dump_object_info(parent);722 dump_object_info(parent);
713 kmem_cache_free(object_cache, object);723 return -EEXIST;
714 goto out;
715 }724 }
716 }725 }
717 rb_link_node(&object->rb_node, rb_parent, link);726 rb_link_node(&object->rb_node, rb_parent, link);
718 rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root :727 rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root :
719 &object_tree_root);728 &object_tree_root);
720 list_add_tail_rcu(&object->object_list, &object_list);729 list_add_tail_rcu(&object->object_list, &object_list);
721out:730
731 return 0;
732}
733
734/*
735 * Create the metadata (struct kmemleak_object) corresponding to an allocated
736 * memory block and add it to the object_list and object_tree_root (or
737 * object_phys_tree_root).
738 */
739static void __create_object(unsigned long ptr, size_t size,
740 int min_count, gfp_t gfp, bool is_phys)
741{
742 struct kmemleak_object *object;
743 unsigned long flags;
744 int ret;
745
746 object = __alloc_object(gfp);
747 if (!object)
748 return;
749
750 raw_spin_lock_irqsave(&kmemleak_lock, flags);
751 ret = __link_object(object, ptr, size, min_count, is_phys);
722 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);752 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
753 if (ret)
754 mem_pool_free(object);
723}755}
724 756
725/* Create kmemleak object which allocated with virtual address. */757/* Create kmemleak object which allocated with virtual address. */
@@ -782,16 +814,25 @@ static void delete_object_full(unsigned long ptr) · 56 unmodified lines
782 */814 */
783static void delete_object_part(unsigned long ptr, size_t size, bool is_phys)815static void delete_object_part(unsigned long ptr, size_t size, bool is_phys)
784{816{
785 struct kmemleak_object *object;817 struct kmemleak_object *object, *object_l, *object_r;
786 unsigned long start, end;818 unsigned long start, end, flags;
819
820 object_l = __alloc_object(GFP_KERNEL);
821 if (!object_l)
822 return;
787 823
788 object = find_and_remove_object(ptr, 1, is_phys);824 object_r = __alloc_object(GFP_KERNEL);
825 if (!object_r)
826 goto out;
827
828 raw_spin_lock_irqsave(&kmemleak_lock, flags);
829 object = __find_and_remove_object(ptr, 1, is_phys);
789 if (!object) {830 if (!object) {
790#ifdef DEBUG831#ifdef DEBUG
791 kmemleak_warn("Partially freeing unknown object at 0x%08lx (size %zu)\n",832 kmemleak_warn("Partially freeing unknown object at 0x%08lx (size %zu)\n",
792 ptr, size);833 ptr, size);
793#endif834#endif
794 return;835 goto unlock;
795 }836 }
796 837
797 /*838 /*
@@ -801,14 +842,25 @@ static void delete_object_part(unsigned long ptr, size_t size, bool is_phys) · 3 unmodified lines
801 */842 */
802 start = object->pointer;843 start = object->pointer;
803 end = object->pointer + object->size;844 end = object->pointer + object->size;
804 if (ptr > start)845 if ((ptr > start) &&
805 __create_object(start, ptr - start, object->min_count,846 !__link_object(object_l, start, ptr - start,
806 GFP_KERNEL, is_phys);847 object->min_count, is_phys))
807 if (ptr + size < end)848 object_l = NULL;
808 __create_object(ptr + size, end - ptr - size, object->min_count,849 if ((ptr + size < end) &&
809 GFP_KERNEL, is_phys);850 !__link_object(object_r, ptr + size, end - ptr - size,
851 object->min_count, is_phys))
852 object_r = NULL;
853
854unlock:
855 raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
856 if (object)
857 __delete_object(object);
810 858
811 __delete_object(object);859out:
860 if (object_l)
861 mem_pool_free(object_l);
862 if (object_r)
863 mem_pool_free(object_r);
812}864}
813 865
814static void __paint_it(struct kmemleak_object *object, int color)866static void __paint_it(struct kmemleak_object *object, int color)
@@ -975,7 +1027,7 @@ static void object_no_scan(unsigned long ptr) · 160 unmodified lines
975void __ref kmemleak_alloc(const void *ptr, size_t size, int min_count,1027void __ref kmemleak_alloc(const void *ptr, size_t size, int min_count,
976 gfp_t gfp)1028 gfp_t gfp)
977{1029{
978 pr_debug("%s(0x%p, %zu, %d)\n", __func__, ptr, size, min_count);1030 pr_debug("%s(0x%px, %zu, %d)\n", __func__, ptr, size, min_count);
979 1031
980 if (kmemleak_enabled && ptr && !IS_ERR(ptr))1032 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
981 create_object((unsigned long)ptr, size, min_count, gfp);1033 create_object((unsigned long)ptr, size, min_count, gfp);
@@ -996,7 +1048,7 @@ void __ref kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, · 14 unmodified lines
996{1048{
997 unsigned int cpu;1049 unsigned int cpu;
998 1050
999 pr_debug("%s(0x%p, %zu)\n", __func__, ptr, size);1051 pr_debug("%s(0x%px, %zu)\n", __func__, ptr, size);
1000 1052
1001 /*1053 /*
1002 * Percpu allocations are only scanned and not reported as leaks1054 * Percpu allocations are only scanned and not reported as leaks
@@ -1020,7 +1072,7 @@ EXPORT_SYMBOL_GPL(kmemleak_alloc_percpu); · 17 unmodified lines
1020 */1072 */
1021void __ref kmemleak_vmalloc(const struct vm_struct *area, size_t size, gfp_t gfp)1073void __ref kmemleak_vmalloc(const struct vm_struct *area, size_t size, gfp_t gfp)
1022{1074{
1023 pr_debug("%s(0x%p, %zu)\n", __func__, area, size);1075 pr_debug("%s(0x%px, %zu)\n", __func__, area, size);
1024 1076
1025 /*1077 /*
1026 * A min_count = 2 is needed because vm_struct contains a reference to1078 * A min_count = 2 is needed because vm_struct contains a reference to
@@ -1043,7 +1095,7 @@ EXPORT_SYMBOL_GPL(kmemleak_vmalloc); · 16 unmodified lines
1043 */1095 */
1044void __ref kmemleak_free(const void *ptr)1096void __ref kmemleak_free(const void *ptr)
1045{1097{
1046 pr_debug("%s(0x%p)\n", __func__, ptr);1098 pr_debug("%s(0x%px)\n", __func__, ptr);
1047 1099
1048 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))1100 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))
1049 delete_object_full((unsigned long)ptr);1101 delete_object_full((unsigned long)ptr);
@@ -1061,7 +1113,7 @@ EXPORT_SYMBOL_GPL(kmemleak_free); · 11 unmodified lines
1061 */1113 */
1062void __ref kmemleak_free_part(const void *ptr, size_t size)1114void __ref kmemleak_free_part(const void *ptr, size_t size)
1063{1115{
1064 pr_debug("%s(0x%p)\n", __func__, ptr);1116 pr_debug("%s(0x%px)\n", __func__, ptr);
1065 1117
1066 if (kmemleak_enabled && ptr && !IS_ERR(ptr))1118 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
1067 delete_object_part((unsigned long)ptr, size, false);1119 delete_object_part((unsigned long)ptr, size, false);
@@ -1079,7 +1131,7 @@ void __ref kmemleak_free_percpu(const void __percpu *ptr) · 11 unmodified lines
1079{1131{
1080 unsigned int cpu;1132 unsigned int cpu;
1081 1133
1082 pr_debug("%s(0x%p)\n", __func__, ptr);1134 pr_debug("%s(0x%px)\n", __func__, ptr);
1083 1135
1084 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))1136 if (kmemleak_free_enabled && ptr && !IS_ERR(ptr))
1085 for_each_possible_cpu(cpu)1137 for_each_possible_cpu(cpu)
@@ -1100,7 +1152,7 @@ void __ref kmemleak_update_trace(const void *ptr) · 14 unmodified lines
1100 struct kmemleak_object *object;1152 struct kmemleak_object *object;
1101 unsigned long flags;1153 unsigned long flags;
1102 1154
1103 pr_debug("%s(0x%p)\n", __func__, ptr);1155 pr_debug("%s(0x%px)\n", __func__, ptr);
1104 1156
1105 if (!kmemleak_enabled || IS_ERR_OR_NULL(ptr))1157 if (!kmemleak_enabled || IS_ERR_OR_NULL(ptr))
1106 return;1158 return;
@@ -1131,7 +1183,7 @@ EXPORT_SYMBOL(kmemleak_update_trace); · 24 unmodified lines
1131 */1183 */
1132void __ref kmemleak_not_leak(const void *ptr)1184void __ref kmemleak_not_leak(const void *ptr)
1133{1185{
1134 pr_debug("%s(0x%p)\n", __func__, ptr);1186 pr_debug("%s(0x%px)\n", __func__, ptr);
1135 1187
1136 if (kmemleak_enabled && ptr && !IS_ERR(ptr))1188 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
1137 make_gray_object((unsigned long)ptr);1189 make_gray_object((unsigned long)ptr);
@@ -1149,7 +1201,7 @@ EXPORT_SYMBOL(kmemleak_not_leak); · 11 unmodified lines
1149 */1201 */
1150void __ref kmemleak_ignore(const void *ptr)1202void __ref kmemleak_ignore(const void *ptr)
1151{1203{
1152 pr_debug("%s(0x%p)\n", __func__, ptr);1204 pr_debug("%s(0x%px)\n", __func__, ptr);
1153 1205
1154 if (kmemleak_enabled && ptr && !IS_ERR(ptr))1206 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
1155 make_black_object((unsigned long)ptr, false);1207 make_black_object((unsigned long)ptr, false);
@@ -1169,7 +1221,7 @@ EXPORT_SYMBOL(kmemleak_ignore); · 13 unmodified lines
1169 */1221 */
1170void __ref kmemleak_scan_area(const void *ptr, size_t size, gfp_t gfp)1222void __ref kmemleak_scan_area(const void *ptr, size_t size, gfp_t gfp)
1171{1223{
1172 pr_debug("%s(0x%p)\n", __func__, ptr);1224 pr_debug("%s(0x%px)\n", __func__, ptr);
1173 1225
1174 if (kmemleak_enabled && ptr && size && !IS_ERR(ptr))1226 if (kmemleak_enabled && ptr && size && !IS_ERR(ptr))
1175 add_scan_area((unsigned long)ptr, size, gfp);1227 add_scan_area((unsigned long)ptr, size, gfp);
@@ -1187,7 +1239,7 @@ EXPORT_SYMBOL(kmemleak_scan_area); · 11 unmodified lines
1187 */1239 */
1188void __ref kmemleak_no_scan(const void *ptr)1240void __ref kmemleak_no_scan(const void *ptr)
1189{1241{
1190 pr_debug("%s(0x%p)\n", __func__, ptr);1242 pr_debug("%s(0x%px)\n", __func__, ptr);
1191 1243
1192 if (kmemleak_enabled && ptr && !IS_ERR(ptr))1244 if (kmemleak_enabled && ptr && !IS_ERR(ptr))
1193 object_no_scan((unsigned long)ptr);1245 object_no_scan((unsigned long)ptr);
@@ -1203,7 +1255,7 @@ EXPORT_SYMBOL(kmemleak_no_scan); · 9 unmodified lines
1203 */1255 */
1204void __ref kmemleak_alloc_phys(phys_addr_t phys, size_t size, gfp_t gfp)1256void __ref kmemleak_alloc_phys(phys_addr_t phys, size_t size, gfp_t gfp)
1205{1257{
1206 pr_debug("%s(0x%pa, %zu)\n", __func__, &phys, size);1258 pr_debug("%s(0x%px, %zu)\n", __func__, &phys, size);
1207 1259
1208 if (kmemleak_enabled)1260 if (kmemleak_enabled)
1209 /*1261 /*
@@ -1223,7 +1275,7 @@ EXPORT_SYMBOL(kmemleak_alloc_phys); · 13 unmodified lines
1223 */1275 */
1224void __ref kmemleak_free_part_phys(phys_addr_t phys, size_t size)1276void __ref kmemleak_free_part_phys(phys_addr_t phys, size_t size)
1225{1277{
1226 pr_debug("%s(0x%pa)\n", __func__, &phys);1278 pr_debug("%s(0x%px)\n", __func__, &phys);
1227 1279
1228 if (kmemleak_enabled)1280 if (kmemleak_enabled)
1229 delete_object_part((unsigned long)phys, size, true);1281 delete_object_part((unsigned long)phys, size, true);
@@ -1237,7 +1289,7 @@ EXPORT_SYMBOL(kmemleak_free_part_phys); · 7 unmodified lines
1237 */1289 */
1238void __ref kmemleak_ignore_phys(phys_addr_t phys)1290void __ref kmemleak_ignore_phys(phys_addr_t phys)
1239{1291{
1240 pr_debug("%s(0x%pa)\n", __func__, &phys);1292 pr_debug("%s(0x%px)\n", __func__, &phys);
1241 1293
1242 if (kmemleak_enabled)1294 if (kmemleak_enabled)
1243 make_black_object((unsigned long)phys, true);1295 make_black_object((unsigned long)phys, true);
mm/kmsan/core.c+31−96
@@ -83,131 +83,66 @@ depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, · 82 unmodified lines
83/* Copy the metadata following the memmove() behavior. */83/* Copy the metadata following the memmove() behavior. */
84void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n)84void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n)
85{85{
86 depot_stack_handle_t prev_old_origin = 0, prev_new_origin = 0;
87 int i, iter, step, src_off, dst_off, oiter_src, oiter_dst;
86 depot_stack_handle_t old_origin = 0, new_origin = 0;88 depot_stack_handle_t old_origin = 0, new_origin = 0;
87 int src_slots, dst_slots, i, iter, step, skip_bits;
88 depot_stack_handle_t *origin_src, *origin_dst;89 depot_stack_handle_t *origin_src, *origin_dst;
89 void *shadow_src, *shadow_dst;90 u8 *shadow_src, *shadow_dst;
90 u32 *align_shadow_src, shadow;91 u32 *align_shadow_dst;
91 bool backwards;92 bool backwards;
92 93
93 shadow_dst = kmsan_get_metadata(dst, KMSAN_META_SHADOW);94 shadow_dst = kmsan_get_metadata(dst, KMSAN_META_SHADOW);
94 if (!shadow_dst)95 if (!shadow_dst)
95 return;96 return;
96 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n));97 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n));
98 align_shadow_dst =
99 (u32 *)ALIGN_DOWN((u64)shadow_dst, KMSAN_ORIGIN_SIZE);
97 100
98 shadow_src = kmsan_get_metadata(src, KMSAN_META_SHADOW);101 shadow_src = kmsan_get_metadata(src, KMSAN_META_SHADOW);
99 if (!shadow_src) {102 if (!shadow_src) {
100 /*103 /* @src is untracked: mark @dst as initialized. */
101 * @src is untracked: zero out destination shadow, ignore the104 kmsan_internal_unpoison_memory(dst, n, /*checked*/ false);
102 * origins, we're done.
103 */
104 __memset(shadow_dst, 0, n);
105 return;105 return;
106 }106 }
107 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n));107 KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n));
108 108
109 __memmove(shadow_dst, shadow_src, n);
110
111 origin_dst = kmsan_get_metadata(dst, KMSAN_META_ORIGIN);109 origin_dst = kmsan_get_metadata(dst, KMSAN_META_ORIGIN);
112 origin_src = kmsan_get_metadata(src, KMSAN_META_ORIGIN);110 origin_src = kmsan_get_metadata(src, KMSAN_META_ORIGIN);
113 KMSAN_WARN_ON(!origin_dst || !origin_src);111 KMSAN_WARN_ON(!origin_dst || !origin_src);
114 src_slots = (ALIGN((u64)src + n, KMSAN_ORIGIN_SIZE) -
115 ALIGN_DOWN((u64)src, KMSAN_ORIGIN_SIZE)) /
116 KMSAN_ORIGIN_SIZE;
117 dst_slots = (ALIGN((u64)dst + n, KMSAN_ORIGIN_SIZE) -
118 ALIGN_DOWN((u64)dst, KMSAN_ORIGIN_SIZE)) /
119 KMSAN_ORIGIN_SIZE;
120 KMSAN_WARN_ON((src_slots < 1) || (dst_slots < 1));
121 KMSAN_WARN_ON((src_slots - dst_slots > 1) ||
122 (dst_slots - src_slots < -1));
123 112
124 backwards = dst > src;113 backwards = dst > src;
125 i = backwards ? min(src_slots, dst_slots) - 1 : 0;114 step = backwards ? -1 : 1;
126 iter = backwards ? -1 : 1;115 iter = backwards ? n - 1 : 0;
127 116 src_off = (u64)src % KMSAN_ORIGIN_SIZE;
128 align_shadow_src =117 dst_off = (u64)dst % KMSAN_ORIGIN_SIZE;
129 (u32 *)ALIGN_DOWN((u64)shadow_src, KMSAN_ORIGIN_SIZE);118
130 for (step = 0; step < min(src_slots, dst_slots); step++, i += iter) {119 /* Copy shadow bytes one by one, updating the origins if necessary. */
131 KMSAN_WARN_ON(i < 0);120 for (i = 0; i < n; i++, iter += step) {
132 shadow = align_shadow_src[i];121 oiter_src = (iter + src_off) / KMSAN_ORIGIN_SIZE;
133 if (i == 0) {122 oiter_dst = (iter + dst_off) / KMSAN_ORIGIN_SIZE;
134 /*123 if (!shadow_src[iter]) {
135 * If @src isn't aligned on KMSAN_ORIGIN_SIZE, don't124 shadow_dst[iter] = 0;
136 * look at the first @src % KMSAN_ORIGIN_SIZE bytes125 if (!align_shadow_dst[oiter_dst])
137 * of the first shadow slot.126 origin_dst[oiter_dst] = 0;
138 */127 continue;
139 skip_bits = ((u64)src % KMSAN_ORIGIN_SIZE) * 8;
140 shadow = (shadow >> skip_bits) << skip_bits;
141 }128 }
142 if (i == src_slots - 1) {129 shadow_dst[iter] = shadow_src[iter];
143 /*130 old_origin = origin_src[oiter_src];
144 * If @src + n isn't aligned on131 if (old_origin == prev_old_origin)
145 * KMSAN_ORIGIN_SIZE, don't look at the last132 new_origin = prev_new_origin;
146 * (@src + n) % KMSAN_ORIGIN_SIZE bytes of the133 else {
147 * last shadow slot.
148 */
149 skip_bits = (((u64)src + n) % KMSAN_ORIGIN_SIZE) * 8;
150 shadow = (shadow << skip_bits) >> skip_bits;
151 }
152 /*
153 * Overwrite the origin only if the corresponding
154 * shadow is nonempty.
155 */
156 if (origin_src[i] && (origin_src[i] != old_origin) && shadow) {
157 old_origin = origin_src[i];
158 new_origin = kmsan_internal_chain_origin(old_origin);
159 /*134 /*
160 * kmsan_internal_chain_origin() may return135 * kmsan_internal_chain_origin() may return
161 * NULL, but we don't want to lose the previous136 * NULL, but we don't want to lose the previous
162 * origin value.137 * origin value.
163 */138 */
139 new_origin = kmsan_internal_chain_origin(old_origin);
164 if (!new_origin)140 if (!new_origin)
165 new_origin = old_origin;141 new_origin = old_origin;
166 }142 }
167 if (shadow)143 origin_dst[oiter_dst] = new_origin;
168 origin_dst[i] = new_origin;144 prev_new_origin = new_origin;
169 else145 prev_old_origin = old_origin;
170 origin_dst[i] = 0;
171 }
172 /*
173 * If dst_slots is greater than src_slots (i.e.
174 * dst_slots == src_slots + 1), there is an extra origin slot at the
175 * beginning or end of the destination buffer, for which we take the
176 * origin from the previous slot.
177 * This is only done if the part of the source shadow corresponding to
178 * slot is non-zero.
179 *
180 * E.g. if we copy 8 aligned bytes that are marked as uninitialized
181 * and have origins o111 and o222, to an unaligned buffer with offset 1,
182 * these two origins are copied to three origin slots, so one of then
183 * needs to be duplicated, depending on the copy direction (@backwards)
184 *
185 * src shadow: |uuuu|uuuu|....|
186 * src origin: |o111|o222|....|
187 *
188 * backwards = 0:
189 * dst shadow: |.uuu|uuuu|u...|
190 * dst origin: |....|o111|o222| - fill the empty slot with o111
191 * backwards = 1:
192 * dst shadow: |.uuu|uuuu|u...|
193 * dst origin: |o111|o222|....| - fill the empty slot with o222
194 */
195 if (src_slots < dst_slots) {
196 if (backwards) {
197 shadow = align_shadow_src[src_slots - 1];
198 skip_bits = (((u64)dst + n) % KMSAN_ORIGIN_SIZE) * 8;
199 shadow = (shadow << skip_bits) >> skip_bits;
200 if (shadow)
201 /* src_slots > 0, therefore dst_slots is at least 2 */
202 origin_dst[dst_slots - 1] =
203 origin_dst[dst_slots - 2];
204 } else {
205 shadow = align_shadow_src[0];
206 skip_bits = ((u64)dst % KMSAN_ORIGIN_SIZE) * 8;
207 shadow = (shadow >> skip_bits) << skip_bits;
208 if (shadow)
209 origin_dst[0] = origin_dst[1];
210 }
211 }146 }
212}147}
213 148
mm/kmsan/kmsan_test.c+71−38
@@ -67,6 +67,17 @@ static bool report_available(void) · 66 unmodified lines
67 return READ_ONCE(observed.available);67 return READ_ONCE(observed.available);
68}68}
69 69
70/* Reset observed.available, so that the test can trigger another report. */
71static void report_reset(void)
72{
73 unsigned long flags;
74
75 spin_lock_irqsave(&observed.lock, flags);
76 WRITE_ONCE(observed.available, false);
77 observed.ignore = false;
78 spin_unlock_irqrestore(&observed.lock, flags);
79}
80
70/* Information we expect in a report. */81/* Information we expect in a report. */
71struct expect_report {82struct expect_report {
72 const char *error_type; /* Error type. */83 const char *error_type; /* Error type. */
@@ -407,33 +418,25 @@ static void test_printk(struct kunit *test) · 334 unmodified lines
407 KUNIT_EXPECT_TRUE(test, report_matches(&expect));418 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
408}419}
409 420
410/*421/* Prevent the compiler from inlining a memcpy() call. */
411 * Prevent the compiler from optimizing @var away. Without this, Clang may422static noinline void *memcpy_noinline(volatile void *dst,
412 * notice that @var is uninitialized and drop memcpy() calls that use it.423 const volatile void *src, size_t size)
413 *424{
414 * There is OPTIMIZER_HIDE_VAR() in linux/compier.h that we cannot use here,425 return memcpy((void *)dst, (const void *)src, size);
415 * because it is implemented as inline assembly receiving @var as a parameter426}
416 * and will enforce a KMSAN check. Same is true for e.g. barrier_data(var).
417 */
418#define DO_NOT_OPTIMIZE(var) barrier()
419 427
420/*428/* Test case: ensure that memcpy() correctly copies initialized values. */
421 * Test case: ensure that memcpy() correctly copies initialized values.
422 * Also serves as a regression test to ensure DO_NOT_OPTIMIZE() does not cause
423 * extra checks.
424 */
425static void test_init_memcpy(struct kunit *test)429static void test_init_memcpy(struct kunit *test)
426{430{
427 EXPECTATION_NO_REPORT(expect);431 EXPECTATION_NO_REPORT(expect);
428 volatile int src;432 volatile long long src;
429 volatile int dst = 0;433 volatile long long dst = 0;
430 434
431 DO_NOT_OPTIMIZE(src);
432 src = 1;435 src = 1;
433 kunit_info(436 kunit_info(
434 test,437 test,
435 "memcpy()ing aligned initialized src to aligned dst (no reports)\n");438 "memcpy()ing aligned initialized src to aligned dst (no reports)\n");
436 memcpy((void *)&dst, (void *)&src, sizeof(src));439 memcpy_noinline((void *)&dst, (void *)&src, sizeof(src));
437 kmsan_check_memory((void *)&dst, sizeof(dst));440 kmsan_check_memory((void *)&dst, sizeof(dst));
438 KUNIT_EXPECT_TRUE(test, report_matches(&expect));441 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
439}442}
@@ -451,8 +454,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 11 unmodified lines
451 kunit_info(454 kunit_info(
452 test,455 test,
453 "memcpy()ing aligned uninit src to aligned dst (UMR report)\n");456 "memcpy()ing aligned uninit src to aligned dst (UMR report)\n");
454 DO_NOT_OPTIMIZE(uninit_src);457 memcpy_noinline((void *)&dst, (void *)&uninit_src, sizeof(uninit_src));
455 memcpy((void *)&dst, (void *)&uninit_src, sizeof(uninit_src));
456 kmsan_check_memory((void *)&dst, sizeof(dst));458 kmsan_check_memory((void *)&dst, sizeof(dst));
457 KUNIT_EXPECT_TRUE(test, report_matches(&expect));459 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
458}460}
@@ -463,7 +465,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 4 unmodified lines
463 *465 *
464 * Copying aligned 4-byte value to an unaligned one leads to touching two466 * Copying aligned 4-byte value to an unaligned one leads to touching two
465 * aligned 4-byte values. This test case checks that KMSAN correctly reports an467 * aligned 4-byte values. This test case checks that KMSAN correctly reports an
466 * error on the first of the two values.468 * error on the mentioned two values.
467 */469 */
468static void test_memcpy_aligned_to_unaligned(struct kunit *test)470static void test_memcpy_aligned_to_unaligned(struct kunit *test)
469{471{
@@ -474,33 +476,65 @@ static void test_memcpy_aligned_to_unaligned(struct kunit *test) · 4 unmodified lines
474 kunit_info(476 kunit_info(
475 test,477 test,
476 "memcpy()ing aligned uninit src to unaligned dst (UMR report)\n");478 "memcpy()ing aligned uninit src to unaligned dst (UMR report)\n");
477 DO_NOT_OPTIMIZE(uninit_src);479 kmsan_check_memory((void *)&uninit_src, sizeof(uninit_src));
478 memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src));480 memcpy_noinline((void *)&dst[1], (void *)&uninit_src,
481 sizeof(uninit_src));
479 kmsan_check_memory((void *)dst, 4);482 kmsan_check_memory((void *)dst, 4);
480 KUNIT_EXPECT_TRUE(test, report_matches(&expect));483 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
484 report_reset();
485 kmsan_check_memory((void *)&dst[4], sizeof(uninit_src));
486 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
481}487}
482 488
483/*489/*
484 * Test case: ensure that memcpy() correctly copies uninitialized values between490 * Test case: ensure that origin slots do not accidentally get overwritten with
485 * aligned `src` and unaligned `dst`.491 * zeroes during memcpy().
486 *492 *
487 * Copying aligned 4-byte value to an unaligned one leads to touching two493 * Previously, when copying memory from an aligned buffer to an unaligned one,
488 * aligned 4-byte values. This test case checks that KMSAN correctly reports an494 * if there were zero origins corresponding to zero shadow values in the source
489 * error on the second of the two values.495 * buffer, they could have ended up being copied to nonzero shadow values in the
496 * destination buffer:
497 *
498 * memcpy(0xffff888080a00000, 0xffff888080900002, 8)
499 *
500 * src (0xffff888080900002): ..xx .... xx..
501 * src origins: o111 0000 o222
502 * dst (0xffff888080a00000): xx.. ..xx
503 * dst origins: o111 0000
504 * (or 0000 o222)
505 *
506 * (here . stands for an initialized byte, and x for an uninitialized one.
507 *
508 * Ensure that this does not happen anymore, and for both destination bytes
509 * the origin is nonzero (i.e. KMSAN reports an error).
490 */510 */
491static void test_memcpy_aligned_to_unaligned2(struct kunit *test)511static void test_memcpy_initialized_gap(struct kunit *test)
492{512{
493 EXPECTATION_UNINIT_VALUE_FN(expect,513 EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_initialized_gap");
494 "test_memcpy_aligned_to_unaligned2");514 volatile char uninit_src[12];
495 volatile int uninit_src;
496 volatile char dst[8] = { 0 };515 volatile char dst[8] = { 0 };
497 516
498 kunit_info(517 kunit_info(
499 test,518 test,
500 "memcpy()ing aligned uninit src to unaligned dst - part 2 (UMR report)\n");519 "unaligned 4-byte initialized value gets a nonzero origin after memcpy() - (2 UMR reports)\n");
501 DO_NOT_OPTIMIZE(uninit_src);520
502 memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src));521 uninit_src[0] = 42;
503 kmsan_check_memory((void *)&dst[4], sizeof(uninit_src));522 uninit_src[1] = 42;
523 uninit_src[4] = 42;
524 uninit_src[5] = 42;
525 uninit_src[6] = 42;
526 uninit_src[7] = 42;
527 uninit_src[10] = 42;
528 uninit_src[11] = 42;
529 memcpy_noinline((void *)&dst[0], (void *)&uninit_src[2], 8);
530
531 kmsan_check_memory((void *)&dst[0], 4);
532 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
533 report_reset();
534 kmsan_check_memory((void *)&dst[2], 4);
535 KUNIT_EXPECT_FALSE(test, report_matches(&expect));
536 report_reset();
537 kmsan_check_memory((void *)&dst[4], 4);
504 KUNIT_EXPECT_TRUE(test, report_matches(&expect));538 KUNIT_EXPECT_TRUE(test, report_matches(&expect));
505}539}
506 540
@@ -513,7 +547,6 @@ static void test_memcpy_aligned_to_unaligned2(struct kunit *test) · 6 unmodified lines
513 \547 \
514 kunit_info(test, \548 kunit_info(test, \
515 "memset" #size "() should initialize memory\n"); \549 "memset" #size "() should initialize memory\n"); \
516 DO_NOT_OPTIMIZE(uninit); \
517 memset##size((uint##size##_t *)&uninit, 0, 1); \550 memset##size((uint##size##_t *)&uninit, 0, 1); \
518 kmsan_check_memory((void *)&uninit, sizeof(uninit)); \551 kmsan_check_memory((void *)&uninit, sizeof(uninit)); \
519 KUNIT_EXPECT_TRUE(test, report_matches(&expect)); \552 KUNIT_EXPECT_TRUE(test, report_matches(&expect)); \
@@ -598,7 +631,7 @@ static struct kunit_case kmsan_test_cases[] = { · 78 unmodified lines
598 KUNIT_CASE(test_init_memcpy),631 KUNIT_CASE(test_init_memcpy),
599 KUNIT_CASE(test_memcpy_aligned_to_aligned),632 KUNIT_CASE(test_memcpy_aligned_to_aligned),
600 KUNIT_CASE(test_memcpy_aligned_to_unaligned),633 KUNIT_CASE(test_memcpy_aligned_to_unaligned),
601 KUNIT_CASE(test_memcpy_aligned_to_unaligned2),634 KUNIT_CASE(test_memcpy_initialized_gap),
602 KUNIT_CASE(test_memset16),635 KUNIT_CASE(test_memset16),
603 KUNIT_CASE(test_memset32),636 KUNIT_CASE(test_memset32),
604 KUNIT_CASE(test_memset64),637 KUNIT_CASE(test_memset64),
mm/kmsan/shadow.c+7−2
@@ -285,12 +285,17 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end) · 284 unmodified lines
285 size = PAGE_ALIGN((u64)end - (u64)start);285 size = PAGE_ALIGN((u64)end - (u64)start);
286 shadow = memblock_alloc(size, PAGE_SIZE);286 shadow = memblock_alloc(size, PAGE_SIZE);
287 origin = memblock_alloc(size, PAGE_SIZE);287 origin = memblock_alloc(size, PAGE_SIZE);
288
289 if (!shadow || !origin)
290 panic("%s: Failed to allocate metadata memory for early boot range of size %llu",
291 __func__, size);
292
288 for (u64 addr = 0; addr < size; addr += PAGE_SIZE) {293 for (u64 addr = 0; addr < size; addr += PAGE_SIZE) {
289 page = virt_to_page_or_null((char *)start + addr);294 page = virt_to_page_or_null((char *)start + addr);
290 shadow_p = virt_to_page_or_null((char *)shadow + addr);295 shadow_p = virt_to_page((char *)shadow + addr);
291 set_no_shadow_origin_page(shadow_p);296 set_no_shadow_origin_page(shadow_p);
292 shadow_page_for(page) = shadow_p;297 shadow_page_for(page) = shadow_p;
293 origin_p = virt_to_page_or_null((char *)origin + addr);298 origin_p = virt_to_page((char *)origin + addr);
294 set_no_shadow_origin_page(origin_p);299 set_no_shadow_origin_page(origin_p);
295 origin_page_for(page) = origin_p;300 origin_page_for(page) = origin_p;
296 }301 }
mm/ksm.c+116−0
@@ -56,6 +56,8 @@ · 55 unmodified lines
56#define DO_NUMA(x) do { } while (0)56#define DO_NUMA(x) do { } while (0)
57#endif57#endif
58 58
59typedef u8 rmap_age_t;
60
59/**61/**
60 * DOC: Overview62 * DOC: Overview
61 *63 *
@@ -193,6 +195,8 @@ struct ksm_stable_node { · 131 unmodified lines
193 * @node: rb node of this rmap_item in the unstable tree195 * @node: rb node of this rmap_item in the unstable tree
194 * @head: pointer to stable_node heading this list in the stable tree196 * @head: pointer to stable_node heading this list in the stable tree
195 * @hlist: link into hlist of rmap_items hanging off that stable_node197 * @hlist: link into hlist of rmap_items hanging off that stable_node
198 * @age: number of scan iterations since creation
199 * @remaining_skips: how many scans to skip
196 */200 */
197struct ksm_rmap_item {201struct ksm_rmap_item {
198 struct ksm_rmap_item *rmap_list;202 struct ksm_rmap_item *rmap_list;
@@ -205,6 +209,8 @@ struct ksm_rmap_item { · 6 unmodified lines
205 struct mm_struct *mm;209 struct mm_struct *mm;
206 unsigned long address; /* + low bits used for flags below */210 unsigned long address; /* + low bits used for flags below */
207 unsigned int oldchecksum; /* when unstable */211 unsigned int oldchecksum; /* when unstable */
212 rmap_age_t age;
213 rmap_age_t remaining_skips;
208 union {214 union {
209 struct rb_node node; /* when node of unstable tree */215 struct rb_node node; /* when node of unstable tree */
210 struct { /* when listed from stable tree */216 struct { /* when listed from stable tree */
@@ -281,9 +287,16 @@ static unsigned int zero_checksum __read_mostly; · 70 unmodified lines
281/* Whether to merge empty (zeroed) pages with actual zero pages */287/* Whether to merge empty (zeroed) pages with actual zero pages */
282static bool ksm_use_zero_pages __read_mostly;288static bool ksm_use_zero_pages __read_mostly;
283 289
290/* Skip pages that couldn't be de-duplicated previously */
291/* Default to true at least temporarily, for testing */
292static bool ksm_smart_scan = true;
293
284/* The number of zero pages which is placed by KSM */294/* The number of zero pages which is placed by KSM */
285unsigned long ksm_zero_pages;295unsigned long ksm_zero_pages;
286 296
297/* The number of pages that have been skipped due to "smart scanning" */
298static unsigned long ksm_pages_skipped;
299
287#ifdef CONFIG_NUMA300#ifdef CONFIG_NUMA
288/* Zeroed when merging across nodes is not allowed */301/* Zeroed when merging across nodes is not allowed */
289static unsigned int ksm_merge_across_nodes = 1;302static unsigned int ksm_merge_across_nodes = 1;
@@ -2305,6 +2318,74 @@ static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, · 2015 unmodified lines
2305 return rmap_item;2318 return rmap_item;
2306}2319}
2307 2320
2321/*
2322 * Calculate skip age for the ksm page age. The age determines how often
2323 * de-duplicating has already been tried unsuccessfully. If the age is
2324 * smaller, the scanning of this page is skipped for less scans.
2325 *
2326 * @age: rmap_item age of page
2327 */
2328static unsigned int skip_age(rmap_age_t age)
2329{
2330 if (age <= 3)
2331 return 1;
2332 if (age <= 5)
2333 return 2;
2334 if (age <= 8)
2335 return 4;
2336
2337 return 8;
2338}
2339
2340/*
2341 * Determines if a page should be skipped for the current scan.
2342 *
2343 * @page: page to check
2344 * @rmap_item: associated rmap_item of page
2345 */
2346static bool should_skip_rmap_item(struct page *page,
2347 struct ksm_rmap_item *rmap_item)
2348{
2349 rmap_age_t age;
2350
2351 if (!ksm_smart_scan)
2352 return false;
2353
2354 /*
2355 * Never skip pages that are already KSM; pages cmp_and_merge_page()
2356 * will essentially ignore them, but we still have to process them
2357 * properly.
2358 */
2359 if (PageKsm(page))
2360 return false;
2361
2362 age = rmap_item->age;
2363 if (age != U8_MAX)
2364 rmap_item->age++;
2365
2366 /*
2367 * Smaller ages are not skipped, they need to get a chance to go
2368 * through the different phases of the KSM merging.
2369 */
2370 if (age < 3)
2371 return false;
2372
2373 /*
2374 * Are we still allowed to skip? If not, then don't skip it
2375 * and determine how much more often we are allowed to skip next.
2376 */
2377 if (!rmap_item->remaining_skips) {
2378 rmap_item->remaining_skips = skip_age(age);
2379 return false;
2380 }
2381
2382 /* Skip this page */
2383 ksm_pages_skipped++;
2384 rmap_item->remaining_skips--;
2385 remove_rmap_item_from_tree(rmap_item);
2386 return true;
2387}
2388
2308static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)2389static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
2309{2390{
2310 struct mm_struct *mm;2391 struct mm_struct *mm;
@@ -2409,6 +2490,10 @@ next_mm: · 98 unmodified lines
2409 if (rmap_item) {2490 if (rmap_item) {
2410 ksm_scan.rmap_list =2491 ksm_scan.rmap_list =
2411 &rmap_item->rmap_list;2492 &rmap_item->rmap_list;
2493
2494 if (should_skip_rmap_item(*page, rmap_item))
2495 goto next_page;
2496
2412 ksm_scan.address += PAGE_SIZE;2497 ksm_scan.address += PAGE_SIZE;
2413 } else2498 } else
2414 put_page(*page);2499 put_page(*page);
@@ -3383,6 +3468,13 @@ static ssize_t pages_volatile_show(struct kobject *kobj, · 968 unmodified lines
3383}3468}
3384KSM_ATTR_RO(pages_volatile);3469KSM_ATTR_RO(pages_volatile);
3385 3470
3471static ssize_t pages_skipped_show(struct kobject *kobj,
3472 struct kobj_attribute *attr, char *buf)
3473{
3474 return sysfs_emit(buf, "%lu\n", ksm_pages_skipped);
3475}
3476KSM_ATTR_RO(pages_skipped);
3477
3386static ssize_t ksm_zero_pages_show(struct kobject *kobj,3478static ssize_t ksm_zero_pages_show(struct kobject *kobj,
3387 struct kobj_attribute *attr, char *buf)3479 struct kobj_attribute *attr, char *buf)
3388{3480{
@@ -3449,6 +3541,28 @@ static ssize_t full_scans_show(struct kobject *kobj, · 60 unmodified lines
3449}3541}
3450KSM_ATTR_RO(full_scans);3542KSM_ATTR_RO(full_scans);
3451 3543
3544static ssize_t smart_scan_show(struct kobject *kobj,
3545 struct kobj_attribute *attr, char *buf)
3546{
3547 return sysfs_emit(buf, "%u\n", ksm_smart_scan);
3548}
3549
3550static ssize_t smart_scan_store(struct kobject *kobj,
3551 struct kobj_attribute *attr,
3552 const char *buf, size_t count)
3553{
3554 int err;
3555 bool value;
3556
3557 err = kstrtobool(buf, &value);
3558 if (err)
3559 return -EINVAL;
3560
3561 ksm_smart_scan = value;
3562 return count;
3563}
3564KSM_ATTR(smart_scan);
3565
3452static struct attribute *ksm_attrs[] = {3566static struct attribute *ksm_attrs[] = {
3453 &sleep_millisecs_attr.attr,3567 &sleep_millisecs_attr.attr,
3454 &pages_to_scan_attr.attr,3568 &pages_to_scan_attr.attr,
@@ -3458,6 +3572,7 @@ static struct attribute *ksm_attrs[] = { · 3 unmodified lines
3458 &pages_sharing_attr.attr,3572 &pages_sharing_attr.attr,
3459 &pages_unshared_attr.attr,3573 &pages_unshared_attr.attr,
3460 &pages_volatile_attr.attr,3574 &pages_volatile_attr.attr,
3575 &pages_skipped_attr.attr,
3461 &ksm_zero_pages_attr.attr,3576 &ksm_zero_pages_attr.attr,
3462 &full_scans_attr.attr,3577 &full_scans_attr.attr,
3463#ifdef CONFIG_NUMA3578#ifdef CONFIG_NUMA
@@ -3469,6 +3584,7 @@ static struct attribute *ksm_attrs[] = { · 5 unmodified lines
3469 &stable_node_chains_prune_millisecs_attr.attr,3584 &stable_node_chains_prune_millisecs_attr.attr,
3470 &use_zero_pages_attr.attr,3585 &use_zero_pages_attr.attr,
3471 &general_profit_attr.attr,3586 &general_profit_attr.attr,
3587 &smart_scan_attr.attr,
3472 NULL,3588 NULL,
3473};3589};
3474 3590
mm/madvise.c+6−27
@@ -141,7 +141,6 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 140 unmodified lines
141{141{
142 struct mm_struct *mm = vma->vm_mm;142 struct mm_struct *mm = vma->vm_mm;
143 int error;143 int error;
144 pgoff_t pgoff;
145 VMA_ITERATOR(vmi, mm, start);144 VMA_ITERATOR(vmi, mm, start);
146 145
147 if (new_flags == vma->vm_flags && anon_vma_name_eq(anon_vma_name(vma), anon_name)) {146 if (new_flags == vma->vm_flags && anon_vma_name_eq(anon_vma_name(vma), anon_name)) {
@@ -149,30 +148,13 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 1 unmodified line
149 return 0;148 return 0;
150 }149 }
151 150
152 pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT);151 vma = vma_modify_flags_name(&vmi, *prev, vma, start, end, new_flags,
153 *prev = vma_merge(&vmi, mm, *prev, start, end, new_flags,152 anon_name);
154 vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma),153 if (IS_ERR(vma))
155 vma->vm_userfaultfd_ctx, anon_name);154 return PTR_ERR(vma);
156 if (*prev) {
157 vma = *prev;
158 goto success;
159 }
160 155
161 *prev = vma;156 *prev = vma;
162 157
163 if (start != vma->vm_start) {
164 error = split_vma(&vmi, vma, start, 1);
165 if (error)
166 return error;
167 }
168
169 if (end != vma->vm_end) {
170 error = split_vma(&vmi, vma, end, 0);
171 if (error)
172 return error;
173 }
174
175success:
176 /* vm_flags is protected by the mmap_lock held in write mode. */158 /* vm_flags is protected by the mmap_lock held in write mode. */
177 vma_start_write(vma);159 vma_start_write(vma);
178 vm_flags_reset(vma, new_flags);160 vm_flags_reset(vma, new_flags);
@@ -746,11 +728,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 567 unmodified lines
746 folio_mark_lazyfree(folio);728 folio_mark_lazyfree(folio);
747 }729 }
748 730
749 if (nr_swap) {731 if (nr_swap)
750 if (current->mm == mm)
751 sync_mm_rss(mm);
752 add_mm_counter(mm, MM_SWAPENTS, nr_swap);732 add_mm_counter(mm, MM_SWAPENTS, nr_swap);
753 }
754 if (start_pte) {733 if (start_pte) {
755 arch_leave_lazy_mmu_mode();734 arch_leave_lazy_mmu_mode();
756 pte_unmap_unlock(start_pte, ptl);735 pte_unmap_unlock(start_pte, ptl);
@@ -991,7 +970,7 @@ static long madvise_remove(struct vm_area_struct *vma, · 234 unmodified lines
991 return -EINVAL;970 return -EINVAL;
992 }971 }
993 972
994 if ((vma->vm_flags & (VM_SHARED|VM_WRITE)) != (VM_SHARED|VM_WRITE))973 if (!vma_is_shared_maywrite(vma))
995 return -EACCES;974 return -EACCES;
996 975
997 offset = (loff_t)(start - vma->vm_start)976 offset = (loff_t)(start - vma->vm_start)
mm/memblock.c+36−13
@@ -892,6 +892,7 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 891 unmodified lines
892 892
893/**893/**
894 * memblock_setclr_flag - set or clear flag for a memory region894 * memblock_setclr_flag - set or clear flag for a memory region
895 * @type: memblock type to set/clear flag for
895 * @base: base address of the region896 * @base: base address of the region
896 * @size: size of the region897 * @size: size of the region
897 * @set: set or clear the flag898 * @set: set or clear the flag
@@ -901,10 +902,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 3 unmodified lines
901 *902 *
902 * Return: 0 on success, -errno on failure.903 * Return: 0 on success, -errno on failure.
903 */904 */
904static int __init_memblock memblock_setclr_flag(phys_addr_t base,905static int __init_memblock memblock_setclr_flag(struct memblock_type *type,
905 phys_addr_t size, int set, int flag)906 phys_addr_t base, phys_addr_t size, int set, int flag)
906{907{
907 struct memblock_type *type = &memblock.memory;
908 int i, ret, start_rgn, end_rgn;908 int i, ret, start_rgn, end_rgn;
909 909
910 ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn);910 ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn);
@@ -933,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, · 22 unmodified lines
933 */933 */
934int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size)934int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size)
935{935{
936 return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG);936 return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG);
937}937}
938 938
939/**939/**
@@ -945,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) · 5 unmodified lines
945 */945 */
946int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)946int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
947{947{
948 return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG);948 return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG);
949}949}
950 950
951/**951/**
@@ -962,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 10 unmodified lines
962 962
963 system_has_some_mirror = true;963 system_has_some_mirror = true;
964 964
965 return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR);965 return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR);
966}966}
967 967
968/**968/**
@@ -982,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 13 unmodified lines
982 */982 */
983int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size)983int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size)
984{984{
985 return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP);985 return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP);
986}986}
987 987
988/**988/**
@@ -994,7 +994,25 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) · 5 unmodified lines
994 */994 */
995int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size)995int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size)
996{996{
997 return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP);997 return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP);
998}
999
1000/**
1001 * memblock_reserved_mark_noinit - Mark a reserved memory region with flag
1002 * MEMBLOCK_RSRV_NOINIT which results in the struct pages not being initialized
1003 * for this region.
1004 * @base: the base phys addr of the region
1005 * @size: the size of the region
1006 *
1007 * struct pages will not be initialized for reserved memory regions marked with
1008 * %MEMBLOCK_RSRV_NOINIT.
1009 *
1010 * Return: 0 on success, -errno on failure.
1011 */
1012int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size)
1013{
1014 return memblock_setclr_flag(&memblock.reserved, base, size, 1,
1015 MEMBLOCK_RSRV_NOINIT);
998}1016}
999 1017
1000static bool should_skip_region(struct memblock_type *type,1018static bool should_skip_region(struct memblock_type *type,
@@ -2113,13 +2131,18 @@ static void __init memmap_init_reserved_pages(void) · 1112 unmodified lines
2113 memblock_set_node(start, end, &memblock.reserved, nid);2131 memblock_set_node(start, end, &memblock.reserved, nid);
2114 }2132 }
2115 2133
2116 /* initialize struct pages for the reserved regions */2134 /*
2135 * initialize struct pages for reserved regions that don't have
2136 * the MEMBLOCK_RSRV_NOINIT flag set
2137 */
2117 for_each_reserved_mem_region(region) {2138 for_each_reserved_mem_region(region) {
2118 nid = memblock_get_region_node(region);2139 if (!memblock_is_reserved_noinit(region)) {
2119 start = region->base;2140 nid = memblock_get_region_node(region);
2120 end = start + region->size;2141 start = region->base;
2142 end = start + region->size;
2121 2143
2122 reserve_bootmem_region(start, end, nid);2144 reserve_bootmem_region(start, end, nid);
2145 }
2123 }2146 }
2124}2147}
2125 2148
mm/memcontrol.c+341−60
@@ -249,6 +249,9 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) · 248 unmodified lines
249 return container_of(vmpr, struct mem_cgroup, vmpressure);249 return container_of(vmpr, struct mem_cgroup, vmpressure);
250}250}
251 251
252#define CURRENT_OBJCG_UPDATE_BIT 0
253#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT)
254
252#ifdef CONFIG_MEMCG_KMEM255#ifdef CONFIG_MEMCG_KMEM
253static DEFINE_SPINLOCK(objcg_lock);256static DEFINE_SPINLOCK(objcg_lock);
254 257
@@ -704,6 +707,8 @@ static const unsigned int memcg_vm_event_stat[] = { · 449 unmodified lines
704#ifdef CONFIG_TRANSPARENT_HUGEPAGE707#ifdef CONFIG_TRANSPARENT_HUGEPAGE
705 THP_FAULT_ALLOC,708 THP_FAULT_ALLOC,
706 THP_COLLAPSE_ALLOC,709 THP_COLLAPSE_ALLOC,
710 THP_SWPOUT,
711 THP_SWPOUT_FALLBACK,
707#endif712#endif
708};713};
709 714
@@ -761,6 +766,22 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) · 51 unmodified lines
761 return x;766 return x;
762}767}
763 768
769static int memcg_page_state_unit(int item);
770
771/*
772 * Normalize the value passed into memcg_rstat_updated() to be in pages. Round
773 * up non-zero sub-page updates to 1 page as zero page updates are ignored.
774 */
775static int memcg_state_val_in_pages(int idx, int val)
776{
777 int unit = memcg_page_state_unit(idx);
778
779 if (!val || unit == PAGE_SIZE)
780 return val;
781 else
782 return max(val * unit / PAGE_SIZE, 1UL);
783}
784
764/**785/**
765 * __mod_memcg_state - update cgroup memory statistics786 * __mod_memcg_state - update cgroup memory statistics
766 * @memcg: the memory cgroup787 * @memcg: the memory cgroup
@@ -773,7 +794,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) · 6 unmodified lines
773 return;794 return;
774 795
775 __this_cpu_add(memcg->vmstats_percpu->state[idx], val);796 __this_cpu_add(memcg->vmstats_percpu->state[idx], val);
776 memcg_rstat_updated(memcg, val);797 memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val));
777}798}
778 799
779/* idx can be of type enum memcg_stat_item or node_stat_item. */800/* idx can be of type enum memcg_stat_item or node_stat_item. */
@@ -798,7 +819,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 18 unmodified lines
798 memcg = pn->memcg;819 memcg = pn->memcg;
799 820
800 /*821 /*
801 * The caller from rmap relay on disabled preemption becase they never822 * The caller from rmap relies on disabled preemption because they never
802 * update their counter from in-interrupt context. For these two823 * update their counter from in-interrupt context. For these two
803 * counters we check that the update is never performed from an824 * counters we check that the update is never performed from an
804 * interrupt context while other caller need to have disabled interrupt.825 * interrupt context while other caller need to have disabled interrupt.
@@ -824,7 +845,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 19 unmodified lines
824 /* Update lruvec */845 /* Update lruvec */
825 __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val);846 __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val);
826 847
827 memcg_rstat_updated(memcg, val);848 memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val));
828 memcg_stats_unlock();849 memcg_stats_unlock();
829}850}
830 851
@@ -1068,17 +1089,25 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 237 unmodified lines
1068}1089}
1069EXPORT_SYMBOL(get_mem_cgroup_from_mm);1090EXPORT_SYMBOL(get_mem_cgroup_from_mm);
1070 1091
1071static __always_inline bool memcg_kmem_bypass(void)1092/**
1093 * get_mem_cgroup_from_current - Obtain a reference on current task's memcg.
1094 */
1095struct mem_cgroup *get_mem_cgroup_from_current(void)
1072{1096{
1073 /* Allow remote memcg charging from any context. */1097 struct mem_cgroup *memcg;
1074 if (unlikely(active_memcg()))
1075 return false;
1076 1098
1077 /* Memcg to charge can't be determined. */1099 if (mem_cgroup_disabled())
1078 if (!in_task() || !current->mm || (current->flags & PF_KTHREAD))1100 return NULL;
1079 return true;
1080 1101
1081 return false;1102again:
1103 rcu_read_lock();
1104 memcg = mem_cgroup_from_task(current);
1105 if (!css_tryget(&memcg->css)) {
1106 rcu_read_unlock();
1107 goto again;
1108 }
1109 rcu_read_unlock();
1110 return memcg;
1082}1111}
1083 1112
1084/**1113/**
@@ -1533,7 +1562,7 @@ static const struct memory_stat memory_stats[] = { · 448 unmodified lines
1533 { "workingset_nodereclaim", WORKINGSET_NODERECLAIM },1562 { "workingset_nodereclaim", WORKINGSET_NODERECLAIM },
1534};1563};
1535 1564
1536/* Translate stat items to the correct unit for memory.stat output */1565/* The actual unit of the state item, not the same as the output unit */
1537static int memcg_page_state_unit(int item)1566static int memcg_page_state_unit(int item)
1538{1567{
1539 switch (item) {1568 switch (item) {
@@ -1541,6 +1570,22 @@ static int memcg_page_state_unit(int item) · 1 unmodified line
1541 case MEMCG_ZSWAP_B:1570 case MEMCG_ZSWAP_B:
1542 case NR_SLAB_RECLAIMABLE_B:1571 case NR_SLAB_RECLAIMABLE_B:
1543 case NR_SLAB_UNRECLAIMABLE_B:1572 case NR_SLAB_UNRECLAIMABLE_B:
1573 return 1;
1574 case NR_KERNEL_STACK_KB:
1575 return SZ_1K;
1576 default:
1577 return PAGE_SIZE;
1578 }
1579}
1580
1581/* Translate stat items to the correct unit for memory.stat output */
1582static int memcg_page_state_output_unit(int item)
1583{
1584 /*
1585 * Workingset state is actually in pages, but we export it to userspace
1586 * as a scalar count of events, so special case it here.
1587 */
1588 switch (item) {
1544 case WORKINGSET_REFAULT_ANON:1589 case WORKINGSET_REFAULT_ANON:
1545 case WORKINGSET_REFAULT_FILE:1590 case WORKINGSET_REFAULT_FILE:
1546 case WORKINGSET_ACTIVATE_ANON:1591 case WORKINGSET_ACTIVATE_ANON:
@@ -1549,17 +1594,23 @@ static int memcg_page_state_unit(int item) · 2 unmodified lines
1549 case WORKINGSET_RESTORE_FILE:1594 case WORKINGSET_RESTORE_FILE:
1550 case WORKINGSET_NODERECLAIM:1595 case WORKINGSET_NODERECLAIM:
1551 return 1;1596 return 1;
1552 case NR_KERNEL_STACK_KB:
1553 return SZ_1K;
1554 default:1597 default:
1555 return PAGE_SIZE;1598 return memcg_page_state_unit(item);
1556 }1599 }
1557}1600}
1558 1601
1559static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg,1602static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg,
1560 int item)1603 int item)
1561{1604{
1562 return memcg_page_state(memcg, item) * memcg_page_state_unit(item);1605 return memcg_page_state(memcg, item) *
1606 memcg_page_state_output_unit(item);
1607}
1608
1609static inline unsigned long memcg_page_state_local_output(
1610 struct mem_cgroup *memcg, int item)
1611{
1612 return memcg_page_state_local(memcg, item) *
1613 memcg_page_state_output_unit(item);
1563}1614}
1564 1615
1565static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)1616static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
@@ -2833,7 +2884,12 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · 1267 unmodified lines
2833 return try_charge_memcg(memcg, gfp_mask, nr_pages);2884 return try_charge_memcg(memcg, gfp_mask, nr_pages);
2834}2885}
2835 2886
2836static inline void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)2887/**
2888 * mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call.
2889 * @memcg: memcg previously charged.
2890 * @nr_pages: number of pages previously charged.
2891 */
2892void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
2837{2893{
2838 if (mem_cgroup_is_root(memcg))2894 if (mem_cgroup_is_root(memcg))
2839 return;2895 return;
@@ -2858,6 +2914,22 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 18 unmodified lines
2858 folio->memcg_data = (unsigned long)memcg;2914 folio->memcg_data = (unsigned long)memcg;
2859}2915}
2860 2916
2917/**
2918 * mem_cgroup_commit_charge - commit a previously successful try_charge().
2919 * @folio: folio to commit the charge to.
2920 * @memcg: memcg previously charged.
2921 */
2922void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg)
2923{
2924 css_get(&memcg->css);
2925 commit_charge(folio, memcg);
2926
2927 local_irq_disable();
2928 mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio));
2929 memcg_check_events(memcg, folio_nid(folio));
2930 local_irq_enable();
2931}
2932
2861#ifdef CONFIG_MEMCG_KMEM2933#ifdef CONFIG_MEMCG_KMEM
2862/*2934/*
2863 * The allocated objcg pointers array is not accounted directly.2935 * The allocated objcg pointers array is not accounted directly.
@@ -3007,28 +3079,105 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) · 143 unmodified lines
3007 3079
3008 for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {3080 for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
3009 objcg = rcu_dereference(memcg->objcg);3081 objcg = rcu_dereference(memcg->objcg);
3010 if (objcg && obj_cgroup_tryget(objcg))3082 if (likely(objcg && obj_cgroup_tryget(objcg)))
3011 break;3083 break;
3012 objcg = NULL;3084 objcg = NULL;
3013 }3085 }
3014 return objcg;3086 return objcg;
3015}3087}
3016 3088
3017__always_inline struct obj_cgroup *get_obj_cgroup_from_current(void)3089static struct obj_cgroup *current_objcg_update(void)
3018{3090{
3019 struct obj_cgroup *objcg = NULL;
3020 struct mem_cgroup *memcg;3091 struct mem_cgroup *memcg;
3092 struct obj_cgroup *old, *objcg = NULL;
3021 3093
3022 if (memcg_kmem_bypass())3094 do {
3023 return NULL;3095 /* Atomically drop the update bit. */
3096 old = xchg(&current->objcg, NULL);
3097 if (old) {
3098 old = (struct obj_cgroup *)
3099 ((unsigned long)old & ~CURRENT_OBJCG_UPDATE_FLAG);
3100 if (old)
3101 obj_cgroup_put(old);
3102
3103 old = NULL;
3104 }
3024 3105
3025 rcu_read_lock();3106 /* If new objcg is NULL, no reason for the second atomic update. */
3026 if (unlikely(active_memcg()))3107 if (!current->mm || (current->flags & PF_KTHREAD))
3027 memcg = active_memcg();3108 return NULL;
3028 else3109
3110 /*
3111 * Release the objcg pointer from the previous iteration,
3112 * if try_cmpxcg() below fails.
3113 */
3114 if (unlikely(objcg)) {
3115 obj_cgroup_put(objcg);
3116 objcg = NULL;
3117 }
3118
3119 /*
3120 * Obtain the new objcg pointer. The current task can be
3121 * asynchronously moved to another memcg and the previous
3122 * memcg can be offlined. So let's get the memcg pointer
3123 * and try get a reference to objcg under a rcu read lock.
3124 */
3125
3126 rcu_read_lock();
3029 memcg = mem_cgroup_from_task(current);3127 memcg = mem_cgroup_from_task(current);
3030 objcg = __get_obj_cgroup_from_memcg(memcg);3128 objcg = __get_obj_cgroup_from_memcg(memcg);
3031 rcu_read_unlock();3129 rcu_read_unlock();
3130
3131 /*
3132 * Try set up a new objcg pointer atomically. If it
3133 * fails, it means the update flag was set concurrently, so
3134 * the whole procedure should be repeated.
3135 */
3136 } while (!try_cmpxchg(&current->objcg, &old, objcg));
3137
3138 return objcg;
3139}
3140
3141__always_inline struct obj_cgroup *current_obj_cgroup(void)
3142{
3143 struct mem_cgroup *memcg;
3144 struct obj_cgroup *objcg;
3145
3146 if (in_task()) {
3147 memcg = current->active_memcg;
3148 if (unlikely(memcg))
3149 goto from_memcg;
3150
3151 objcg = READ_ONCE(current->objcg);
3152 if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG))
3153 objcg = current_objcg_update();
3154 /*
3155 * Objcg reference is kept by the task, so it's safe
3156 * to use the objcg by the current task.
3157 */
3158 return objcg;
3159 }
3160
3161 memcg = this_cpu_read(int_active_memcg);
3162 if (unlikely(memcg))
3163 goto from_memcg;
3164
3165 return NULL;
3166
3167from_memcg:
3168 for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
3169 /*
3170 * Memcg pointer is protected by scope (see set_active_memcg())
3171 * and is pinning the corresponding objcg, so objcg can't go
3172 * away and can be used within the scope without any additional
3173 * protection.
3174 */
3175 objcg = rcu_dereference_check(memcg->objcg, 1);
3176 if (likely(objcg))
3177 break;
3178 objcg = NULL;
3179 }
3180
3032 return objcg;3181 return objcg;
3033}3182}
3034 3183
@@ -3126,15 +3275,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) · 91 unmodified lines
3126 struct obj_cgroup *objcg;3275 struct obj_cgroup *objcg;
3127 int ret = 0;3276 int ret = 0;
3128 3277
3129 objcg = get_obj_cgroup_from_current();3278 objcg = current_obj_cgroup();
3130 if (objcg) {3279 if (objcg) {
3131 ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order);3280 ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order);
3132 if (!ret) {3281 if (!ret) {
3282 obj_cgroup_get(objcg);
3133 page->memcg_data = (unsigned long)objcg |3283 page->memcg_data = (unsigned long)objcg |
3134 MEMCG_DATA_KMEM;3284 MEMCG_DATA_KMEM;
3135 return 0;3285 return 0;
3136 }3286 }
3137 obj_cgroup_put(objcg);
3138 }3287 }
3139 return ret;3288 return ret;
3140}3289}
@@ -3761,6 +3910,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) · 620 unmodified lines
3761 3910
3762 objcg->memcg = memcg;3911 objcg->memcg = memcg;
3763 rcu_assign_pointer(memcg->objcg, objcg);3912 rcu_assign_pointer(memcg->objcg, objcg);
3913 obj_cgroup_get(objcg);
3914 memcg->orig_objcg = objcg;
3764 3915
3765 static_branch_enable(&memcg_kmem_online_key);3916 static_branch_enable(&memcg_kmem_online_key);
3766 3917
@@ -4066,7 +4217,10 @@ static const unsigned int memcg1_stats[] = { · 299 unmodified lines
4066 NR_WRITEBACK,4217 NR_WRITEBACK,
4067 WORKINGSET_REFAULT_ANON,4218 WORKINGSET_REFAULT_ANON,
4068 WORKINGSET_REFAULT_FILE,4219 WORKINGSET_REFAULT_FILE,
4220#ifdef CONFIG_SWAP
4069 MEMCG_SWAP,4221 MEMCG_SWAP,
4222 NR_SWAPCACHE,
4223#endif
4070};4224};
4071 4225
4072static const char *const memcg1_stat_names[] = {4226static const char *const memcg1_stat_names[] = {
@@ -4081,7 +4235,10 @@ static const char *const memcg1_stat_names[] = { · 8 unmodified lines
4081 "writeback",4235 "writeback",
4082 "workingset_refault_anon",4236 "workingset_refault_anon",
4083 "workingset_refault_file",4237 "workingset_refault_file",
4238#ifdef CONFIG_SWAP
4084 "swap",4239 "swap",
4240 "swapcached",
4241#endif
4085};4242};
4086 4243
4087/* Universal VM events cgroup1 shows, original sort order */4244/* Universal VM events cgroup1 shows, original sort order */
@@ -4105,11 +4262,8 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 17 unmodified lines
4105 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {4262 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
4106 unsigned long nr;4263 unsigned long nr;
4107 4264
4108 if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account())4265 nr = memcg_page_state_local_output(memcg, memcg1_stats[i]);
4109 continue;4266 seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr);
4110 nr = memcg_page_state_local(memcg, memcg1_stats[i]);
4111 seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i],
4112 nr * memcg_page_state_unit(memcg1_stats[i]));
4113 }4267 }
4114 4268
4115 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)4269 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
@@ -4129,18 +4283,15 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 13 unmodified lines
4129 }4283 }
4130 seq_buf_printf(s, "hierarchical_memory_limit %llu\n",4284 seq_buf_printf(s, "hierarchical_memory_limit %llu\n",
4131 (u64)memory * PAGE_SIZE);4285 (u64)memory * PAGE_SIZE);
4132 if (do_memsw_account())4286 seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",
4133 seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",4287 (u64)memsw * PAGE_SIZE);
4134 (u64)memsw * PAGE_SIZE);
4135 4288
4136 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {4289 for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
4137 unsigned long nr;4290 unsigned long nr;
4138 4291
4139 if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account())4292 nr = memcg_page_state_output(memcg, memcg1_stats[i]);
4140 continue;
4141 nr = memcg_page_state(memcg, memcg1_stats[i]);
4142 seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i],4293 seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i],
4143 (u64)nr * memcg_page_state_unit(memcg1_stats[i]));4294 (u64)nr);
4144 }4295 }
4145 4296
4146 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)4297 for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
@@ -5268,6 +5419,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) · 1121 unmodified lines
5268{5419{
5269 int node;5420 int node;
5270 5421
5422 if (memcg->orig_objcg)
5423 obj_cgroup_put(memcg->orig_objcg);
5424
5271 for_each_node(node)5425 for_each_node(node)
5272 free_mem_cgroup_per_node_info(memcg, node);5426 free_mem_cgroup_per_node_info(memcg, node);
5273 kfree(memcg->vmstats);5427 kfree(memcg->vmstats);
@@ -6076,7 +6230,7 @@ static void __mem_cgroup_clear_mc(void) · 802 unmodified lines
6076 6230
6077 /* we must uncharge all the leftover precharges from mc.to */6231 /* we must uncharge all the leftover precharges from mc.to */
6078 if (mc.precharge) {6232 if (mc.precharge) {
6079 cancel_charge(mc.to, mc.precharge);6233 mem_cgroup_cancel_charge(mc.to, mc.precharge);
6080 mc.precharge = 0;6234 mc.precharge = 0;
6081 }6235 }
6082 /*6236 /*
@@ -6084,7 +6238,7 @@ static void __mem_cgroup_clear_mc(void) · 1 unmodified line
6084 * we must uncharge here.6238 * we must uncharge here.
6085 */6239 */
6086 if (mc.moved_charge) {6240 if (mc.moved_charge) {
6087 cancel_charge(mc.from, mc.moved_charge);6241 mem_cgroup_cancel_charge(mc.from, mc.moved_charge);
6088 mc.moved_charge = 0;6242 mc.moved_charge = 0;
6089 }6243 }
6090 /* we must fixup refcnts and charges */6244 /* we must fixup refcnts and charges */
@@ -6364,6 +6518,7 @@ static void mem_cgroup_move_task(void) · 273 unmodified lines
6364 mem_cgroup_clear_mc();6518 mem_cgroup_clear_mc();
6365 }6519 }
6366}6520}
6521
6367#else /* !CONFIG_MMU */6522#else /* !CONFIG_MMU */
6368static int mem_cgroup_can_attach(struct cgroup_taskset *tset)6523static int mem_cgroup_can_attach(struct cgroup_taskset *tset)
6369{6524{
@@ -6377,8 +6532,39 @@ static void mem_cgroup_move_task(void) · 7 unmodified lines
6377}6532}
6378#endif6533#endif
6379 6534
6535#ifdef CONFIG_MEMCG_KMEM
6536static void mem_cgroup_fork(struct task_struct *task)
6537{
6538 /*
6539 * Set the update flag to cause task->objcg to be initialized lazily
6540 * on the first allocation. It can be done without any synchronization
6541 * because it's always performed on the current task, so does
6542 * current_objcg_update().
6543 */
6544 task->objcg = (struct obj_cgroup *)CURRENT_OBJCG_UPDATE_FLAG;
6545}
6546
6547static void mem_cgroup_exit(struct task_struct *task)
6548{
6549 struct obj_cgroup *objcg = task->objcg;
6550
6551 objcg = (struct obj_cgroup *)
6552 ((unsigned long)objcg & ~CURRENT_OBJCG_UPDATE_FLAG);
6553 if (objcg)
6554 obj_cgroup_put(objcg);
6555
6556 /*
6557 * Some kernel allocations can happen after this point,
6558 * but let's ignore them. It can be done without any synchronization
6559 * because it's always performed on the current task, so does
6560 * current_objcg_update().
6561 */
6562 task->objcg = NULL;
6563}
6564#endif
6565
6380#ifdef CONFIG_LRU_GEN6566#ifdef CONFIG_LRU_GEN
6381static void mem_cgroup_attach(struct cgroup_taskset *tset)6567static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset)
6382{6568{
6383 struct task_struct *task;6569 struct task_struct *task;
6384 struct cgroup_subsys_state *css;6570 struct cgroup_subsys_state *css;
@@ -6396,10 +6582,31 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) · 11 unmodified lines
6396 task_unlock(task);6582 task_unlock(task);
6397}6583}
6398#else6584#else
6585static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {}
6586#endif /* CONFIG_LRU_GEN */
6587
6588#ifdef CONFIG_MEMCG_KMEM
6589static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset)
6590{
6591 struct task_struct *task;
6592 struct cgroup_subsys_state *css;
6593
6594 cgroup_taskset_for_each(task, css, tset) {
6595 /* atomically set the update bit */
6596 set_bit(CURRENT_OBJCG_UPDATE_BIT, (unsigned long *)&task->objcg);
6597 }
6598}
6599#else
6600static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {}
6601#endif /* CONFIG_MEMCG_KMEM */
6602
6603#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM)
6399static void mem_cgroup_attach(struct cgroup_taskset *tset)6604static void mem_cgroup_attach(struct cgroup_taskset *tset)
6400{6605{
6606 mem_cgroup_lru_gen_attach(tset);
6607 mem_cgroup_kmem_attach(tset);
6401}6608}
6402#endif /* CONFIG_LRU_GEN */6609#endif
6403 6610
6404static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value)6611static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value)
6405{6612{
@@ -6622,7 +6829,8 @@ static int memory_stat_show(struct seq_file *m, void *v) · 216 unmodified lines
6622static inline unsigned long lruvec_page_state_output(struct lruvec *lruvec,6829static inline unsigned long lruvec_page_state_output(struct lruvec *lruvec,
6623 int item)6830 int item)
6624{6831{
6625 return lruvec_page_state(lruvec, item) * memcg_page_state_unit(item);6832 return lruvec_page_state(lruvec, item) *
6833 memcg_page_state_output_unit(item);
6626}6834}
6627 6835
6628static int memory_numa_stat_show(struct seq_file *m, void *v)6836static int memory_numa_stat_show(struct seq_file *m, void *v)
@@ -6808,9 +7016,15 @@ struct cgroup_subsys memory_cgrp_subsys = { · 179 unmodified lines
6808 .css_reset = mem_cgroup_css_reset,7016 .css_reset = mem_cgroup_css_reset,
6809 .css_rstat_flush = mem_cgroup_css_rstat_flush,7017 .css_rstat_flush = mem_cgroup_css_rstat_flush,
6810 .can_attach = mem_cgroup_can_attach,7018 .can_attach = mem_cgroup_can_attach,
7019#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM)
6811 .attach = mem_cgroup_attach,7020 .attach = mem_cgroup_attach,
7021#endif
6812 .cancel_attach = mem_cgroup_cancel_attach,7022 .cancel_attach = mem_cgroup_cancel_attach,
6813 .post_attach = mem_cgroup_move_task,7023 .post_attach = mem_cgroup_move_task,
7024#ifdef CONFIG_MEMCG_KMEM
7025 .fork = mem_cgroup_fork,
7026 .exit = mem_cgroup_exit,
7027#endif
6814 .dfl_cftypes = memory_files,7028 .dfl_cftypes = memory_files,
6815 .legacy_cftypes = mem_cgroup_legacy_files,7029 .legacy_cftypes = mem_cgroup_legacy_files,
6816 .early_init = 0,7030 .early_init = 0,
@@ -6990,20 +7204,13 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, · 173 unmodified lines
6990static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,7204static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
6991 gfp_t gfp)7205 gfp_t gfp)
6992{7206{
6993 long nr_pages = folio_nr_pages(folio);
6994 int ret;7207 int ret;
6995 7208
6996 ret = try_charge(memcg, gfp, nr_pages);7209 ret = try_charge(memcg, gfp, folio_nr_pages(folio));
6997 if (ret)7210 if (ret)
6998 goto out;7211 goto out;
6999 7212
7000 css_get(&memcg->css);7213 mem_cgroup_commit_charge(folio, memcg);
7001 commit_charge(folio, memcg);
7002
7003 local_irq_disable();
7004 mem_cgroup_charge_statistics(memcg, nr_pages);
7005 memcg_check_events(memcg, folio_nid(folio));
7006 local_irq_enable();
7007out:7214out:
7008 return ret;7215 return ret;
7009}7216}
@@ -7020,6 +7227,41 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) · 10 unmodified lines
7020 return ret;7227 return ret;
7021}7228}
7022 7229
7230/**
7231 * mem_cgroup_hugetlb_try_charge - try to charge the memcg for a hugetlb folio
7232 * @memcg: memcg to charge.
7233 * @gfp: reclaim mode.
7234 * @nr_pages: number of pages to charge.
7235 *
7236 * This function is called when allocating a huge page folio to determine if
7237 * the memcg has the capacity for it. It does not commit the charge yet,
7238 * as the hugetlb folio itself has not been obtained from the hugetlb pool.
7239 *
7240 * Once we have obtained the hugetlb folio, we can call
7241 * mem_cgroup_commit_charge() to commit the charge. If we fail to obtain the
7242 * folio, we should instead call mem_cgroup_cancel_charge() to undo the effect
7243 * of try_charge().
7244 *
7245 * Returns 0 on success. Otherwise, an error code is returned.
7246 */
7247int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp,
7248 long nr_pages)
7249{
7250 /*
7251 * If hugetlb memcg charging is not enabled, do not fail hugetlb allocation,
7252 * but do not attempt to commit charge later (or cancel on error) either.
7253 */
7254 if (mem_cgroup_disabled() || !memcg ||
7255 !cgroup_subsys_on_dfl(memory_cgrp_subsys) ||
7256 !(cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING))
7257 return -EOPNOTSUPP;
7258
7259 if (try_charge(memcg, gfp, nr_pages))
7260 return -ENOMEM;
7261
7262 return 0;
7263}
7264
7023/**7265/**
7024 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.7266 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
7025 * @folio: folio to charge.7267 * @folio: folio to charge.
@@ -7216,16 +7458,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) · 190 unmodified lines
7216}7458}
7217 7459
7218/**7460/**
7219 * mem_cgroup_migrate - Charge a folio's replacement.7461 * mem_cgroup_replace_folio - Charge a folio's replacement.
7220 * @old: Currently circulating folio.7462 * @old: Currently circulating folio.
7221 * @new: Replacement folio.7463 * @new: Replacement folio.
7222 *7464 *
7223 * Charge @new as a replacement folio for @old. @old will7465 * Charge @new as a replacement folio for @old. @old will
7224 * be uncharged upon free.7466 * be uncharged upon free. This is only used by the page cache
7467 * (in replace_page_cache_folio()).
7225 *7468 *
7226 * Both folios must be locked, @new->mapping must be set up.7469 * Both folios must be locked, @new->mapping must be set up.
7227 */7470 */
7228void mem_cgroup_migrate(struct folio *old, struct folio *new)7471void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
7229{7472{
7230 struct mem_cgroup *memcg;7473 struct mem_cgroup *memcg;
7231 long nr_pages = folio_nr_pages(new);7474 long nr_pages = folio_nr_pages(new);
@@ -7264,6 +7507,44 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) · 32 unmodified lines
7264 local_irq_restore(flags);7507 local_irq_restore(flags);
7265}7508}
7266 7509
7510/**
7511 * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
7512 * @old: Currently circulating folio.
7513 * @new: Replacement folio.
7514 *
7515 * Transfer the memcg data from the old folio to the new folio for migration.
7516 * The old folio's data info will be cleared. Note that the memory counters
7517 * will remain unchanged throughout the process.
7518 *
7519 * Both folios must be locked, @new->mapping must be set up.
7520 */
7521void mem_cgroup_migrate(struct folio *old, struct folio *new)
7522{
7523 struct mem_cgroup *memcg;
7524
7525 VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
7526 VM_BUG_ON_FOLIO(!folio_test_locked(new), new);
7527 VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new);
7528 VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new);
7529
7530 if (mem_cgroup_disabled())
7531 return;
7532
7533 memcg = folio_memcg(old);
7534 /*
7535 * Note that it is normal to see !memcg for a hugetlb folio.
7536 * For e.g, itt could have been allocated when memory_hugetlb_accounting
7537 * was not selected.
7538 */
7539 VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old);
7540 if (!memcg)
7541 return;
7542
7543 /* Transfer the charge and the css ref */
7544 commit_charge(new, memcg);
7545 old->memcg_data = 0;
7546}
7547
7267DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key);7548DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key);
7268EXPORT_SYMBOL(memcg_sockets_enabled_key);7549EXPORT_SYMBOL(memcg_sockets_enabled_key);
7269 7550
@@ -7763,7 +8044,7 @@ static struct cftype memsw_files[] = { · 493 unmodified lines
7763 *8044 *
7764 * This doesn't check for specific headroom, and it is not atomic8045 * This doesn't check for specific headroom, and it is not atomic
7765 * either. But with zswap, the size of the allocation is only known8046 * either. But with zswap, the size of the allocation is only known
7766 * once compression has occured, and this optimistic pre-check avoids8047 * once compression has occurred, and this optimistic pre-check avoids
7767 * spending cycles on compression when there is already no room left8048 * spending cycles on compression when there is already no room left
7768 * or zswap is disabled altogether somewhere in the hierarchy.8049 * or zswap is disabled altogether somewhere in the hierarchy.
7769 */8050 */
mm/memfd.c+0−6
@@ -315,12 +315,6 @@ SYSCALL_DEFINE2(memfd_create, · 314 unmodified lines
315 if ((flags & MFD_EXEC) && (flags & MFD_NOEXEC_SEAL))315 if ((flags & MFD_EXEC) && (flags & MFD_NOEXEC_SEAL))
316 return -EINVAL;316 return -EINVAL;
317 317
318 if (!(flags & (MFD_EXEC | MFD_NOEXEC_SEAL))) {
319 pr_warn_once(
320 "%s[%d]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set\n",
321 current->comm, task_pid_nr(current));
322 }
323
324 error = check_sysctl_memfd_noexec(&flags);318 error = check_sysctl_memfd_noexec(&flags);
325 if (error < 0)319 if (error < 0)
326 return error;320 return error;
mm/memory-failure.c+16−13
@@ -1713,20 +1713,23 @@ static void unmap_and_kill(struct list_head *to_kill, unsigned long pfn, · 1712 unmodified lines
1713 kill_procs(to_kill, flags & MF_MUST_KILL, false, pfn, flags);1713 kill_procs(to_kill, flags & MF_MUST_KILL, false, pfn, flags);
1714}1714}
1715 1715
1716/*
1717 * Only dev_pagemap pages get here, such as fsdax when the filesystem
1718 * either do not claim or fails to claim a hwpoison event, or devdax.
1719 * The fsdax pages are initialized per base page, and the devdax pages
1720 * could be initialized either as base pages, or as compound pages with
1721 * vmemmap optimization enabled. Devdax is simplistic in its dealing with
1722 * hwpoison, such that, if a subpage of a compound page is poisoned,
1723 * simply mark the compound head page is by far sufficient.
1724 */
1716static int mf_generic_kill_procs(unsigned long long pfn, int flags,1725static int mf_generic_kill_procs(unsigned long long pfn, int flags,
1717 struct dev_pagemap *pgmap)1726 struct dev_pagemap *pgmap)
1718{1727{
1719 struct page *page = pfn_to_page(pfn);1728 struct folio *folio = pfn_folio(pfn);
1720 LIST_HEAD(to_kill);1729 LIST_HEAD(to_kill);
1721 dax_entry_t cookie;1730 dax_entry_t cookie;
1722 int rc = 0;1731 int rc = 0;
1723 1732
1724 /*
1725 * Pages instantiated by device-dax (not filesystem-dax)
1726 * may be compound pages.
1727 */
1728 page = compound_head(page);
1729
1730 /*1733 /*
1731 * Prevent the inode from being freed while we are interrogating1734 * Prevent the inode from being freed while we are interrogating
1732 * the address_space, typically this would be handled by1735 * the address_space, typically this would be handled by
@@ -1734,11 +1737,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 1 unmodified line
1734 * also prevents changes to the mapping of this pfn until1737 * also prevents changes to the mapping of this pfn until
1735 * poison signaling is complete.1738 * poison signaling is complete.
1736 */1739 */
1737 cookie = dax_lock_page(page);1740 cookie = dax_lock_folio(folio);
1738 if (!cookie)1741 if (!cookie)
1739 return -EBUSY;1742 return -EBUSY;
1740 1743
1741 if (hwpoison_filter(page)) {1744 if (hwpoison_filter(&folio->page)) {
1742 rc = -EOPNOTSUPP;1745 rc = -EOPNOTSUPP;
1743 goto unlock;1746 goto unlock;
1744 }1747 }
@@ -1760,7 +1763,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 15 unmodified lines
1760 * Use this flag as an indication that the dax page has been1763 * Use this flag as an indication that the dax page has been
1761 * remapped UC to prevent speculative consumption of poison.1764 * remapped UC to prevent speculative consumption of poison.
1762 */1765 */
1763 SetPageHWPoison(page);1766 SetPageHWPoison(&folio->page);
1764 1767
1765 /*1768 /*
1766 * Unlike System-RAM there is no possibility to swap in a1769 * Unlike System-RAM there is no possibility to swap in a
@@ -1769,11 +1772,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 2 unmodified lines
1769 * SIGBUS (i.e. MF_MUST_KILL)1772 * SIGBUS (i.e. MF_MUST_KILL)
1770 */1773 */
1771 flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;1774 flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
1772 collect_procs(page, &to_kill, true);1775 collect_procs(&folio->page, &to_kill, true);
1773 1776
1774 unmap_and_kill(&to_kill, pfn, page->mapping, page->index, flags);1777 unmap_and_kill(&to_kill, pfn, folio->mapping, folio->index, flags);
1775unlock:1778unlock:
1776 dax_unlock_page(page, cookie);1779 dax_unlock_folio(folio, cookie);
1777 return rc;1780 return rc;
1778}1781}
1779 1782
mm/memory-tiers.c+168−7
@@ -5,6 +5,7 @@ · 4 unmodified lines
5#include <linux/kobject.h>5#include <linux/kobject.h>
6#include <linux/memory.h>6#include <linux/memory.h>
7#include <linux/memory-tiers.h>7#include <linux/memory-tiers.h>
8#include <linux/notifier.h>
8 9
9#include "internal.h"10#include "internal.h"
10 11
@@ -36,7 +37,7 @@ struct node_memory_type_map { · 25 unmodified lines
36static DEFINE_MUTEX(memory_tier_lock);37static DEFINE_MUTEX(memory_tier_lock);
37static LIST_HEAD(memory_tiers);38static LIST_HEAD(memory_tiers);
38static struct node_memory_type_map node_memory_types[MAX_NUMNODES];39static struct node_memory_type_map node_memory_types[MAX_NUMNODES];
39static struct memory_dev_type *default_dram_type;40struct memory_dev_type *default_dram_type;
40 41
41static struct bus_type memory_tier_subsys = {42static struct bus_type memory_tier_subsys = {
42 .name = "memory_tiering",43 .name = "memory_tiering",
@@ -105,6 +106,13 @@ static int top_tier_adistance; · 62 unmodified lines
105static struct demotion_nodes *node_demotion __read_mostly;106static struct demotion_nodes *node_demotion __read_mostly;
106#endif /* CONFIG_MIGRATION */107#endif /* CONFIG_MIGRATION */
107 108
109static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms);
110
111static bool default_dram_perf_error;
112static struct node_hmem_attrs default_dram_perf;
113static int default_dram_perf_ref_nid = NUMA_NO_NODE;
114static const char *default_dram_perf_ref_source;
115
108static inline struct memory_tier *to_memory_tier(struct device *device)116static inline struct memory_tier *to_memory_tier(struct device *device)
109{117{
110 return container_of(device, struct memory_tier, dev);118 return container_of(device, struct memory_tier, dev);
@@ -115,7 +123,7 @@ static __always_inline nodemask_t get_memtier_nodemask(struct memory_tier *memti · 4 unmodified lines
115 nodemask_t nodes = NODE_MASK_NONE;123 nodemask_t nodes = NODE_MASK_NONE;
116 struct memory_dev_type *memtype;124 struct memory_dev_type *memtype;
117 125
118 list_for_each_entry(memtype, &memtier->memory_types, tier_sibiling)126 list_for_each_entry(memtype, &memtier->memory_types, tier_sibling)
119 nodes_or(nodes, nodes, memtype->nodes);127 nodes_or(nodes, nodes, memtype->nodes);
120 128
121 return nodes;129 return nodes;
@@ -174,7 +182,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 52 unmodified lines
174 * If the memtype is already part of a memory tier,182 * If the memtype is already part of a memory tier,
175 * just return that.183 * just return that.
176 */184 */
177 if (!list_empty(&memtype->tier_sibiling)) {185 if (!list_empty(&memtype->tier_sibling)) {
178 list_for_each_entry(memtier, &memory_tiers, list) {186 list_for_each_entry(memtier, &memory_tiers, list) {
179 if (adistance == memtier->adistance_start)187 if (adistance == memtier->adistance_start)
180 return memtier;188 return memtier;
@@ -218,7 +226,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 37 unmodified lines
218 memtier = new_memtier;226 memtier = new_memtier;
219 227
220link_memtype:228link_memtype:
221 list_add(&memtype->tier_sibiling, &memtier->memory_types);229 list_add(&memtype->tier_sibling, &memtier->memory_types);
222 return memtier;230 return memtier;
223}231}
224 232
@@ -527,7 +535,7 @@ static bool clear_node_memory_tier(int node) · 302 unmodified lines
527 memtype = node_memory_types[node].memtype;535 memtype = node_memory_types[node].memtype;
528 node_clear(node, memtype->nodes);536 node_clear(node, memtype->nodes);
529 if (nodes_empty(memtype->nodes)) {537 if (nodes_empty(memtype->nodes)) {
530 list_del_init(&memtype->tier_sibiling);538 list_del_init(&memtype->tier_sibling);
531 if (list_empty(&memtier->memory_types))539 if (list_empty(&memtier->memory_types))
532 destroy_memory_tier(memtier);540 destroy_memory_tier(memtier);
533 }541 }
@@ -553,7 +561,7 @@ struct memory_dev_type *alloc_memory_type(int adistance) · 19 unmodified lines
553 return ERR_PTR(-ENOMEM);561 return ERR_PTR(-ENOMEM);
554 562
555 memtype->adistance = adistance;563 memtype->adistance = adistance;
556 INIT_LIST_HEAD(&memtype->tier_sibiling);564 INIT_LIST_HEAD(&memtype->tier_sibling);
557 memtype->nodes = NODE_MASK_NONE;565 memtype->nodes = NODE_MASK_NONE;
558 kref_init(&memtype->kref);566 kref_init(&memtype->kref);
559 return memtype;567 return memtype;
@@ -578,13 +586,14 @@ EXPORT_SYMBOL_GPL(init_node_memory_type); · 18 unmodified lines
578void clear_node_memory_type(int node, struct memory_dev_type *memtype)586void clear_node_memory_type(int node, struct memory_dev_type *memtype)
579{587{
580 mutex_lock(&memory_tier_lock);588 mutex_lock(&memory_tier_lock);
581 if (node_memory_types[node].memtype == memtype)589 if (node_memory_types[node].memtype == memtype || !memtype)
582 node_memory_types[node].map_count--;590 node_memory_types[node].map_count--;
583 /*591 /*
584 * If we umapped all the attached devices to this node,592 * If we umapped all the attached devices to this node,
585 * clear the node memory type.593 * clear the node memory type.
586 */594 */
587 if (!node_memory_types[node].map_count) {595 if (!node_memory_types[node].map_count) {
596 memtype = node_memory_types[node].memtype;
588 node_memory_types[node].memtype = NULL;597 node_memory_types[node].memtype = NULL;
589 put_memory_type(memtype);598 put_memory_type(memtype);
590 }599 }
@@ -592,6 +601,158 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) · 1 unmodified line
592}601}
593EXPORT_SYMBOL_GPL(clear_node_memory_type);602EXPORT_SYMBOL_GPL(clear_node_memory_type);
594 603
604static void dump_hmem_attrs(struct node_hmem_attrs *attrs, const char *prefix)
605{
606 pr_info(
607"%sread_latency: %u, write_latency: %u, read_bandwidth: %u, write_bandwidth: %u\n",
608 prefix, attrs->read_latency, attrs->write_latency,
609 attrs->read_bandwidth, attrs->write_bandwidth);
610}
611
612int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf,
613 const char *source)
614{
615 int rc = 0;
616
617 mutex_lock(&memory_tier_lock);
618 if (default_dram_perf_error) {
619 rc = -EIO;
620 goto out;
621 }
622
623 if (perf->read_latency + perf->write_latency == 0 ||
624 perf->read_bandwidth + perf->write_bandwidth == 0) {
625 rc = -EINVAL;
626 goto out;
627 }
628
629 if (default_dram_perf_ref_nid == NUMA_NO_NODE) {
630 default_dram_perf = *perf;
631 default_dram_perf_ref_nid = nid;
632 default_dram_perf_ref_source = kstrdup(source, GFP_KERNEL);
633 goto out;
634 }
635
636 /*
637 * The performance of all default DRAM nodes is expected to be
638 * same (that is, the variation is less than 10%). And it
639 * will be used as base to calculate the abstract distance of
640 * other memory nodes.
641 */
642 if (abs(perf->read_latency - default_dram_perf.read_latency) * 10 >
643 default_dram_perf.read_latency ||
644 abs(perf->write_latency - default_dram_perf.write_latency) * 10 >
645 default_dram_perf.write_latency ||
646 abs(perf->read_bandwidth - default_dram_perf.read_bandwidth) * 10 >
647 default_dram_perf.read_bandwidth ||
648 abs(perf->write_bandwidth - default_dram_perf.write_bandwidth) * 10 >
649 default_dram_perf.write_bandwidth) {
650 pr_info(
651"memory-tiers: the performance of DRAM node %d mismatches that of the reference\n"
652"DRAM node %d.\n", nid, default_dram_perf_ref_nid);
653 pr_info(" performance of reference DRAM node %d:\n",
654 default_dram_perf_ref_nid);
655 dump_hmem_attrs(&default_dram_perf, " ");
656 pr_info(" performance of DRAM node %d:\n", nid);
657 dump_hmem_attrs(perf, " ");
658 pr_info(
659" disable default DRAM node performance based abstract distance algorithm.\n");
660 default_dram_perf_error = true;
661 rc = -EINVAL;
662 }
663
664out:
665 mutex_unlock(&memory_tier_lock);
666 return rc;
667}
668
669int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist)
670{
671 if (default_dram_perf_error)
672 return -EIO;
673
674 if (default_dram_perf_ref_nid == NUMA_NO_NODE)
675 return -ENOENT;
676
677 if (perf->read_latency + perf->write_latency == 0 ||
678 perf->read_bandwidth + perf->write_bandwidth == 0)
679 return -EINVAL;
680
681 mutex_lock(&memory_tier_lock);
682 /*
683 * The abstract distance of a memory node is in direct proportion to
684 * its memory latency (read + write) and inversely proportional to its
685 * memory bandwidth (read + write). The abstract distance, memory
686 * latency, and memory bandwidth of the default DRAM nodes are used as
687 * the base.
688 */
689 *adist = MEMTIER_ADISTANCE_DRAM *
690 (perf->read_latency + perf->write_latency) /
691 (default_dram_perf.read_latency + default_dram_perf.write_latency) *
692 (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) /
693 (perf->read_bandwidth + perf->write_bandwidth);
694 mutex_unlock(&memory_tier_lock);
695
696 return 0;
697}
698EXPORT_SYMBOL_GPL(mt_perf_to_adistance);
699
700/**
701 * register_mt_adistance_algorithm() - Register memory tiering abstract distance algorithm
702 * @nb: The notifier block which describe the algorithm
703 *
704 * Return: 0 on success, errno on error.
705 *
706 * Every memory tiering abstract distance algorithm provider needs to
707 * register the algorithm with register_mt_adistance_algorithm(). To
708 * calculate the abstract distance for a specified memory node, the
709 * notifier function will be called unless some high priority
710 * algorithm has provided result. The prototype of the notifier
711 * function is as follows,
712 *
713 * int (*algorithm_notifier)(struct notifier_block *nb,
714 * unsigned long nid, void *data);
715 *
716 * Where "nid" specifies the memory node, "data" is the pointer to the
717 * returned abstract distance (that is, "int *adist"). If the
718 * algorithm provides the result, NOTIFY_STOP should be returned.
719 * Otherwise, return_value & %NOTIFY_STOP_MASK == 0 to allow the next
720 * algorithm in the chain to provide the result.
721 */
722int register_mt_adistance_algorithm(struct notifier_block *nb)
723{
724 return blocking_notifier_chain_register(&mt_adistance_algorithms, nb);
725}
726EXPORT_SYMBOL_GPL(register_mt_adistance_algorithm);
727
728/**
729 * unregister_mt_adistance_algorithm() - Unregister memory tiering abstract distance algorithm
730 * @nb: the notifier block which describe the algorithm
731 *
732 * Return: 0 on success, errno on error.
733 */
734int unregister_mt_adistance_algorithm(struct notifier_block *nb)
735{
736 return blocking_notifier_chain_unregister(&mt_adistance_algorithms, nb);
737}
738EXPORT_SYMBOL_GPL(unregister_mt_adistance_algorithm);
739
740/**
741 * mt_calc_adistance() - Calculate abstract distance with registered algorithms
742 * @node: the node to calculate abstract distance for
743 * @adist: the returned abstract distance
744 *
745 * Return: if return_value & %NOTIFY_STOP_MASK != 0, then some
746 * abstract distance algorithm provides the result, and return it via
747 * @adist. Otherwise, no algorithm can provide the result and @adist
748 * will be kept as it is.
749 */
750int mt_calc_adistance(int node, int *adist)
751{
752 return blocking_notifier_call_chain(&mt_adistance_algorithms, node, adist);
753}
754EXPORT_SYMBOL_GPL(mt_calc_adistance);
755
595static int __meminit memtier_hotplug_callback(struct notifier_block *self,756static int __meminit memtier_hotplug_callback(struct notifier_block *self,
596 unsigned long action, void *_arg)757 unsigned long action, void *_arg)
597{758{
mm/memory.c+179−123
@@ -1,3 +1,4 @@
1
1// SPDX-License-Identifier: GPL-2.0-only2// SPDX-License-Identifier: GPL-2.0-only
2/*3/*
3 * linux/mm/memory.c4 * linux/mm/memory.c
@@ -471,8 +472,6 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) · 467 unmodified lines
471{472{
472 int i;473 int i;
473 474
474 if (current->mm == mm)
475 sync_mm_rss(mm);
476 for (i = 0; i < NR_MM_COUNTERS; i++)475 for (i = 0; i < NR_MM_COUNTERS; i++)
477 if (rss[i])476 if (rss[i])
478 add_mm_counter(mm, i, rss[i]);477 add_mm_counter(mm, i, rss[i]);
@@ -691,6 +690,16 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, · 212 unmodified lines
691out:690out:
692 return pfn_to_page(pfn);691 return pfn_to_page(pfn);
693}692}
693
694struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
695 unsigned long addr, pmd_t pmd)
696{
697 struct page *page = vm_normal_page_pmd(vma, addr, pmd);
698
699 if (page)
700 return page_folio(page);
701 return NULL;
702}
694#endif703#endif
695 704
696static void restore_exclusive_pte(struct vm_area_struct *vma,705static void restore_exclusive_pte(struct vm_area_struct *vma,
@@ -3006,23 +3015,24 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · 2309 unmodified lines
3006 * case, all we need to do here is to mark the page as writable and update3015 * case, all we need to do here is to mark the page as writable and update
3007 * any related book-keeping.3016 * any related book-keeping.
3008 */3017 */
3009static inline void wp_page_reuse(struct vm_fault *vmf)3018static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio)
3010 __releases(vmf->ptl)3019 __releases(vmf->ptl)
3011{3020{
3012 struct vm_area_struct *vma = vmf->vma;3021 struct vm_area_struct *vma = vmf->vma;
3013 struct page *page = vmf->page;
3014 pte_t entry;3022 pte_t entry;
3015 3023
3016 VM_BUG_ON(!(vmf->flags & FAULT_FLAG_WRITE));3024 VM_BUG_ON(!(vmf->flags & FAULT_FLAG_WRITE));
3017 VM_BUG_ON(page && PageAnon(page) && !PageAnonExclusive(page));
3018 3025
3019 /*3026 if (folio) {
3020 * Clear the pages cpupid information as the existing3027 VM_BUG_ON(folio_test_anon(folio) &&
3021 * information potentially belongs to a now completely3028 !PageAnonExclusive(vmf->page));
3022 * unrelated process.3029 /*
3023 */3030 * Clear the folio's cpupid information as the existing
3024 if (page)3031 * information potentially belongs to a now completely
3025 page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);3032 * unrelated process.
3033 */
3034 folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1);
3035 }
3026 3036
3027 flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));3037 flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
3028 entry = pte_mkyoung(vmf->orig_pte);3038 entry = pte_mkyoung(vmf->orig_pte);
@@ -3033,6 +3043,36 @@ static inline void wp_page_reuse(struct vm_fault *vmf) · 4 unmodified lines
3033 count_vm_event(PGREUSE);3043 count_vm_event(PGREUSE);
3034}3044}
3035 3045
3046/*
3047 * We could add a bitflag somewhere, but for now, we know that all
3048 * vm_ops that have a ->map_pages have been audited and don't need
3049 * the mmap_lock to be held.
3050 */
3051static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf)
3052{
3053 struct vm_area_struct *vma = vmf->vma;
3054
3055 if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK))
3056 return 0;
3057 vma_end_read(vma);
3058 return VM_FAULT_RETRY;
3059}
3060
3061static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf)
3062{
3063 struct vm_area_struct *vma = vmf->vma;
3064
3065 if (likely(vma->anon_vma))
3066 return 0;
3067 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {
3068 vma_end_read(vma);
3069 return VM_FAULT_RETRY;
3070 }
3071 if (__anon_vma_prepare(vma))
3072 return VM_FAULT_OOM;
3073 return 0;
3074}
3075
3036/*3076/*
3037 * Handle the case of a page which we actually need to copy to a new page,3077 * Handle the case of a page which we actually need to copy to a new page,
3038 * either due to COW or unsharing.3078 * either due to COW or unsharing.
@@ -3060,27 +3100,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 21 unmodified lines
3060 pte_t entry;3100 pte_t entry;
3061 int page_copied = 0;3101 int page_copied = 0;
3062 struct mmu_notifier_range range;3102 struct mmu_notifier_range range;
3063 int ret;3103 vm_fault_t ret;
3064 3104
3065 delayacct_wpcopy_start();3105 delayacct_wpcopy_start();
3066 3106
3067 if (vmf->page)3107 if (vmf->page)
3068 old_folio = page_folio(vmf->page);3108 old_folio = page_folio(vmf->page);
3069 if (unlikely(anon_vma_prepare(vma)))3109 ret = vmf_anon_prepare(vmf);
3070 goto oom;3110 if (unlikely(ret))
3111 goto out;
3071 3112
3072 if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {3113 if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
3073 new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);3114 new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
3074 if (!new_folio)3115 if (!new_folio)
3075 goto oom;3116 goto oom;
3076 } else {3117 } else {
3118 int err;
3077 new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma,3119 new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma,
3078 vmf->address, false);3120 vmf->address, false);
3079 if (!new_folio)3121 if (!new_folio)
3080 goto oom;3122 goto oom;
3081 3123
3082 ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf);3124 err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf);
3083 if (ret) {3125 if (err) {
3084 /*3126 /*
3085 * COW failed, if the fault was solved by other,3127 * COW failed, if the fault was solved by other,
3086 * it's fine. If not, userspace would re-fault on3128 * it's fine. If not, userspace would re-fault on
@@ -3093,7 +3135,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 6 unmodified lines
3093 folio_put(old_folio);3135 folio_put(old_folio);
3094 3136
3095 delayacct_wpcopy_end();3137 delayacct_wpcopy_end();
3096 return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0;3138 return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0;
3097 }3139 }
3098 kmsan_copy_page_meta(&new_folio->page, vmf->page);3140 kmsan_copy_page_meta(&new_folio->page, vmf->page);
3099 }3141 }
@@ -3203,11 +3245,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 103 unmodified lines
3203oom_free_new:3245oom_free_new:
3204 folio_put(new_folio);3246 folio_put(new_folio);
3205oom:3247oom:
3248 ret = VM_FAULT_OOM;
3249out:
3206 if (old_folio)3250 if (old_folio)
3207 folio_put(old_folio);3251 folio_put(old_folio);
3208 3252
3209 delayacct_wpcopy_end();3253 delayacct_wpcopy_end();
3210 return VM_FAULT_OOM;3254 return ret;
3211}3255}
3212 3256
3213/**3257/**
@@ -3215,6 +3259,7 @@ oom: · 1 unmodified line
3215 * writeable once the page is prepared3259 * writeable once the page is prepared
3216 *3260 *
3217 * @vmf: structure describing the fault3261 * @vmf: structure describing the fault
3262 * @folio: the folio of vmf->page
3218 *3263 *
3219 * This function handles all that is needed to finish a write page fault in a3264 * This function handles all that is needed to finish a write page fault in a
3220 * shared mapping due to PTE being read-only once the mapped page is prepared.3265 * shared mapping due to PTE being read-only once the mapped page is prepared.
@@ -3226,7 +3271,7 @@ oom: · 5 unmodified lines
3226 * Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before3271 * Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before
3227 * we acquired PTE lock.3272 * we acquired PTE lock.
3228 */3273 */
3229vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf)3274static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, struct folio *folio)
3230{3275{
3231 WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));3276 WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
3232 vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,3277 vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
@@ -3242,7 +3287,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) · 9 unmodified lines
3242 pte_unmap_unlock(vmf->pte, vmf->ptl);3287 pte_unmap_unlock(vmf->pte, vmf->ptl);
3243 return VM_FAULT_NOPAGE;3288 return VM_FAULT_NOPAGE;
3244 }3289 }
3245 wp_page_reuse(vmf);3290 wp_page_reuse(vmf, folio);
3246 return 0;3291 return 0;
3247}3292}
3248 3293
@@ -3258,18 +3303,17 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) · 9 unmodified lines
3258 vm_fault_t ret;3303 vm_fault_t ret;
3259 3304
3260 pte_unmap_unlock(vmf->pte, vmf->ptl);3305 pte_unmap_unlock(vmf->pte, vmf->ptl);
3261 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {3306 ret = vmf_can_call_fault(vmf);
3262 vma_end_read(vmf->vma);3307 if (ret)
3263 return VM_FAULT_RETRY;3308 return ret;
3264 }
3265 3309
3266 vmf->flags |= FAULT_FLAG_MKWRITE;3310 vmf->flags |= FAULT_FLAG_MKWRITE;
3267 ret = vma->vm_ops->pfn_mkwrite(vmf);3311 ret = vma->vm_ops->pfn_mkwrite(vmf);
3268 if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))3312 if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))
3269 return ret;3313 return ret;
3270 return finish_mkwrite_fault(vmf);3314 return finish_mkwrite_fault(vmf, NULL);
3271 }3315 }
3272 wp_page_reuse(vmf);3316 wp_page_reuse(vmf, NULL);
3273 return 0;3317 return 0;
3274}3318}
3275 3319
@@ -3285,10 +3329,10 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 9 unmodified lines
3285 vm_fault_t tmp;3329 vm_fault_t tmp;
3286 3330
3287 pte_unmap_unlock(vmf->pte, vmf->ptl);3331 pte_unmap_unlock(vmf->pte, vmf->ptl);
3288 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {3332 tmp = vmf_can_call_fault(vmf);
3333 if (tmp) {
3289 folio_put(folio);3334 folio_put(folio);
3290 vma_end_read(vmf->vma);3335 return tmp;
3291 return VM_FAULT_RETRY;
3292 }3336 }
3293 3337
3294 tmp = do_page_mkwrite(vmf, folio);3338 tmp = do_page_mkwrite(vmf, folio);
@@ -3297,14 +3341,14 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines
3297 folio_put(folio);3341 folio_put(folio);
3298 return tmp;3342 return tmp;
3299 }3343 }
3300 tmp = finish_mkwrite_fault(vmf);3344 tmp = finish_mkwrite_fault(vmf, folio);
3301 if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {3345 if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {
3302 folio_unlock(folio);3346 folio_unlock(folio);
3303 folio_put(folio);3347 folio_put(folio);
3304 return tmp;3348 return tmp;
3305 }3349 }
3306 } else {3350 } else {
3307 wp_page_reuse(vmf);3351 wp_page_reuse(vmf, folio);
3308 folio_lock(folio);3352 folio_lock(folio);
3309 }3353 }
3310 ret |= fault_dirty_shared_page(vmf);3354 ret |= fault_dirty_shared_page(vmf);
@@ -3313,6 +3357,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines
3313 return ret;3357 return ret;
3314}3358}
3315 3359
3360static bool wp_can_reuse_anon_folio(struct folio *folio,
3361 struct vm_area_struct *vma)
3362{
3363 /*
3364 * We have to verify under folio lock: these early checks are
3365 * just an optimization to avoid locking the folio and freeing
3366 * the swapcache if there is little hope that we can reuse.
3367 *
3368 * KSM doesn't necessarily raise the folio refcount.
3369 */
3370 if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
3371 return false;
3372 if (!folio_test_lru(folio))
3373 /*
3374 * We cannot easily detect+handle references from
3375 * remote LRU caches or references to LRU folios.
3376 */
3377 lru_add_drain();
3378 if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
3379 return false;
3380 if (!folio_trylock(folio))
3381 return false;
3382 if (folio_test_swapcache(folio))
3383 folio_free_swap(folio);
3384 if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) {
3385 folio_unlock(folio);
3386 return false;
3387 }
3388 /*
3389 * Ok, we've got the only folio reference from our mapping
3390 * and the folio is locked, it's dark out, and we're wearing
3391 * sunglasses. Hit it.
3392 */
3393 folio_move_anon_rmap(folio, vma);
3394 folio_unlock(folio);
3395 return true;
3396}
3397
3316/*3398/*
3317 * This routine handles present pages, when3399 * This routine handles present pages, when
3318 * * users try to write to a shared page (FAULT_FLAG_WRITE)3400 * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -3341,11 +3423,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 22 unmodified lines
3341 const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;3423 const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
3342 struct vm_area_struct *vma = vmf->vma;3424 struct vm_area_struct *vma = vmf->vma;
3343 struct folio *folio = NULL;3425 struct folio *folio = NULL;
3426 pte_t pte;
3344 3427
3345 if (likely(!unshare)) {3428 if (likely(!unshare)) {
3346 if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) {3429 if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) {
3347 pte_unmap_unlock(vmf->pte, vmf->ptl);3430 if (!userfaultfd_wp_async(vma)) {
3348 return handle_userfault(vmf, VM_UFFD_WP);3431 pte_unmap_unlock(vmf->pte, vmf->ptl);
3432 return handle_userfault(vmf, VM_UFFD_WP);
3433 }
3434
3435 /*
3436 * Nothing needed (cache flush, TLB invalidations,
3437 * etc.) because we're only removing the uffd-wp bit,
3438 * which is completely invisible to the user.
3439 */
3440 pte = pte_clear_uffd_wp(ptep_get(vmf->pte));
3441
3442 set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
3443 /*
3444 * Update this to be prepared for following up CoW
3445 * handling
3446 */
3447 vmf->orig_pte = pte;
3349 }3448 }
3350 3449
3351 /*3450 /*
@@ -3382,62 +3481,21 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 30 unmodified lines
3382 /*3481 /*
3383 * Private mapping: create an exclusive anonymous page copy if reuse3482 * Private mapping: create an exclusive anonymous page copy if reuse
3384 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.3483 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.
3484 *
3485 * If we encounter a page that is marked exclusive, we must reuse
3486 * the page without further checks.
3385 */3487 */
3386 if (folio && folio_test_anon(folio)) {3488 if (folio && folio_test_anon(folio) &&
3387 /*3489 (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
3388 * If the page is exclusive to this process we must reuse the3490 if (!PageAnonExclusive(vmf->page))
3389 * page without further checks.3491 SetPageAnonExclusive(vmf->page);
3390 */
3391 if (PageAnonExclusive(vmf->page))
3392 goto reuse;
3393
3394 /*
3395 * We have to verify under folio lock: these early checks are
3396 * just an optimization to avoid locking the folio and freeing
3397 * the swapcache if there is little hope that we can reuse.
3398 *
3399 * KSM doesn't necessarily raise the folio refcount.
3400 */
3401 if (folio_test_ksm(folio) || folio_ref_count(folio) > 3)
3402 goto copy;
3403 if (!folio_test_lru(folio))
3404 /*
3405 * We cannot easily detect+handle references from
3406 * remote LRU caches or references to LRU folios.
3407 */
3408 lru_add_drain();
3409 if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio))
3410 goto copy;
3411 if (!folio_trylock(folio))
3412 goto copy;
3413 if (folio_test_swapcache(folio))
3414 folio_free_swap(folio);
3415 if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) {
3416 folio_unlock(folio);
3417 goto copy;
3418 }
3419 /*
3420 * Ok, we've got the only folio reference from our mapping
3421 * and the folio is locked, it's dark out, and we're wearing
3422 * sunglasses. Hit it.
3423 */
3424 page_move_anon_rmap(vmf->page, vma);
3425 folio_unlock(folio);
3426reuse:
3427 if (unlikely(unshare)) {3492 if (unlikely(unshare)) {
3428 pte_unmap_unlock(vmf->pte, vmf->ptl);3493 pte_unmap_unlock(vmf->pte, vmf->ptl);
3429 return 0;3494 return 0;
3430 }3495 }
3431 wp_page_reuse(vmf);3496 wp_page_reuse(vmf, folio);
3432 return 0;3497 return 0;
3433 }3498 }
3434copy:
3435 if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) {
3436 pte_unmap_unlock(vmf->pte, vmf->ptl);
3437 vma_end_read(vmf->vma);
3438 return VM_FAULT_RETRY;
3439 }
3440
3441 /*3499 /*
3442 * Ok, we need to copy. Oh, well..3500 * Ok, we need to copy. Oh, well..
3443 */3501 */
@@ -4563,10 +4621,9 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) · 1119 unmodified lines
4563 return ret;4621 return ret;
4564 }4622 }
4565 4623
4566 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {4624 ret = vmf_can_call_fault(vmf);
4567 vma_end_read(vmf->vma);4625 if (ret)
4568 return VM_FAULT_RETRY;4626 return ret;
4569 }
4570 4627
4571 ret = __do_fault(vmf);4628 ret = __do_fault(vmf);
4572 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))4629 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
@@ -4585,13 +4642,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) · 12 unmodified lines
4585 struct vm_area_struct *vma = vmf->vma;4642 struct vm_area_struct *vma = vmf->vma;
4586 vm_fault_t ret;4643 vm_fault_t ret;
4587 4644
4588 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {4645 ret = vmf_can_call_fault(vmf);
4589 vma_end_read(vma);4646 if (!ret)
4590 return VM_FAULT_RETRY;4647 ret = vmf_anon_prepare(vmf);
4591 }4648 if (ret)
4592 4649 return ret;
4593 if (unlikely(anon_vma_prepare(vma)))
4594 return VM_FAULT_OOM;
4595 4650
4596 vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);4651 vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
4597 if (!vmf->cow_page)4652 if (!vmf->cow_page)
@@ -4630,10 +4685,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) · 32 unmodified lines
4630 vm_fault_t ret, tmp;4685 vm_fault_t ret, tmp;
4631 struct folio *folio;4686 struct folio *folio;
4632 4687
4633 if (vmf->flags & FAULT_FLAG_VMA_LOCK) {4688 ret = vmf_can_call_fault(vmf);
4634 vma_end_read(vma);4689 if (ret)
4635 return VM_FAULT_RETRY;4690 return ret;
4636 }
4637 4691
4638 ret = __do_fault(vmf);4692 ret = __do_fault(vmf);
4639 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))4693 if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
@@ -4719,10 +4773,10 @@ static vm_fault_t do_fault(struct vm_fault *vmf) · 79 unmodified lines
4719 return ret;4773 return ret;
4720}4774}
4721 4775
4722int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,4776int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma,
4723 unsigned long addr, int page_nid, int *flags)4777 unsigned long addr, int page_nid, int *flags)
4724{4778{
4725 get_page(page);4779 folio_get(folio);
4726 4780
4727 /* Record the current PID acceesing VMA */4781 /* Record the current PID acceesing VMA */
4728 vma_set_access_pid_bit(vma);4782 vma_set_access_pid_bit(vma);
@@ -4733,14 +4787,14 @@ int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, · 4 unmodified lines
4733 *flags |= TNF_FAULT_LOCAL;4787 *flags |= TNF_FAULT_LOCAL;
4734 }4788 }
4735 4789
4736 return mpol_misplaced(page, vma, addr);4790 return mpol_misplaced(folio, vma, addr);
4737}4791}
4738 4792
4739static vm_fault_t do_numa_page(struct vm_fault *vmf)4793static vm_fault_t do_numa_page(struct vm_fault *vmf)
4740{4794{
4741 struct vm_area_struct *vma = vmf->vma;4795 struct vm_area_struct *vma = vmf->vma;
4742 struct page *page = NULL;4796 struct folio *folio = NULL;
4743 int page_nid = NUMA_NO_NODE;4797 int nid = NUMA_NO_NODE;
4744 bool writable = false;4798 bool writable = false;
4745 int last_cpupid;4799 int last_cpupid;
4746 int target_nid;4800 int target_nid;
@@ -4771,12 +4825,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 24 unmodified lines
4771 can_change_pte_writable(vma, vmf->address, pte))4825 can_change_pte_writable(vma, vmf->address, pte))
4772 writable = true;4826 writable = true;
4773 4827
4774 page = vm_normal_page(vma, vmf->address, pte);4828 folio = vm_normal_folio(vma, vmf->address, pte);
4775 if (!page || is_zone_device_page(page))4829 if (!folio || folio_is_zone_device(folio))
4776 goto out_map;4830 goto out_map;
4777 4831
4778 /* TODO: handle PTE-mapped THP */4832 /* TODO: handle PTE-mapped THP */
4779 if (PageCompound(page))4833 if (folio_test_large(folio))
4780 goto out_map;4834 goto out_map;
4781 4835
4782 /*4836 /*
@@ -4791,34 +4845,33 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 8 unmodified lines
4791 flags |= TNF_NO_GROUP;4845 flags |= TNF_NO_GROUP;
4792 4846
4793 /*4847 /*
4794 * Flag if the page is shared between multiple address spaces. This4848 * Flag if the folio is shared between multiple address spaces. This
4795 * is later used when determining whether to group tasks together4849 * is later used when determining whether to group tasks together
4796 */4850 */
4797 if (page_mapcount(page) > 1 && (vma->vm_flags & VM_SHARED))4851 if (folio_estimated_sharers(folio) > 1 && (vma->vm_flags & VM_SHARED))
4798 flags |= TNF_SHARED;4852 flags |= TNF_SHARED;
4799 4853
4800 page_nid = page_to_nid(page);4854 nid = folio_nid(folio);
4801 /*4855 /*
4802 * For memory tiering mode, cpupid of slow memory page is used4856 * For memory tiering mode, cpupid of slow memory page is used
4803 * to record page access time. So use default value.4857 * to record page access time. So use default value.
4804 */4858 */
4805 if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&4859 if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
4806 !node_is_toptier(page_nid))4860 !node_is_toptier(nid))
4807 last_cpupid = (-1 & LAST_CPUPID_MASK);4861 last_cpupid = (-1 & LAST_CPUPID_MASK);
4808 else4862 else
4809 last_cpupid = page_cpupid_last(page);4863 last_cpupid = folio_last_cpupid(folio);
4810 target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,4864 target_nid = numa_migrate_prep(folio, vma, vmf->address, nid, &flags);
4811 &flags);
4812 if (target_nid == NUMA_NO_NODE) {4865 if (target_nid == NUMA_NO_NODE) {
4813 put_page(page);4866 folio_put(folio);
4814 goto out_map;4867 goto out_map;
4815 }4868 }
4816 pte_unmap_unlock(vmf->pte, vmf->ptl);4869 pte_unmap_unlock(vmf->pte, vmf->ptl);
4817 writable = false;4870 writable = false;
4818 4871
4819 /* Migrate to the requested node */4872 /* Migrate to the requested node */
4820 if (migrate_misplaced_page(page, vma, target_nid)) {4873 if (migrate_misplaced_folio(folio, vma, target_nid)) {
4821 page_nid = target_nid;4874 nid = target_nid;
4822 flags |= TNF_MIGRATED;4875 flags |= TNF_MIGRATED;
4823 } else {4876 } else {
4824 flags |= TNF_MIGRATE_FAIL;4877 flags |= TNF_MIGRATE_FAIL;
@@ -4834,8 +4887,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 9 unmodified lines
4834 }4887 }
4835 4888
4836out:4889out:
4837 if (page_nid != NUMA_NO_NODE)4890 if (nid != NUMA_NO_NODE)
4838 task_numa_fault(last_cpupid, page_nid, 1, flags);4891 task_numa_fault(last_cpupid, nid, 1, flags);
4839 return 0;4892 return 0;
4840out_map:4893out_map:
4841 /*4894 /*
@@ -4872,8 +4925,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 30 unmodified lines
4872 4925
4873 if (vma_is_anonymous(vma)) {4926 if (vma_is_anonymous(vma)) {
4874 if (likely(!unshare) &&4927 if (likely(!unshare) &&
4875 userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd))4928 userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) {
4929 if (userfaultfd_wp_async(vmf->vma))
4930 goto split;
4876 return handle_userfault(vmf, VM_UFFD_WP);4931 return handle_userfault(vmf, VM_UFFD_WP);
4932 }
4877 return do_huge_pmd_wp_page(vmf);4933 return do_huge_pmd_wp_page(vmf);
4878 }4934 }
4879 4935
@@ -4885,6 +4941,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 5 unmodified lines
4885 }4941 }
4886 }4942 }
4887 4943
4944split:
4888 /* COW or write-notify handled on pte level: split pmd. */4945 /* COW or write-notify handled on pte level: split pmd. */
4889 __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL);4946 __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL);
4890 4947
@@ -5736,8 +5793,8 @@ EXPORT_SYMBOL_GPL(generic_access_phys); · 845 unmodified lines
5736/*5793/*
5737 * Access another process' address space as given in mm.5794 * Access another process' address space as given in mm.
5738 */5795 */
5739int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf,5796static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
5740 int len, unsigned int gup_flags)5797 void *buf, int len, unsigned int gup_flags)
5741{5798{
5742 void *old_buf = buf;5799 void *old_buf = buf;
5743 int write = gup_flags & FOLL_WRITE;5800 int write = gup_flags & FOLL_WRITE;
@@ -5760,7 +5817,7 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 16 unmodified lines
5760 struct page *page = get_user_page_vma_remote(mm, addr,5817 struct page *page = get_user_page_vma_remote(mm, addr,
5761 gup_flags, &vma);5818 gup_flags, &vma);
5762 5819
5763 if (IS_ERR_OR_NULL(page)) {5820 if (IS_ERR(page)) {
5764 /* We might need to expand the stack to access it */5821 /* We might need to expand the stack to access it */
5765 vma = vma_lookup(mm, addr);5822 vma = vma_lookup(mm, addr);
5766 if (!vma) {5823 if (!vma) {
@@ -5774,7 +5831,6 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 7 unmodified lines
5774 continue;5831 continue;
5775 }5832 }
5776 5833
5777
5778 /*5834 /*
5779 * Check if this is a VM_IO | VM_PFNMAP VMA, which5835 * Check if this is a VM_IO | VM_PFNMAP VMA, which
5780 * we can access using slightly different code.5836 * we can access using slightly different code.
mm/memory_hotplug.c+6−2
@@ -1689,7 +1689,7 @@ static int scan_movable_pages(unsigned long start, unsigned long end, · 1688 unmodified lines
1689 */1689 */
1690 if (HPageMigratable(head))1690 if (HPageMigratable(head))
1691 goto found;1691 goto found;
1692 skip = compound_nr(head) - (page - head);1692 skip = compound_nr(head) - (pfn - page_to_pfn(head));
1693 pfn += skip - 1;1693 pfn += skip - 1;
1694 }1694 }
1695 return -ENOENT;1695 return -ENOENT;
@@ -2012,12 +2012,16 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, · 316 unmodified lines
2012 /* reinitialise watermarks and update pcp limits */2012 /* reinitialise watermarks and update pcp limits */
2013 init_per_zone_wmark_min();2013 init_per_zone_wmark_min();
2014 2014
2015 /*
2016 * Make sure to mark the node as memory-less before rebuilding the zone
2017 * list. Otherwise this node would still appear in the fallback lists.
2018 */
2019 node_states_clear_node(node, &arg);
2015 if (!populated_zone(zone)) {2020 if (!populated_zone(zone)) {
2016 zone_pcp_reset(zone);2021 zone_pcp_reset(zone);
2017 build_all_zonelists(NULL);2022 build_all_zonelists(NULL);
2018 }2023 }
2019 2024
2020 node_states_clear_node(node, &arg);
2021 if (arg.status_change_nid >= 0) {2025 if (arg.status_change_nid >= 0) {
2022 kcompactd_stop(node);2026 kcompactd_stop(node);
2023 kswapd_stop(node);2027 kswapd_stop(node);
mm/mempolicy.c+455−571
@@ -25,7 +25,7 @@ · 24 unmodified lines
25 * to the last. It would be better if bind would truly restrict25 * to the last. It would be better if bind would truly restrict
26 * the allocation to memory nodes instead26 * the allocation to memory nodes instead
27 *27 *
28 * preferred Try a specific node first before normal fallback.28 * preferred Try a specific node first before normal fallback.
29 * As a special case NUMA_NO_NODE here means do the allocation29 * As a special case NUMA_NO_NODE here means do the allocation
30 * on the local CPU. This is normally identical to default,30 * on the local CPU. This is normally identical to default,
31 * but useful to set in a VMA when you have a non default31 * but useful to set in a VMA when you have a non default
@@ -52,7 +52,7 @@ · 20 unmodified lines
52 * on systems with highmem kernel lowmem allocation don't get policied.52 * on systems with highmem kernel lowmem allocation don't get policied.
53 * Same with GFP_DMA allocations.53 * Same with GFP_DMA allocations.
54 *54 *
55 * For shmfs/tmpfs/hugetlbfs shared memory the policy is shared between55 * For shmem/tmpfs shared memory the policy is shared between
56 * all users and remembered even when nobody has memory mapped.56 * all users and remembered even when nobody has memory mapped.
57 */57 */
58 58
@@ -111,7 +111,8 @@ · 52 unmodified lines
111 111
112/* Internal flags */112/* Internal flags */
113#define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0) /* Skip checks for continuous vmas */113#define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0) /* Skip checks for continuous vmas */
114#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */114#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */
115#define MPOL_MF_WRLOCK (MPOL_MF_INTERNAL << 2) /* Write-lock walked vmas */
115 116
116static struct kmem_cache *policy_cache;117static struct kmem_cache *policy_cache;
117static struct kmem_cache *sn_cache;118static struct kmem_cache *sn_cache;
@@ -267,9 +268,6 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 149 unmodified lines
267{268{
268 struct mempolicy *policy;269 struct mempolicy *policy;
269 270
270 pr_debug("setting mode %d flags %d nodes[0] %lx\n",
271 mode, flags, nodes ? nodes_addr(*nodes)[0] : NUMA_NO_NODE);
272
273 if (mode == MPOL_DEFAULT) {271 if (mode == MPOL_DEFAULT) {
274 if (nodes && !nodes_empty(*nodes))272 if (nodes && !nodes_empty(*nodes))
275 return ERR_PTR(-EINVAL);273 return ERR_PTR(-EINVAL);
@@ -297,6 +295,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 21 unmodified lines
297 return ERR_PTR(-EINVAL);295 return ERR_PTR(-EINVAL);
298 } else if (nodes_empty(*nodes))296 } else if (nodes_empty(*nodes))
299 return ERR_PTR(-EINVAL);297 return ERR_PTR(-EINVAL);
298
300 policy = kmem_cache_alloc(policy_cache, GFP_KERNEL);299 policy = kmem_cache_alloc(policy_cache, GFP_KERNEL);
301 if (!policy)300 if (!policy)
302 return ERR_PTR(-ENOMEM);301 return ERR_PTR(-ENOMEM);
@@ -309,11 +308,11 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 6 unmodified lines
309}308}
310 309
311/* Slow path of a mpol destructor. */310/* Slow path of a mpol destructor. */
312void __mpol_put(struct mempolicy *p)311void __mpol_put(struct mempolicy *pol)
313{312{
314 if (!atomic_dec_and_test(&p->refcnt))313 if (!atomic_dec_and_test(&pol->refcnt))
315 return;314 return;
316 kmem_cache_free(policy_cache, p);315 kmem_cache_free(policy_cache, pol);
317}316}
318 317
319static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)318static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
@@ -370,7 +369,6 @@ static void mpol_rebind_policy(struct mempolicy *pol, const nodemask_t *newmask) · 50 unmodified lines
370 *369 *
371 * Called with task's alloc_lock held.370 * Called with task's alloc_lock held.
372 */371 */
373
374void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new)372void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new)
375{373{
376 mpol_rebind_policy(tsk->mempolicy, new);374 mpol_rebind_policy(tsk->mempolicy, new);
@@ -381,7 +379,6 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) · 4 unmodified lines
381 *379 *
382 * Call holding a reference to mm. Takes mm->mmap_lock during call.380 * Call holding a reference to mm. Takes mm->mmap_lock during call.
383 */381 */
384
385void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)382void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)
386{383{
387 struct vm_area_struct *vma;384 struct vm_area_struct *vma;
@@ -420,8 +417,25 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { · 32 unmodified lines
420 },417 },
421};418};
422 419
423static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,420static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
424 unsigned long flags);421 unsigned long flags);
422static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
423 pgoff_t ilx, int *nid);
424
425static bool strictly_unmovable(unsigned long flags)
426{
427 /*
428 * STRICT without MOVE flags lets do_mbind() fail immediately with -EIO
429 * if any misplaced page is found.
430 */
431 return (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ==
432 MPOL_MF_STRICT;
433}
434
435struct migration_mpol { /* for alloc_migration_target_by_mpol() */
436 struct mempolicy *pol;
437 pgoff_t ilx;
438};
425 439
426struct queue_pages {440struct queue_pages {
427 struct list_head *pagelist;441 struct list_head *pagelist;
@@ -430,7 +444,8 @@ struct queue_pages { · 2 unmodified lines
430 unsigned long start;444 unsigned long start;
431 unsigned long end;445 unsigned long end;
432 struct vm_area_struct *first;446 struct vm_area_struct *first;
433 bool has_unmovable;447 struct folio *large; /* note last large folio encountered */
448 long nr_failed; /* could not be isolated at this time */
434};449};
435 450
436/*451/*
@@ -448,61 +463,37 @@ static inline bool queue_folio_required(struct folio *folio, · 11 unmodified lines
448 return node_isset(nid, *qp->nmask) == !(flags & MPOL_MF_INVERT);463 return node_isset(nid, *qp->nmask) == !(flags & MPOL_MF_INVERT);
449}464}
450 465
451/*466static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
452 * queue_folios_pmd() has three possible return values:
453 * 0 - folios are placed on the right node or queued successfully, or
454 * special page is met, i.e. zero page, or unmovable page is found
455 * but continue walking (indicated by queue_pages.has_unmovable).
456 * -EIO - is migration entry or only MPOL_MF_STRICT was specified and an
457 * existing folio was already on a node that does not follow the
458 * policy.
459 */
460static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,
461 unsigned long end, struct mm_walk *walk)
462 __releases(ptl)
463{467{
464 int ret = 0;
465 struct folio *folio;468 struct folio *folio;
466 struct queue_pages *qp = walk->private;469 struct queue_pages *qp = walk->private;
467 unsigned long flags;
468 470
469 if (unlikely(is_pmd_migration_entry(*pmd))) {471 if (unlikely(is_pmd_migration_entry(*pmd))) {
470 ret = -EIO;472 qp->nr_failed++;
471 goto unlock;473 return;
472 }474 }
473 folio = pfn_folio(pmd_pfn(*pmd));475 folio = pfn_folio(pmd_pfn(*pmd));
474 if (is_huge_zero_page(&folio->page)) {476 if (is_huge_zero_page(&folio->page)) {
475 walk->action = ACTION_CONTINUE;477 walk->action = ACTION_CONTINUE;
476 goto unlock;478 return;
477 }479 }
478 if (!queue_folio_required(folio, qp))480 if (!queue_folio_required(folio, qp))
479 goto unlock;481 return;
480 482 if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
481 flags = qp->flags;483 !vma_migratable(walk->vma) ||
482 /* go to folio migration */484 !migrate_folio_add(folio, qp->pagelist, qp->flags))
483 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {485 qp->nr_failed++;
484 if (!vma_migratable(walk->vma) ||
485 migrate_folio_add(folio, qp->pagelist, flags)) {
486 qp->has_unmovable = true;
487 goto unlock;
488 }
489 } else
490 ret = -EIO;
491unlock:
492 spin_unlock(ptl);
493 return ret;
494}486}
495 487
496/*488/*
497 * Scan through pages checking if pages follow certain conditions,489 * Scan through folios, checking if they satisfy the required conditions,
498 * and move them to the pagelist if they do.490 * moving them from LRU to local pagelist for migration if they do (or not).
499 *491 *
500 * queue_folios_pte_range() has three possible return values:492 * queue_folios_pte_range() has two possible return values:
501 * 0 - folios are placed on the right node or queued successfully, or493 * 0 - continue walking to scan for more, even if an existing folio on the
502 * special page is met, i.e. zero page, or unmovable page is found494 * wrong node could not be isolated and queued for migration.
503 * but continue walking (indicated by queue_pages.has_unmovable).495 * -EIO - only MPOL_MF_STRICT was specified, without MPOL_MF_MOVE or ..._ALL,
504 * -EIO - only MPOL_MF_STRICT was specified and an existing folio was already496 * and an existing folio was on a node that does not follow the policy.
505 * on a node that does not follow the policy.
506 */497 */
507static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,498static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
508 unsigned long end, struct mm_walk *walk)499 unsigned long end, struct mm_walk *walk)
@@ -516,8 +507,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 7 unmodified lines
516 spinlock_t *ptl;507 spinlock_t *ptl;
517 508
518 ptl = pmd_trans_huge_lock(pmd, vma);509 ptl = pmd_trans_huge_lock(pmd, vma);
519 if (ptl)510 if (ptl) {
520 return queue_folios_pmd(pmd, ptl, addr, end, walk);511 queue_folios_pmd(pmd, walk);
512 spin_unlock(ptl);
513 goto out;
514 }
521 515
522 mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);516 mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
523 if (!pte) {517 if (!pte) {
@@ -526,8 +520,13 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 2 unmodified lines
526 }520 }
527 for (; addr != end; pte++, addr += PAGE_SIZE) {521 for (; addr != end; pte++, addr += PAGE_SIZE) {
528 ptent = ptep_get(pte);522 ptent = ptep_get(pte);
529 if (!pte_present(ptent))523 if (pte_none(ptent))
524 continue;
525 if (!pte_present(ptent)) {
526 if (is_migration_entry(pte_to_swp_entry(ptent)))
527 qp->nr_failed++;
530 continue;528 continue;
529 }
531 folio = vm_normal_folio(vma, addr, ptent);530 folio = vm_normal_folio(vma, addr, ptent);
532 if (!folio || folio_is_zone_device(folio))531 if (!folio || folio_is_zone_device(folio))
533 continue;532 continue;
@@ -539,95 +538,87 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 5 unmodified lines
539 continue;538 continue;
540 if (!queue_folio_required(folio, qp))539 if (!queue_folio_required(folio, qp))
541 continue;540 continue;
542 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {541 if (folio_test_large(folio)) {
543 /*
544 * MPOL_MF_STRICT must be specified if we get here.
545 * Continue walking vmas due to MPOL_MF_MOVE* flags.
546 */
547 if (!vma_migratable(vma))
548 qp->has_unmovable = true;
549
550 /*542 /*
551 * Do not abort immediately since there may be543 * A large folio can only be isolated from LRU once,
552 * temporary off LRU pages in the range. Still544 * but may be mapped by many PTEs (and Copy-On-Write may
553 * need migrate other LRU pages.545 * intersperse PTEs of other, order 0, folios). This is
546 * a common case, so don't mistake it for failure (but
547 * there can be other cases of multi-mapped pages which
548 * this quick check does not help to filter out - and a
549 * search of the pagelist might grow to be prohibitive).
550 *
551 * migrate_pages(&pagelist) returns nr_failed folios, so
552 * check "large" now so that queue_pages_range() returns
553 * a comparable nr_failed folios. This does imply that
554 * if folio could not be isolated for some racy reason
555 * at its first PTE, later PTEs will not give it another
556 * chance of isolation; but keeps the accounting simple.
554 */557 */
555 if (migrate_folio_add(folio, qp->pagelist, flags))558 if (folio == qp->large)
556 qp->has_unmovable = true;559 continue;
557 } else560 qp->large = folio;
558 break;561 }
562 if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
563 !vma_migratable(vma) ||
564 !migrate_folio_add(folio, qp->pagelist, flags)) {
565 qp->nr_failed++;
566 if (strictly_unmovable(flags))
567 break;
568 }
559 }569 }
560 pte_unmap_unlock(mapped_pte, ptl);570 pte_unmap_unlock(mapped_pte, ptl);
561 cond_resched();571 cond_resched();
562 572out:
563 return addr != end ? -EIO : 0;573 if (qp->nr_failed && strictly_unmovable(flags))
574 return -EIO;
575 return 0;
564}576}
565 577
566static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,578static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
567 unsigned long addr, unsigned long end,579 unsigned long addr, unsigned long end,
568 struct mm_walk *walk)580 struct mm_walk *walk)
569{581{
570 int ret = 0;
571#ifdef CONFIG_HUGETLB_PAGE582#ifdef CONFIG_HUGETLB_PAGE
572 struct queue_pages *qp = walk->private;583 struct queue_pages *qp = walk->private;
573 unsigned long flags = (qp->flags & MPOL_MF_VALID);584 unsigned long flags = qp->flags;
574 struct folio *folio;585 struct folio *folio;
575 spinlock_t *ptl;586 spinlock_t *ptl;
576 pte_t entry;587 pte_t entry;
577 588
578 ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte);589 ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte);
579 entry = huge_ptep_get(pte);590 entry = huge_ptep_get(pte);
580 if (!pte_present(entry))591 if (!pte_present(entry)) {
592 if (unlikely(is_hugetlb_entry_migration(entry)))
593 qp->nr_failed++;
581 goto unlock;594 goto unlock;
595 }
582 folio = pfn_folio(pte_pfn(entry));596 folio = pfn_folio(pte_pfn(entry));
583 if (!queue_folio_required(folio, qp))597 if (!queue_folio_required(folio, qp))
584 goto unlock;598 goto unlock;
585 599 if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
586 if (flags == MPOL_MF_STRICT) {600 !vma_migratable(walk->vma)) {
587 /*601 qp->nr_failed++;
588 * STRICT alone means only detecting misplaced folio and no
589 * need to further check other vma.
590 */
591 ret = -EIO;
592 goto unlock;602 goto unlock;
593 }603 }
594
595 if (!vma_migratable(walk->vma)) {
596 /*
597 * Must be STRICT with MOVE*, otherwise .test_walk() have
598 * stopped walking current vma.
599 * Detecting misplaced folio but allow migrating folios which
600 * have been queued.
601 */
602 qp->has_unmovable = true;
603 goto unlock;
604 }
605
606 /*604 /*
607 * With MPOL_MF_MOVE, we try to migrate only unshared folios. If it605 * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio.
608 * is shared it is likely not worth migrating.606 * Choosing not to migrate a shared folio is not counted as a failure.
609 *607 *
610 * To check if the folio is shared, ideally we want to make sure608 * To check if the folio is shared, ideally we want to make sure
611 * every page is mapped to the same process. Doing that is very609 * every page is mapped to the same process. Doing that is very
612 * expensive, so check the estimated mapcount of the folio instead.610 * expensive, so check the estimated sharers of the folio instead.
613 */611 */
614 if (flags & (MPOL_MF_MOVE_ALL) ||612 if ((flags & MPOL_MF_MOVE_ALL) ||
615 (flags & MPOL_MF_MOVE && folio_estimated_sharers(folio) == 1 &&613 (folio_estimated_sharers(folio) == 1 && !hugetlb_pmd_shared(pte)))
616 !hugetlb_pmd_shared(pte))) {614 if (!isolate_hugetlb(folio, qp->pagelist))
617 if (!isolate_hugetlb(folio, qp->pagelist) &&615 qp->nr_failed++;
618 (flags & MPOL_MF_STRICT))
619 /*
620 * Failed to isolate folio but allow migrating pages
621 * which have been queued.
622 */
623 qp->has_unmovable = true;
624 }
625unlock:616unlock:
626 spin_unlock(ptl);617 spin_unlock(ptl);
627#else618 if (qp->nr_failed && strictly_unmovable(flags))
628 BUG();619 return -EIO;
629#endif620#endif
630 return ret;621 return 0;
631}622}
632 623
633#ifdef CONFIG_NUMA_BALANCING624#ifdef CONFIG_NUMA_BALANCING
@@ -656,12 +647,6 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, · 22 unmodified lines
656 647
657 return nr_updated;648 return nr_updated;
658}649}
659#else
660static unsigned long change_prot_numa(struct vm_area_struct *vma,
661 unsigned long addr, unsigned long end)
662{
663 return 0;
664}
665#endif /* CONFIG_NUMA_BALANCING */650#endif /* CONFIG_NUMA_BALANCING */
666 651
667static int queue_pages_test_walk(unsigned long start, unsigned long end,652static int queue_pages_test_walk(unsigned long start, unsigned long end,
@@ -700,16 +685,11 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, · 32 unmodified lines
700 if (endvma > end)685 if (endvma > end)
701 endvma = end;686 endvma = end;
702 687
703 if (flags & MPOL_MF_LAZY) {688 /*
704 /* Similar to task_numa_work, skip inaccessible VMAs */689 * Check page nodes, and queue pages to move, in the current vma.
705 if (!is_vm_hugetlb_page(vma) && vma_is_accessible(vma) &&690 * But if no moving, and no strict checking, the scan can be skipped.
706 !(vma->vm_flags & VM_MIXEDMAP))691 */
707 change_prot_numa(vma, start, endvma);692 if (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
708 return 1;
709 }
710
711 /* queue pages from current vma */
712 if (flags & MPOL_MF_VALID)
713 return 0;693 return 0;
714 return 1;694 return 1;
715}695}
@@ -731,22 +711,21 @@ static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { · 15 unmodified lines
731/*711/*
732 * Walk through page tables and collect pages to be migrated.712 * Walk through page tables and collect pages to be migrated.
733 *713 *
734 * If pages found in a given range are on a set of nodes (determined by714 * If pages found in a given range are not on the required set of @nodes,
735 * @nodes and @flags,) it's isolated and queued to the pagelist which is715 * and migration is allowed, they are isolated and queued to @pagelist.
736 * passed via @private.
737 *716 *
738 * queue_pages_range() has three possible return values:717 * queue_pages_range() may return:
739 * 1 - there is unmovable page, but MPOL_MF_MOVE* & MPOL_MF_STRICT were718 * 0 - all pages already on the right node, or successfully queued for moving
740 * specified.719 * (or neither strict checking nor moving requested: only range checking).
741 * 0 - queue pages successfully or no misplaced page.720 * >0 - this number of misplaced folios could not be queued for moving
742 * errno - i.e. misplaced pages with MPOL_MF_STRICT specified (-EIO) or721 * (a hugetlbfs page or a transparent huge page being counted as 1).
743 * memory range specified by nodemask and maxnode points outside722 * -EIO - a misplaced page found, when MPOL_MF_STRICT specified without MOVEs.
744 * your accessible address space (-EFAULT)723 * -EFAULT - a hole in the memory range, when MPOL_MF_DISCONTIG_OK unspecified.
745 */724 */
746static int725static long
747queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end,726queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end,
748 nodemask_t *nodes, unsigned long flags,727 nodemask_t *nodes, unsigned long flags,
749 struct list_head *pagelist, bool lock_vma)728 struct list_head *pagelist)
750{729{
751 int err;730 int err;
752 struct queue_pages qp = {731 struct queue_pages qp = {
@@ -756,20 +735,17 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 3 unmodified lines
756 .start = start,735 .start = start,
757 .end = end,736 .end = end,
758 .first = NULL,737 .first = NULL,
759 .has_unmovable = false,
760 };738 };
761 const struct mm_walk_ops *ops = lock_vma ?739 const struct mm_walk_ops *ops = (flags & MPOL_MF_WRLOCK) ?
762 &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops;740 &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops;
763 741
764 err = walk_page_range(mm, start, end, ops, &qp);742 err = walk_page_range(mm, start, end, ops, &qp);
765 743
766 if (qp.has_unmovable)
767 err = 1;
768 if (!qp.first)744 if (!qp.first)
769 /* whole range in hole */745 /* whole range in hole */
770 err = -EFAULT;746 err = -EFAULT;
771 747
772 return err;748 return err ? : qp.nr_failed;
773}749}
774 750
775/*751/*
@@ -777,7 +753,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 1 unmodified line
777 * This must be called with the mmap_lock held for writing.753 * This must be called with the mmap_lock held for writing.
778 */754 */
779static int vma_replace_policy(struct vm_area_struct *vma,755static int vma_replace_policy(struct vm_area_struct *vma,
780 struct mempolicy *pol)756 struct mempolicy *pol)
781{757{
782 int err;758 int err;
783 struct mempolicy *old;759 struct mempolicy *old;
@@ -785,11 +761,6 @@ static int vma_replace_policy(struct vm_area_struct *vma, · 1 unmodified line
785 761
786 vma_assert_write_locked(vma);762 vma_assert_write_locked(vma);
787 763
788 pr_debug("vma %lx-%lx/%lx vm_ops %p vm_file %p set_policy %p\n",
789 vma->vm_start, vma->vm_end, vma->vm_pgoff,
790 vma->vm_ops, vma->vm_file,
791 vma->vm_ops ? vma->vm_ops->set_policy : NULL);
792
793 new = mpol_dup(pol);764 new = mpol_dup(pol);
794 if (IS_ERR(new))765 if (IS_ERR(new))
795 return PTR_ERR(new);766 return PTR_ERR(new);
@@ -815,10 +786,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 19 unmodified lines
815 struct vm_area_struct **prev, unsigned long start,786 struct vm_area_struct **prev, unsigned long start,
816 unsigned long end, struct mempolicy *new_pol)787 unsigned long end, struct mempolicy *new_pol)
817{788{
818 struct vm_area_struct *merged;
819 unsigned long vmstart, vmend;789 unsigned long vmstart, vmend;
820 pgoff_t pgoff;
821 int err;
822 790
823 vmend = min(end, vma->vm_end);791 vmend = min(end, vma->vm_end);
824 if (start > vma->vm_start) {792 if (start > vma->vm_start) {
@@ -828,31 +796,14 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 3 unmodified lines
828 vmstart = vma->vm_start;796 vmstart = vma->vm_start;
829 }797 }
830 798
831 if (mpol_equal(vma_policy(vma), new_pol)) {799 if (mpol_equal(vma->vm_policy, new_pol)) {
832 *prev = vma;800 *prev = vma;
833 return 0;801 return 0;
834 }802 }
835 803
836 pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT);804 vma = vma_modify_policy(vmi, *prev, vma, vmstart, vmend, new_pol);
837 merged = vma_merge(vmi, vma->vm_mm, *prev, vmstart, vmend, vma->vm_flags,805 if (IS_ERR(vma))
838 vma->anon_vma, vma->vm_file, pgoff, new_pol,806 return PTR_ERR(vma);
839 vma->vm_userfaultfd_ctx, anon_vma_name(vma));
840 if (merged) {
841 *prev = merged;
842 return vma_replace_policy(merged, new_pol);
843 }
844
845 if (vma->vm_start != vmstart) {
846 err = split_vma(vmi, vma, vmstart, 1);
847 if (err)
848 return err;
849 }
850
851 if (vma->vm_end != vmend) {
852 err = split_vma(vmi, vma, vmend, 0);
853 if (err)
854 return err;
855 }
856 807
857 *prev = vma;808 *prev = vma;
858 return vma_replace_policy(vma, new_pol);809 return vma_replace_policy(vma, new_pol);
@@ -900,18 +851,18 @@ out: · 41 unmodified lines
900 *851 *
901 * Called with task's alloc_lock held852 * Called with task's alloc_lock held
902 */853 */
903static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)854static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes)
904{855{
905 nodes_clear(*nodes);856 nodes_clear(*nodes);
906 if (p == &default_policy)857 if (pol == &default_policy)
907 return;858 return;
908 859
909 switch (p->mode) {860 switch (pol->mode) {
910 case MPOL_BIND:861 case MPOL_BIND:
911 case MPOL_INTERLEAVE:862 case MPOL_INTERLEAVE:
912 case MPOL_PREFERRED:863 case MPOL_PREFERRED:
913 case MPOL_PREFERRED_MANY:864 case MPOL_PREFERRED_MANY:
914 *nodes = p->nodes;865 *nodes = pol->nodes;
915 break;866 break;
916 case MPOL_LOCAL:867 case MPOL_LOCAL:
917 /* return empty node mask for local allocation */868 /* return empty node mask for local allocation */
@@ -958,6 +909,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 40 unmodified lines
958 }909 }
959 910
960 if (flags & MPOL_F_ADDR) {911 if (flags & MPOL_F_ADDR) {
912 pgoff_t ilx; /* ignored here */
961 /*913 /*
962 * Do NOT fall back to task policy if the914 * Do NOT fall back to task policy if the
963 * vma/shared policy at addr is NULL. We915 * vma/shared policy at addr is NULL. We
@@ -969,10 +921,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 5 unmodified lines
969 mmap_read_unlock(mm);921 mmap_read_unlock(mm);
970 return -EFAULT;922 return -EFAULT;
971 }923 }
972 if (vma->vm_ops && vma->vm_ops->get_policy)924 pol = __get_vma_policy(vma, addr, &ilx);
973 pol = vma->vm_ops->get_policy(vma, addr);
974 else
975 pol = vma->vm_policy;
976 } else if (addr)925 } else if (addr)
977 return -EINVAL;926 return -EINVAL;
978 927
@@ -1032,16 +981,16 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 53 unmodified lines
1032}981}
1033 982
1034#ifdef CONFIG_MIGRATION983#ifdef CONFIG_MIGRATION
1035static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,984static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
1036 unsigned long flags)985 unsigned long flags)
1037{986{
1038 /*987 /*
1039 * We try to migrate only unshared folios. If it is shared it988 * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio.
1040 * is likely not worth migrating.989 * Choosing not to migrate a shared folio is not counted as a failure.
1041 *990 *
1042 * To check if the folio is shared, ideally we want to make sure991 * To check if the folio is shared, ideally we want to make sure
1043 * every page is mapped to the same process. Doing that is very992 * every page is mapped to the same process. Doing that is very
1044 * expensive, so check the estimated mapcount of the folio instead.993 * expensive, so check the estimated sharers of the folio instead.
1045 */994 */
1046 if ((flags & MPOL_MF_MOVE_ALL) || folio_estimated_sharers(folio) == 1) {995 if ((flags & MPOL_MF_MOVE_ALL) || folio_estimated_sharers(folio) == 1) {
1047 if (folio_isolate_lru(folio)) {996 if (folio_isolate_lru(folio)) {
@@ -1049,32 +998,31 @@ static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, · 1 unmodified line
1049 node_stat_mod_folio(folio,998 node_stat_mod_folio(folio,
1050 NR_ISOLATED_ANON + folio_is_file_lru(folio),999 NR_ISOLATED_ANON + folio_is_file_lru(folio),
1051 folio_nr_pages(folio));1000 folio_nr_pages(folio));
1052 } else if (flags & MPOL_MF_STRICT) {1001 } else {
1053 /*1002 /*
1054 * Non-movable folio may reach here. And, there may be1003 * Non-movable folio may reach here. And, there may be
1055 * temporary off LRU folios or non-LRU movable folios.1004 * temporary off LRU folios or non-LRU movable folios.
1056 * Treat them as unmovable folios since they can't be1005 * Treat them as unmovable folios since they can't be
1057 * isolated, so they can't be moved at the moment. It1006 * isolated, so they can't be moved at the moment.
1058 * should return -EIO for this case too.
1059 */1007 */
1060 return -EIO;1008 return false;
1061 }1009 }
1062 }1010 }
1063 1011 return true;
1064 return 0;
1065}1012}
1066 1013
1067/*1014/*
1068 * Migrate pages from one node to a target node.1015 * Migrate pages from one node to a target node.
1069 * Returns error or the number of pages not migrated.1016 * Returns error or the number of pages not migrated.
1070 */1017 */
1071static int migrate_to_node(struct mm_struct *mm, int source, int dest,1018static long migrate_to_node(struct mm_struct *mm, int source, int dest,
1072 int flags)1019 int flags)
1073{1020{
1074 nodemask_t nmask;1021 nodemask_t nmask;
1075 struct vm_area_struct *vma;1022 struct vm_area_struct *vma;
1076 LIST_HEAD(pagelist);1023 LIST_HEAD(pagelist);
1077 int err = 0;1024 long nr_failed;
1025 long err = 0;
1078 struct migration_target_control mtc = {1026 struct migration_target_control mtc = {
1079 .nid = dest,1027 .nid = dest,
1080 .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,1028 .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
@@ -1083,23 +1031,30 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 2 unmodified lines
1083 nodes_clear(nmask);1031 nodes_clear(nmask);
1084 node_set(source, nmask);1032 node_set(source, nmask);
1085 1033
1034 VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)));
1035
1036 mmap_read_lock(mm);
1037 vma = find_vma(mm, 0);
1038
1086 /*1039 /*
1087 * This does not "check" the range but isolates all pages that1040 * This does not migrate the range, but isolates all pages that
1088 * need migration. Between passing in the full user address1041 * need migration. Between passing in the full user address
1089 * space range and MPOL_MF_DISCONTIG_OK, this call can not fail.1042 * space range and MPOL_MF_DISCONTIG_OK, this call cannot fail,
1043 * but passes back the count of pages which could not be isolated.
1090 */1044 */
1091 vma = find_vma(mm, 0);1045 nr_failed = queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask,
1092 VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)));1046 flags | MPOL_MF_DISCONTIG_OK, &pagelist);
1093 queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask,1047 mmap_read_unlock(mm);
1094 flags | MPOL_MF_DISCONTIG_OK, &pagelist, false);
1095 1048
1096 if (!list_empty(&pagelist)) {1049 if (!list_empty(&pagelist)) {
1097 err = migrate_pages(&pagelist, alloc_migration_target, NULL,1050 err = migrate_pages(&pagelist, alloc_migration_target, NULL,
1098 (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);1051 (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL);
1099 if (err)1052 if (err)
1100 putback_movable_pages(&pagelist);1053 putback_movable_pages(&pagelist);
1101 }1054 }
1102 1055
1056 if (err >= 0)
1057 err += nr_failed;
1103 return err;1058 return err;
1104}1059}
1105 1060
@@ -1112,14 +1067,12 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 6 unmodified lines
1112int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,1067int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,
1113 const nodemask_t *to, int flags)1068 const nodemask_t *to, int flags)
1114{1069{
1115 int busy = 0;1070 long nr_failed = 0;
1116 int err = 0;1071 long err = 0;
1117 nodemask_t tmp;1072 nodemask_t tmp;
1118 1073
1119 lru_cache_disable();1074 lru_cache_disable();
1120 1075
1121 mmap_read_lock(mm);
1122
1123 /*1076 /*
1124 * Find a 'source' bit set in 'tmp' whose corresponding 'dest'1077 * Find a 'source' bit set in 'tmp' whose corresponding 'dest'
1125 * bit in 'to' is not also set in 'tmp'. Clear the found 'source'1078 * bit in 'to' is not also set in 'tmp'. Clear the found 'source'
@@ -1195,59 +1148,58 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 69 unmodified lines
1195 node_clear(source, tmp);1148 node_clear(source, tmp);
1196 err = migrate_to_node(mm, source, dest, flags);1149 err = migrate_to_node(mm, source, dest, flags);
1197 if (err > 0)1150 if (err > 0)
1198 busy += err;1151 nr_failed += err;
1199 if (err < 0)1152 if (err < 0)
1200 break;1153 break;
1201 }1154 }
1202 mmap_read_unlock(mm);
1203 1155
1204 lru_cache_enable();1156 lru_cache_enable();
1205 if (err < 0)1157 if (err < 0)
1206 return err;1158 return err;
1207 return busy;1159 return (nr_failed < INT_MAX) ? nr_failed : INT_MAX;
1208
1209}1160}
1210 1161
1211/*1162/*
1212 * Allocate a new page for page migration based on vma policy.1163 * Allocate a new folio for page migration, according to NUMA mempolicy.
1213 * Start by assuming the page is mapped by the same vma as contains @start.
1214 * Search forward from there, if not. N.B., this assumes that the
1215 * list of pages handed to migrate_pages()--which is how we get here--
1216 * is in virtual address order.
1217 */1164 */
1218static struct folio *new_folio(struct folio *src, unsigned long start)1165static struct folio *alloc_migration_target_by_mpol(struct folio *src,
1166 unsigned long private)
1219{1167{
1220 struct vm_area_struct *vma;1168 struct migration_mpol *mmpol = (struct migration_mpol *)private;
1221 unsigned long address;1169 struct mempolicy *pol = mmpol->pol;
1222 VMA_ITERATOR(vmi, current->mm, start);1170 pgoff_t ilx = mmpol->ilx;
1223 gfp_t gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL;1171 struct page *page;
1172 unsigned int order;
1173 int nid = numa_node_id();
1174 gfp_t gfp;
1224 1175
1225 for_each_vma(vmi, vma) {1176 order = folio_order(src);
1226 address = page_address_in_vma(&src->page, vma);1177 ilx += src->index >> order;
1227 if (address != -EFAULT)
1228 break;
1229 }
1230 1178
1231 if (folio_test_hugetlb(src)) {1179 if (folio_test_hugetlb(src)) {
1232 return alloc_hugetlb_folio_vma(folio_hstate(src),1180 nodemask_t *nodemask;
1233 vma, address);1181 struct hstate *h;
1182
1183 h = folio_hstate(src);
1184 gfp = htlb_alloc_mask(h);
1185 nodemask = policy_nodemask(gfp, pol, ilx, &nid);
1186 return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp);
1234 }1187 }
1235 1188
1236 if (folio_test_large(src))1189 if (folio_test_large(src))
1237 gfp = GFP_TRANSHUGE;1190 gfp = GFP_TRANSHUGE;
1191 else
1192 gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP;
1238 1193
1239 /*1194 page = alloc_pages_mpol(gfp, order, pol, ilx, nid);
1240 * if !vma, vma_alloc_folio() will use task or system default policy1195 return page_rmappable_folio(page);
1241 */
1242 return vma_alloc_folio(gfp, folio_order(src), vma, address,
1243 folio_test_large(src));
1244}1196}
1245#else1197#else
1246 1198
1247static int migrate_folio_add(struct folio *folio, struct list_head *foliolist,1199static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
1248 unsigned long flags)1200 unsigned long flags)
1249{1201{
1250 return -EIO;1202 return false;
1251}1203}
1252 1204
1253int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,1205int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,
@@ -1256,7 +1208,8 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 2 unmodified lines
1256 return -ENOSYS;1208 return -ENOSYS;
1257}1209}
1258 1210
1259static struct folio *new_folio(struct folio *src, unsigned long start)1211static struct folio *alloc_migration_target_by_mpol(struct folio *src,
1212 unsigned long private)
1260{1213{
1261 return NULL;1214 return NULL;
1262}1215}
@@ -1269,10 +1222,11 @@ static long do_mbind(unsigned long start, unsigned long len, · 6 unmodified lines
1269 struct mm_struct *mm = current->mm;1222 struct mm_struct *mm = current->mm;
1270 struct vm_area_struct *vma, *prev;1223 struct vm_area_struct *vma, *prev;
1271 struct vma_iterator vmi;1224 struct vma_iterator vmi;
1225 struct migration_mpol mmpol;
1272 struct mempolicy *new;1226 struct mempolicy *new;
1273 unsigned long end;1227 unsigned long end;
1274 int err;1228 long err;
1275 int ret;1229 long nr_failed;
1276 LIST_HEAD(pagelist);1230 LIST_HEAD(pagelist);
1277 1231
1278 if (flags & ~(unsigned long)MPOL_MF_VALID)1232 if (flags & ~(unsigned long)MPOL_MF_VALID)
@@ -1298,9 +1252,6 @@ static long do_mbind(unsigned long start, unsigned long len, · 19 unmodified lines
1298 if (IS_ERR(new))1252 if (IS_ERR(new))
1299 return PTR_ERR(new);1253 return PTR_ERR(new);
1300 1254
1301 if (flags & MPOL_MF_LAZY)
1302 new->flags |= MPOL_F_MOF;
1303
1304 /*1255 /*
1305 * If we are using the default policy then operation1256 * If we are using the default policy then operation
1306 * on discontinuous address spaces is okay after all1257 * on discontinuous address spaces is okay after all
@@ -1308,14 +1259,8 @@ static long do_mbind(unsigned long start, unsigned long len, · 1 unmodified line
1308 if (!new)1259 if (!new)
1309 flags |= MPOL_MF_DISCONTIG_OK;1260 flags |= MPOL_MF_DISCONTIG_OK;
1310 1261
1311 pr_debug("mbind %lx-%lx mode:%d flags:%d nodes:%lx\n",1262 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
1312 start, start + len, mode, mode_flags,
1313 nmask ? nodes_addr(*nmask)[0] : NUMA_NO_NODE);
1314
1315 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
1316
1317 lru_cache_disable();1263 lru_cache_disable();
1318 }
1319 {1264 {
1320 NODEMASK_SCRATCH(scratch);1265 NODEMASK_SCRATCH(scratch);
1321 if (scratch) {1266 if (scratch) {
@@ -1331,45 +1276,81 @@ static long do_mbind(unsigned long start, unsigned long len, · 9 unmodified lines
1331 goto mpol_out;1276 goto mpol_out;
1332 1277
1333 /*1278 /*
1334 * Lock the VMAs before scanning for pages to migrate, to ensure we don't1279 * Lock the VMAs before scanning for pages to migrate,
1335 * miss a concurrently inserted page.1280 * to ensure we don't miss a concurrently inserted page.
1336 */1281 */
1337 ret = queue_pages_range(mm, start, end, nmask,1282 nr_failed = queue_pages_range(mm, start, end, nmask,
1338 flags | MPOL_MF_INVERT, &pagelist, true);1283 flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist);
1339 1284
1340 if (ret < 0) {1285 if (nr_failed < 0) {
1341 err = ret;1286 err = nr_failed;
1342 goto up_out;1287 nr_failed = 0;
1343 }1288 } else {
1344 1289 vma_iter_init(&vmi, mm, start);
1345 vma_iter_init(&vmi, mm, start);1290 prev = vma_prev(&vmi);
1346 prev = vma_prev(&vmi);1291 for_each_vma_range(vmi, vma, end) {
1347 for_each_vma_range(vmi, vma, end) {1292 err = mbind_range(&vmi, vma, &prev, start, end, new);
1348 err = mbind_range(&vmi, vma, &prev, start, end, new);1293 if (err)
1349 if (err)1294 break;
1350 break;1295 }
1351 }1296 }
1352 1297
1353 if (!err) {1298 if (!err && !list_empty(&pagelist)) {
1354 int nr_failed = 0;1299 /* Convert MPOL_DEFAULT's NULL to task or default policy */
1355 1300 if (!new) {
1356 if (!list_empty(&pagelist)) {1301 new = get_task_policy(current);
1357 WARN_ON_ONCE(flags & MPOL_MF_LAZY);1302 mpol_get(new);
1358 nr_failed = migrate_pages(&pagelist, new_folio, NULL,
1359 start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL);
1360 if (nr_failed)
1361 putback_movable_pages(&pagelist);
1362 }1303 }
1304 mmpol.pol = new;
1305 mmpol.ilx = 0;
1363 1306
1364 if (((ret > 0) || nr_failed) && (flags & MPOL_MF_STRICT))1307 /*
1365 err = -EIO;1308 * In the interleaved case, attempt to allocate on exactly the
1366 } else {1309 * targeted nodes, for the first VMA to be migrated; for later
1367up_out:1310 * VMAs, the nodes will still be interleaved from the targeted
1368 if (!list_empty(&pagelist))1311 * nodemask, but one by one may be selected differently.
1369 putback_movable_pages(&pagelist);1312 */
1313 if (new->mode == MPOL_INTERLEAVE) {
1314 struct page *page;
1315 unsigned int order;
1316 unsigned long addr = -EFAULT;
1317
1318 list_for_each_entry(page, &pagelist, lru) {
1319 if (!PageKsm(page))
1320 break;
1321 }
1322 if (!list_entry_is_head(page, &pagelist, lru)) {
1323 vma_iter_init(&vmi, mm, start);
1324 for_each_vma_range(vmi, vma, end) {
1325 addr = page_address_in_vma(page, vma);
1326 if (addr != -EFAULT)
1327 break;
1328 }
1329 }
1330 if (addr != -EFAULT) {
1331 order = compound_order(page);
1332 /* We already know the pol, but not the ilx */
1333 mpol_cond_put(get_vma_policy(vma, addr, order,
1334 &mmpol.ilx));
1335 /* Set base from which to increment by index */
1336 mmpol.ilx -= page->index >> order;
1337 }
1338 }
1370 }1339 }
1371 1340
1372 mmap_write_unlock(mm);1341 mmap_write_unlock(mm);
1342
1343 if (!err && !list_empty(&pagelist)) {
1344 nr_failed |= migrate_pages(&pagelist,
1345 alloc_migration_target_by_mpol, NULL,
1346 (unsigned long)&mmpol, MIGRATE_SYNC,
1347 MR_MEMPOLICY_MBIND, NULL);
1348 }
1349
1350 if (nr_failed && (flags & MPOL_MF_STRICT))
1351 err = -EIO;
1352 if (!list_empty(&pagelist))
1353 putback_movable_pages(&pagelist);
1373mpol_out:1354mpol_out:
1374 mpol_put(new);1355 mpol_put(new);
1375 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))1356 if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
@@ -1690,7 +1671,6 @@ out: · 314 unmodified lines
1690out_put:1671out_put:
1691 put_task_struct(task);1672 put_task_struct(task);
1692 goto out;1673 goto out;
1693
1694}1674}
1695 1675
1696SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode,1676SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode,
@@ -1700,7 +1680,6 @@ SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, · 3 unmodified lines
1700 return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes);1680 return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes);
1701}1681}
1702 1682
1703
1704/* Retrieve NUMA policy */1683/* Retrieve NUMA policy */
1705static int kernel_get_mempolicy(int __user *policy,1684static int kernel_get_mempolicy(int __user *policy,
1706 unsigned long __user *nmask,1685 unsigned long __user *nmask,
@@ -1767,34 +1746,19 @@ bool vma_migratable(struct vm_area_struct *vma) · 60 unmodified lines
1767}1746}
1768 1747
1769struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,1748struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
1770 unsigned long addr)1749 unsigned long addr, pgoff_t *ilx)
1771{1750{
1772 struct mempolicy *pol = NULL;1751 *ilx = 0;
1773 1752 return (vma->vm_ops && vma->vm_ops->get_policy) ?
1774 if (vma) {1753 vma->vm_ops->get_policy(vma, addr, ilx) : vma->vm_policy;
1775 if (vma->vm_ops && vma->vm_ops->get_policy) {
1776 pol = vma->vm_ops->get_policy(vma, addr);
1777 } else if (vma->vm_policy) {
1778 pol = vma->vm_policy;
1779
1780 /*
1781 * shmem_alloc_page() passes MPOL_F_SHARED policy with
1782 * a pseudo vma whose vma->vm_ops=NULL. Take a reference
1783 * count on these policies which will be dropped by
1784 * mpol_cond_put() later
1785 */
1786 if (mpol_needs_cond_ref(pol))
1787 mpol_get(pol);
1788 }
1789 }
1790
1791 return pol;
1792}1754}
1793 1755
1794/*1756/*
1795 * get_vma_policy(@vma, @addr)1757 * get_vma_policy(@vma, @addr, @order, @ilx)
1796 * @vma: virtual memory area whose policy is sought1758 * @vma: virtual memory area whose policy is sought
1797 * @addr: address in @vma for shared policy lookup1759 * @addr: address in @vma for shared policy lookup
1760 * @order: 0, or appropriate huge_page_order for interleaving
1761 * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE
1798 *1762 *
1799 * Returns effective policy for a VMA at specified address.1763 * Returns effective policy for a VMA at specified address.
1800 * Falls back to current->mempolicy or system default policy, as necessary.1764 * Falls back to current->mempolicy or system default policy, as necessary.
@@ -1803,14 +1767,18 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, · 2 unmodified lines
1803 * freeing by another task. It is the caller's responsibility to free the1767 * freeing by another task. It is the caller's responsibility to free the
1804 * extra reference for shared policies.1768 * extra reference for shared policies.
1805 */1769 */
1806static struct mempolicy *get_vma_policy(struct vm_area_struct *vma,1770struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
1807 unsigned long addr)1771 unsigned long addr, int order, pgoff_t *ilx)
1808{1772{
1809 struct mempolicy *pol = __get_vma_policy(vma, addr);1773 struct mempolicy *pol;
1810 1774
1775 pol = __get_vma_policy(vma, addr, ilx);
1811 if (!pol)1776 if (!pol)
1812 pol = get_task_policy(current);1777 pol = get_task_policy(current);
1813 1778 if (pol->mode == MPOL_INTERLEAVE) {
1779 *ilx += vma->vm_pgoff >> order;
1780 *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
1781 }
1814 return pol;1782 return pol;
1815}1783}
1816 1784
@@ -1820,8 +1788,9 @@ bool vma_policy_mof(struct vm_area_struct *vma) · 3 unmodified lines
1820 1788
1821 if (vma->vm_ops && vma->vm_ops->get_policy) {1789 if (vma->vm_ops && vma->vm_ops->get_policy) {
1822 bool ret = false;1790 bool ret = false;
1791 pgoff_t ilx; /* ignored here */
1823 1792
1824 pol = vma->vm_ops->get_policy(vma, vma->vm_start);1793 pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx);
1825 if (pol && (pol->flags & MPOL_F_MOF))1794 if (pol && (pol->flags & MPOL_F_MOF))
1826 ret = true;1795 ret = true;
1827 mpol_cond_put(pol);1796 mpol_cond_put(pol);
@@ -1856,64 +1825,15 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) · 28 unmodified lines
1856 return zone >= dynamic_policy_zone;1825 return zone >= dynamic_policy_zone;
1857}1826}
1858 1827
1859/*
1860 * Return a nodemask representing a mempolicy for filtering nodes for
1861 * page allocation
1862 */
1863nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy)
1864{
1865 int mode = policy->mode;
1866
1867 /* Lower zones don't get a nodemask applied for MPOL_BIND */
1868 if (unlikely(mode == MPOL_BIND) &&
1869 apply_policy_zone(policy, gfp_zone(gfp)) &&
1870 cpuset_nodemask_valid_mems_allowed(&policy->nodes))
1871 return &policy->nodes;
1872
1873 if (mode == MPOL_PREFERRED_MANY)
1874 return &policy->nodes;
1875
1876 return NULL;
1877}
1878
1879/*
1880 * Return the preferred node id for 'prefer' mempolicy, and return
1881 * the given id for all other policies.
1882 *
1883 * policy_node() is always coupled with policy_nodemask(), which
1884 * secures the nodemask limit for 'bind' and 'prefer-many' policy.
1885 */
1886static int policy_node(gfp_t gfp, struct mempolicy *policy, int nd)
1887{
1888 if (policy->mode == MPOL_PREFERRED) {
1889 nd = first_node(policy->nodes);
1890 } else {
1891 /*
1892 * __GFP_THISNODE shouldn't even be used with the bind policy
1893 * because we might easily break the expectation to stay on the
1894 * requested node and not break the policy.
1895 */
1896 WARN_ON_ONCE(policy->mode == MPOL_BIND && (gfp & __GFP_THISNODE));
1897 }
1898
1899 if ((policy->mode == MPOL_BIND ||
1900 policy->mode == MPOL_PREFERRED_MANY) &&
1901 policy->home_node != NUMA_NO_NODE)
1902 return policy->home_node;
1903
1904 return nd;
1905}
1906
1907/* Do dynamic interleaving for a process */1828/* Do dynamic interleaving for a process */
1908static unsigned interleave_nodes(struct mempolicy *policy)1829static unsigned int interleave_nodes(struct mempolicy *policy)
1909{1830{
1910 unsigned next;1831 unsigned int nid;
1911 struct task_struct *me = current;
1912 1832
1913 next = next_node_in(me->il_prev, policy->nodes);1833 nid = next_node_in(current->il_prev, policy->nodes);
1914 if (next < MAX_NUMNODES)1834 if (nid < MAX_NUMNODES)
1915 me->il_prev = next;1835 current->il_prev = nid;
1916 return next;1836 return nid;
1917}1837}
1918 1838
1919/*1839/*
@@ -1964,11 +1884,11 @@ unsigned int mempolicy_slab_node(void) · 44 unmodified lines
1964}1884}
1965 1885
1966/*1886/*
1967 * Do static interleaving for a VMA with known offset @n. Returns the n'th1887 * Do static interleaving for interleave index @ilx. Returns the ilx'th
1968 * node in pol->nodes (starting from n=0), wrapping around if n exceeds the1888 * node in pol->nodes (starting from ilx=0), wrapping around if ilx
1969 * number of present nodes.1889 * exceeds the number of present nodes.
1970 */1890 */
1971static unsigned offset_il_node(struct mempolicy *pol, unsigned long n)1891static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx)
1972{1892{
1973 nodemask_t nodemask = pol->nodes;1893 nodemask_t nodemask = pol->nodes;
1974 unsigned int target, nnodes;1894 unsigned int target, nnodes;
@@ -1986,33 +1906,54 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) · 11 unmodified lines
1986 nnodes = nodes_weight(nodemask);1906 nnodes = nodes_weight(nodemask);
1987 if (!nnodes)1907 if (!nnodes)
1988 return numa_node_id();1908 return numa_node_id();
1989 target = (unsigned int)n % nnodes;1909 target = ilx % nnodes;
1990 nid = first_node(nodemask);1910 nid = first_node(nodemask);
1991 for (i = 0; i < target; i++)1911 for (i = 0; i < target; i++)
1992 nid = next_node(nid, nodemask);1912 nid = next_node(nid, nodemask);
1993 return nid;1913 return nid;
1994}1914}
1995 1915
1996/* Determine a node number for interleave */1916/*
1997static inline unsigned interleave_nid(struct mempolicy *pol,1917 * Return a nodemask representing a mempolicy for filtering nodes for
1998 struct vm_area_struct *vma, unsigned long addr, int shift)1918 * page allocation, together with preferred node id (or the input node id).
1919 */
1920static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol,
1921 pgoff_t ilx, int *nid)
1999{1922{
2000 if (vma) {1923 nodemask_t *nodemask = NULL;
2001 unsigned long off;
2002 1924
1925 switch (pol->mode) {
1926 case MPOL_PREFERRED:
1927 /* Override input node id */
1928 *nid = first_node(pol->nodes);
1929 break;
1930 case MPOL_PREFERRED_MANY:
1931 nodemask = &pol->nodes;
1932 if (pol->home_node != NUMA_NO_NODE)
1933 *nid = pol->home_node;
1934 break;
1935 case MPOL_BIND:
1936 /* Restrict to nodemask (but not on lower zones) */
1937 if (apply_policy_zone(pol, gfp_zone(gfp)) &&
1938 cpuset_nodemask_valid_mems_allowed(&pol->nodes))
1939 nodemask = &pol->nodes;
1940 if (pol->home_node != NUMA_NO_NODE)
1941 *nid = pol->home_node;
2003 /*1942 /*
2004 * for small pages, there is no difference between1943 * __GFP_THISNODE shouldn't even be used with the bind policy
2005 * shift and PAGE_SHIFT, so the bit-shift is safe.1944 * because we might easily break the expectation to stay on the
2006 * for huge pages, since vm_pgoff is in units of small1945 * requested node and not break the policy.
2007 * pages, we need to shift off the always 0 bits to get
2008 * a useful offset.
2009 */1946 */
2010 BUG_ON(shift < PAGE_SHIFT);1947 WARN_ON_ONCE(gfp & __GFP_THISNODE);
2011 off = vma->vm_pgoff >> (shift - PAGE_SHIFT);1948 break;
2012 off += (addr - vma->vm_start) >> shift;1949 case MPOL_INTERLEAVE:
2013 return offset_il_node(pol, off);1950 /* Override input node id */
2014 } else1951 *nid = (ilx == NO_INTERLEAVE_INDEX) ?
2015 return interleave_nodes(pol);1952 interleave_nodes(pol) : interleave_nid(pol, ilx);
1953 break;
1954 }
1955
1956 return nodemask;
2016}1957}
2017 1958
2018#ifdef CONFIG_HUGETLBFS1959#ifdef CONFIG_HUGETLBFS
@@ -2028,27 +1969,16 @@ static inline unsigned interleave_nid(struct mempolicy *pol, · 9 unmodified lines
2028 * to the struct mempolicy for conditional unref after allocation.1969 * to the struct mempolicy for conditional unref after allocation.
2029 * If the effective policy is 'bind' or 'prefer-many', returns a pointer1970 * If the effective policy is 'bind' or 'prefer-many', returns a pointer
2030 * to the mempolicy's @nodemask for filtering the zonelist.1971 * to the mempolicy's @nodemask for filtering the zonelist.
2031 *
2032 * Must be protected by read_mems_allowed_begin()
2033 */1972 */
2034int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags,1973int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags,
2035 struct mempolicy **mpol, nodemask_t **nodemask)1974 struct mempolicy **mpol, nodemask_t **nodemask)
2036{1975{
1976 pgoff_t ilx;
2037 int nid;1977 int nid;
2038 int mode;
2039
2040 *mpol = get_vma_policy(vma, addr);
2041 *nodemask = NULL;
2042 mode = (*mpol)->mode;
2043 1978
2044 if (unlikely(mode == MPOL_INTERLEAVE)) {1979 nid = numa_node_id();
2045 nid = interleave_nid(*mpol, vma, addr,1980 *mpol = get_vma_policy(vma, addr, hstate_vma(vma)->order, &ilx);
2046 huge_page_shift(hstate_vma(vma)));1981 *nodemask = policy_nodemask(gfp_flags, *mpol, ilx, &nid);
2047 } else {
2048 nid = policy_node(gfp_flags, *mpol, numa_node_id());
2049 if (mode == MPOL_BIND || mode == MPOL_PREFERRED_MANY)
2050 *nodemask = &(*mpol)->nodes;
2051 }
2052 return nid;1982 return nid;
2053}1983}
2054 1984
@@ -2126,27 +2056,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk, · 71 unmodified lines
2126 return ret;2056 return ret;
2127}2057}
2128 2058
2129/* Allocate a page in interleaved policy.
2130 Own path because it needs to do special accounting. */
2131static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
2132 unsigned nid)
2133{
2134 struct page *page;
2135
2136 page = __alloc_pages(gfp, order, nid, NULL);
2137 /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
2138 if (!static_branch_likely(&vm_numa_stat_key))
2139 return page;
2140 if (page && page_to_nid(page) == nid) {
2141 preempt_disable();
2142 __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT);
2143 preempt_enable();
2144 }
2145 return page;
2146}
2147
2148static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,2059static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
2149 int nid, struct mempolicy *pol)2060 int nid, nodemask_t *nodemask)
2150{2061{
2151 struct page *page;2062 struct page *page;
2152 gfp_t preferred_gfp;2063 gfp_t preferred_gfp;
@@ -2159,7 +2070,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 6 unmodified lines
2159 */2070 */
2160 preferred_gfp = gfp | __GFP_NOWARN;2071 preferred_gfp = gfp | __GFP_NOWARN;
2161 preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);2072 preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
2162 page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes);2073 page = __alloc_pages(preferred_gfp, order, nid, nodemask);
2163 if (!page)2074 if (!page)
2164 page = __alloc_pages(gfp, order, nid, NULL);2075 page = __alloc_pages(gfp, order, nid, NULL);
2165 2076
@@ -2167,61 +2078,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 1 unmodified line
2167}2078}
2168 2079
2169/**2080/**
2170 * vma_alloc_folio - Allocate a folio for a VMA.2081 * alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
2171 * @gfp: GFP flags.2082 * @gfp: GFP flags.
2172 * @order: Order of the folio.2083 * @order: Order of the page allocation.
2173 * @vma: Pointer to VMA or NULL if not available.2084 * @pol: Pointer to the NUMA mempolicy.
2174 * @addr: Virtual address of the allocation. Must be inside @vma.2085 * @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()).
2175 * @hugepage: For hugepages try only the preferred node if possible.2086 * @nid: Preferred node (usually numa_node_id() but @mpol may override it).
2176 *2087 *
2177 * Allocate a folio for a specific address in @vma, using the appropriate2088 * Return: The page on success or NULL if allocation fails.
2178 * NUMA policy. When @vma is not NULL the caller must hold the mmap_lock
2179 * of the mm_struct of the VMA to prevent it from going away. Should be
2180 * used for all allocations for folios that will be mapped into user space.
2181 *
2182 * Return: The folio on success or NULL if allocation fails.
2183 */2089 */
2184struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,2090struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
2185 unsigned long addr, bool hugepage)2091 struct mempolicy *pol, pgoff_t ilx, int nid)
2186{2092{
2187 struct mempolicy *pol;2093 nodemask_t *nodemask;
2188 int node = numa_node_id();2094 struct page *page;
2189 struct folio *folio;
2190 int preferred_nid;
2191 nodemask_t *nmask;
2192
2193 pol = get_vma_policy(vma, addr);
2194
2195 if (pol->mode == MPOL_INTERLEAVE) {
2196 struct page *page;
2197 unsigned nid;
2198
2199 nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order);
2200 mpol_cond_put(pol);
2201 gfp |= __GFP_COMP;
2202 page = alloc_page_interleave(gfp, order, nid);
2203 folio = (struct folio *)page;
2204 if (folio && order > 1)
2205 folio_prep_large_rmappable(folio);
2206 goto out;
2207 }
2208
2209 if (pol->mode == MPOL_PREFERRED_MANY) {
2210 struct page *page;
2211 2095
2212 node = policy_node(gfp, pol, node);2096 nodemask = policy_nodemask(gfp, pol, ilx, &nid);
2213 gfp |= __GFP_COMP;
2214 page = alloc_pages_preferred_many(gfp, order, node, pol);
2215 mpol_cond_put(pol);
2216 folio = (struct folio *)page;
2217 if (folio && order > 1)
2218 folio_prep_large_rmappable(folio);
2219 goto out;
2220 }
2221 2097
2222 if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) {2098 if (pol->mode == MPOL_PREFERRED_MANY)
2223 int hpage_node = node;2099 return alloc_pages_preferred_many(gfp, order, nid, nodemask);
2224 2100
2101 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
2102 /* filter "hugepage" allocation, unless from alloc_pages() */
2103 order == HPAGE_PMD_ORDER && ilx != NO_INTERLEAVE_INDEX) {
2225 /*2104 /*
2226 * For hugepage allocation and non-interleave policy which2105 * For hugepage allocation and non-interleave policy which
2227 * allows the current node (or other explicitly preferred2106 * allows the current node (or other explicitly preferred
@@ -2232,39 +2111,68 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, · 4 unmodified lines
2232 * If the policy is interleave or does not allow the current2111 * If the policy is interleave or does not allow the current
2233 * node in its nodemask, we allocate the standard way.2112 * node in its nodemask, we allocate the standard way.
2234 */2113 */
2235 if (pol->mode == MPOL_PREFERRED)2114 if (pol->mode != MPOL_INTERLEAVE &&
2236 hpage_node = first_node(pol->nodes);2115 (!nodemask || node_isset(nid, *nodemask))) {
2237
2238 nmask = policy_nodemask(gfp, pol);
2239 if (!nmask || node_isset(hpage_node, *nmask)) {
2240 mpol_cond_put(pol);
2241 /*2116 /*
2242 * First, try to allocate THP only on local node, but2117 * First, try to allocate THP only on local node, but
2243 * don't reclaim unnecessarily, just compact.2118 * don't reclaim unnecessarily, just compact.
2244 */2119 */
2245 folio = __folio_alloc_node(gfp | __GFP_THISNODE |2120 page = __alloc_pages_node(nid,
2246 __GFP_NORETRY, order, hpage_node);2121 gfp | __GFP_THISNODE | __GFP_NORETRY, order);
2247 2122 if (page || !(gfp & __GFP_DIRECT_RECLAIM))
2123 return page;
2248 /*2124 /*
2249 * If hugepage allocations are configured to always2125 * If hugepage allocations are configured to always
2250 * synchronous compact or the vma has been madvised2126 * synchronous compact or the vma has been madvised
2251 * to prefer hugepage backing, retry allowing remote2127 * to prefer hugepage backing, retry allowing remote
2252 * memory with both reclaim and compact as well.2128 * memory with both reclaim and compact as well.
2253 */2129 */
2254 if (!folio && (gfp & __GFP_DIRECT_RECLAIM))2130 }
2255 folio = __folio_alloc(gfp, order, hpage_node,2131 }
2256 nmask);
2257 2132
2258 goto out;2133 page = __alloc_pages(gfp, order, nid, nodemask);
2134
2135 if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) {
2136 /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */
2137 if (static_branch_likely(&vm_numa_stat_key) &&
2138 page_to_nid(page) == nid) {
2139 preempt_disable();
2140 __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT);
2141 preempt_enable();
2259 }2142 }
2260 }2143 }
2261 2144
2262 nmask = policy_nodemask(gfp, pol);2145 return page;
2263 preferred_nid = policy_node(gfp, pol, node);2146}
2264 folio = __folio_alloc(gfp, order, preferred_nid, nmask);2147
2148/**
2149 * vma_alloc_folio - Allocate a folio for a VMA.
2150 * @gfp: GFP flags.
2151 * @order: Order of the folio.
2152 * @vma: Pointer to VMA.
2153 * @addr: Virtual address of the allocation. Must be inside @vma.
2154 * @hugepage: Unused (was: For hugepages try only preferred node if possible).
2155 *
2156 * Allocate a folio for a specific address in @vma, using the appropriate
2157 * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the
2158 * VMA to prevent it from going away. Should be used for all allocations
2159 * for folios that will be mapped into user space, excepting hugetlbfs, and
2160 * excepting where direct use of alloc_pages_mpol() is more appropriate.
2161 *
2162 * Return: The folio on success or NULL if allocation fails.
2163 */
2164struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma,
2165 unsigned long addr, bool hugepage)
2166{
2167 struct mempolicy *pol;
2168 pgoff_t ilx;
2169 struct page *page;
2170
2171 pol = get_vma_policy(vma, addr, order, &ilx);
2172 page = alloc_pages_mpol(gfp | __GFP_COMP, order,
2173 pol, ilx, numa_node_id());
2265 mpol_cond_put(pol);2174 mpol_cond_put(pol);
2266out:2175 return page_rmappable_folio(page);
2267 return folio;
2268}2176}
2269EXPORT_SYMBOL(vma_alloc_folio);2177EXPORT_SYMBOL(vma_alloc_folio);
2270 2178
@@ -2282,40 +2190,25 @@ EXPORT_SYMBOL(vma_alloc_folio); · 11 unmodified lines
2282 * flags are used.2190 * flags are used.
2283 * Return: The page on success or NULL if allocation fails.2191 * Return: The page on success or NULL if allocation fails.
2284 */2192 */
2285struct page *alloc_pages(gfp_t gfp, unsigned order)2193struct page *alloc_pages(gfp_t gfp, unsigned int order)
2286{2194{
2287 struct mempolicy *pol = &default_policy;2195 struct mempolicy *pol = &default_policy;
2288 struct page *page;
2289
2290 if (!in_interrupt() && !(gfp & __GFP_THISNODE))
2291 pol = get_task_policy(current);
2292 2196
2293 /*2197 /*
2294 * No reference counting needed for current->mempolicy2198 * No reference counting needed for current->mempolicy
2295 * nor system default_policy2199 * nor system default_policy
2296 */2200 */
2297 if (pol->mode == MPOL_INTERLEAVE)2201 if (!in_interrupt() && !(gfp & __GFP_THISNODE))
2298 page = alloc_page_interleave(gfp, order, interleave_nodes(pol));2202 pol = get_task_policy(current);
2299 else if (pol->mode == MPOL_PREFERRED_MANY)
2300 page = alloc_pages_preferred_many(gfp, order,
2301 policy_node(gfp, pol, numa_node_id()), pol);
2302 else
2303 page = __alloc_pages(gfp, order,
2304 policy_node(gfp, pol, numa_node_id()),
2305 policy_nodemask(gfp, pol));
2306 2203
2307 return page;2204 return alloc_pages_mpol(gfp, order,
2205 pol, NO_INTERLEAVE_INDEX, numa_node_id());
2308}2206}
2309EXPORT_SYMBOL(alloc_pages);2207EXPORT_SYMBOL(alloc_pages);
2310 2208
2311struct folio *folio_alloc(gfp_t gfp, unsigned order)2209struct folio *folio_alloc(gfp_t gfp, unsigned int order)
2312{2210{
2313 struct page *page = alloc_pages(gfp | __GFP_COMP, order);2211 return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order));
2314 struct folio *folio = (struct folio *)page;
2315
2316 if (folio && order > 1)
2317 folio_prep_large_rmappable(folio);
2318 return folio;
2319}2212}
2320EXPORT_SYMBOL(folio_alloc);2213EXPORT_SYMBOL(folio_alloc);
2321 2214
@@ -2384,6 +2277,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 62 unmodified lines
2384 unsigned long nr_pages, struct page **page_array)2277 unsigned long nr_pages, struct page **page_array)
2385{2278{
2386 struct mempolicy *pol = &default_policy;2279 struct mempolicy *pol = &default_policy;
2280 nodemask_t *nodemask;
2281 int nid;
2387 2282
2388 if (!in_interrupt() && !(gfp & __GFP_THISNODE))2283 if (!in_interrupt() && !(gfp & __GFP_THISNODE))
2389 pol = get_task_policy(current);2284 pol = get_task_policy(current);
@@ -2396,14 +2291,15 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 6 unmodified lines
2396 return alloc_pages_bulk_array_preferred_many(gfp,2291 return alloc_pages_bulk_array_preferred_many(gfp,
2397 numa_node_id(), pol, nr_pages, page_array);2292 numa_node_id(), pol, nr_pages, page_array);
2398 2293
2399 return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()),2294 nid = numa_node_id();
2400 policy_nodemask(gfp, pol), nr_pages, NULL,2295 nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid);
2401 page_array);2296 return __alloc_pages_bulk(gfp, nid, nodemask,
2297 nr_pages, NULL, page_array);
2402}2298}
2403 2299
2404int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)2300int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
2405{2301{
2406 struct mempolicy *pol = mpol_dup(vma_policy(src));2302 struct mempolicy *pol = mpol_dup(src->vm_policy);
2407 2303
2408 if (IS_ERR(pol))2304 if (IS_ERR(pol))
2409 return PTR_ERR(pol);2305 return PTR_ERR(pol);
@@ -2488,8 +2384,8 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 78 unmodified lines
2488 * lookup first element intersecting start-end. Caller holds sp->lock for2384 * lookup first element intersecting start-end. Caller holds sp->lock for
2489 * reading or for writing2385 * reading or for writing
2490 */2386 */
2491static struct sp_node *2387static struct sp_node *sp_lookup(struct shared_policy *sp,
2492sp_lookup(struct shared_policy *sp, unsigned long start, unsigned long end)2388 pgoff_t start, pgoff_t end)
2493{2389{
2494 struct rb_node *n = sp->root.rb_node;2390 struct rb_node *n = sp->root.rb_node;
2495 2391
@@ -2540,13 +2436,11 @@ static void sp_insert(struct shared_policy *sp, struct sp_node *new) · 44 unmodified lines
2540 }2436 }
2541 rb_link_node(&new->nd, parent, p);2437 rb_link_node(&new->nd, parent, p);
2542 rb_insert_color(&new->nd, &sp->root);2438 rb_insert_color(&new->nd, &sp->root);
2543 pr_debug("inserting %lx-%lx: %d\n", new->start, new->end,
2544 new->policy ? new->policy->mode : 0);
2545}2439}
2546 2440
2547/* Find shared policy intersecting idx */2441/* Find shared policy intersecting idx */
2548struct mempolicy *2442struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
2549mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx)2443 pgoff_t idx)
2550{2444{
2551 struct mempolicy *pol = NULL;2445 struct mempolicy *pol = NULL;
2552 struct sp_node *sn;2446 struct sp_node *sn;
@@ -2570,39 +2464,38 @@ static void sp_free(struct sp_node *n) · 17 unmodified lines
2570}2464}
2571 2465
2572/**2466/**
2573 * mpol_misplaced - check whether current page node is valid in policy2467 * mpol_misplaced - check whether current folio node is valid in policy
2574 *2468 *
2575 * @page: page to be checked2469 * @folio: folio to be checked
2576 * @vma: vm area where page mapped2470 * @vma: vm area where folio mapped
2577 * @addr: virtual address where page mapped2471 * @addr: virtual address in @vma for shared policy lookup and interleave policy
2578 *2472 *
2579 * Lookup current policy node id for vma,addr and "compare to" page's2473 * Lookup current policy node id for vma,addr and "compare to" folio's
2580 * node id. Policy determination "mimics" alloc_page_vma().2474 * node id. Policy determination "mimics" alloc_page_vma().
2581 * Called from fault path where we know the vma and faulting address.2475 * Called from fault path where we know the vma and faulting address.
2582 *2476 *
2583 * Return: NUMA_NO_NODE if the page is in a node that is valid for this2477 * Return: NUMA_NO_NODE if the page is in a node that is valid for this
2584 * policy, or a suitable node ID to allocate a replacement page from.2478 * policy, or a suitable node ID to allocate a replacement folio from.
2585 */2479 */
2586int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long addr)2480int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma,
2481 unsigned long addr)
2587{2482{
2588 struct mempolicy *pol;2483 struct mempolicy *pol;
2484 pgoff_t ilx;
2589 struct zoneref *z;2485 struct zoneref *z;
2590 int curnid = page_to_nid(page);2486 int curnid = folio_nid(folio);
2591 unsigned long pgoff;
2592 int thiscpu = raw_smp_processor_id();2487 int thiscpu = raw_smp_processor_id();
2593 int thisnid = cpu_to_node(thiscpu);2488 int thisnid = cpu_to_node(thiscpu);
2594 int polnid = NUMA_NO_NODE;2489 int polnid = NUMA_NO_NODE;
2595 int ret = NUMA_NO_NODE;2490 int ret = NUMA_NO_NODE;
2596 2491
2597 pol = get_vma_policy(vma, addr);2492 pol = get_vma_policy(vma, addr, folio_order(folio), &ilx);
2598 if (!(pol->flags & MPOL_F_MOF))2493 if (!(pol->flags & MPOL_F_MOF))
2599 goto out;2494 goto out;
2600 2495
2601 switch (pol->mode) {2496 switch (pol->mode) {
2602 case MPOL_INTERLEAVE:2497 case MPOL_INTERLEAVE:
2603 pgoff = vma->vm_pgoff;2498 polnid = interleave_nid(pol, ilx);
2604 pgoff += (addr - vma->vm_start) >> PAGE_SHIFT;
2605 polnid = offset_il_node(pol, pgoff);
2606 break;2499 break;
2607 2500
2608 case MPOL_PREFERRED:2501 case MPOL_PREFERRED:
@@ -2643,11 +2536,12 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long · 34 unmodified lines
2643 BUG();2536 BUG();
2644 }2537 }
2645 2538
2646 /* Migrate the page towards the node whose CPU is referencing it */2539 /* Migrate the folio towards the node whose CPU is referencing it */
2647 if (pol->flags & MPOL_F_MORON) {2540 if (pol->flags & MPOL_F_MORON) {
2648 polnid = thisnid;2541 polnid = thisnid;
2649 2542
2650 if (!should_numa_migrate_memory(current, page, curnid, thiscpu))2543 if (!should_numa_migrate_memory(current, folio, curnid,
2544 thiscpu))
2651 goto out;2545 goto out;
2652 }2546 }
2653 2547
@@ -2678,7 +2572,6 @@ void mpol_put_task_policy(struct task_struct *task) · 24 unmodified lines
2678 2572
2679static void sp_delete(struct shared_policy *sp, struct sp_node *n)2573static void sp_delete(struct shared_policy *sp, struct sp_node *n)
2680{2574{
2681 pr_debug("deleting %lx-l%lx\n", n->start, n->end);
2682 rb_erase(&n->nd, &sp->root);2575 rb_erase(&n->nd, &sp->root);
2683 sp_free(n);2576 sp_free(n);
2684}2577}
@@ -2713,8 +2606,8 @@ static struct sp_node *sp_alloc(unsigned long start, unsigned long end, · 28 unmodified lines
2713}2606}
2714 2607
2715/* Replace a policy range. */2608/* Replace a policy range. */
2716static int shared_policy_replace(struct shared_policy *sp, unsigned long start,2609static int shared_policy_replace(struct shared_policy *sp, pgoff_t start,
2717 unsigned long end, struct sp_node *new)2610 pgoff_t end, struct sp_node *new)
2718{2611{
2719 struct sp_node *n;2612 struct sp_node *n;
2720 struct sp_node *n_new = NULL;2613 struct sp_node *n_new = NULL;
@@ -2797,30 +2690,30 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) · 76 unmodified lines
2797 rwlock_init(&sp->lock);2690 rwlock_init(&sp->lock);
2798 2691
2799 if (mpol) {2692 if (mpol) {
2800 struct vm_area_struct pvma;2693 struct sp_node *sn;
2801 struct mempolicy *new;2694 struct mempolicy *npol;
2802 NODEMASK_SCRATCH(scratch);2695 NODEMASK_SCRATCH(scratch);
2803 2696
2804 if (!scratch)2697 if (!scratch)
2805 goto put_mpol;2698 goto put_mpol;
2806 /* contextualize the tmpfs mount point mempolicy */2699
2807 new = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);2700 /* contextualize the tmpfs mount point mempolicy to this file */
2808 if (IS_ERR(new))2701 npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask);
2702 if (IS_ERR(npol))
2809 goto free_scratch; /* no valid nodemask intersection */2703 goto free_scratch; /* no valid nodemask intersection */
2810 2704
2811 task_lock(current);2705 task_lock(current);
2812 ret = mpol_set_nodemask(new, &mpol->w.user_nodemask, scratch);2706 ret = mpol_set_nodemask(npol, &mpol->w.user_nodemask, scratch);
2813 task_unlock(current);2707 task_unlock(current);
2814 if (ret)2708 if (ret)
2815 goto put_new;2709 goto put_npol;
2816 2710
2817 /* Create pseudo-vma that contains just the policy */2711 /* alloc node covering entire file; adds ref to file's npol */
2818 vma_init(&pvma, NULL);2712 sn = sp_alloc(0, MAX_LFS_FILESIZE >> PAGE_SHIFT, npol);
2819 pvma.vm_end = TASK_SIZE; /* policy covers entire file */2713 if (sn)
2820 mpol_set_shared_policy(sp, &pvma, new); /* adds ref */2714 sp_insert(sp, sn);
2821 2715put_npol:
2822put_new:2716 mpol_put(npol); /* drop initial ref on file's npol */
2823 mpol_put(new); /* drop initial ref */
2824free_scratch:2717free_scratch:
2825 NODEMASK_SCRATCH_FREE(scratch);2718 NODEMASK_SCRATCH_FREE(scratch);
2826put_mpol:2719put_mpol:
@@ -2828,46 +2721,40 @@ put_mpol: · 1 unmodified line
2828 }2721 }
2829}2722}
2830 2723
2831int mpol_set_shared_policy(struct shared_policy *info,2724int mpol_set_shared_policy(struct shared_policy *sp,
2832 struct vm_area_struct *vma, struct mempolicy *npol)2725 struct vm_area_struct *vma, struct mempolicy *pol)
2833{2726{
2834 int err;2727 int err;
2835 struct sp_node *new = NULL;2728 struct sp_node *new = NULL;
2836 unsigned long sz = vma_pages(vma);2729 unsigned long sz = vma_pages(vma);
2837 2730
2838 pr_debug("set_shared_policy %lx sz %lu %d %d %lx\n",2731 if (pol) {
2839 vma->vm_pgoff,2732 new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, pol);
2840 sz, npol ? npol->mode : -1,
2841 npol ? npol->flags : -1,
2842 npol ? nodes_addr(npol->nodes)[0] : NUMA_NO_NODE);
2843
2844 if (npol) {
2845 new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, npol);
2846 if (!new)2733 if (!new)
2847 return -ENOMEM;2734 return -ENOMEM;
2848 }2735 }
2849 err = shared_policy_replace(info, vma->vm_pgoff, vma->vm_pgoff+sz, new);2736 err = shared_policy_replace(sp, vma->vm_pgoff, vma->vm_pgoff + sz, new);
2850 if (err && new)2737 if (err && new)
2851 sp_free(new);2738 sp_free(new);
2852 return err;2739 return err;
2853}2740}
2854 2741
2855/* Free a backing policy store on inode delete. */2742/* Free a backing policy store on inode delete. */
2856void mpol_free_shared_policy(struct shared_policy *p)2743void mpol_free_shared_policy(struct shared_policy *sp)
2857{2744{
2858 struct sp_node *n;2745 struct sp_node *n;
2859 struct rb_node *next;2746 struct rb_node *next;
2860 2747
2861 if (!p->root.rb_node)2748 if (!sp->root.rb_node)
2862 return;2749 return;
2863 write_lock(&p->lock);2750 write_lock(&sp->lock);
2864 next = rb_first(&p->root);2751 next = rb_first(&sp->root);
2865 while (next) {2752 while (next) {
2866 n = rb_entry(next, struct sp_node, nd);2753 n = rb_entry(next, struct sp_node, nd);
2867 next = rb_next(&n->nd);2754 next = rb_next(&n->nd);
2868 sp_delete(p, n);2755 sp_delete(sp, n);
2869 }2756 }
2870 write_unlock(&p->lock);2757 write_unlock(&sp->lock);
2871}2758}
2872 2759
2873#ifdef CONFIG_NUMA_BALANCING2760#ifdef CONFIG_NUMA_BALANCING
@@ -2917,7 +2804,6 @@ static inline void __init check_numabalancing_enable(void) · 43 unmodified lines
2917}2804}
2918#endif /* CONFIG_NUMA_BALANCING */2805#endif /* CONFIG_NUMA_BALANCING */
2919 2806
2920/* assumes fs == KERNEL_DS */
2921void __init numa_policy_init(void)2807void __init numa_policy_init(void)
2922{2808{
2923 nodemask_t interleave_nodes;2809 nodemask_t interleave_nodes;
@@ -2980,7 +2866,6 @@ void numa_default_policy(void) · 56 unmodified lines
2980/*2866/*
2981 * Parse and format mempolicy from/to strings2867 * Parse and format mempolicy from/to strings
2982 */2868 */
2983
2984static const char * const policy_modes[] =2869static const char * const policy_modes[] =
2985{2870{
2986 [MPOL_DEFAULT] = "default",2871 [MPOL_DEFAULT] = "default",
@@ -2991,7 +2876,6 @@ static const char * const policy_modes[] = · 4 unmodified lines
2991 [MPOL_PREFERRED_MANY] = "prefer (many)",2876 [MPOL_PREFERRED_MANY] = "prefer (many)",
2992};2877};
2993 2878
2994
2995#ifdef CONFIG_TMPFS2879#ifdef CONFIG_TMPFS
2996/**2880/**
2997 * mpol_parse_str - parse string to mempolicy, for tmpfs mpol mount option.2881 * mpol_parse_str - parse string to mempolicy, for tmpfs mpol mount option.
mm/migrate.c+117−108
@@ -157,8 +157,8 @@ void putback_movable_pages(struct list_head *l) · 156 unmodified lines
157 list_del(&folio->lru);157 list_del(&folio->lru);
158 /*158 /*
159 * We isolated non-lru movable folio so here we can use159 * We isolated non-lru movable folio so here we can use
160 * __PageMovable because LRU folio's mapping cannot have160 * __folio_test_movable because LRU folio's mapping cannot
161 * PAGE_MAPPING_MOVABLE.161 * have PAGE_MAPPING_MOVABLE.
162 */162 */
163 if (unlikely(__folio_test_movable(folio))) {163 if (unlikely(__folio_test_movable(folio))) {
164 VM_BUG_ON_FOLIO(!folio_test_isolated(folio), folio);164 VM_BUG_ON_FOLIO(!folio_test_isolated(folio), folio);
@@ -249,7 +249,7 @@ static bool remove_migration_pte(struct folio *folio, · 84 unmodified lines
249 249
250 pte = arch_make_huge_pte(pte, shift, vma->vm_flags);250 pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
251 if (folio_test_anon(folio))251 if (folio_test_anon(folio))
252 hugepage_add_anon_rmap(new, vma, pvmw.address,252 hugepage_add_anon_rmap(folio, vma, pvmw.address,
253 rmap_flags);253 rmap_flags);
254 else254 else
255 page_dup_file_rmap(new, true);255 page_dup_file_rmap(new, true);
@@ -524,7 +524,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 268 unmodified lines
524 int expected_count;524 int expected_count;
525 525
526 xas_lock_irq(&xas);526 xas_lock_irq(&xas);
527 expected_count = 2 + folio_has_private(src);527 expected_count = folio_expected_refs(mapping, src);
528 if (!folio_ref_freeze(src, expected_count)) {528 if (!folio_ref_freeze(src, expected_count)) {
529 xas_unlock_irq(&xas);529 xas_unlock_irq(&xas);
530 return -EAGAIN;530 return -EAGAIN;
@@ -533,11 +533,11 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 2 unmodified lines
533 dst->index = src->index;533 dst->index = src->index;
534 dst->mapping = src->mapping;534 dst->mapping = src->mapping;
535 535
536 folio_get(dst);536 folio_ref_add(dst, folio_nr_pages(dst));
537 537
538 xas_store(&xas, dst);538 xas_store(&xas, dst);
539 539
540 folio_ref_unfreeze(src, expected_count - 1);540 folio_ref_unfreeze(src, expected_count - folio_nr_pages(src));
541 541
542 xas_unlock_irq(&xas);542 xas_unlock_irq(&xas);
543 543
@@ -588,20 +588,20 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 44 unmodified lines
588 * Copy NUMA information to the new page, to prevent over-eager588 * Copy NUMA information to the new page, to prevent over-eager
589 * future migrations of this same page.589 * future migrations of this same page.
590 */590 */
591 cpupid = page_cpupid_xchg_last(&folio->page, -1);591 cpupid = folio_xchg_last_cpupid(folio, -1);
592 /*592 /*
593 * For memory tiering mode, when migrate between slow and fast593 * For memory tiering mode, when migrate between slow and fast
594 * memory node, reset cpupid, because that is used to record594 * memory node, reset cpupid, because that is used to record
595 * page access time in slow memory node.595 * page access time in slow memory node.
596 */596 */
597 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) {597 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) {
598 bool f_toptier = node_is_toptier(page_to_nid(&folio->page));598 bool f_toptier = node_is_toptier(folio_nid(folio));
599 bool t_toptier = node_is_toptier(page_to_nid(&newfolio->page));599 bool t_toptier = node_is_toptier(folio_nid(newfolio));
600 600
601 if (f_toptier != t_toptier)601 if (f_toptier != t_toptier)
602 cpupid = -1;602 cpupid = -1;
603 }603 }
604 page_cpupid_xchg_last(&newfolio->page, cpupid);604 folio_xchg_last_cpupid(newfolio, cpupid);
605 605
606 folio_migrate_ksm(newfolio, folio);606 folio_migrate_ksm(newfolio, folio);
607 /*607 /*
@@ -633,8 +633,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 25 unmodified lines
633 633
634 folio_copy_owner(newfolio, folio);634 folio_copy_owner(newfolio, folio);
635 635
636 if (!folio_test_hugetlb(folio))636 mem_cgroup_migrate(folio, newfolio);
637 mem_cgroup_migrate(folio, newfolio);
638}637}
639EXPORT_SYMBOL(folio_migrate_flags);638EXPORT_SYMBOL(folio_migrate_flags);
640 639
@@ -946,7 +945,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 305 unmodified lines
946 enum migrate_mode mode)945 enum migrate_mode mode)
947{946{
948 int rc = -EAGAIN;947 int rc = -EAGAIN;
949 bool is_lru = !__PageMovable(&src->page);948 bool is_lru = !__folio_test_movable(src);
950 949
951 VM_BUG_ON_FOLIO(!folio_test_locked(src), src);950 VM_BUG_ON_FOLIO(!folio_test_locked(src), src);
952 VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);951 VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst);
@@ -993,7 +992,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 40 unmodified lines
993 * src is freed; but stats require that PageAnon be left as PageAnon.992 * src is freed; but stats require that PageAnon be left as PageAnon.
994 */993 */
995 if (rc == MIGRATEPAGE_SUCCESS) {994 if (rc == MIGRATEPAGE_SUCCESS) {
996 if (__PageMovable(&src->page)) {995 if (__folio_test_movable(src)) {
997 VM_BUG_ON_FOLIO(!folio_test_isolated(src), src);996 VM_BUG_ON_FOLIO(!folio_test_isolated(src), src);
998 997
999 /*998 /*
@@ -1028,22 +1027,28 @@ union migration_ptr { · 28 unmodified lines
1028 struct anon_vma *anon_vma;1027 struct anon_vma *anon_vma;
1029 struct address_space *mapping;1028 struct address_space *mapping;
1030};1029};
1030
1031enum {
1032 PAGE_WAS_MAPPED = BIT(0),
1033 PAGE_WAS_MLOCKED = BIT(1),
1034};
1035
1031static void __migrate_folio_record(struct folio *dst,1036static void __migrate_folio_record(struct folio *dst,
1032 unsigned long page_was_mapped,1037 unsigned long old_page_state,
1033 struct anon_vma *anon_vma)1038 struct anon_vma *anon_vma)
1034{1039{
1035 union migration_ptr ptr = { .anon_vma = anon_vma };1040 union migration_ptr ptr = { .anon_vma = anon_vma };
1036 dst->mapping = ptr.mapping;1041 dst->mapping = ptr.mapping;
1037 dst->private = (void *)page_was_mapped;1042 dst->private = (void *)old_page_state;
1038}1043}
1039 1044
1040static void __migrate_folio_extract(struct folio *dst,1045static void __migrate_folio_extract(struct folio *dst,
1041 int *page_was_mappedp,1046 int *old_page_state,
1042 struct anon_vma **anon_vmap)1047 struct anon_vma **anon_vmap)
1043{1048{
1044 union migration_ptr ptr = { .mapping = dst->mapping };1049 union migration_ptr ptr = { .mapping = dst->mapping };
1045 *anon_vmap = ptr.anon_vma;1050 *anon_vmap = ptr.anon_vma;
1046 *page_was_mappedp = (unsigned long)dst->private;1051 *old_page_state = (unsigned long)dst->private;
1047 dst->mapping = NULL;1052 dst->mapping = NULL;
1048 dst->private = NULL;1053 dst->private = NULL;
1049}1054}
@@ -1085,7 +1090,7 @@ static void migrate_folio_done(struct folio *src, · 35 unmodified lines
1085 /*1090 /*
1086 * Compaction can migrate also non-LRU pages which are1091 * Compaction can migrate also non-LRU pages which are
1087 * not accounted to NR_ISOLATED_*. They can be recognized1092 * not accounted to NR_ISOLATED_*. They can be recognized
1088 * as __PageMovable1093 * as __folio_test_movable
1089 */1094 */
1090 if (likely(!__folio_test_movable(src)))1095 if (likely(!__folio_test_movable(src)))
1091 mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +1096 mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
@@ -1104,9 +1109,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 12 unmodified lines
1104{1109{
1105 struct folio *dst;1110 struct folio *dst;
1106 int rc = -EAGAIN;1111 int rc = -EAGAIN;
1107 int page_was_mapped = 0;1112 int old_page_state = 0;
1108 struct anon_vma *anon_vma = NULL;1113 struct anon_vma *anon_vma = NULL;
1109 bool is_lru = !__PageMovable(&src->page);1114 bool is_lru = !__folio_test_movable(src);
1110 bool locked = false;1115 bool locked = false;
1111 bool dst_locked = false;1116 bool dst_locked = false;
1112 1117
@@ -1158,6 +1163,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 45 unmodified lines
1158 folio_lock(src);1163 folio_lock(src);
1159 }1164 }
1160 locked = true;1165 locked = true;
1166 if (folio_test_mlocked(src))
1167 old_page_state |= PAGE_WAS_MLOCKED;
1161 1168
1162 if (folio_test_writeback(src)) {1169 if (folio_test_writeback(src)) {
1163 /*1170 /*
@@ -1207,7 +1214,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 43 unmodified lines
1207 dst_locked = true;1214 dst_locked = true;
1208 1215
1209 if (unlikely(!is_lru)) {1216 if (unlikely(!is_lru)) {
1210 __migrate_folio_record(dst, page_was_mapped, anon_vma);1217 __migrate_folio_record(dst, old_page_state, anon_vma);
1211 return MIGRATEPAGE_UNMAP;1218 return MIGRATEPAGE_UNMAP;
1212 }1219 }
1213 1220
@@ -1233,11 +1240,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 19 unmodified lines
1233 VM_BUG_ON_FOLIO(folio_test_anon(src) &&1240 VM_BUG_ON_FOLIO(folio_test_anon(src) &&
1234 !folio_test_ksm(src) && !anon_vma, src);1241 !folio_test_ksm(src) && !anon_vma, src);
1235 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);1242 try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
1236 page_was_mapped = 1;1243 old_page_state |= PAGE_WAS_MAPPED;
1237 }1244 }
1238 1245
1239 if (!folio_mapped(src)) {1246 if (!folio_mapped(src)) {
1240 __migrate_folio_record(dst, page_was_mapped, anon_vma);1247 __migrate_folio_record(dst, old_page_state, anon_vma);
1241 return MIGRATEPAGE_UNMAP;1248 return MIGRATEPAGE_UNMAP;
1242 }1249 }
1243 1250
@@ -1249,7 +1256,8 @@ out: · 5 unmodified lines
1249 if (rc == -EAGAIN)1256 if (rc == -EAGAIN)
1250 ret = NULL;1257 ret = NULL;
1251 1258
1252 migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret);1259 migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1260 anon_vma, locked, ret);
1253 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);1261 migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private);
1254 1262
1255 return rc;1263 return rc;
@@ -1262,12 +1270,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 6 unmodified lines
1262 struct list_head *ret)1270 struct list_head *ret)
1263{1271{
1264 int rc;1272 int rc;
1265 int page_was_mapped = 0;1273 int old_page_state = 0;
1266 struct anon_vma *anon_vma = NULL;1274 struct anon_vma *anon_vma = NULL;
1267 bool is_lru = !__PageMovable(&src->page);1275 bool is_lru = !__folio_test_movable(src);
1268 struct list_head *prev;1276 struct list_head *prev;
1269 1277
1270 __migrate_folio_extract(dst, &page_was_mapped, &anon_vma);1278 __migrate_folio_extract(dst, &old_page_state, &anon_vma);
1271 prev = dst->lru.prev;1279 prev = dst->lru.prev;
1272 list_del(&dst->lru);1280 list_del(&dst->lru);
1273 1281
@@ -1288,10 +1296,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 14 unmodified lines
1288 * isolated from the unevictable LRU: but this case is the easiest.1296 * isolated from the unevictable LRU: but this case is the easiest.
1289 */1297 */
1290 folio_add_lru(dst);1298 folio_add_lru(dst);
1291 if (page_was_mapped)1299 if (old_page_state & PAGE_WAS_MLOCKED)
1292 lru_add_drain();1300 lru_add_drain();
1293 1301
1294 if (page_was_mapped)1302 if (old_page_state & PAGE_WAS_MAPPED)
1295 remove_migration_ptes(src, dst, false);1303 remove_migration_ptes(src, dst, false);
1296 1304
1297out_unlock_both:1305out_unlock_both:
@@ -1323,11 +1331,12 @@ out: · 25 unmodified lines
1323 */1331 */
1324 if (rc == -EAGAIN) {1332 if (rc == -EAGAIN) {
1325 list_add(&dst->lru, prev);1333 list_add(&dst->lru, prev);
1326 __migrate_folio_record(dst, page_was_mapped, anon_vma);1334 __migrate_folio_record(dst, old_page_state, anon_vma);
1327 return rc;1335 return rc;
1328 }1336 }
1329 1337
1330 migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret);1338 migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED,
1339 anon_vma, true, ret);
1331 migrate_folio_undo_dst(dst, true, put_new_folio, private);1340 migrate_folio_undo_dst(dst, true, put_new_folio, private);
1332 1341
1333 return rc;1342 return rc;
@@ -1496,6 +1505,7 @@ struct migrate_pages_stats { · 162 unmodified lines
1496 int nr_thp_succeeded; /* THP migrated successfully */1505 int nr_thp_succeeded; /* THP migrated successfully */
1497 int nr_thp_failed; /* THP failed to be migrated */1506 int nr_thp_failed; /* THP failed to be migrated */
1498 int nr_thp_split; /* THP split before migrating */1507 int nr_thp_split; /* THP split before migrating */
1508 int nr_split; /* Large folio (include THP) split before migrating */
1499};1509};
1500 1510
1501/*1511/*
@@ -1615,6 +1625,7 @@ static int migrate_pages_batch(struct list_head *from, · 113 unmodified lines
1615 int nr_retry_pages = 0;1625 int nr_retry_pages = 0;
1616 int pass = 0;1626 int pass = 0;
1617 bool is_thp = false;1627 bool is_thp = false;
1628 bool is_large = false;
1618 struct folio *folio, *folio2, *dst = NULL, *dst2;1629 struct folio *folio, *folio2, *dst = NULL, *dst2;
1619 int rc, rc_saved = 0, nr_pages;1630 int rc, rc_saved = 0, nr_pages;
1620 LIST_HEAD(unmap_folios);1631 LIST_HEAD(unmap_folios);
@@ -1630,7 +1641,8 @@ static int migrate_pages_batch(struct list_head *from, · 9 unmodified lines
1630 nr_retry_pages = 0;1641 nr_retry_pages = 0;
1631 1642
1632 list_for_each_entry_safe(folio, folio2, from, lru) {1643 list_for_each_entry_safe(folio, folio2, from, lru) {
1633 is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);1644 is_large = folio_test_large(folio);
1645 is_thp = is_large && folio_test_pmd_mappable(folio);
1634 nr_pages = folio_nr_pages(folio);1646 nr_pages = folio_nr_pages(folio);
1635 1647
1636 cond_resched();1648 cond_resched();
@@ -1650,6 +1662,7 @@ static int migrate_pages_batch(struct list_head *from, · 13 unmodified lines
1650 stats->nr_thp_failed++;1662 stats->nr_thp_failed++;
1651 if (!try_split_folio(folio, split_folios)) {1663 if (!try_split_folio(folio, split_folios)) {
1652 stats->nr_thp_split++;1664 stats->nr_thp_split++;
1665 stats->nr_split++;
1653 continue;1666 continue;
1654 }1667 }
1655 stats->nr_failed_pages += nr_pages;1668 stats->nr_failed_pages += nr_pages;
@@ -1678,11 +1691,12 @@ static int migrate_pages_batch(struct list_head *from, · 22 unmodified lines
1678 nr_failed++;1691 nr_failed++;
1679 stats->nr_thp_failed += is_thp;1692 stats->nr_thp_failed += is_thp;
1680 /* Large folio NUMA faulting doesn't split to retry. */1693 /* Large folio NUMA faulting doesn't split to retry. */
1681 if (folio_test_large(folio) && !nosplit) {1694 if (is_large && !nosplit) {
1682 int ret = try_split_folio(folio, split_folios);1695 int ret = try_split_folio(folio, split_folios);
1683 1696
1684 if (!ret) {1697 if (!ret) {
1685 stats->nr_thp_split += is_thp;1698 stats->nr_thp_split += is_thp;
1699 stats->nr_split++;
1686 break;1700 break;
1687 } else if (reason == MR_LONGTERM_PIN &&1701 } else if (reason == MR_LONGTERM_PIN &&
1688 ret == -EAGAIN) {1702 ret == -EAGAIN) {
@@ -1795,12 +1809,12 @@ out: · 106 unmodified lines
1795 dst = list_first_entry(&dst_folios, struct folio, lru);1809 dst = list_first_entry(&dst_folios, struct folio, lru);
1796 dst2 = list_next_entry(dst, lru);1810 dst2 = list_next_entry(dst, lru);
1797 list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) {1811 list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) {
1798 int page_was_mapped = 0;1812 int old_page_state = 0;
1799 struct anon_vma *anon_vma = NULL;1813 struct anon_vma *anon_vma = NULL;
1800 1814
1801 __migrate_folio_extract(dst, &page_was_mapped, &anon_vma);1815 __migrate_folio_extract(dst, &old_page_state, &anon_vma);
1802 migrate_folio_undo_src(folio, page_was_mapped, anon_vma,1816 migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
1803 true, ret_folios);1817 anon_vma, true, ret_folios);
1804 list_del(&dst->lru);1818 list_del(&dst->lru);
1805 migrate_folio_undo_dst(dst, true, put_new_folio, private);1819 migrate_folio_undo_dst(dst, true, put_new_folio, private);
1806 dst = dst2;1820 dst = dst2;
@@ -1828,6 +1842,7 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 21 unmodified lines
1828 stats->nr_succeeded += astats.nr_succeeded;1842 stats->nr_succeeded += astats.nr_succeeded;
1829 stats->nr_thp_succeeded += astats.nr_thp_succeeded;1843 stats->nr_thp_succeeded += astats.nr_thp_succeeded;
1830 stats->nr_thp_split += astats.nr_thp_split;1844 stats->nr_thp_split += astats.nr_thp_split;
1845 stats->nr_split += astats.nr_split;
1831 if (rc < 0) {1846 if (rc < 0) {
1832 stats->nr_failed_pages += astats.nr_failed_pages;1847 stats->nr_failed_pages += astats.nr_failed_pages;
1833 stats->nr_thp_failed += astats.nr_thp_failed;1848 stats->nr_thp_failed += astats.nr_thp_failed;
@@ -1835,7 +1850,11 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 1 unmodified line
1835 return rc;1850 return rc;
1836 }1851 }
1837 stats->nr_thp_failed += astats.nr_thp_split;1852 stats->nr_thp_failed += astats.nr_thp_split;
1838 nr_failed += astats.nr_thp_split;1853 /*
1854 * Do not count rc, as pages will be retried below.
1855 * Count nr_split only, since it includes nr_thp_split.
1856 */
1857 nr_failed += astats.nr_split;
1839 /*1858 /*
1840 * Fall back to migrate all failed folios one by one synchronously. All1859 * Fall back to migrate all failed folios one by one synchronously. All
1841 * failed folios except split THPs will be retried, so their failure1860 * failed folios except split THPs will be retried, so their failure
@@ -1970,7 +1989,8 @@ out: · 128 unmodified lines
1970 count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);1989 count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);
1971 trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,1990 trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,
1972 stats.nr_thp_succeeded, stats.nr_thp_failed,1991 stats.nr_thp_succeeded, stats.nr_thp_failed,
1973 stats.nr_thp_split, mode, reason);1992 stats.nr_thp_split, stats.nr_split, mode,
1993 reason);
1974 1994
1975 if (ret_succeeded)1995 if (ret_succeeded)
1976 *ret_succeeded = stats.nr_succeeded;1996 *ret_succeeded = stats.nr_succeeded;
@@ -2029,8 +2049,7 @@ static int store_status(int __user *status, int start, int value, int nr) · 52 unmodified lines
2029 return 0;2049 return 0;
2030}2050}
2031 2051
2032static int do_move_pages_to_node(struct mm_struct *mm,2052static int do_move_pages_to_node(struct list_head *pagelist, int node)
2033 struct list_head *pagelist, int node)
2034{2053{
2035 int err;2054 int err;
2036 struct migration_target_control mtc = {2055 struct migration_target_control mtc = {
@@ -2060,8 +2079,8 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 23 unmodified lines
2060 struct vm_area_struct *vma;2079 struct vm_area_struct *vma;
2061 unsigned long addr;2080 unsigned long addr;
2062 struct page *page;2081 struct page *page;
2082 struct folio *folio;
2063 int err;2083 int err;
2064 bool isolated;
2065 2084
2066 mmap_read_lock(mm);2085 mmap_read_lock(mm);
2067 addr = (unsigned long)untagged_addr_remote(mm, p);2086 addr = (unsigned long)untagged_addr_remote(mm, p);
@@ -2082,51 +2101,44 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 14 unmodified lines
2082 if (!page)2101 if (!page)
2083 goto out;2102 goto out;
2084 2103
2085 if (is_zone_device_page(page))2104 folio = page_folio(page);
2086 goto out_putpage;2105 if (folio_is_zone_device(folio))
2106 goto out_putfolio;
2087 2107
2088 err = 0;2108 err = 0;
2089 if (page_to_nid(page) == node)2109 if (folio_nid(folio) == node)
2090 goto out_putpage;2110 goto out_putfolio;
2091 2111
2092 err = -EACCES;2112 err = -EACCES;
2093 if (page_mapcount(page) > 1 && !migrate_all)2113 if (page_mapcount(page) > 1 && !migrate_all)
2094 goto out_putpage;2114 goto out_putfolio;
2095 2115
2096 if (PageHuge(page)) {2116 err = -EBUSY;
2097 if (PageHead(page)) {2117 if (folio_test_hugetlb(folio)) {
2098 isolated = isolate_hugetlb(page_folio(page), pagelist);2118 if (isolate_hugetlb(folio, pagelist))
2099 err = isolated ? 1 : -EBUSY;2119 err = 1;
2100 }
2101 } else {2120 } else {
2102 struct page *head;2121 if (!folio_isolate_lru(folio))
2103 2122 goto out_putfolio;
2104 head = compound_head(page);
2105 isolated = isolate_lru_page(head);
2106 if (!isolated) {
2107 err = -EBUSY;
2108 goto out_putpage;
2109 }
2110 2123
2111 err = 1;2124 err = 1;
2112 list_add_tail(&head->lru, pagelist);2125 list_add_tail(&folio->lru, pagelist);
2113 mod_node_page_state(page_pgdat(head),2126 node_stat_mod_folio(folio,
2114 NR_ISOLATED_ANON + page_is_file_lru(head),2127 NR_ISOLATED_ANON + folio_is_file_lru(folio),
2115 thp_nr_pages(head));2128 folio_nr_pages(folio));
2116 }2129 }
2117out_putpage:2130out_putfolio:
2118 /*2131 /*
2119 * Either remove the duplicate refcount from2132 * Either remove the duplicate refcount from folio_isolate_lru()
2120 * isolate_lru_page() or drop the page ref if it was2133 * or drop the folio ref if it was not isolated.
2121 * not isolated.
2122 */2134 */
2123 put_page(page);2135 folio_put(folio);
2124out:2136out:
2125 mmap_read_unlock(mm);2137 mmap_read_unlock(mm);
2126 return err;2138 return err;
2127}2139}
2128 2140
2129static int move_pages_and_store_status(struct mm_struct *mm, int node,2141static int move_pages_and_store_status(int node,
2130 struct list_head *pagelist, int __user *status,2142 struct list_head *pagelist, int __user *status,
2131 int start, int i, unsigned long nr_pages)2143 int start, int i, unsigned long nr_pages)
2132{2144{
@@ -2135,7 +2147,7 @@ static int move_pages_and_store_status(struct mm_struct *mm, int node, · 2 unmodified lines
2135 if (list_empty(pagelist))2147 if (list_empty(pagelist))
2136 return 0;2148 return 0;
2137 2149
2138 err = do_move_pages_to_node(mm, pagelist, node);2150 err = do_move_pages_to_node(pagelist, node);
2139 if (err) {2151 if (err) {
2140 /*2152 /*
2141 * Positive err means the number of failed2153 * Positive err means the number of failed
@@ -2203,7 +2215,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 61 unmodified lines
2203 current_node = node;2215 current_node = node;
2204 start = i;2216 start = i;
2205 } else if (node != current_node) {2217 } else if (node != current_node) {
2206 err = move_pages_and_store_status(mm, current_node,2218 err = move_pages_and_store_status(current_node,
2207 &pagelist, status, start, i, nr_pages);2219 &pagelist, status, start, i, nr_pages);
2208 if (err)2220 if (err)
2209 goto out;2221 goto out;
@@ -2238,7 +2250,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 28 unmodified lines
2238 if (err)2250 if (err)
2239 goto out_flush;2251 goto out_flush;
2240 2252
2241 err = move_pages_and_store_status(mm, current_node, &pagelist,2253 err = move_pages_and_store_status(current_node, &pagelist,
2242 status, start, i, nr_pages);2254 status, start, i, nr_pages);
2243 if (err) {2255 if (err) {
2244 /* We have accounted for page i */2256 /* We have accounted for page i */
@@ -2250,7 +2262,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 5 unmodified lines
2250 }2262 }
2251out_flush:2263out_flush:
2252 /* Make sure we do not overwrite the existing error */2264 /* Make sure we do not overwrite the existing error */
2253 err1 = move_pages_and_store_status(mm, current_node, &pagelist,2265 err1 = move_pages_and_store_status(current_node, &pagelist,
2254 status, start, i, nr_pages);2266 status, start, i, nr_pages);
2255 if (err >= 0)2267 if (err >= 0)
2256 err = err1;2268 err = err1;
@@ -2491,16 +2503,9 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src, · 234 unmodified lines
2491 return __folio_alloc_node(gfp, order, nid);2503 return __folio_alloc_node(gfp, order, nid);
2492}2504}
2493 2505
2494static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page)2506static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
2495{2507{
2496 int nr_pages = thp_nr_pages(page);2508 int nr_pages = folio_nr_pages(folio);
2497 int order = compound_order(page);
2498
2499 VM_BUG_ON_PAGE(order && !PageTransHuge(page), page);
2500
2501 /* Do not migrate THP mapped by multiple processes */
2502 if (PageTransHuge(page) && total_mapcount(page) > 1)
2503 return 0;
2504 2509
2505 /* Avoid migrating to a node that is nearly full */2510 /* Avoid migrating to a node that is nearly full */
2506 if (!migrate_balanced_pgdat(pgdat, nr_pages)) {2511 if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
@@ -2512,75 +2517,79 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · 5 unmodified lines
2512 if (managed_zone(pgdat->node_zones + z))2517 if (managed_zone(pgdat->node_zones + z))
2513 break;2518 break;
2514 }2519 }
2515 wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE);2520 wakeup_kswapd(pgdat->node_zones + z, 0,
2521 folio_order(folio), ZONE_MOVABLE);
2516 return 0;2522 return 0;
2517 }2523 }
2518 2524
2519 if (!isolate_lru_page(page))2525 if (!folio_isolate_lru(folio))
2520 return 0;2526 return 0;
2521 2527
2522 mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_is_file_lru(page),2528 node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
2523 nr_pages);2529 nr_pages);
2524 2530
2525 /*2531 /*
2526 * Isolating the page has taken another reference, so the2532 * Isolating the folio has taken another reference, so the
2527 * caller's reference can be safely dropped without the page2533 * caller's reference can be safely dropped without the folio
2528 * disappearing underneath us during migration.2534 * disappearing underneath us during migration.
2529 */2535 */
2530 put_page(page);2536 folio_put(folio);
2531 return 1;2537 return 1;
2532}2538}
2533 2539
2534/*2540/*
2535 * Attempt to migrate a misplaced page to the specified destination2541 * Attempt to migrate a misplaced folio to the specified destination
2536 * node. Caller is expected to have an elevated reference count on2542 * node. Caller is expected to have an elevated reference count on
2537 * the page that will be dropped by this function before returning.2543 * the folio that will be dropped by this function before returning.
2538 */2544 */
2539int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,2545int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
2540 int node)2546 int node)
2541{2547{
2542 pg_data_t *pgdat = NODE_DATA(node);2548 pg_data_t *pgdat = NODE_DATA(node);
2543 int isolated;2549 int isolated;
2544 int nr_remaining;2550 int nr_remaining;
2545 unsigned int nr_succeeded;2551 unsigned int nr_succeeded;
2546 LIST_HEAD(migratepages);2552 LIST_HEAD(migratepages);
2547 int nr_pages = thp_nr_pages(page);2553 int nr_pages = folio_nr_pages(folio);
2548 2554
2549 /*2555 /*
2550 * Don't migrate file pages that are mapped in multiple processes2556 * Don't migrate file folios that are mapped in multiple processes
2551 * with execute permissions as they are probably shared libraries.2557 * with execute permissions as they are probably shared libraries.
2558 * To check if the folio is shared, ideally we want to make sure
2559 * every page is mapped to the same process. Doing that is very
2560 * expensive, so check the estimated mapcount of the folio instead.
2552 */2561 */
2553 if (page_mapcount(page) != 1 && page_is_file_lru(page) &&2562 if (folio_estimated_sharers(folio) != 1 && folio_is_file_lru(folio) &&
2554 (vma->vm_flags & VM_EXEC))2563 (vma->vm_flags & VM_EXEC))
2555 goto out;2564 goto out;
2556 2565
2557 /*2566 /*
2558 * Also do not migrate dirty pages as not all filesystems can move2567 * Also do not migrate dirty folios as not all filesystems can move
2559 * dirty pages in MIGRATE_ASYNC mode which is a waste of cycles.2568 * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles.
2560 */2569 */
2561 if (page_is_file_lru(page) && PageDirty(page))2570 if (folio_is_file_lru(folio) && folio_test_dirty(folio))
2562 goto out;2571 goto out;
2563 2572
2564 isolated = numamigrate_isolate_page(pgdat, page);2573 isolated = numamigrate_isolate_folio(pgdat, folio);
2565 if (!isolated)2574 if (!isolated)
2566 goto out;2575 goto out;
2567 2576
2568 list_add(&page->lru, &migratepages);2577 list_add(&folio->lru, &migratepages);
2569 nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,2578 nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
2570 NULL, node, MIGRATE_ASYNC,2579 NULL, node, MIGRATE_ASYNC,
2571 MR_NUMA_MISPLACED, &nr_succeeded);2580 MR_NUMA_MISPLACED, &nr_succeeded);
2572 if (nr_remaining) {2581 if (nr_remaining) {
2573 if (!list_empty(&migratepages)) {2582 if (!list_empty(&migratepages)) {
2574 list_del(&page->lru);2583 list_del(&folio->lru);
2575 mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +2584 node_stat_mod_folio(folio, NR_ISOLATED_ANON +
2576 page_is_file_lru(page), -nr_pages);2585 folio_is_file_lru(folio), -nr_pages);
2577 putback_lru_page(page);2586 folio_putback_lru(folio);
2578 }2587 }
2579 isolated = 0;2588 isolated = 0;
2580 }2589 }
2581 if (nr_succeeded) {2590 if (nr_succeeded) {
2582 count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);2591 count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
2583 if (!node_is_toptier(page_to_nid(page)) && node_is_toptier(node))2592 if (!node_is_toptier(folio_nid(folio)) && node_is_toptier(node))
2584 mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,2593 mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,
2585 nr_succeeded);2594 nr_succeeded);
2586 }2595 }
@@ -2588,7 +2597,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, · 1 unmodified line
2588 return isolated;2597 return isolated;
2589 2598
2590out:2599out:
2591 put_page(page);2600 folio_put(folio);
2592 return 0;2601 return 0;
2593}2602}
2594#endif /* CONFIG_NUMA_BALANCING */2603#endif /* CONFIG_NUMA_BALANCING */
mm/mlock.c+72−23
@@ -305,6 +305,62 @@ void munlock_folio(struct folio *folio) · 304 unmodified lines
305 local_unlock(&mlock_fbatch.lock);305 local_unlock(&mlock_fbatch.lock);
306}306}
307 307
308static inline unsigned int folio_mlock_step(struct folio *folio,
309 pte_t *pte, unsigned long addr, unsigned long end)
310{
311 unsigned int count, i, nr = folio_nr_pages(folio);
312 unsigned long pfn = folio_pfn(folio);
313 pte_t ptent = ptep_get(pte);
314
315 if (!folio_test_large(folio))
316 return 1;
317
318 count = pfn + nr - pte_pfn(ptent);
319 count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT);
320
321 for (i = 0; i < count; i++, pte++) {
322 pte_t entry = ptep_get(pte);
323
324 if (!pte_present(entry))
325 break;
326 if (pte_pfn(entry) - pfn >= nr)
327 break;
328 }
329
330 return i;
331}
332
333static inline bool allow_mlock_munlock(struct folio *folio,
334 struct vm_area_struct *vma, unsigned long start,
335 unsigned long end, unsigned int step)
336{
337 /*
338 * For unlock, allow munlock large folio which is partially
339 * mapped to VMA. As it's possible that large folio is
340 * mlocked and VMA is split later.
341 *
342 * During memory pressure, such kind of large folio can
343 * be split. And the pages are not in VM_LOCKed VMA
344 * can be reclaimed.
345 */

101 more lines aren’t shown here.

mm/mm_init.c+1−3

This diff is large: its 18 lines aren’t shown here.

mm/mmap.c+116−33

This diff is large: its 273 lines aren’t shown here.

mm/mmzone.c+3−3

This diff is large: its 23 lines aren’t shown here.

mm/mprotect.c+24−34

This diff is large: its 130 lines aren’t shown here.

mm/mremap.c+86−21

This diff is large: its 164 lines aren’t shown here.

mm/nommu.c+4−4

This diff is large: its 22 lines aren’t shown here.

mm/oom_kill.c+8−8

This diff is large: its 58 lines aren’t shown here.

mm/page-writeback.c+17−20

This diff is large: its 60 lines aren’t shown here.

mm/page_alloc.c+267−76

This diff is large: its 593 lines aren’t shown here.

mm/page_io.c+4−4

This diff is large: its 30 lines aren’t shown here.

mm/page_owner.c+2−2

This diff is large: its 14 lines aren’t shown here.

mm/percpu.c+3−5

This diff is large: its 33 lines aren’t shown here.

mm/rmap.c+101−63

This diff is large: its 290 lines aren’t shown here.

mm/shmem.c+294−294

This diff is large: its 1055 lines aren’t shown here.

mm/show_mem.c+4−7

This diff is large: its 34 lines aren’t shown here.

mm/shrinker.c+809−0

This diff is large: its 810 lines aren’t shown here.

mm/shrinker_debug.c+10−25

This diff is large: its 124 lines aren’t shown here.

mm/slab.h+8−7

This diff is large: its 42 lines aren’t shown here.

mm/swap.h+5−4

This diff is large: its 32 lines aren’t shown here.

mm/swap_state.c+57−35

This diff is large: its 222 lines aren’t shown here.

mm/util.c+1−0

This diff is large: its 8 lines aren’t shown here.

mm/vmalloc.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/vmscan.c+22−712

This diff is large: its 868 lines aren’t shown here.

mm/vmstat.c+17−13

This diff is large: its 112 lines aren’t shown here.

mm/workingset.c+16−13

This diff is large: its 58 lines aren’t shown here.

mm/zsmalloc.c+16−13

This diff is large: its 69 lines aren’t shown here.

mm/zswap.c+12−3

This diff is large: its 58 lines aren’t shown here.

net/sunrpc/auth.c+12−8

This diff is large: its 40 lines aren’t shown here.

tools/include/uapi/linux/fs.h+59−0

This diff is large: its 64 lines aren’t shown here.

tools/include/uapi/linux/prctl.h+2−1

This diff is large: its 10 lines aren’t shown here.

tools/mm/page_owner_sort.c+103−114

This diff is large: its 398 lines aren’t shown here.

tools/testing/radix-tree/linux.c+2−2

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/cgroup/.gitignore+1−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/cgroup/Makefile+2−0

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/cgroup/test_hugetlb_memcg.c+234−0

This diff is large: its 235 lines aren’t shown here.

tools/testing/selftests/cgroup/test_zswap.c+48−0

This diff is large: its 69 lines aren’t shown here.

tools/testing/selftests/clone3/clone3.c+13−0

This diff is large: its 27 lines aren’t shown here.

tools/testing/selftests/damon/sysfs.sh+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/.gitignore+2−0

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+3−1

This diff is large: its 25 lines aren’t shown here.

tools/testing/selftests/mm/config+1−0

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/mm/gup_longterm.c+2−1

This diff is large: its 13 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb-madvise.c+0−19

This diff is large: its 26 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb_fault_after_madv.c+73−0

This diff is large: its 74 lines aren’t shown here.

tools/testing/selftests/mm/ksm_functional_tests.c+65−1

This diff is large: its 96 lines aren’t shown here.

tools/testing/selftests/mm/mdwe_test.c+122−15

This diff is large: its 219 lines aren’t shown here.

tools/testing/selftests/mm/mremap_test.c+257−44

This diff is large: its 437 lines aren’t shown here.

tools/testing/selftests/mm/pagemap_ioctl.c+1660−0

This diff is large: its 1661 lines aren’t shown here.

tools/testing/selftests/mm/run_vmtests.sh+8−0

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/mm/vm_util.c+19−0

This diff is large: its 23 lines aren’t shown here.

tools/testing/selftests/mm/vm_util.h+1−0

This diff is large: its 8 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t