History

Merge tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Pull MM updates from Andrew Morton: - "selftests/mm: clean up build output and verbosity" (Li Wang) Remove some noise from the MM selftests build - "mm: Free contiguous order-0 pages efficiently" (Ryan Roberts) Speed up the freeing of a batch of 0-order pages by first scanning them for coalescing opportunities. This is applicable to vfree() and to the releasing of frozen pages - "mm/damon: introduce DAMOS failed region quota charge ratio" (SeongJae Park) Address a DAMOS usability issue: The DAMOS quota often exhausts prematurely because it charges for all memory attempted, causing slow and inconsistent performance when actions fail on unreclaimable memory. To fix this, a new feature lets users set a smaller, flexible quota charge ratio (via a numerator and denominator) for failed regions. Since failed actions cause less overhead, reducing their quota cost ensures more predictable and efficient DAMOS processing - "selftests/cgroup: improve zswap tests robustness and support large page sizes" (Li Wang) Fix various spurious failures and improves the overall robustness of the cgroup zswap selftests - "fix MAP_DROPPABLE not supported errno" (Anthony Yznaga) Fix an issue in the mlock selftests on arm32 - "mm: huge_memory: clean up defrag sysfs with shared" (Breno Leitao) Some maintenance work in the huge_memory code - "treewide: fixup gfp_t printks" (Brendan Jackman) Use the special vprintf() gfp_t conversion in various places - "mm: Fix vmemmap optimization accounting and initialization" (Muchun Song) Fix several bugs in the vmemmap optimization, mainly around incorrect page accounting and memmap initialization in the DAX and memory hotplug paths. It also fixes pageblock migratetype initialization and struct page initialization for ZONE_DEVICE compound pages - "mm/damon: repost non-hotfix reviewed patches in damon/next tree" A sprinkle of unrelated minor bugfixes for DAMON - "mm: remove page_mapped()" (David Hildenbrand) Remove this function from the tree, replacing it with folio_mapped() - "mm/damon: let DAMON be paused and resumed" (SeongJae Park) Allow DAMON to be paused and resumed without losing its current state - "kasan: hw_tags: Disable tagging for stack and page-tables" (Muhammad Usama Anjum) Simplify and speed up kasan by removing its ineffective tagging of stacks and page tables - "mm/damon/reclaim,lru_sort: monitor all system rams by default" (SeongJae Park) Simplify deployment on diverse hardware like NUMA systems by updating DAMON_RECLAIM and DAMON_LRU_SORT to automatically monitor the physical address range covering all System RAM areas by default, replacing the overly restrictive behavior that only targeted the single largest memory block to save on negligible overhead - "mm/damon/sysfs: document filters/ directory as deprecated" (SeongJae Park) Update some DAMON docs - "mm: use spinlock guards for zone lock" (Dmitry Ilvokhin) Switch zone->lock handling over to using the guard() mechanisms - "mm/filemap: tighten mmap_miss hit accounting" (fujunjie) Fix a flaw where the mmap_miss counter over-credited page cache hits during fault-arounds and page-fault retries. This results in significant reduction of redundant synchronous mmap readahead I/O, drastically cutting down execution time and gigabytes read for sparse random or strided memory access workloads - "selftests/cgroup: Fix false positive failures in test_percpu_basic" (Li Wang) Fix a couple of false-positives in the cgroup kmem selftests - "mm/damon/reclaim: support monitoring intervals auto-tuning" (SeongJae Park) Add a new parameter to DAMON permitting DAMON_RECLAIM to automatically tune DAMON's sampling and aggregation intervals - "mm/damon/stat: add kdamond_pid parameter" (SeongJae Park) Change DAMON_STAT to provide the pid of its kdamond - "mm/kmemleak: dedupe verbose scan output" (Breno Leitao) Remove large amounts of duplicated backtraces from the verbose-mode kmemleak output - "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)" (David Hildenbrand) Reduce our use of CONFIG_HAVE_BOOTMEM_INFO_NODE, with a view to removing it entirely in a later series - "mm/damon: validate min_region_size to be power of 2" (Liew Rui Yan) Prevent users from passing a non-power-of-2 value of `addr_unit', as this later results in undesirable behavior - "mm: document read_pages and simplify usage" (Frederick Mayle) - "tools/mm/page-types: Fix misc bugs" (Ye Liu) Fix three issues in tools/mm/page-types.c - "mm: misc cleanups from __GFP_UNMAPPED series" (Brendan Jackman) Implement several cleanups in the page allocator and related code - "mm, swap: swap table phase IV: unify allocation" (Kairui Song) Unify the allocation and charging of anon and shmem swap in folios, provides better synchronization, consolidates the metadata management, hence dropping the static array and map, and improves performance - "mm/damon: introduce data attributes monitoring" (SeongJae Park( Extend DAMON to monitor general data attributes other than accesses - "mm/vmalloc: free unused pages on vrealloc() shrink" (Shivam Kalra) Implement the TODO in vrealloc() to unmap and free unused pages when shrinking across a page boundary - "mm/damon: documentation and comment fixes" (niecheng) - "remove mmap_action success, error hooks" (Lorenzo Stoakes) Eliminate custom hooks from mmap_action by removing the problematic success_hook which allowed drivers to improperly access uninitialized VMAs. It replaces the error_hook with a simple error-code field and updates the memory char driver accordingly - "mm/damon: minor improvements for code readability and tests" (SeongJae Park) - "mm/damon: fix macro arguments and clarify quota goals doc" (Maksym Shcherba) - "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c" (Mike Rapoport) - "mm/mglru: improve reclaim loop and dirty folio" (Kairui Song and others) Clean up and slightly improves MGLRU's reclaim loop and dirty writeback handling. Large performance improvements are measured - "use vma locks for proc/pid/{smaps|numa_maps} reads" (Suren Baghdasaryan) Use per-vma locks when reading /proc/pid/smaps and numa_maps similar to reduce contention on central mmap_lock - "refactors thpsize_shmem_enabled_store() and thpsize_shmem_enabled_show()" (Ran Xiaokai) Some cleanup work in the THP code - "selftests/memfd: fix compilation warnings" (Konstantin Khorenko) Fix a few build glitches in the memfd selftest code. - "memcg: shrink obj_stock_pcp and cache multiple objcgs" (Shakeel Butt) Resolve a 68% performance regression caused by NUMA-node cache thrashing around struct obj_stock_pcp by shrinking its existing fields and expanding it into a multi-slot array that caches up to five obj_cgroup pointers per CPU, allowing per-node variants of the same memcg to coexist within a single 64-byte cache line. - "zram: writeback fixes" (Sergey Senozhatsky) address a couple of unrelated zram writeback issues - "mm: switch THP shrinker to list_lru" (Johannes Weiner) Resolve NUMA-awareness issues and streamlines callsite interaction by refactoring and extending the list_lru API to completely replace the complex, open-coded deferred split queue for Transparent Huge Pages - "mm: improve large folio readahead for exec memory" (Usama Arif) Improve large-folio readahead on systems like 64K-page arm64 by preventing the mmap_miss check from permanently disabling target-oriented VM_EXEC readahead, and by generalizing the force_thp_readahead gate to support mappings with any usefully large maximum folio order under the cache cap. - "userfaultfd/pagemap: pre-existing fixes" (Kiryl Shutsemau) Fix a bunch of minor issues in the userfaultfd/pagemap, all of which were flagged by Sashiko review of proposed new material - "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and vmemmap_check_pmd()" (Muchun Song) Provide generic versions of these two functions so the four arch-specific implementations can be removed. - "mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device" (Youngjun Park) Address a uswsusp-vs-swapoff race and reduces the swap device reference taking/releasing frequency. - "mm/hmm: A fix and a selftest" (Dev Jain) * tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits) selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries fs/proc/task_mmu: do not warn on seeing non-migration pmd entry lib/test_hmm: check alloc_page_vma() return value and handle OOM mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX mm/swap: remove redundant swap device reference in alloc/free mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device mm/filemap: use folio_next_index() for start vmalloc: fix NULL pointer dereference in is_vm_area_hugepages() sparc/mm: drop vmemmap_check_pmd helper and use generic code loongarch/mm: drop vmemmap_check_pmd helper and use generic code riscv/mm: drop vmemmap_pmd helpers and use generic code arm64/mm: drop vmemmap_pmd helpers and use generic code mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd() rust: page: mark Page::nid as inline userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks userfaultfd: gate must_wait writability check on pte_present() mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole() fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry() fs/proc/task_mmu: fix make_uffd_wp_huge_pte() prot-update race ...

Linus Torvalds committed 4 months ago

Commit a552c812 parents c98d767 e3d8707169 files · +9215 −5618Browse files

Showing the changes to include (29 of 169 files). Show all

Unified · Split

29 files
  1. include/asm-generic/pgalloc.h +1 −1
  2. include/linux/bootmem_info.h +0 −1
  3. include/linux/compaction.h +5 −5
  4. include/linux/damon.h +111 −25
  5. include/linux/gfp.h +4 −0
  6. include/linux/gfp_types.h +3 −3
  7. include/linux/highmem-internal.h +1 −1
  8. include/linux/huge_mm.h +42 −5
  9. include/linux/list_lru.h +70 −0
  10. include/linux/memcontrol.h +17 −14
  11. include/linux/memory.h +5 −2
  12. include/linux/memory_hotplug.h +5 −3
  13. include/linux/mm.h +44 −18
  14. include/linux/mm_inline.h +1 −1
  15. include/linux/mm_types.h +3 −16
  16. include/linux/mmu_notifier.h +2 −2
  17. include/linux/mmzone.h +6 −17
  18. include/linux/nodemask.h +9 −9
  19. include/linux/page_ref.h +18 −0
  20. include/linux/pageblock-flags.h +3 −3
  21. include/linux/pagemap.h +2 −0
  22. include/linux/swap.h +12 −12
  23. include/linux/swap_cgroup.h +0 −47
  24. include/linux/thread_info.h +1 −1
  25. include/linux/userfaultfd_k.h +2 −38
  26. include/linux/vmalloc.h +3 −1
  27. include/linux/vmpressure.h +5 −4
  28. include/trace/events/damon.h +38 −0
  29. include/trace/events/vmscan.h +52 −0
include/asm-generic/pgalloc.h+1−1
@@ -4,7 +4,7 @@ · 3 unmodified lines
4 4
5#ifdef CONFIG_MMU5#ifdef CONFIG_MMU
6 6
7#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO)7#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
8#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)8#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)
9 9
10/**10/**
include/linux/bootmem_info.h+0−1
@@ -82,7 +82,6 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 81 unmodified lines
82 82
83static inline void free_bootmem_page(struct page *page)83static inline void free_bootmem_page(struct page *page)
84{84{
85 kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
86 free_reserved_page(page);85 free_reserved_page(page);
87}86}
88#endif87#endif
include/linux/compaction.h+5−5
@@ -2,6 +2,8 @@ · 1 unmodified line
2#ifndef _LINUX_COMPACTION_H2#ifndef _LINUX_COMPACTION_H
3#define _LINUX_COMPACTION_H3#define _LINUX_COMPACTION_H
4 4
5#include <linux/swap.h>
6
5/*7/*
6 * Determines how hard direct compaction should try to succeed.8 * Determines how hard direct compaction should try to succeed.
7 * Lower value means higher priority, analogically to reclaim priority.9 * Lower value means higher priority, analogically to reclaim priority.
@@ -73,11 +75,9 @@ static inline unsigned long compact_gap(unsigned int order) · 65 unmodified lines
73 * effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum75 * effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum
74 * that the migrate scanner can have isolated on migrate list, and free76 * that the migrate scanner can have isolated on migrate list, and free
75 * scanner is only invoked when the number of isolated free pages is77 * scanner is only invoked when the number of isolated free pages is
76 * lower than that. But it's not worth to complicate the formula here78 * lower than that.
77 * as a bigger gap for higher orders than strictly necessary can also
78 * improve chances of compaction success.
79 */79 */
80 return 2UL << order;80 return min(2UL << order, COMPACT_CLUSTER_MAX);
81}81}
82 82
83static inline int current_is_kcompactd(void)83static inline int current_is_kcompactd(void)
@@ -101,7 +101,7 @@ extern void compaction_defer_reset(struct zone *zone, int order, · 17 unmodified lines
101 bool alloc_success);101 bool alloc_success);
102 102
103bool compaction_zonelist_suitable(struct alloc_context *ac, int order,103bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
104 int alloc_flags);104 int alloc_flags, gfp_t gfp_mask);
105 105
106extern void __meminit kcompactd_run(int nid);106extern void __meminit kcompactd_run(int nid);
107extern void __meminit kcompactd_stop(int nid);107extern void __meminit kcompactd_stop(int nid);
include/linux/damon.h+111−25
@@ -8,23 +8,20 @@ · 7 unmodified lines
8#ifndef _DAMON_H_8#ifndef _DAMON_H_
9#define _DAMON_H_9#define _DAMON_H_
10 10
11#include <linux/math64.h>
11#include <linux/memcontrol.h>12#include <linux/memcontrol.h>
12#include <linux/mutex.h>13#include <linux/mutex.h>
14#include <linux/prandom.h>
13#include <linux/time64.h>15#include <linux/time64.h>
14#include <linux/types.h>16#include <linux/types.h>
15#include <linux/random.h>
16 17
17/* Minimal region size. Every damon_region is aligned by this. */18/* Minimal region size. Every damon_region is aligned by this. */
18#define DAMON_MIN_REGION_SZ PAGE_SIZE19#define DAMON_MIN_REGION_SZ PAGE_SIZE
20/* Maximum number of monitoring probes. */
21#define DAMON_MAX_PROBES (4)
19/* Max priority score for DAMON-based operation schemes */22/* Max priority score for DAMON-based operation schemes */
20#define DAMOS_MAX_SCORE (99)23#define DAMOS_MAX_SCORE (99)
21 24
22/* Get a random number in [l, r) */
23static inline unsigned long damon_rand(unsigned long l, unsigned long r)
24{
25 return l + get_random_u32_below(r - l);
26}
27
28/**25/**
29 * struct damon_addr_range - Represents an address region of [@start, @end).26 * struct damon_addr_range - Represents an address region of [@start, @end).
30 * @start: Start address of the region (inclusive).27 * @start: Start address of the region (inclusive).
@@ -52,6 +49,7 @@ struct damon_size_range { · 21 unmodified lines
52 * @nr_accesses: Access frequency of this region.49 * @nr_accesses: Access frequency of this region.
53 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for50 * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
54 * each sampling interval.51 * each sampling interval.
52 * @probe_hits: Number of probe-positive region samples.
55 * @list: List head for siblings.53 * @list: List head for siblings.
56 * @age: Age of this region.54 * @age: Age of this region.
57 *55 *
@@ -80,6 +78,7 @@ struct damon_region { · 22 unmodified lines
80 unsigned long sampling_addr;78 unsigned long sampling_addr;
81 unsigned int nr_accesses;79 unsigned int nr_accesses;
82 unsigned int nr_accesses_bp;80 unsigned int nr_accesses_bp;
81 unsigned char probe_hits[DAMON_MAX_PROBES];
83 struct list_head list;82 struct list_head list;
84 83
85 unsigned int age;84 unsigned int age;
@@ -121,6 +120,7 @@ struct damon_target { · 35 unmodified lines
121 * @DAMOS_PAGEOUT: Reclaim the region.120 * @DAMOS_PAGEOUT: Reclaim the region.
122 * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.121 * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
123 * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.122 * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
123 * @DAMOS_COLLAPSE: Call ``madvise()`` for the region with MADV_COLLAPSE.
124 * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.124 * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
125 * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.125 * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.
126 * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.126 * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.
@@ -140,6 +140,7 @@ enum damos_action { · 13 unmodified lines
140 DAMOS_PAGEOUT,140 DAMOS_PAGEOUT,
141 DAMOS_HUGEPAGE,141 DAMOS_HUGEPAGE,
142 DAMOS_NOHUGEPAGE,142 DAMOS_NOHUGEPAGE,
143 DAMOS_COLLAPSE,
143 DAMOS_LRU_PRIO,144 DAMOS_LRU_PRIO,
144 DAMOS_LRU_DEPRIO,145 DAMOS_LRU_DEPRIO,
145 DAMOS_MIGRATE_HOT,146 DAMOS_MIGRATE_HOT,
@@ -159,6 +160,8 @@ enum damos_action { · 13 unmodified lines
159 * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.160 * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.
160 * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.161 * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.
161 * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.162 * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.
163 * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a
164 * node in basis points (0-10000).
162 * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.165 * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.
163 *166 *
164 * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.167 * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@@ -172,6 +175,7 @@ enum damos_quota_goal_metric { · 7 unmodified lines
172 DAMOS_QUOTA_NODE_MEMCG_FREE_BP,175 DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
173 DAMOS_QUOTA_ACTIVE_MEM_BP,176 DAMOS_QUOTA_ACTIVE_MEM_BP,
174 DAMOS_QUOTA_INACTIVE_MEM_BP,177 DAMOS_QUOTA_INACTIVE_MEM_BP,
178 DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
175 NR_DAMOS_QUOTA_GOAL_METRICS,179 NR_DAMOS_QUOTA_GOAL_METRICS,
176};180};
177 181
@@ -233,6 +237,8 @@ enum damos_quota_goal_tuner { · 55 unmodified lines
233 * @goals: Head of quota tuning goals (&damos_quota_goal) list.237 * @goals: Head of quota tuning goals (&damos_quota_goal) list.
234 * @goal_tuner: Goal-based @esz tuning algorithm to use.238 * @goal_tuner: Goal-based @esz tuning algorithm to use.
235 * @esz: Effective size quota in bytes.239 * @esz: Effective size quota in bytes.
240 * @fail_charge_num: Failed regions charge rate numerator.
241 * @fail_charge_denom: Failed regions charge rate denominator.
236 *242 *
237 * @weight_sz: Weight of the region's size for prioritization.243 * @weight_sz: Weight of the region's size for prioritization.
238 * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.244 * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.
@@ -262,6 +268,10 @@ enum damos_quota_goal_tuner { · 23 unmodified lines
262 *268 *
263 * The resulting effective size quota in bytes is set to @esz.269 * The resulting effective size quota in bytes is set to @esz.
264 *270 *
271 * For DAMOS action applying failed amount of regions, charging those same to
272 * those that the action has successfully applied may be unfair. For the
273 * reason, 'the size * @fail_charge_num / @fail_charge_denom' is charged.
274 *
265 * For selecting regions within the quota, DAMON prioritizes current scheme's275 * For selecting regions within the quota, DAMON prioritizes current scheme's
266 * target memory regions using the &struct damon_operations->get_scheme_score.276 * target memory regions using the &struct damon_operations->get_scheme_score.
267 * You could customize the prioritization logic by setting &weight_sz,277 * You could customize the prioritization logic by setting &weight_sz,
@@ -276,6 +286,9 @@ struct damos_quota { · 8 unmodified lines
276 enum damos_quota_goal_tuner goal_tuner;286 enum damos_quota_goal_tuner goal_tuner;
277 unsigned long esz;287 unsigned long esz;
278 288
289 unsigned int fail_charge_num;
290 unsigned int fail_charge_denom;
291
279 unsigned int weight_sz;292 unsigned int weight_sz;
280 unsigned int weight_nr_accesses;293 unsigned int weight_nr_accesses;
281 unsigned int weight_age;294 unsigned int weight_age;
@@ -617,6 +630,7 @@ enum damon_ops_id { · 335 unmodified lines
617 * @update: Update operations-related data structures.630 * @update: Update operations-related data structures.
618 * @prepare_access_checks: Prepare next access check of target regions.631 * @prepare_access_checks: Prepare next access check of target regions.
619 * @check_accesses: Check the accesses to target regions.632 * @check_accesses: Check the accesses to target regions.
633 * @apply_probes: Apply probes for each region.
620 * @get_scheme_score: Get the score of a region for a scheme.634 * @get_scheme_score: Get the score of a region for a scheme.
621 * @apply_scheme: Apply a DAMON-based operation scheme.635 * @apply_scheme: Apply a DAMON-based operation scheme.
622 * @target_valid: Determine if the target is valid.636 * @target_valid: Determine if the target is valid.
@@ -643,6 +657,8 @@ enum damon_ops_id { · 20 unmodified lines
643 * last preparation and update the number of observed accesses of each region.657 * last preparation and update the number of observed accesses of each region.
644 * It should also return max number of observed accesses that made as a result658 * It should also return max number of observed accesses that made as a result
645 * of its update. The value will be used for regions adjustment threshold.659 * of its update. The value will be used for regions adjustment threshold.
660 * @apply_probes should apply the data attribute probes to each region and
661 * accordingly update the probe hits counter of the region.
646 * @get_scheme_score should return the priority score of a region for a scheme662 * @get_scheme_score should return the priority score of a region for a scheme
647 * as an integer in [0, &DAMOS_MAX_SCORE].663 * as an integer in [0, &DAMOS_MAX_SCORE].
648 * @apply_scheme is called from @kdamond when a region for user provided664 * @apply_scheme is called from @kdamond when a region for user provided
@@ -660,6 +676,7 @@ struct damon_operations { · 11 unmodified lines
660 void (*update)(struct damon_ctx *context);676 void (*update)(struct damon_ctx *context);
661 void (*prepare_access_checks)(struct damon_ctx *context);677 void (*prepare_access_checks)(struct damon_ctx *context);
662 unsigned int (*check_accesses)(struct damon_ctx *context);678 unsigned int (*check_accesses)(struct damon_ctx *context);
679 void (*apply_probes)(struct damon_ctx *context);
663 int (*get_scheme_score)(struct damon_ctx *context,680 int (*get_scheme_score)(struct damon_ctx *context,
664 struct damon_region *r, struct damos *scheme);681 struct damon_region *r, struct damos *scheme);
665 unsigned long (*apply_scheme)(struct damon_ctx *context,682 unsigned long (*apply_scheme)(struct damon_ctx *context,
@@ -721,6 +738,47 @@ struct damon_intervals_goal { · 55 unmodified lines
721 unsigned long max_sample_us;738 unsigned long max_sample_us;
722};739};
723 740
741/**
742 * enum damon_filter_type - Type of &struct damon_filter
743 *
744 * @DAMON_FILTER_TYPE_ANON: Anonymous pages.
745 * @DAMON_FILTER_TYPE_MEMCG: Specific memcg's pages.
746 */
747enum damon_filter_type {
748 DAMON_FILTER_TYPE_ANON,
749 DAMON_FILTER_TYPE_MEMCG,
750};
751
752/**
753 * struct damon_filter - DAMON region filter for &struct damon_probe.
754 *
755 * @type: Type of the region.
756 * @matching: Whether this filter is for the type-matching ones.
757 * @allow: Whether the @type-@matching ones should pass this filter.
758 * @memcg_id: Memcg id of the question if @type is DAMON_FILTER_MEMCG.
759 * @list: Siblings list.
760 */
761struct damon_filter {
762 enum damon_filter_type type;
763 bool matching;
764 bool allow;
765 union {
766 u64 memcg_id;
767 };
768 struct list_head list;
769};
770
771/**
772 * struct damon_probe - Data region attribute probe.
773 *
774 * @filters: Filters for assessing if a given region is for this probe.
775 * @list: Siblings list.
776 */
777struct damon_probe {
778 struct list_head filters;
779 struct list_head list;
780};
781
724/**782/**
725 * struct damon_attrs - Monitoring attributes for accuracy/overhead control.783 * struct damon_attrs - Monitoring attributes for accuracy/overhead control.
726 *784 *
@@ -787,6 +845,7 @@ struct damon_attrs { · 60 unmodified lines
787 * @ops: Set of monitoring operations for given use cases.845 * @ops: Set of monitoring operations for given use cases.
788 * @addr_unit: Scale factor for core to ops address conversion.846 * @addr_unit: Scale factor for core to ops address conversion.
789 * @min_region_sz: Minimum region size.847 * @min_region_sz: Minimum region size.
848 * @pause: Pause kdamond main loop.
790 * @adaptive_targets: Head of monitoring targets (&damon_target) list.849 * @adaptive_targets: Head of monitoring targets (&damon_target) list.
791 * @schemes: Head of schemes (&damos) list.850 * @schemes: Head of schemes (&damos) list.
792 */851 */
@@ -838,13 +897,34 @@ struct damon_ctx { · 45 unmodified lines
838 897
839/* public: */898/* public: */
840 struct damon_operations ops;899 struct damon_operations ops;
900 struct list_head probes;
841 unsigned long addr_unit;901 unsigned long addr_unit;
842 unsigned long min_region_sz;902 unsigned long min_region_sz;
903 bool pause;
843 904
844 struct list_head adaptive_targets;905 struct list_head adaptive_targets;
845 struct list_head schemes;906 struct list_head schemes;
907
908 /* Per-ctx PRNG state for damon_rand(); kdamond is the sole consumer. */
909 struct rnd_state rnd_state;
846};910};
847 911
912/* Get a random number in [@l, @r) using @ctx's lockless PRNG. */
913static inline unsigned long damon_rand(struct damon_ctx *ctx,
914 unsigned long l, unsigned long r)
915{
916 unsigned long span = r - l;
917 u64 rnd;
918
919 if (span <= U32_MAX) {
920 rnd = prandom_u32_state(&ctx->rnd_state);
921 return l + (unsigned long)((rnd * span) >> 32);
922 }
923 rnd = ((u64)prandom_u32_state(&ctx->rnd_state) << 32) |
924 prandom_u32_state(&ctx->rnd_state);
925 return l + mul_u64_u64_shr(rnd, span, 64);
926}
927
848static inline struct damon_region *damon_next_region(struct damon_region *r)928static inline struct damon_region *damon_next_region(struct damon_region *r)
849{929{
850 return container_of(r->list.next, struct damon_region, list);930 return container_of(r->list.next, struct damon_region, list);
@@ -870,15 +950,26 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 19 unmodified lines
870 return r->ar.end - r->ar.start;950 return r->ar.end - r->ar.start;
871}951}
872 952
953#define damon_for_each_filter(f, p) \
954 list_for_each_entry(f, &(p)->filters, list)
955
956#define damon_for_each_filter_safe(f, next, p) \
957 list_for_each_entry_safe(f, next, &(p)->filters, list)
958
959#define damon_for_each_probe(p, ctx) \
960 list_for_each_entry(p, &(ctx)->probes, list)
961
962#define damon_for_each_probe_safe(p, next, ctx) \
963 list_for_each_entry_safe(p, next, &(ctx)->probes, list)
873 964
874#define damon_for_each_region(r, t) \965#define damon_for_each_region(r, t) \
875 list_for_each_entry(r, &t->regions_list, list)966 list_for_each_entry(r, &(t)->regions_list, list)
876 967
877#define damon_for_each_region_from(r, t) \968#define damon_for_each_region_from(r, t) \
878 list_for_each_entry_from(r, &t->regions_list, list)969 list_for_each_entry_from(r, &(t)->regions_list, list)
879 970
880#define damon_for_each_region_safe(r, next, t) \971#define damon_for_each_region_safe(r, next, t) \
881 list_for_each_entry_safe(r, next, &t->regions_list, list)972 list_for_each_entry_safe(r, next, &(t)->regions_list, list)
882 973
883#define damon_for_each_target(t, ctx) \974#define damon_for_each_target(t, ctx) \
884 list_for_each_entry(t, &(ctx)->adaptive_targets, list)975 list_for_each_entry(t, &(ctx)->adaptive_targets, list)
@@ -893,7 +984,7 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 8 unmodified lines
893 list_for_each_entry_safe(s, next, &(ctx)->schemes, list)984 list_for_each_entry_safe(s, next, &(ctx)->schemes, list)
894 985
895#define damos_for_each_quota_goal(goal, quota) \986#define damos_for_each_quota_goal(goal, quota) \
896 list_for_each_entry(goal, &quota->goals, list)987 list_for_each_entry(goal, &(quota)->goals, list)
897 988
898#define damos_for_each_quota_goal_safe(goal, next, quota) \989#define damos_for_each_quota_goal_safe(goal, next, quota) \
899 list_for_each_entry_safe(goal, next, &(quota)->goals, list)990 list_for_each_entry_safe(goal, next, &(quota)->goals, list)
@@ -912,21 +1003,16 @@ static inline unsigned long damon_sz_region(struct damon_region *r) · 12 unmodified lines
912 1003
913#ifdef CONFIG_DAMON1004#ifdef CONFIG_DAMON
914 1005
915struct damon_region *damon_new_region(unsigned long start, unsigned long end);1006struct damon_filter *damon_new_filter(enum damon_filter_type type,
1007 bool matching, bool allow);
1008void damon_add_filter(struct damon_probe *probe, struct damon_filter *f);
1009void damon_destroy_filter(struct damon_filter *f);
916 1010
917/*1011struct damon_probe *damon_new_probe(void);
918 * Add a region between two other regions1012void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe);
919 */1013
920static inline void damon_insert_region(struct damon_region *r,1014struct damon_region *damon_new_region(unsigned long start, unsigned long end);
921 struct damon_region *prev, struct damon_region *next,
922 struct damon_target *t)
923{
924 __list_add(&r->list, &prev->list, &next->list);
925 t->nr_regions++;
926}
927 1015
928void damon_add_region(struct damon_region *r, struct damon_target *t);
929void damon_destroy_region(struct damon_region *r, struct damon_target *t);
930int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,1016int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
931 unsigned int nr_ranges, unsigned long min_region_sz);1017 unsigned int nr_ranges, unsigned long min_region_sz);
932void damon_update_region_access_rate(struct damon_region *r, bool accessed,1018void damon_update_region_access_rate(struct damon_region *r, bool accessed,
@@ -994,7 +1080,7 @@ int damon_kdamond_pid(struct damon_ctx *ctx); · 61 unmodified lines
994int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);1080int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
995int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);1081int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
996 1082
997int damon_set_region_biggest_system_ram_default(struct damon_target *t,1083int damon_set_region_system_rams_default(struct damon_target *t,
998 unsigned long *start, unsigned long *end,1084 unsigned long *start, unsigned long *end,
999 unsigned long addr_unit,1085 unsigned long addr_unit,
1000 unsigned long min_region_sz);1086 unsigned long min_region_sz);
include/linux/gfp.h+4−0
@@ -239,6 +239,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · 238 unmodified lines
239 struct page **page_array);239 struct page **page_array);
240#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))240#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))
241 241
242void free_pages_bulk(struct page **page_array, unsigned long nr_pages);
243
242unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,244unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
243 unsigned long nr_pages,245 unsigned long nr_pages,
244 struct page **page_array);246 struct page **page_array);
@@ -467,6 +469,8 @@ void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages); · 222 unmodified lines
467void free_contig_range(unsigned long pfn, unsigned long nr_pages);469void free_contig_range(unsigned long pfn, unsigned long nr_pages);
468#endif470#endif
469 471
472void __free_contig_range(unsigned long pfn, unsigned long nr_pages);
473
470DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))474DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))
471 475
472#endif /* __LINUX_GFP_H */476#endif /* __LINUX_GFP_H */
include/linux/gfp_types.h+3−3
@@ -281,9 +281,9 @@ enum { · 280 unmodified lines
281 *281 *
282 * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.282 * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.
283 * Used for userspace and vmalloc pages; the latter are unpoisoned by283 * Used for userspace and vmalloc pages; the latter are unpoisoned by
284 * kasan_unpoison_vmalloc instead. For userspace pages, results in284 * kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc
285 * poisoning being skipped as well, see should_skip_kasan_poison for285 * is skipped too. For userspace pages, results in poisoning being skipped as
286 * details. Only effective in HW_TAGS mode.286 * well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode.
287 */287 */
288#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)288#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
289#define __GFP_COMP ((__force gfp_t)___GFP_COMP)289#define __GFP_COMP ((__force gfp_t)___GFP_COMP)
include/linux/highmem-internal.h+1−1
@@ -262,7 +262,7 @@ static inline bool is_kmap_addr(const void *x) · 261 unmodified lines
262 * @__addr: Virtual address to be unmapped262 * @__addr: Virtual address to be unmapped
263 *263 *
264 * Unmaps an address previously mapped by kmap_atomic() and re-enables264 * Unmaps an address previously mapped by kmap_atomic() and re-enables
265 * pagefaults. Depending on PREEMP_RT configuration, re-enables also265 * pagefaults. Depending on PREEMPT_RT configuration, re-enables also
266 * migration and preemption. Users should not count on these side effects.266 * migration and preemption. Users should not count on these side effects.
267 *267 *
268 * Mappings should be unmapped in the reverse order that they were mapped.268 * Mappings should be unmapped in the reverse order that they were mapped.
include/linux/huge_mm.h+42−5
@@ -237,6 +237,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 236 unmodified lines
237 return true;237 return true;
238}238}
239 239
240/*
241 * Make sure huge_gfp is always more limited than limit_gfp.
242 * Some shmem users want THP allocation to be done less aggressively
243 * and only in certain zone.
244 */
245static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
246{
247 gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
248 gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
249 gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
250 gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
251
252 /* Allow allocations only from the originally specified zones. */
253 result |= zoneflags;
254
255 /*
256 * Minimize the result gfp by taking the union with the deny flags,
257 * and the intersection of the allow flags.
258 */
259 result |= (limit_gfp & denyflags);
260 result |= (huge_gfp & limit_gfp) & allowflags;
261
262 return result;
263}
264
240/*265/*
241 * Filter the bitfield of input orders to the ones suitable for use in the vma.266 * Filter the bitfield of input orders to the ones suitable for use in the vma.
242 * See thp_vma_suitable_order().267 * See thp_vma_suitable_order().
@@ -414,10 +439,10 @@ static inline int split_huge_page(struct page *page) · 171 unmodified lines
414{439{
415 return split_huge_page_to_list_to_order(page, NULL, 0);440 return split_huge_page_to_list_to_order(page, NULL, 0);
416}441}
442
443int folio_memcg_alloc_deferred(struct folio *folio);
444
417void deferred_split_folio(struct folio *folio, bool partially_mapped);445void deferred_split_folio(struct folio *folio, bool partially_mapped);
418#ifdef CONFIG_MEMCG
419void reparent_deferred_split_queue(struct mem_cgroup *memcg);
420#endif
421 446
422void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,447void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
423 unsigned long address, bool freeze);448 unsigned long address, bool freeze);
@@ -581,6 +606,11 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma, · 157 unmodified lines
581 return false;606 return false;
582}607}
583 608
609static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
610{
611 return huge_gfp;
612}
613
584static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,614static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
585 unsigned long addr, unsigned long orders)615 unsigned long addr, unsigned long orders)
586{616{
@@ -649,8 +679,15 @@ static inline int try_folio_split_to_order(struct folio *folio, · 62 unmodified lines
649 return -EINVAL;679 return -EINVAL;
650}680}
651 681
652static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}682static inline int folio_memcg_alloc_deferred(struct folio *folio)
653static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}683{
684 return 0;
685}
686
687static inline void deferred_split_folio(struct folio *folio, bool partially_mapped)
688{
689}
690
654#define split_huge_pmd(__vma, __pmd, __address) \691#define split_huge_pmd(__vma, __pmd, __address) \
655 do { } while (0)692 do { } while (0)
656 693
include/linux/list_lru.h+70−0
@@ -81,8 +81,75 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker · 80 unmodified lines
81 81
82int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,82int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
83 gfp_t gfp);83 gfp_t gfp);
84
85#ifdef CONFIG_MEMCG
86/**
87 * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
88 * @folio: the newly allocated & charged folio
89 * @lru: the list_lru this might be queued on
90 * @gfp: gfp mask
91 *
92 * Allocate list_lru heads (per-memcg, per-node) needed to queue this
93 * particular folio down the line.
94 *
95 * This does memcg_list_lru_alloc(), but on the memcg that @folio is
96 * associated with. Handles folio_memcg() access rules in the fast
97 * path (list_lru heads allocated) and the allocation slowpath.
98 *
99 * Returns 0 on success, a negative error value otherwise.
100 */
101int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
102 gfp_t gfp);
103#else
104static inline int folio_memcg_list_lru_alloc(struct folio *folio,
105 struct list_lru *lru, gfp_t gfp)
106{
107 return 0;
108}
109#endif
110
84void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);111void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
85 112
113/**
114 * list_lru_lock: lock the sublist for the given node and memcg
115 * @lru: the lru pointer
116 * @nid: the node id of the sublist to lock.
117 * @memcg: pointer to the cgroup of the sublist to lock. On return,
118 * updated to the cgroup whose sublist was actually locked,
119 * which may be an ancestor if the original memcg was dying.
120 *
121 * Returns the locked list_lru_one sublist. The caller must call
122 * list_lru_unlock() when done.
123 *
124 * You must ensure that the memcg is not freed during this call (e.g., with
125 * rcu or by taking a css refcnt).
126 *
127 * Return: the locked list_lru_one, or NULL on failure
128 */
129struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
130 struct mem_cgroup **memcg);
131
132/**
133 * list_lru_unlock: unlock a sublist locked by list_lru_lock()
134 * @l: the list_lru_one to unlock
135 */
136void list_lru_unlock(struct list_lru_one *l);
137
138struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
139 struct mem_cgroup **memcg);
140void list_lru_unlock_irq(struct list_lru_one *l);
141
142struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
143 struct mem_cgroup **memcg, unsigned long *irq_flags);
144void list_lru_unlock_irqrestore(struct list_lru_one *l,
145 unsigned long *irq_flags);
146
147/* Caller-locked variants, see list_lru_add() etc for documentation */
148bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
149 struct list_head *item, int nid, struct mem_cgroup *memcg);
150bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
151 struct list_head *item, int nid);
152
86/**153/**
87 * list_lru_add: add an element to the lru list's tail154 * list_lru_add: add an element to the lru list's tail
88 * @lru: the lru pointer155 * @lru: the lru pointer
@@ -115,6 +182,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren · 26 unmodified lines
115bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,182bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
116 struct mem_cgroup *memcg);183 struct mem_cgroup *memcg);
117 184
185bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
186 struct mem_cgroup *memcg);
187
118/**188/**
119 * list_lru_add_obj: add an element to the lru list's tail189 * list_lru_add_obj: add an element to the lru list's tail
120 * @lru: the lru pointer190 * @lru: the lru pointer
include/linux/memcontrol.h+17−14
@@ -29,6 +29,7 @@ struct obj_cgroup; · 28 unmodified lines
29struct page;29struct page;
30struct mm_struct;30struct mm_struct;
31struct kmem_cache;31struct kmem_cache;
32struct swap_cluster_info;
32 33
33/* Cgroup-specific page state, on top of universal node page state */34/* Cgroup-specific page state, on top of universal node page state */
34enum memcg_stat_item {35enum memcg_stat_item {
@@ -277,10 +278,6 @@ struct mem_cgroup { · 242 unmodified lines
277 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];278 struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
278#endif279#endif
279 280
280#ifdef CONFIG_TRANSPARENT_HUGEPAGE
281 struct deferred_split deferred_split_queue;
282#endif
283
284#ifdef CONFIG_LRU_GEN_WALKS_MMU281#ifdef CONFIG_LRU_GEN_WALKS_MMU
285 /* per-memcg mm_struct list */282 /* per-memcg mm_struct list */
286 struct lru_gen_mm_list mm_list;283 struct lru_gen_mm_list mm_list;
@@ -646,8 +643,8 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 359 unmodified lines
646 643
647int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);644int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
648 645
649int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,646int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
650 gfp_t gfp, swp_entry_t entry);647 struct mm_struct *mm, gfp_t gfp);
651 648
652void __mem_cgroup_uncharge(struct folio *folio);649void __mem_cgroup_uncharge(struct folio *folio);
653 650
@@ -1137,7 +1134,7 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp) · 483 unmodified lines
1137}1134}
1138 1135
1139static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,1136static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
1140 struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)1137 unsigned short id, struct mm_struct *mm, gfp_t gfp)
1141{1138{
1142 return 0;1139 return 0;
1143}1140}
@@ -1899,9 +1896,6 @@ static inline void mem_cgroup_exit_user_fault(void) · 755 unmodified lines
1899 current->in_user_fault = 0;1896 current->in_user_fault = 0;
1900}1897}
1901 1898
1902void memcg1_swapout(struct folio *folio, swp_entry_t entry);
1903void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages);
1904
1905#else /* CONFIG_MEMCG_V1 */1899#else /* CONFIG_MEMCG_V1 */
1906static inline1900static inline
1907unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,1901unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
@@ -1929,14 +1923,23 @@ static inline void mem_cgroup_exit_user_fault(void) · 21 unmodified lines
1929{1923{
1930}1924}
1931 1925
1932static inline void memcg1_swapout(struct folio *folio, swp_entry_t entry)1926#endif /* CONFIG_MEMCG_V1 */
1927
1928#if defined(CONFIG_MEMCG_V1) && defined(CONFIG_SWAP)
1929
1930void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci);
1931void memcg1_swapin(struct folio *folio);
1932
1933#else
1934
1935static inline void __memcg1_swapout(struct folio *folio,
1936 struct swap_cluster_info *ci)
1933{1937{
1934}1938}
1935 1939
1936static inline void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)1940static inline void memcg1_swapin(struct folio *folio)
1937{1941{
1938}1942}
1939 1943#endif
1940#endif /* CONFIG_MEMCG_V1 */
1941 1944
1942#endif /* _LINUX_MEMCONTROL_H */1945#endif /* _LINUX_MEMCONTROL_H */
include/linux/memory.h+5−2
@@ -158,7 +158,11 @@ int create_memory_block_devices(unsigned long start, unsigned long size, · 157 unmodified lines
158void remove_memory_block_devices(unsigned long start, unsigned long size);158void remove_memory_block_devices(unsigned long start, unsigned long size);
159extern void memory_dev_init(void);159extern void memory_dev_init(void);
160extern int memory_notify(enum memory_block_state state, void *v);160extern int memory_notify(enum memory_block_state state, void *v);
161extern struct memory_block *find_memory_block(unsigned long section_nr);161struct memory_block *memory_block_get(unsigned long block_id);
162static inline void memory_block_put(struct memory_block *mem)
163{
164 put_device(&mem->dev);
165}
162typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);166typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);
163extern int walk_memory_blocks(unsigned long start, unsigned long size,167extern int walk_memory_blocks(unsigned long start, unsigned long size,
164 void *arg, walk_memory_blocks_func_t func);168 void *arg, walk_memory_blocks_func_t func);
@@ -171,7 +175,6 @@ struct memory_group *memory_group_find_by_id(int mgid); · 6 unmodified lines
171typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);175typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);
172int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,176int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
173 struct memory_group *excluded, void *arg);177 struct memory_group *excluded, void *arg);
174struct memory_block *find_memory_block_by_id(unsigned long block_id);
175#define hotplug_memory_notifier(fn, pri) ({ \178#define hotplug_memory_notifier(fn, pri) ({ \
176 static __meminitdata struct notifier_block fn##_mem_nb =\179 static __meminitdata struct notifier_block fn##_mem_nb =\
177 { .notifier_call = fn, .priority = pri };\180 { .notifier_call = fn, .priority = pri };\
include/linux/memory_hotplug.h+5−3
@@ -135,9 +135,10 @@ static inline bool movable_node_is_enabled(void) · 134 unmodified lines
135 return movable_node_enabled;135 return movable_node_enabled;
136}136}
137 137
138extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap);138extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
139 struct dev_pagemap *pgmap);
139extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,140extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
140 struct vmem_altmap *altmap);141 struct vmem_altmap *altmap, struct dev_pagemap *pgmap);
141 142
142/* reasonably generic interface to expand the physical pages */143/* reasonably generic interface to expand the physical pages */
143extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,144extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -307,7 +308,8 @@ extern int sparse_add_section(int nid, unsigned long pfn, · 163 unmodified lines
307 unsigned long nr_pages, struct vmem_altmap *altmap,308 unsigned long nr_pages, struct vmem_altmap *altmap,
308 struct dev_pagemap *pgmap);309 struct dev_pagemap *pgmap);
309extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,310extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
310 struct vmem_altmap *altmap);311 struct vmem_altmap *altmap,
312 struct dev_pagemap *pgmap);
311extern struct zone *zone_for_pfn_range(enum mmop online_type,313extern struct zone *zone_for_pfn_range(enum mmop online_type,
312 int nid, struct memory_group *group, unsigned long start_pfn,314 int nid, struct memory_group *group, unsigned long start_pfn,
313 unsigned long nr_pages);315 unsigned long nr_pages);
include/linux/mm.h+44−18
@@ -496,6 +496,21 @@ enum { · 495 unmodified lines
496#else496#else
497#define VM_UFFD_MINOR VM_NONE497#define VM_UFFD_MINOR VM_NONE
498#endif498#endif
499
500/*
501 * vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on
502 * the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits
503 * -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose
504 * bitmap cannot hold it.
505 */
506#define VMA_UFFD_MISSING mk_vma_flags(VMA_UFFD_MISSING_BIT)
507#define VMA_UFFD_WP mk_vma_flags(VMA_UFFD_WP_BIT)
508#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
509#define VMA_UFFD_MINOR mk_vma_flags(VMA_UFFD_MINOR_BIT)
510#else
511#define VMA_UFFD_MINOR EMPTY_VMA_FLAGS
512#endif
513
499#ifdef CONFIG_64BIT514#ifdef CONFIG_64BIT
500#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)515#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
501#define VM_SEALED INIT_VM_FLAG(SEALED)516#define VM_SEALED INIT_VM_FLAG(SEALED)
@@ -1238,6 +1253,30 @@ static __always_inline void vma_flags_set_mask(vma_flags_t *flags, · 736 unmodified lines
1238#define vma_flags_set(flags, ...) \1253#define vma_flags_set(flags, ...) \
1239 vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))1254 vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))
1240 1255
1256static __always_inline vma_flags_t __mk_vma_flags_from_masks(size_t count,
1257 const vma_flags_t *masks)
1258{
1259 vma_flags_t flags = EMPTY_VMA_FLAGS;
1260 size_t i;
1261
1262 for (i = 0; i < count; i++)
1263 vma_flags_set_mask(&flags, masks[i]);
1264 return flags;
1265}
1266
1267/*
1268 * Combine pre-computed vma_flags_t masks into one value, e.g.:
1269 *
1270 * vma_flags_t flags = mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR);
1271 *
1272 * Unlike mk_vma_flags(), which takes bit numbers, this takes whole masks --
1273 * each of which may be EMPTY_VMA_FLAGS when its feature is unavailable -- so a
1274 * bit that does not exist on the current build is never materialised.
1275 */
1276#define mk_vma_flags_from_masks(...) \
1277 __mk_vma_flags_from_masks(COUNT_ARGS(__VA_ARGS__), \
1278 (const vma_flags_t []){__VA_ARGS__})
1279
1241/* Clear all of the to-clear flags in flags, non-atomically. */1280/* Clear all of the to-clear flags in flags, non-atomically. */
1242static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,1281static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,
1243 vma_flags_t to_clear)1282 vma_flags_t to_clear)
@@ -1489,6 +1528,11 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma) · 245 unmodified lines
1489 vma->vm_ops = NULL;1528 vma->vm_ops = NULL;
1490}1529}
1491 1530
1531static inline void vma_desc_set_anonymous(struct vm_area_desc *desc)
1532{
1533 desc->vm_ops = NULL;
1534}
1535
1492static inline bool vma_is_anonymous(struct vm_area_struct *vma)1536static inline bool vma_is_anonymous(struct vm_area_struct *vma)
1493{1537{
1494 return !vma->vm_ops;1538 return !vma->vm_ops;
@@ -1888,16 +1932,6 @@ static inline bool folio_mapped(const struct folio *folio) · 393 unmodified lines
1888 return folio_mapcount(folio) >= 1;1932 return folio_mapcount(folio) >= 1;
1889}1933}
1890 1934
1891/*
1892 * Return true if this page is mapped into pagetables.
1893 * For compound page it returns true if any sub-page of compound page is mapped,
1894 * even if this particular sub-page is not itself mapped by any PTE or PMD.
1895 */
1896static inline bool page_mapped(const struct page *page)
1897{
1898 return folio_mapped(page_folio(page));
1899}
1900
1901static inline struct page *virt_to_head_page(const void *x)1935static inline struct page *virt_to_head_page(const void *x)
1902{1936{
1903 struct page *page = virt_to_page(x);1937 struct page *page = virt_to_page(x);
@@ -4855,18 +4889,10 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) · 2951 unmodified lines
4855}4889}
4856#endif4890#endif
4857 4891
4858void *sparse_buffer_alloc(unsigned long size);
4859unsigned long section_map_size(void);4892unsigned long section_map_size(void);
4860struct page * __populate_section_memmap(unsigned long pfn,4893struct page * __populate_section_memmap(unsigned long pfn,
4861 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,4894 unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
4862 struct dev_pagemap *pgmap);4895 struct dev_pagemap *pgmap);
4863pgd_t *vmemmap_pgd_populate(unsigned long addr, int node);
4864p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node);
4865pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node);
4866pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node);
4867pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
4868 struct vmem_altmap *altmap, unsigned long ptpfn,
4869 unsigned long flags);
4870void *vmemmap_alloc_block(unsigned long size, int node);4896void *vmemmap_alloc_block(unsigned long size, int node);
4871struct vmem_altmap;4897struct vmem_altmap;
4872void *vmemmap_alloc_block_buf(unsigned long size, int node,4898void *vmemmap_alloc_block_buf(unsigned long size, int node,
include/linux/mm_inline.h+1−1
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec, · 246 unmodified lines
247 (folio_test_dirty(folio) || folio_test_writeback(folio))))247 (folio_test_dirty(folio) || folio_test_writeback(folio))))
248 gen = MIN_NR_GENS;248 gen = MIN_NR_GENS;
249 else249 else
250 gen = MAX_NR_GENS - folio_test_workingset(folio);250 gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
251 251
252 return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));252 return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
253}253}
include/linux/mm_types.h+3−16
@@ -845,23 +845,10 @@ struct mmap_action { · 844 unmodified lines
845 enum mmap_action_type type;845 enum mmap_action_type type;
846 846
847 /*847 /*
848 * If specified, this hook is invoked after the selected action has been848 * If non-zero, replace errors that arise from mmap actions with this
849 * successfully completed. Note that the VMA write lock still held.849 * value instead. Only valid error codes may be specified.
850 *
851 * The absolute minimum ought to be done here.
852 *
853 * Returns 0 on success, or an error code.
854 */
855 int (*success_hook)(const struct vm_area_struct *vma);
856
857 /*
858 * If specified, this hook is invoked when an error occurred when
859 * attempting the selected action.
860 *
861 * The hook can return an error code in order to filter the error, but
862 * it is not valid to clear the error here.
863 */850 */
864 int (*error_hook)(int err);851 int error_override;
865 852
866 /*853 /*
867 * This should be set in rare instances where the operation required854 * This should be set in rare instances where the operation required
include/linux/mmu_notifier.h+2−2
@@ -134,8 +134,8 @@ struct mmu_notifier_ops { · 133 unmodified lines
134 * Invalidation of multiple concurrent ranges may be134 * Invalidation of multiple concurrent ranges may be
135 * optionally permitted by the driver. Either way the135 * optionally permitted by the driver. Either way the
136 * establishment of sptes is forbidden in the range passed to136 * establishment of sptes is forbidden in the range passed to
137 * invalidate_range_begin/end for the whole duration of the137 * invalidate_range_start/end for the whole duration of the
138 * invalidate_range_begin/end critical section.138 * invalidate_range_start/end critical section.
139 *139 *
140 * invalidate_range_start() is called when all pages in the140 * invalidate_range_start() is called when all pages in the
141 * range are still mapped and have at least a refcount of one.141 * range are still mapped and have at least a refcount of one.
include/linux/mmzone.h+6−17
@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt) · 176 unmodified lines
177 return mt < MIGRATE_PCPTYPES;177 return mt < MIGRATE_PCPTYPES;
178}178}
179 179
180#define for_each_migratetype_order(order, type) \180#define for_each_free_list(list, zone, order) \
181 for (order = 0; order < NR_PAGE_ORDERS; order++) \181 for (order = 0; order < NR_PAGE_ORDERS; order++) \
182 for (type = 0; type < MIGRATE_TYPES; type++)182 for (unsigned int __type = 0; \
183 __type < MIGRATE_TYPES && \
184 (list = &(zone)->free_area[order].free_list[__type], 1); \
185 __type++)
183 186
184extern int page_group_by_mobility_disabled;187extern int page_group_by_mobility_disabled;
185 188
@@ -211,7 +214,6 @@ enum numa_stat_item { · 25 unmodified lines
211#endif214#endif
212 215
213enum zone_stat_item {216enum zone_stat_item {
214 /* First 128 byte cacheline (assuming 64 bit words) */
215 NR_FREE_PAGES,217 NR_FREE_PAGES,
216 NR_FREE_PAGES_BLOCKS,218 NR_FREE_PAGES_BLOCKS,
217 NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */219 NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */
@@ -222,7 +224,6 @@ enum zone_stat_item { · 4 unmodified lines
222 NR_ZONE_UNEVICTABLE,224 NR_ZONE_UNEVICTABLE,
223 NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */225 NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */
224 NR_MLOCK, /* mlock()ed pages found and moved off LRU */226 NR_MLOCK, /* mlock()ed pages found and moved off LRU */
225 /* Second 128 byte cacheline */
226#if IS_ENABLED(CONFIG_ZSMALLOC)227#if IS_ENABLED(CONFIG_ZSMALLOC)
227 NR_ZSPAGES, /* allocated in zsmalloc */228 NR_ZSPAGES, /* allocated in zsmalloc */
228#endif229#endif
@@ -1428,14 +1429,6 @@ struct zonelist { · 1199 unmodified lines
1428 */1429 */
1429extern struct page *mem_map;1430extern struct page *mem_map;
1430 1431
1431#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1432struct deferred_split {
1433 spinlock_t split_queue_lock;
1434 struct list_head split_queue;
1435 unsigned long split_queue_len;
1436};
1437#endif
1438
1439#ifdef CONFIG_MEMORY_FAILURE1432#ifdef CONFIG_MEMORY_FAILURE
1440/*1433/*
1441 * Per NUMA node memory failure handling statistics.1434 * Per NUMA node memory failure handling statistics.
@@ -1561,10 +1554,6 @@ typedef struct pglist_data { · 119 unmodified lines
1561 unsigned long first_deferred_pfn;1554 unsigned long first_deferred_pfn;
1562#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */1555#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
1563 1556
1564#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1565 struct deferred_split deferred_split_queue;
1566#endif
1567
1568#ifdef CONFIG_NUMA_BALANCING1557#ifdef CONFIG_NUMA_BALANCING
1569 /* start time in ms of current promote rate limit period */1558 /* start time in ms of current promote rate limit period */
1570 unsigned int nbp_rl_start;1559 unsigned int nbp_rl_start;
include/linux/nodemask.h+9−9
@@ -24,23 +24,23 @@ · 23 unmodified lines
24 * void nodes_setall(mask) set all bits24 * void nodes_setall(mask) set all bits
25 * void nodes_clear(mask) clear all bits25 * void nodes_clear(mask) clear all bits
26 * int node_isset(node, mask) true iff bit 'node' set in mask26 * int node_isset(node, mask) true iff bit 'node' set in mask
27 * int node_test_and_set(node, mask) test and set bit 'node' in mask27 * bool node_test_and_set(node, mask) test and set bit 'node' in mask
28 *28 *
29 * void nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]29 * bool nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
30 * void nodes_or(dst, src1, src2) dst = src1 | src2 [union]30 * void nodes_or(dst, src1, src2) dst = src1 | src2 [union]
31 * void nodes_xor(dst, src1, src2) dst = src1 ^ src231 * void nodes_xor(dst, src1, src2) dst = src1 ^ src2
32 * void nodes_andnot(dst, src1, src2) dst = src1 & ~src232 * bool nodes_andnot(dst, src1, src2) dst = src1 & ~src2
33 * void nodes_complement(dst, src) dst = ~src33 * void nodes_complement(dst, src) dst = ~src
34 *34 *
35 * int nodes_equal(mask1, mask2) Does mask1 == mask2?35 * bool nodes_equal(mask1, mask2) Does mask1 == mask2?
36 * int nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?36 * bool nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
37 * int nodes_subset(mask1, mask2) Is mask1 a subset of mask2?37 * bool nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
38 * int nodes_empty(mask) Is mask empty (no bits sets)?38 * bool nodes_empty(mask) Is mask empty (no bits sets)?
39 * int nodes_full(mask) Is mask full (all bits sets)?39 * bool nodes_full(mask) Is mask full (all bits sets)?
40 * int nodes_weight(mask) Hamming weight - number of set bits40 * int nodes_weight(mask) Hamming weight - number of set bits
41 *41 *
42 * unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES42 * unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES
43 * unsigend int next_node(node, mask) Next node past 'node', or MAX_NUMNODES43 * unsigned int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
44 * unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,44 * unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,
45 * or MAX_NUMNODES45 * or MAX_NUMNODES
46 * unsigned int first_unset_node(mask) First node not set in mask, or46 * unsigned int first_unset_node(mask) First node not set in mask, or
include/linux/page_ref.h+18−0
@@ -71,6 +71,12 @@ static inline int page_ref_count(const struct page *page) · 70 unmodified lines
71 * folio_ref_count - The reference count on this folio.71 * folio_ref_count - The reference count on this folio.
72 * @folio: The folio.72 * @folio: The folio.
73 *73 *
74 * Folios contain a reference count. When that reference count reaches
75 * zero, the folio is referred to as frozen. At this point, it will
76 * usually be returned to the memory allocator, but some parts of the
77 * kernel freeze folios in order to perform unusual operations on them
78 * such as splitting or migration.
79 *
74 * The refcount is usually incremented by calls to folio_get() and80 * The refcount is usually incremented by calls to folio_get() and
75 * decremented by calls to folio_put(). Some typical users of the81 * decremented by calls to folio_put(). Some typical users of the
76 * folio refcount:82 * folio refcount:
@@ -82,6 +88,18 @@ static inline int page_ref_count(const struct page *page) · 5 unmodified lines
82 * - Pipes88 * - Pipes
83 * - Direct IO which references this page in the process address space89 * - Direct IO which references this page in the process address space
84 *90 *
91 * The reference count has three components: expected, temporary and
92 * spurious. The expected reference count of a folio is that which
93 * we would logically expect it to be from just reading the code.
94 * Temporary refcounts are gained by threads which need a temporary
95 * reference to make sure the folio isn't reallocated while they use it.
96 * Spurious refcounts are gained by threads which, thanks to RCU walks
97 * of the page tables or file cache, find a stale pointer to a folio.
98 * These threads will drop the refcount after discoveering the pointer
99 * is stale, but it can surprise other users to see the spurious refcount
100 * on a freshly allocated folio (eg they may see a refcount of 2 instead
101 * of 1).
102 *
85 * Return: The number of references to this folio.103 * Return: The number of references to this folio.
86 */104 */
87static inline int folio_ref_count(const struct folio *folio)105static inline int folio_ref_count(const struct folio *folio)
include/linux/pageblock-flags.h+3−3
@@ -36,12 +36,12 @@ enum pageblock_bits { · 35 unmodified lines
36 36
37#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))37#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))
38 38
39#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))39#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
40 40
41#ifdef CONFIG_MEMORY_ISOLATION41#ifdef CONFIG_MEMORY_ISOLATION
42#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate))42#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
43#else43#else
44#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK44#define PAGEBLOCK_ISO_MASK 0
45#endif45#endif
46 46
47#if defined(CONFIG_HUGETLB_PAGE)47#if defined(CONFIG_HUGETLB_PAGE)
include/linux/pagemap.h+2−0
@@ -1350,6 +1350,7 @@ struct readahead_control { · 1349 unmodified lines
1350 struct file_ra_state *ra;1350 struct file_ra_state *ra;
1351/* private: use the readahead_* accessors instead */1351/* private: use the readahead_* accessors instead */
1352 pgoff_t _index;1352 pgoff_t _index;
1353 pgoff_t _max_index; /* limit readahead to _max_index, inclusive */
1353 unsigned int _nr_pages;1354 unsigned int _nr_pages;
1354 unsigned int _batch_count;1355 unsigned int _batch_count;
1355 bool dropbehind;1356 bool dropbehind;
@@ -1363,6 +1364,7 @@ struct readahead_control { · 7 unmodified lines
1363 .mapping = m, \1364 .mapping = m, \
1364 .ra = r, \1365 .ra = r, \
1365 ._index = i, \1366 ._index = i, \
1367 ._max_index = ULONG_MAX, \
1366 }1368 }
1367 1369
1368#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)1370#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)
include/linux/swap.h+12−12
@@ -213,6 +213,7 @@ enum { · 212 unmodified lines
213 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */213 SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
214 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */214 SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
215 SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */215 SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
216 SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
216 /* add others here before... */217 /* add others here before... */
217};218};
218 219
@@ -252,7 +253,6 @@ struct swap_info_struct { · 33 unmodified lines
252 struct plist_node list; /* entry in swap_active_head */253 struct plist_node list; /* entry in swap_active_head */
253 signed char type; /* strange name for an index */254 signed char type; /* strange name for an index */
254 unsigned int max; /* size of this swap device */255 unsigned int max; /* size of this swap device */
255 unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
256 struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */256 struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
257 struct list_head free_clusters; /* free clusters list */257 struct list_head free_clusters; /* free clusters list */
258 struct list_head full_clusters; /* full clusters list */258 struct list_head full_clusters; /* full clusters list */
@@ -433,7 +433,9 @@ static inline long get_nr_swap_pages(void) · 174 unmodified lines
433}433}
434 434
435extern void si_swapinfo(struct sysinfo *);435extern void si_swapinfo(struct sysinfo *);
436int swap_type_of(dev_t device, sector_t offset);436extern int pin_hibernation_swap_type(dev_t device, sector_t offset);
437extern void unpin_hibernation_swap_type(int type);
438extern int find_hibernation_swap_type(dev_t device, sector_t offset);
437int find_first_swap(dev_t *device);439int find_first_swap(dev_t *device);
438extern unsigned int count_swap_pages(int, int);440extern unsigned int count_swap_pages(int, int);
439extern sector_t swapdev_block(int, pgoff_t);441extern sector_t swapdev_block(int, pgoff_t);
@@ -571,33 +573,31 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp) · 131 unmodified lines
571#endif573#endif
572 574
573#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)575#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
574int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);576int __mem_cgroup_try_charge_swap(struct folio *folio);
575static inline int mem_cgroup_try_charge_swap(struct folio *folio,577static inline int mem_cgroup_try_charge_swap(struct folio *folio)
576 swp_entry_t entry)
577{578{
578 if (mem_cgroup_disabled())579 if (mem_cgroup_disabled())
579 return 0;580 return 0;
580 return __mem_cgroup_try_charge_swap(folio, entry);581 return __mem_cgroup_try_charge_swap(folio);
581}582}
582 583
583extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);584extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
584static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)585static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
585{586{
586 if (mem_cgroup_disabled())587 if (mem_cgroup_disabled())
587 return;588 return;
588 __mem_cgroup_uncharge_swap(entry, nr_pages);589 __mem_cgroup_uncharge_swap(id, nr_pages);
589}590}
590 591
591extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);592extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
592extern bool mem_cgroup_swap_full(struct folio *folio);593extern bool mem_cgroup_swap_full(struct folio *folio);
593#else594#else
594static inline int mem_cgroup_try_charge_swap(struct folio *folio,595static inline int mem_cgroup_try_charge_swap(struct folio *folio)
595 swp_entry_t entry)
596{596{
597 return 0;597 return 0;
598}598}
599 599
600static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,600static inline void mem_cgroup_uncharge_swap(unsigned short id,
601 unsigned int nr_pages)601 unsigned int nr_pages)
602{602{
603}603}
include/linux/swap_cgroup.h+0−47
@@ -1,47 +0,0 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2#ifndef __LINUX_SWAP_CGROUP_H
3#define __LINUX_SWAP_CGROUP_H
4
5#include <linux/swap.h>
6
7#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
8
9extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
10extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
11extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
12extern int swap_cgroup_swapon(int type, unsigned long max_pages);
13extern void swap_cgroup_swapoff(int type);
14
15#else
16
17static inline
18void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
19{
20}
21
22static inline
23unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
24{
25 return 0;
26}
27
28static inline
29unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
30{
31 return 0;
32}
33
34static inline int
35swap_cgroup_swapon(int type, unsigned long max_pages)
36{
37 return 0;
38}
39
40static inline void swap_cgroup_swapoff(int type)
41{
42 return;
43}
44
45#endif
46
47#endif /* __LINUX_SWAP_CGROUP_H */
include/linux/thread_info.h+1−1
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart, · 91 unmodified lines
92#define THREAD_ALIGN THREAD_SIZE92#define THREAD_ALIGN THREAD_SIZE
93#endif93#endif
94 94
95#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO)95#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)
96 96
97/*97/*
98 * flag set/clear/test wrappers98 * flag set/clear/test wrappers
include/linux/userfaultfd_k.h+2−38
@@ -23,8 +23,8 @@ · 22 unmodified lines
23/* The set of all possible UFFD-related VM flags. */23/* The set of all possible UFFD-related VM flags. */
24#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)24#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
25 25
26#define __VMA_UFFD_FLAGS mk_vma_flags(VMA_UFFD_MISSING_BIT, VMA_UFFD_WP_BIT, \26#define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \
27 VMA_UFFD_MINOR_BIT)27 VMA_UFFD_MINOR)
28 28
29/*29/*
30 * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining30 * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -147,26 +147,12 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at · 116 unmodified lines
147/* Flags controlling behavior. These behavior changes are mode-independent. */147/* Flags controlling behavior. These behavior changes are mode-independent. */
148#define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)148#define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
149 149
150extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
151 unsigned long src_start, unsigned long len,
152 uffd_flags_t flags);
153extern ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
154 unsigned long dst_start,
155 unsigned long len);
156extern ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long dst_start,
157 unsigned long len, uffd_flags_t flags);
158extern ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
159 unsigned long len, uffd_flags_t flags);
160extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
161 unsigned long len, bool enable_wp);
162extern long uffd_wp_range(struct vm_area_struct *vma,150extern long uffd_wp_range(struct vm_area_struct *vma,
163 unsigned long start, unsigned long len, bool enable_wp);151 unsigned long start, unsigned long len, bool enable_wp);
164 152
165/* move_pages */153/* move_pages */
166void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);154void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
167void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);155void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
168ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
169 unsigned long src_start, unsigned long len, __u64 flags);
170int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,156int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,
171 struct vm_area_struct *dst_vma,157 struct vm_area_struct *dst_vma,
172 struct vm_area_struct *src_vma,158 struct vm_area_struct *src_vma,
@@ -239,9 +225,6 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 66 unmodified lines
239 return vma->vm_flags & __VM_UFFD_FLAGS;225 return vma->vm_flags & __VM_UFFD_FLAGS;
240}226}
241 227
242bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
243 bool wp_async);
244
245static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)228static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
246{229{
247 struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;230 struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
@@ -271,25 +254,6 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm, · 23 unmodified lines
271extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);254extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
272extern bool userfaultfd_wp_async(struct vm_area_struct *vma);255extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
273 256
274void userfaultfd_reset_ctx(struct vm_area_struct *vma);
275
276struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
277 struct vm_area_struct *prev,
278 struct vm_area_struct *vma,
279 unsigned long start,
280 unsigned long end);
281
282int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
283 struct vm_area_struct *vma,
284 vm_flags_t vm_flags,
285 unsigned long start, unsigned long end,
286 bool wp_async);
287
288void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
289
290void userfaultfd_release_all(struct mm_struct *mm,
291 struct userfaultfd_ctx *ctx);
292
293static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)257static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
294{258{
295 /* Only wr-protect mode uses pte markers */259 /* Only wr-protect mode uses pte markers */
include/linux/vmalloc.h+3−1
@@ -265,7 +265,9 @@ static inline bool is_vm_area_hugepages(const void *addr) · 264 unmodified lines
265 * allocated in the vmalloc layer.265 * allocated in the vmalloc layer.
266 */266 */
267#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC267#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
268 return find_vm_area(addr)->page_order > 0;268 struct vm_struct *area = find_vm_area(addr);
269
270 return area && area->page_order > 0;
269#else271#else
270 return false;272 return false;
271#endif273#endif
include/linux/vmpressure.h+5−4
@@ -30,8 +30,8 @@ struct vmpressure { · 29 unmodified lines
30struct mem_cgroup;30struct mem_cgroup;
31 31
32#ifdef CONFIG_MEMCG32#ifdef CONFIG_MEMCG
33extern void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,33void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
34 unsigned long scanned, unsigned long reclaimed);34 unsigned long scanned, unsigned long reclaimed);
35extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);35extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);
36 36
37extern void vmpressure_init(struct vmpressure *vmpr);37extern void vmpressure_init(struct vmpressure *vmpr);
@@ -44,8 +44,9 @@ extern int vmpressure_register_event(struct mem_cgroup *memcg, · 6 unmodified lines
44extern void vmpressure_unregister_event(struct mem_cgroup *memcg,44extern void vmpressure_unregister_event(struct mem_cgroup *memcg,
45 struct eventfd_ctx *eventfd);45 struct eventfd_ctx *eventfd);
46#else46#else
47static inline void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,47static inline void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg,
48 unsigned long scanned, unsigned long reclaimed) {}48 bool tree, unsigned long scanned,
49 unsigned long reclaimed) {}
49static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,50static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,
50 int prio) {}51 int prio) {}
51#endif /* CONFIG_MEMCG */52#endif /* CONFIG_MEMCG */
include/trace/events/damon.h+38−0
@@ -130,6 +130,44 @@ TRACE_EVENT(damon_monitor_intervals_tune, · 129 unmodified lines
130 TP_printk("sample_us=%lu", __entry->sample_us)130 TP_printk("sample_us=%lu", __entry->sample_us)
131);131);
132 132
133TRACE_EVENT_CONDITION(damon_region_aggregated,
134
135 TP_PROTO(unsigned int target_id, struct damon_region *r,
136 unsigned int nr_regions, unsigned int nr_probes),
137
138 TP_ARGS(target_id, r, nr_regions, nr_probes),
139
140 TP_CONDITION(nr_probes > 0),
141
142 TP_STRUCT__entry(
143 __field(unsigned long, target_id)
144 __field(unsigned long, start)
145 __field(unsigned long, end)
146 __field(unsigned int, nr_regions)
147 __field(unsigned int, nr_accesses)
148 __field(unsigned int, age)
149 __dynamic_array(unsigned char, probe_hits, nr_probes)
150 ),
151
152 TP_fast_assign(
153 __entry->target_id = target_id;
154 __entry->start = r->ar.start;
155 __entry->end = r->ar.end;
156 __entry->nr_regions = nr_regions;
157 __entry->nr_accesses = r->nr_accesses;
158 __entry->age = r->age;
159 memcpy(__get_dynamic_array(probe_hits), r->probe_hits,
160 sizeof(*r->probe_hits) * nr_probes);
161 ),
162
163 TP_printk("target_id=%lu nr_regions=%u %lu-%lu: %u %u probe_hits=%s",
164 __entry->target_id, __entry->nr_regions,
165 __entry->start, __entry->end,
166 __entry->nr_accesses, __entry->age,
167 __print_hex(__get_dynamic_array(probe_hits),
168 __get_dynamic_array_len(probe_hits)))
169);
170
133TRACE_EVENT(damon_aggregated,171TRACE_EVENT(damon_aggregated,
134 172
135 TP_PROTO(unsigned int target_id, struct damon_region *r,173 TP_PROTO(unsigned int target_id, struct damon_region *r,
include/trace/events/vmscan.h+52−0
@@ -96,6 +96,58 @@ TRACE_EVENT(mm_vmscan_kswapd_wake, · 95 unmodified lines
96 __entry->order)96 __entry->order)
97);97);
98 98
99TRACE_EVENT(mm_vmscan_balance_pgdat_begin,
100
101 TP_PROTO(int nid, int order, int highest_zoneidx),
102
103 TP_ARGS(nid, order, highest_zoneidx),
104
105 TP_STRUCT__entry(
106 __field(int, nid)
107 __field(int, order)
108 __field(int, highest_zoneidx)
109 ),
110
111 TP_fast_assign(
112 __entry->nid = nid;
113 __entry->order = order;
114 __entry->highest_zoneidx = highest_zoneidx;
115 ),
116
117 TP_printk("nid=%d order=%d highest_zoneidx=%-8s",
118 __entry->nid,
119 __entry->order,
120 __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE))
121);
122
123TRACE_EVENT(mm_vmscan_balance_pgdat_end,
124
125 TP_PROTO(int nid, int order, int highest_zoneidx,
126 unsigned long nr_reclaimed),
127
128 TP_ARGS(nid, order, highest_zoneidx, nr_reclaimed),
129
130 TP_STRUCT__entry(
131 __field(int, nid)
132 __field(int, order)
133 __field(int, highest_zoneidx)
134 __field(unsigned long, nr_reclaimed)
135 ),
136
137 TP_fast_assign(
138 __entry->nid = nid;
139 __entry->order = order;
140 __entry->highest_zoneidx = highest_zoneidx;
141 __entry->nr_reclaimed = nr_reclaimed;
142 ),
143
144 TP_printk("nid=%d order=%d highest_zoneidx=%-8s nr_reclaimed=%lu",
145 __entry->nid,
146 __entry->order,
147 __print_symbolic(__entry->highest_zoneidx, ZONE_TYPE),
148 __entry->nr_reclaimed)
149);
150
99TRACE_EVENT(mm_vmscan_wakeup_kswapd,151TRACE_EVENT(mm_vmscan_wakeup_kswapd,
100 152
101 TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),153 TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t