lael-labs/linux

Documentation/ABI/testing/sysfs-kernel-mm-damon+6−0

 Description:	Writing to and reading from this file sets and gets the action
 		of the scheme.
+What:		/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/target_nid
+Date:		Jun 2024
+Contact:	SeongJae Park <sj@kernel.org>
+Description:	Action's target NUMA node id.  Supported by only relevant
+		actions.
 What:		/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us
 Date:		Sep 2023
 Contact:	SeongJae Park <sj@kernel.org>

Documentation/admin-guide/cgroup-v2.rst+11−7

 	This is a simple interface to trigger memory reclaim in the
 	target cgroup.
-	This file accepts a single key, the number of bytes to reclaim.
-	No nested keys are currently supported.
 	Example::
 	  echo "1G" > memory.reclaim
-	The interface can be later extended with nested keys to
-	configure the reclaim behavior. For example, specify the
-	type of memory to reclaim from (anon, file, ..).
 	Please note that the kernel can over or under reclaim from
 	the target cgroup. If less bytes are reclaimed than the
 	specified amount, -EAGAIN is returned.
 	This means that the networking layer will not adapt based on
 	reclaim induced by memory.reclaim.
+The following nested keys are defined.
+	  ==========            ================================
+	  swappiness            Swappiness value to reclaim with
+	  ==========            ================================
+	Specifying a swappiness value instructs the kernel to perform
+	the reclaim with that swappiness value. Note that this has the
+	same semantics as vm.swappiness applied to memcg reclaim with
+	all the existing limitations and potential future extensions.
   memory.peak
 	A read-only single value file which exists on non-root
 	cgroups.

Documentation/admin-guide/kernel-parameters.txt+6−3

 	vmalloc=nn[KMG]	[KNL,BOOT,EARLY] Forces the vmalloc area to have an
 			exact size of <nn>. This can be used to increase
-			the minimum size (128MB on x86). It can also be
+			the minimum size (128MB on x86, arm32 platforms).
-			used to decrease the size and leave more room
+			It can also be used to decrease the size and leave more room
-			for directly mapped kernel RAM.
+			for directly mapped kernel RAM. Note that this parameter does
+			not exist on many other platforms (including arm64, alpha,
+			loongarch, arc, csky, hexagon, microblaze, mips, nios2, openrisc,
+			parisc, m64k, powerpc, riscv, sh, um, xtensa, s390, sparc).
 	vmcp_cma=nn[MG]	[KNL,S390,EARLY]
 			Sets the memory size reserved for contiguous memory

Documentation/admin-guide/mm/damon/start.rst+42−4

@@ -34,18 +34,56 @@ detail) of DAMON, you should ensure :doc:`sysfs </filesystems/sysfs>` is · 33 unmodified lines
34	mounted.	34	mounted.
35		35
36		36
		37	Snapshot Data Access Patterns
		38	=============================
		39
		40	The commands below show the memory access pattern of a program at the moment of
		41	the execution. ::
		42
		43	$ git clone https://github.com/sjp38/masim; cd masim; make
		44	$ sudo damo start "./masim ./configs/stairs.cfg --quiet"
		45	$ sudo ./damo show
		46	0 addr [85.541 TiB , 85.541 TiB ) (57.707 MiB ) access 0 % age 10.400 s
		47	1 addr [85.541 TiB , 85.542 TiB ) (413.285 MiB) access 0 % age 11.400 s
		48	2 addr [127.649 TiB , 127.649 TiB) (57.500 MiB ) access 0 % age 1.600 s
		49	3 addr [127.649 TiB , 127.649 TiB) (32.500 MiB ) access 0 % age 500 ms
		50	4 addr [127.649 TiB , 127.649 TiB) (9.535 MiB ) access 100 % age 300 ms
		51	5 addr [127.649 TiB , 127.649 TiB) (8.000 KiB ) access 60 % age 0 ns
		52	6 addr [127.649 TiB , 127.649 TiB) (6.926 MiB ) access 0 % age 1 s
		53	7 addr [127.998 TiB , 127.998 TiB) (120.000 KiB) access 0 % age 11.100 s
		54	8 addr [127.998 TiB , 127.998 TiB) (8.000 KiB ) access 40 % age 100 ms
		55	9 addr [127.998 TiB , 127.998 TiB) (4.000 KiB ) access 0 % age 11 s
		56	total size: 577.590 MiB
		57	$ sudo ./damo stop
		58
		59	The first command of the above example downloads and builds an artificial
		60	memory access generator program called ``masim``. The second command asks DAMO
		61	to execute the artificial generator process start via the given command and
		62	make DAMON monitors the generator process. The third command retrieves the
		63	current snapshot of the monitored access pattern of the process from DAMON and
		64	shows the pattern in a human readable format.
		65
		66	Each line of the output shows which virtual address range (``addr [XX, XX)``)
		67	of the process is how frequently (``access XX %``) accessed for how long time
		68	(``age XX``). For example, the fifth region of ~9 MiB size is being most
		69	frequently accessed for last 300 milliseconds. Finally, the fourth command
		70	stops DAMON.
		71
		72	Note that DAMON can monitor not only virtual address spaces but multiple types
		73	of address spaces including the physical address space.
		74
		75
37	Recording Data Access Patterns	76	Recording Data Access Patterns
38	==============================	77	==============================
39		78
40	The commands below record the memory access patterns of a program and save the	79	The commands below record the memory access patterns of a program and save the
41	monitoring results to a file. ::	80	monitoring results to a file. ::
42		81
43	$ git clone https://github.com/sjp38/masim	82	$ ./masim ./configs/zigzag.cfg &
44	$ cd masim; make; ./masim ./configs/zigzag.cfg &
45	$ sudo damo record -o damon.data $(pidof masim)	83	$ sudo damo record -o damon.data $(pidof masim)
46		84
47	The first two lines of the commands download an artificial memory access	85	The line of the commands run the artificial memory access
48	generator program and run it in the background. The generator will repeatedly	86	generator program again. The generator will repeatedly
49	access two 100 MiB sized memory regions one by one. You can substitute this	87	access two 100 MiB sized memory regions one by one. You can substitute this
50	with your real workload. The last line asks ``damo`` to record the access	88	with your real workload. The last line asks ``damo`` to record the access
51	pattern in the ``damon.data`` file.	89	pattern in the ``damon.data`` file.

Documentation/admin-guide/mm/damon/usage.rst+7−3

     │ │ │ │ │ │ │ │ ...
     │ │ │ │ │ │ ...
     │ │ │ │ │ :ref:`schemes <sysfs_schemes>`/nr_schemes
-    │ │ │ │ │ │ :ref:`0 <sysfs_scheme>`/action,apply_interval_us
+    │ │ │ │ │ │ :ref:`0 <sysfs_scheme>`/action,target_nid,apply_interval_us
     │ │ │ │ │ │ │ :ref:`access_pattern <sysfs_access_pattern>`/
     │ │ │ │ │ │ │ │ sz/min,max
     │ │ │ │ │ │ │ │ nr_accesses/min,max
 ------------
 In each scheme directory, five directories (``access_pattern``, ``quotas``,
-``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files
+``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and three files
-(``action`` and ``apply_interval``) exist.
+(``action``, ``target_nid`` and ``apply_interval``) exist.
 The ``action`` file is for setting and getting the scheme's :ref:`action
 <damon_design_damos_action>`.  The keywords that can be written to and read
 from the file and their meaning are same to those of the list on
 :ref:`design doc <damon_design_damos_action>`.
+The ``target_nid`` file is for setting the migration target node, which is
+only meaningful when the ``action`` is either ``migrate_hot`` or
+``migrate_cold``.
 The ``apply_interval_us`` file is for setting and getting the scheme's
 :ref:`apply_interval <damon_design_damos>` in microseconds.

Documentation/admin-guide/mm/pagemap.rst+2−23

@@ -118,7 +118,7 @@ Short descriptions to the page flags · 117 unmodified lines
118	21 - KSM	118	21 - KSM
119	Identical memory pages dynamically shared between one or more processes.	119	Identical memory pages dynamically shared between one or more processes.
120	22 - THP	120	22 - THP
121	Contiguous pages which construct transparent hugepages.	121	Contiguous pages which construct THP of any size and mapped by any granularity.
122	23 - OFFLINE	122	23 - OFFLINE
123	The page is logically offline.	123	The page is logically offline.
124	24 - ZERO_PAGE	124	24 - ZERO_PAGE
@@ -173,27 +173,6 @@ LRU related page flags · 48 unmodified lines
173	The page-types tool in the tools/mm directory can be used to query the	173	The page-types tool in the tools/mm directory can be used to query the
174	above flags.	174	above flags.
175		175
176	Using pagemap to do something useful
177	====================================
178
179	The general procedure for using pagemap to find out about a process' memory
180	usage goes like this:
181
182	1. Read ``/proc/pid/maps`` to determine which parts of the memory space are
183	mapped to what.
184	2. Select the maps you are interested in -- all of them, or a particular
185	library, or the stack or the heap, etc.
186	3. Open ``/proc/pid/pagemap`` and seek to the pages you would like to examine.
187	4. Read a u64 for each page from pagemap.
188	5. Open ``/proc/kpagecount`` and/or ``/proc/kpageflags``. For each PFN you
189	just read, seek to that entry in the file, and read the data you want.
190
191	For example, to find the "unique set size" (USS), which is the amount of
192	memory that a process is using that is not shared with any other process,
193	you can go through every map in the process, find the PFNs, look those up
194	in kpagecount, and tally up the number of pages that are only referenced
195	once.
196
197	Exceptions for Shared Memory	176	Exceptions for Shared Memory
198	============================	177	============================
199		178
@@ -252,7 +231,7 @@ Following flags about pages are currently supported: · 52 unmodified lines
252	- ``PAGE_IS_PRESENT`` - Page is present in the memory	231	- ``PAGE_IS_PRESENT`` - Page is present in the memory
253	- ``PAGE_IS_SWAPPED`` - Page is in swapped	232	- ``PAGE_IS_SWAPPED`` - Page is in swapped
254	- ``PAGE_IS_PFNZERO`` - Page has zero PFN	233	- ``PAGE_IS_PFNZERO`` - Page has zero PFN
255	- ``PAGE_IS_HUGE`` - Page is THP or Hugetlb backed	234	- ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed
256	- ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty	235	- ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty
257		236
258	The ``struct pm_scan_arg`` is used as the argument of the IOCTL.	237	The ``struct pm_scan_arg`` is used as the argument of the IOCTL.

Documentation/admin-guide/mm/transhuge.rst+68−17

@@ -202,12 +202,11 @@ PMD-mappable transparent hugepage:: · 201 unmodified lines
202		202
203	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size	203	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
204		204
205	khugepaged will be automatically started when one or more hugepage	205	khugepaged will be automatically started when PMD-sized THP is enabled
206	sizes are enabled (either by directly setting "always" or "madvise",	206	(either of the per-size anon control or the top-level control are set
207	or by setting "inherit" while the top-level enabled is set to "always"	207	to "always" or "madvise"), and it'll be automatically shutdown when
208	or "madvise"), and it'll be automatically shutdown when the last	208	PMD-sized THP is disabled (when both the per-size anon control and the
209	hugepage size is disabled (either by directly setting "never", or by	209	top-level control are "never")
210	setting "inherit" while the top-level enabled is set to "never").
211		210
212	Khugepaged controls	211	Khugepaged controls
213	-------------------	212	-------------------
@@ -332,6 +331,31 @@ deny · 118 unmodified lines
332	force	331	force
333	Force the huge option on for all - very useful for testing;	332	Force the huge option on for all - very useful for testing;
334		333
		334	Shmem can also use "multi-size THP" (mTHP) by adding a new sysfs knob to
		335	control mTHP allocation:
		336	'/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/shmem_enabled',
		337	and its value for each mTHP is essentially consistent with the global
		338	setting. An 'inherit' option is added to ensure compatibility with these
		339	global settings. Conversely, the options 'force' and 'deny' are dropped,
		340	which are rather testing artifacts from the old ages.
		341
		342	always
		343	Attempt to allocate <size> huge pages every time we need a new page;
		344
		345	inherit
		346	Inherit the top-level "shmem_enabled" value. By default, PMD-sized hugepages
		347	have enabled="inherit" and all other hugepage sizes have enabled="never";
		348
		349	never
		350	Do not allocate <size> huge pages;
		351
		352	within_size
		353	Only allocate <size> huge page if it will be fully within i_size.
		354	Also respect fadvise()/madvise() hints;
		355
		356	advise
		357	Only allocate <size> huge pages if requested with fadvise()/madvise();
		358
335	Need of application restart	359	Need of application restart
336	===========================	360	===========================
337		361
@@ -344,10 +368,6 @@ also applies to the regions registered in khugepaged. · 6 unmodified lines
344	Monitoring usage	368	Monitoring usage
345	================	369	================
346		370
347	.. note::
348	Currently the below counters only record events relating to
349	PMD-sized THP. Events relating to other THP sizes are not included.
350
351	The number of PMD-sized anonymous transparent huge pages currently used by the	371	The number of PMD-sized anonymous transparent huge pages currently used by the
352	system is available by reading the AnonHugePages field in ``/proc/meminfo``.	372	system is available by reading the AnonHugePages field in ``/proc/meminfo``.
353	To identify what applications are using PMD-sized anonymous transparent huge	373	To identify what applications are using PMD-sized anonymous transparent huge
@@ -392,20 +412,23 @@ thp_collapse_alloc_failed · 38 unmodified lines
392	the allocation.	412	the allocation.
393		413
394	thp_file_alloc	414	thp_file_alloc
395	is incremented every time a file huge page is successfully	415	is incremented every time a shmem huge page is successfully
396	allocated.	416	allocated (Note that despite being named after "file", the counter
		417	measures only shmem).
397		418
398	thp_file_fallback	419	thp_file_fallback
399	is incremented if a file huge page is attempted to be allocated	420	is incremented if a shmem huge page is attempted to be allocated
400	but fails and instead falls back to using small pages.	421	but fails and instead falls back to using small pages. (Note that
		422	despite being named after "file", the counter measures only shmem).
401		423
402	thp_file_fallback_charge	424	thp_file_fallback_charge
403	is incremented if a file huge page cannot be charged and instead	425	is incremented if a shmem huge page cannot be charged and instead
404	falls back to using small pages even though the allocation was	426	falls back to using small pages even though the allocation was
405	successful.	427	successful. (Note that despite being named after "file", the
		428	counter measures only shmem).
406		429
407	thp_file_mapped	430	thp_file_mapped
408	is incremented every time a file huge page is mapped into	431	is incremented every time a file or shmem huge page is mapped into
409	user address space.	432	user address space.
410		433
411	thp_split_page	434	thp_split_page
@@ -476,6 +499,34 @@ swpout_fallback · 64 unmodified lines
476	Usually because failed to allocate some continuous swap space	499	Usually because failed to allocate some continuous swap space
477	for the huge page.	500	for the huge page.
478		501
		502	shmem_alloc
		503	is incremented every time a shmem huge page is successfully
		504	allocated.
		505
		506	shmem_fallback
		507	is incremented if a shmem huge page is attempted to be allocated
		508	but fails and instead falls back to using small pages.
		509
		510	shmem_fallback_charge
		511	is incremented if a shmem huge page cannot be charged and instead
		512	falls back to using small pages even though the allocation was
		513	successful.
		514
		515	split
		516	is incremented every time a huge page is successfully split into
		517	smaller orders. This can happen for a variety of reasons but a
		518	common reason is that a huge page is old and is being reclaimed.
		519
		520	split_failed
		521	is incremented if kernel fails to split huge
		522	page. This can happen if the page was pinned by somebody.
		523
		524	split_deferred
		525	is incremented when a huge page is put onto split queue.
		526	This happens when a huge page is partially unmapped and splitting
		527	it would free up some memory. Pages on split queue are going to
		528	be split under memory pressure, if splitting is possible.
		529
479	As the system ages, allocating huge pages may be expensive as the	530	As the system ages, allocating huge pages may be expensive as the
480	system uses memory compaction to copy data around memory to free a	531	system uses memory compaction to copy data around memory to free a
481	huge page for use. There are some counters in ``/proc/vmstat`` to help	532	huge page for use. There are some counters in ``/proc/vmstat`` to help

Documentation/admin-guide/sysctl/vm.rst+38−0

@@ -36,6 +36,7 @@ Currently, these files are in /proc/sys/vm: · 35 unmodified lines
36	- dirtytime_expire_seconds	36	- dirtytime_expire_seconds
37	- dirty_writeback_centisecs	37	- dirty_writeback_centisecs
38	- drop_caches	38	- drop_caches
		39	- enable_soft_offline
39	- extfrag_threshold	40	- extfrag_threshold
40	- highmem_is_dirtyable	41	- highmem_is_dirtyable
41	- hugetlb_shm_group	42	- hugetlb_shm_group
@@ -267,6 +268,43 @@ used:: · 225 unmodified lines
267	These are informational only. They do not mean that anything is wrong	268	These are informational only. They do not mean that anything is wrong
268	with your system. To disable them, echo 4 (bit 2) into drop_caches.	269	with your system. To disable them, echo 4 (bit 2) into drop_caches.
269		270
		271	enable_soft_offline
		272	===================
		273	Correctable memory errors are very common on servers. Soft-offline is kernel's
		274	solution for memory pages having (excessive) corrected memory errors.
		275
		276	For different types of page, soft-offline has different behaviors / costs.
		277
		278	- For a raw error page, soft-offline migrates the in-use page's content to
		279	a new raw page.
		280
		281	- For a page that is part of a transparent hugepage, soft-offline splits the
		282	transparent hugepage into raw pages, then migrates only the raw error page.
		283	As a result, user is transparently backed by 1 less hugepage, impacting
		284	memory access performance.
		285
		286	- For a page that is part of a HugeTLB hugepage, soft-offline first migrates
		287	the entire HugeTLB hugepage, during which a free hugepage will be consumed
		288	as migration target. Then the original hugepage is dissolved into raw
		289	pages without compensation, reducing the capacity of the HugeTLB pool by 1.
		290
		291	It is user's call to choose between reliability (staying away from fragile
		292	physical memory) vs performance / capacity implications in transparent and
		293	HugeTLB cases.
		294
		295	For all architectures, enable_soft_offline controls whether to soft offline
		296	memory pages. When set to 1, kernel attempts to soft offline the pages
		297	whenever it thinks needed. When set to 0, kernel returns EOPNOTSUPP to
		298	the request to soft offline the pages. Its default value is 1.
		299
		300	It is worth mentioning that after setting enable_soft_offline to 0, the
		301	following requests to soft offline pages will not be performed:
		302
		303	- Request to soft offline pages from RAS Correctable Errors Collector.
		304
		305	- On ARM, the request to soft offline pages from GHES driver.
		306
		307	- On PARISC, the request to soft offline pages from Page Deallocation Table.
270		308
271	extfrag_threshold	309	extfrag_threshold
272	=================	310	=================

Documentation/core-api/pin_user_pages.rst+9−9

Documentation/dev-tools/kmsan.rst+9−2

 function in the file or directory. Most users won't need KMSAN_SANITIZE, unless
 their code gets broken by KMSAN (e.g. runs at early boot time).
+KMSAN checks can also be temporarily disabled for the current task using
+``kmsan_disable_current()`` and ``kmsan_enable_current()`` calls. Each
+``kmsan_enable_current()`` call must be preceded by a
+``kmsan_disable_current()`` call; these call pairs may be nested. One needs to
+be careful with these calls, keeping the regions short and preferring other
+ways to disable instrumentation, where possible.
 Support
 =======
 ~~~~~~~~~~~~~~~~~~~~
 Every task_struct has an associated KMSAN task state that holds the KMSAN
-context (see above) and a per-task flag disallowing KMSAN reports::
+context (see above) and a per-task counter disallowing KMSAN reports::
   struct kmsan_context {
     ...
-    bool allow_reporting;
+    unsigned int depth;
     struct kmsan_context_state cstate;
     ...
   }

Documentation/filesystems/proc.rst+9−0

@@ -443,6 +443,15 @@ is not associated with a file: · 442 unmodified lines
443		443
444	or if empty, the mapping is anonymous.	444	or if empty, the mapping is anonymous.
445		445
		446	Starting with 6.11 kernel, /proc/PID/maps provides an alternative
		447	ioctl()-based API that gives ability to flexibly and efficiently query and
		448	filter individual VMAs. This interface is binary and is meant for more
		449	efficient and easy programmatic use. `struct procmap_query`, defined in
		450	linux/fs.h UAPI header, serves as an input/output argument to the
		451	`PROCMAP_QUERY` ioctl() command. See comments in linus/fs.h UAPI header for
		452	details on query semantics, supported flags, data returned, and general API
		453	usage information.
		454
446	The /proc/PID/smaps is an extension based on maps, showing the memory	455	The /proc/PID/smaps is an extension based on maps, showing the memory
447	consumption for each of the process's mappings. For each mapping (aka Virtual	456	consumption for each of the process's mappings. For each mapping (aka Virtual
448	Memory Area, or VMA) there is a series of lines such as the following::	457	Memory Area, or VMA) there is a series of lines such as the following::

Documentation/mm/arch_pgtable_helpers.rst+0−4

 +---------------------------+--------------------------------------------------+
 | pmd_leaf                  | Tests a leaf mapped PMD                          |
 +---------------------------+--------------------------------------------------+
-| pmd_huge                  | Tests a HugeTLB mapped PMD                       |
-+---------------------------+--------------------------------------------------+
 | pmd_trans_huge            | Tests a Transparent Huge Page (THP) at PMD       |
 +---------------------------+--------------------------------------------------+
 | pmd_present               | Tests whether pmd_page() points to valid memory  |
 +---------------------------+--------------------------------------------------+
 | pud_leaf                  | Tests a leaf mapped PUD                          |
 +---------------------------+--------------------------------------------------+
-| pud_huge                  | Tests a HugeTLB mapped PUD                       |
-+---------------------------+--------------------------------------------------+
 | pud_trans_huge            | Tests a Transparent Huge Page (THP) at PUD       |
 +---------------------------+--------------------------------------------------+
 | pud_present               | Tests a valid mapped PUD                         |

Documentation/mm/damon/design.rst+101−48

@@ -16,53 +16,24 @@ called DAMON ``context``. DAMON executes each context with a kernel thread · 15 unmodified lines
16	called ``kdamond``. Multiple kdamonds could run in parallel, for different	16	called ``kdamond``. Multiple kdamonds could run in parallel, for different
17	types of monitoring.	17	types of monitoring.
18		18
		19	To know how user-space can do the configurations and start/stop DAMON, refer to
		20	:ref:`DAMON sysfs interface <sysfs_interface>` documentation.
		21
19		22
20	Overall Architecture	23	Overall Architecture
21	====================	24	====================
22		25
23	DAMON subsystem is configured with three layers including	26	DAMON subsystem is configured with three layers including
24		27
25	- Operations Set: Implements fundamental operations for DAMON that depends on	28	- :ref:`Operations Set <damon_operations_set>`: Implements fundamental
26	the given monitoring target address-space and available set of	29	operations for DAMON that depends on the given monitoring target
27	software/hardware primitives,	30	address-space and available set of software/hardware primitives,
28	- Core: Implements core logics including monitoring overhead/accurach control	31	- :ref:`Core <damon_core_logic>`: Implements core logics including monitoring
29	and access-aware system operations on top of the operations set layer, and	32	overhead/accuracy control and access-aware system operations on top of the
30	- Modules: Implements kernel modules for various purposes that provides	33	operations set layer, and
31	interfaces for the user space, on top of the core layer.	34	- :ref:`Modules <damon_modules>`: Implements kernel modules for various
32		35	purposes that provides interfaces for the user space, on top of the core
33		36	layer.
34	.. _damon_design_configurable_operations_set:
35
36	Configurable Operations Set
37	---------------------------
38
39	For data access monitoring and additional low level work, DAMON needs a set of
40	implementations for specific operations that are dependent on and optimized for
41	the given target address space. On the other hand, the accuracy and overhead
42	tradeoff mechanism, which is the core logic of DAMON, is in the pure logic
43	space. DAMON separates the two parts in different layers, namely DAMON
44	Operations Set and DAMON Core Logics Layers, respectively. It further defines
45	the interface between the layers to allow various operations sets to be
46	configured with the core logic.
47
48	Due to this design, users can extend DAMON for any address space by configuring
49	the core logic to use the appropriate operations set. If any appropriate set
50	is unavailable, users can implement one on their own.
51
52	For example, physical memory, virtual memory, swap space, those for specific
53	processes, NUMA nodes, files, and backing memory devices would be supportable.
54	Also, if some architectures or devices supporting special optimized access
55	check primitives, those will be easily configurable.
56
57
58	Programmable Modules
59	--------------------
60
61	Core layer of DAMON is implemented as a framework, and exposes its application
62	programming interface to all kernel space components such as subsystems and
63	modules. For common use cases of DAMON, DAMON subsystem provides kernel
64	modules that built on top of the core layer using the API, which can be easily
65	used by the user space end users.
66		37
67		38
68	.. _damon_operations_set:	39	.. _damon_operations_set:
@@ -70,11 +41,32 @@ used by the user space end users. · 1 unmodified line
70	Operations Set Layer	41	Operations Set Layer
71	====================	42	====================
72		43
73	The monitoring operations are defined in two parts:	44	.. _damon_design_configurable_operations_set:
		45
		46	For data access monitoring and additional low level work, DAMON needs a set of
		47	implementations for specific operations that are dependent on and optimized for
		48	the given target address space. For example, below two operations for access
		49	monitoring are address-space dependent.
74		50
75	1. Identification of the monitoring target address range for the address space.	51	1. Identification of the monitoring target address range for the address space.
76	2. Access check of specific address range in the target space.	52	2. Access check of specific address range in the target space.
77		53
		54	DAMON consolidates these implementations in a layer called DAMON Operations
		55	Set, and defines the interface between it and the upper layer. The upper layer
		56	is dedicated for DAMON's core logics including the mechanism for control of the
		57	monitoring accruracy and the overhead.
		58
		59	Hence, DAMON can easily be extended for any address space and/or available
		60	hardware features by configuring the core logic to use the appropriate
		61	operations set. If there is no available operations set for a given purpose, a
		62	new operations set can be implemented following the interface between the
		63	layers.
		64
		65	For example, physical memory, virtual memory, swap space, those for specific
		66	processes, NUMA nodes, files, and backing memory devices would be supportable.
		67	Also, if some architectures or devices support special optimized access check
		68	features, those will be easily configurable.
		69
78	DAMON currently provides below three operation sets. Below two subsections	70	DAMON currently provides below three operation sets. Below two subsections
79	describe how those work.	71	describe how those work.
80		72
@@ -82,6 +74,10 @@ describe how those work. · 1 unmodified line
82	- fvaddr: Monitor fixed virtual address ranges	74	- fvaddr: Monitor fixed virtual address ranges
83	- paddr: Monitor the physical address space of the system	75	- paddr: Monitor the physical address space of the system
84		76
		77	To know how user-space can do the configuration via :ref:`DAMON sysfs interface
		78	<sysfs_interface>`, refer to :ref:`operations <sysfs_context>` file part of the
		79	documentation.
		80
85		81
86	.. _damon_design_vaddr_target_regions_construction:	82	.. _damon_design_vaddr_target_regions_construction:
87		83
@@ -140,9 +136,12 @@ conflict with the reclaim logic using ``PG_idle`` and ``PG_young`` page flags, · 52 unmodified lines
140	as Idle page tracking does.	136	as Idle page tracking does.
141		137
142		138
		139	.. _damon_core_logic:
		140
143	Core Logics	141	Core Logics
144	===========	142	===========
145		143
		144	.. _damon_design_monitoring:
146		145
147	Monitoring	146	Monitoring
148	----------	147	----------
@@ -152,6 +151,10 @@ monitoring attributes, ``sampling interval``, ``aggregation interval``, · 3 unmodified lines
152	``update interval``, ``minimum number of regions``, and ``maximum number of	151	``update interval``, ``minimum number of regions``, and ``maximum number of
153	regions``.	152	regions``.
154		153
		154	To know how user-space can set the attributes via :ref:`DAMON sysfs interface
		155	<sysfs_interface>`, refer to :ref:`monitoring_attrs <sysfs_monitoring_attrs>`
		156	part of the documentation.
		157
155		158
156	Access Frequency Monitoring	159	Access Frequency Monitoring
157	~~~~~~~~~~~~~~~~~~~~~~~~~~~	160	~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -192,7 +195,7 @@ one page in the region is required to be checked. Thus, for each ``sampling · 34 unmodified lines
192	interval``, DAMON randomly picks one page in each region, waits for one	195	interval``, DAMON randomly picks one page in each region, waits for one
193	``sampling interval``, checks whether the page is accessed meanwhile, and	196	``sampling interval``, checks whether the page is accessed meanwhile, and
194	increases the access frequency counter of the region if so. The counter is	197	increases the access frequency counter of the region if so. The counter is
195	called ``nr_regions`` of the region. Therefore, the monitoring overhead is	198	called ``nr_accesses`` of the region. Therefore, the monitoring overhead is
196	controllable by setting the number of regions. DAMON allows users to set the	199	controllable by setting the number of regions. DAMON allows users to set the
197	minimum and the maximum number of regions for the trade-off.	200	minimum and the maximum number of regions for the trade-off.
198		201
@@ -209,11 +212,18 @@ the data access pattern can be dynamically changed. This will result in low · 10 unmodified lines
209	monitoring quality. To keep the assumption as much as possible, DAMON	212	monitoring quality. To keep the assumption as much as possible, DAMON
210	adaptively merges and splits each region based on their access frequency.	213	adaptively merges and splits each region based on their access frequency.
211		214
212	For each ``aggregation interval``, it compares the access frequencies of	215	For each ``aggregation interval``, it compares the access frequencies
213	adjacent regions and merges those if the frequency difference is small. Then,	216	(``nr_accesses``) of adjacent regions. If the difference is small, and if the
214	after it reports and clears the aggregated access frequency of each region, it	217	sum of the two regions' sizes is smaller than the size of total regions divided
215	splits each region into two or three regions if the total number of regions	218	by the ``minimum number of regions``, DAMON merges the two regions. If the
216	will not exceed the user-specified maximum number of regions after the split.	219	resulting number of total regions is still higher than ``maximum number of
		220	regions``, it repeats the merging with increasing access frequenceis difference
		221	threshold until the upper-limit of the number of regions is met, or the
		222	threshold becomes higher than possible maximum value (``aggregation interval``
		223	divided by ``sampling interval``). Then, after it reports and clears the
		224	aggregated access frequency of each region, it splits each region into two or
		225	three regions if the total number of regions will not exceed the user-specified
		226	maximum number of regions after the split.
217		227
218	In this way, DAMON provides its best-effort quality and minimal overhead while	228	In this way, DAMON provides its best-effort quality and minimal overhead while
219	keeping the bounds users set for their trade-off.	229	keeping the bounds users set for their trade-off.
@@ -248,6 +258,11 @@ and applies it to monitoring operations-related data structures such as the · 28 unmodified lines
248	abstracted monitoring target memory area only for each of a user-specified time	258	abstracted monitoring target memory area only for each of a user-specified time
249	interval (``update interval``).	259	interval (``update interval``).
250		260
		261	User-space can get the monitoring results via DAMON sysfs interface and/or
		262	tracepoints. For more details, please refer to the documentations for
		263	:ref:`DAMOS tried regions <sysfs_schemes_tried_regions>` and :ref:`tracepoint`,
		264	respectively.
		265
251		266
252	.. _damon_design_damos:	267	.. _damon_design_damos:
253		268
@@ -288,6 +303,10 @@ the access pattern of interest, and applies the user-desired operation actions · 34 unmodified lines
288	to the regions, for every user-specified time interval called	303	to the regions, for every user-specified time interval called
289	``apply_interval``.	304	``apply_interval``.
290		305
		306	To know how user-space can set ``apply_interval`` via :ref:`DAMON sysfs
		307	interface <sysfs_interface>`, refer to :ref:`apply_interval_us <sysfs_scheme>`
		308	part of the documentation.
		309
291		310
292	.. _damon_design_damos_action:	311	.. _damon_design_damos_action:
293		312
@@ -325,6 +344,10 @@ that supports each action are as below. · 31 unmodified lines
325	Supported by ``paddr`` operations set.	344	Supported by ``paddr`` operations set.
326	- ``lru_deprio``: Deprioritize the region on its LRU lists.	345	- ``lru_deprio``: Deprioritize the region on its LRU lists.
327	Supported by ``paddr`` operations set.	346	Supported by ``paddr`` operations set.
		347	- ``migrate_hot``: Migrate the regions prioritizing warmer regions.
		348	Supported by ``paddr`` operations set.
		349	- ``migrate_cold``: Migrate the regions prioritizing colder regions.
		350	Supported by ``paddr`` operations set.
328	- ``stat``: Do nothing but count the statistics.	351	- ``stat``: Do nothing but count the statistics.
329	Supported by all operations sets.	352	Supported by all operations sets.
330		353
@@ -332,6 +355,10 @@ Applying the actions except ``stat`` to a region is considered as changing the · 1 unmodified line
332	region's characteristics. Hence, DAMOS resets the age of regions when any such	355	region's characteristics. Hence, DAMOS resets the age of regions when any such
333	actions are applied to those.	356	actions are applied to those.
334		357
		358	To know how user-space can set the action via :ref:`DAMON sysfs interface
		359	<sysfs_interface>`, refer to :ref:`action <sysfs_scheme>` part of the
		360	documentation.
		361
335		362
336	.. _damon_design_damos_access_pattern:	363	.. _damon_design_damos_access_pattern:
337		364
@@ -345,6 +372,10 @@ interest by setting minimum and maximum values of the three properties. If a · 7 unmodified lines
345	region's three properties are in the ranges, DAMOS classifies it as one of the	372	region's three properties are in the ranges, DAMOS classifies it as one of the
346	regions that the scheme is having an interest in.	373	regions that the scheme is having an interest in.
347		374
		375	To know how user-space can set the access pattern via :ref:`DAMON sysfs
		376	interface <sysfs_interface>`, refer to :ref:`access_pattern
		377	<sysfs_access_pattern>` part of the documentation.
		378
348		379
349	.. _damon_design_damos_quotas:	380	.. _damon_design_damos_quotas:
350		381
@@ -364,6 +395,10 @@ feature called quotas. It lets users specify an upper limit of time that DAMOS · 13 unmodified lines
364	can use for applying the action, and/or a maximum bytes of memory regions that	395	can use for applying the action, and/or a maximum bytes of memory regions that
365	the action can be applied within a user-specified time duration.	396	the action can be applied within a user-specified time duration.
366		397
		398	To know how user-space can set the basic quotas via :ref:`DAMON sysfs interface
		399	<sysfs_interface>`, refer to :ref:`quotas <sysfs_quotas>` part of the
		400	documentation.
		401
367		402
368	.. _damon_design_damos_quotas_prioritization:	403	.. _damon_design_damos_quotas_prioritization:
369		404
@@ -391,6 +426,10 @@ information to the underlying mechanism. Nevertheless, how and even whether · 21 unmodified lines
391	the weight will be respected are up to the underlying prioritization mechanism	426	the weight will be respected are up to the underlying prioritization mechanism
392	implementation.	427	implementation.
393		428
		429	To know how user-space can set the prioritization weights via :ref:`DAMON sysfs
		430	interface <sysfs_interface>`, refer to :ref:`weights <sysfs_quotas>` part of
		431	the documentation.
		432
394		433
395	.. _damon_design_damos_quotas_auto_tuning:	434	.. _damon_design_damos_quotas_auto_tuning:
396		435
@@ -420,6 +459,10 @@ Currently, two ``target_metric`` are provided. · 23 unmodified lines
420	DAMOS does the measurement on its own, so only ``target_value`` need to be	459	DAMOS does the measurement on its own, so only ``target_value`` need to be
421	set by users at the initial time. In other words, DAMOS does self-feedback.	460	set by users at the initial time. In other words, DAMOS does self-feedback.
422		461
		462	To know how user-space can set the tuning goal metric, the target value, and/or
		463	the current value via :ref:`DAMON sysfs interface <sysfs_interface>`, refer to
		464	:ref:`quota goals <sysfs_schemes_quota_goals>` part of the documentation.
		465
423		466
424	.. _damon_design_damos_watermarks:	467	.. _damon_design_damos_watermarks:
425		468
@@ -442,6 +485,10 @@ is activated. If all schemes are deactivated by the watermarks, the monitoring · 16 unmodified lines
442	is also deactivated. In this case, the DAMON worker thread only periodically	485	is also deactivated. In this case, the DAMON worker thread only periodically
443	checks the watermarks and therefore incurs nearly zero overhead.	486	checks the watermarks and therefore incurs nearly zero overhead.
444		487
		488	To know how user-space can set the watermarks via :ref:`DAMON sysfs interface
		489	<sysfs_interface>`, refer to :ref:`watermarks <sysfs_watermarks>` part of the
		490	documentation.
		491
445		492
446	.. _damon_design_damos_filters:	493	.. _damon_design_damos_filters:
447		494
@@ -488,6 +535,10 @@ Below types of filters are currently supported. · 40 unmodified lines
488	- Applied to pages that belonging to a given DAMON monitoring target.	535	- Applied to pages that belonging to a given DAMON monitoring target.
489	- Handled by the core logic.	536	- Handled by the core logic.
490		537
		538	To know how user-space can set the watermarks via :ref:`DAMON sysfs interface
		539	<sysfs_interface>`, refer to :ref:`filters <sysfs_filters>` part of the
		540	documentation.
		541
491		542
492	Application Programming Interface	543	Application Programming Interface
493	---------------------------------	544	---------------------------------
@@ -501,6 +552,8 @@ interface, namely ``include/linux/damon.h``. Please refer to the API · 7 unmodified lines
501	:doc:`document </mm/damon/api>` for details of the interface.	552	:doc:`document </mm/damon/api>` for details of the interface.
502		553
503		554
		555	.. _damon_modules:
		556
504	Modules	557	Modules
505	=======	558	=======
506		559

Documentation/mm/damon/index.rst+13−9

 DAMON is a Linux kernel subsystem that provides a framework for data access
 monitoring and the monitoring results based system operations.  The core
-monitoring mechanisms of DAMON (refer to :doc:`design` for the detail) make it
+monitoring :ref:`mechanisms <damon_design_monitoring>` of DAMON make it
  - *accurate* (the monitoring output is useful enough for DRAM level memory
    management; It might not appropriate for CPU Cache levels, though),
    of the size of target workloads).
 Using this framework, therefore, the kernel can operate system in an
-access-aware fashion.  Because the features are also exposed to the user space,
+access-aware fashion.  Because the features are also exposed to the :doc:`user
-users who have special information about their workloads can write personalized
+space </admin-guide/mm/damon/index>`, users who have special information about
-applications for better understanding and optimizations of their workloads and
+their workloads can write personalized applications for better understanding
-systems.
+and optimizations of their workloads and systems.
-For easier development of such systems, DAMON provides a feature called DAMOS
+For easier development of such systems, DAMON provides a feature called
-(DAMon-based Operation Schemes) in addition to the monitoring.  Using the
+:ref:`DAMOS <damon_design_damos>` (DAMon-based Operation Schemes) in addition
-feature, DAMON users in both kernel and user spaces can do access-aware system
+to the monitoring.  Using the feature, DAMON users in both kernel and :doc:`user
-operations with no code but simple configurations.
+spaces </admin-guide/mm/damon/index>` can do access-aware system operations
+with no code but simple configurations.
 .. toctree::
    :maxdepth: 2
    design
    api
    maintainer-profile
+To utilize and control DAMON from the user-space, please refer to the
+administration :doc:`guide </admin-guide/mm/damon/index>`.

Documentation/mm/damon/maintainer-profile.rst+36−0

@@ -53,6 +53,40 @@ Mon-Fri) in PT (Pacific Time). The response to patches will occasionally be · 52 unmodified lines
53	slow. Do not hesitate to send a ping if you have not heard back within a week	53	slow. Do not hesitate to send a ping if you have not heard back within a week
54	of sending a patch.	54	of sending a patch.
55		55
		56	Mailing tool
		57	------------
		58
		59	Like many other Linux kernel subsystems, DAMON uses the mailing lists
		60	(damon@lists.linux.dev and linux-mm@kvack.org) as the major communication
		61	channel. There is a simple tool called HacKerMaiL (``hkml``) [8]_ , which is
		62	for people who are not very familiar with the mailing lists based
		63	communication. The tool could be particularly helpful for DAMON community
		64	members since it is developed and maintained by DAMON maintainer. The tool is
		65	also officially announced to support DAMON and general Linux kernel development
		66	workflow.
		67
		68	In other words, ``hkml`` [8]_ is a mailing tool for DAMON community, which
		69	DAMON maintainer is committed to support. Please feel free to try and report
		70	issues or feature requests for the tool to the maintainer.
		71
		72	Community meetup
		73	----------------
		74
		75	DAMON community is maintaining two bi-weekly meetup series for community
		76	members who prefer synchronous conversations over mails.
		77
		78	The first one is for any discussion between every community member. No
		79	reservation is needed.
		80
		81	The seconds one is for discussions on specific topics between restricted
		82	members including the maintainer. The maintainer shares the available time
		83	slots, and attendees should reserve one of those at least 24 hours before the
		84	time slot, by reaching out to the maintainer.
		85
		86	Schedules and available reservation time slots are available at the Google doc
		87	[9]_ . DAMON maintainer will also provide periodic reminder to the mailing
		88	list (damon@lists.linux.dev).
		89
56		90
57	.. [1] https://git.kernel.org/akpm/mm/h/mm-unstable	91	.. [1] https://git.kernel.org/akpm/mm/h/mm-unstable
58	.. [2] https://git.kernel.org/sj/h/damon/next	92	.. [2] https://git.kernel.org/sj/h/damon/next
@@ -61,3 +95,5 @@ of sending a patch. · 2 unmodified lines
61	.. [5] https://github.com/awslabs/damon-tests/blob/master/corr/tests/kunit.sh	95	.. [5] https://github.com/awslabs/damon-tests/blob/master/corr/tests/kunit.sh
62	.. [6] https://github.com/awslabs/damon-tests/tree/master/corr	96	.. [6] https://github.com/awslabs/damon-tests/tree/master/corr
63	.. [7] https://github.com/awslabs/damon-tests/tree/master/perf	97	.. [7] https://github.com/awslabs/damon-tests/tree/master/perf
		98	.. [8] https://github.com/damonitor/hackermail
		99	.. [9] https://docs.google.com/document/d/1v43Kcj3ly4CYqmAkMaZzLiM2GEnWfgdGbZAH3mi2vpM/edit?usp=sharing

Documentation/mm/unevictable-lru.rst+5−5

MAINTAINERS+2−0

@@ -5701,6 +5701,8 @@ L: linux-mm@kvack.org · 5700 unmodified lines
5701	S: Maintained	5701	S: Maintained
5702	F: include/linux/memcontrol.h	5702	F: include/linux/memcontrol.h
5703	F: mm/memcontrol.c	5703	F: mm/memcontrol.c
		5704	F: mm/memcontrol-v1.c
		5705	F: mm/memcontrol-v1.h
5704	F: mm/swap_cgroup.c	5706	F: mm/swap_cgroup.c
5705	F: samples/cgroup/*	5707	F: samples/cgroup/*
5706	F: tools/testing/selftests/cgroup/memcg_protection.m	5708	F: tools/testing/selftests/cgroup/memcg_protection.m

arch/arm/include/asm/cacheflush.h+1−1

@@ -283,7 +283,7 @@ void flush_cache_pages(struct vm_area_struct *vma, unsigned long user_addr, · 282 unmodified lines
283	* flush_dcache_page is used when the kernel has written to the page	283	* flush_dcache_page is used when the kernel has written to the page
284	* cache page at virtual address page->virtual.	284	* cache page at virtual address page->virtual.
285	*	285	*
286	* If this page isn't mapped (ie, page_mapping == NULL), or it might	286	* If this page isn't mapped (ie, folio_mapping == NULL), or it might
287	* have userspace mappings, then we _must_ always clean + invalidate	287	* have userspace mappings, then we _must_ always clean + invalidate
288	* the dcache entries associated with the kernel mapping.	288	* the dcache entries associated with the kernel mapping.
289	*	289	*

arch/arm/include/asm/hugetlb-3level.h+2−2

@@ -13,12 +13,12 @@ · 12 unmodified lines
13		13
14	/*	14	/*
15	* If our huge pte is non-zero then mark the valid bit.	15	* If our huge pte is non-zero then mark the valid bit.
16	* This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero	16	* This allows pte_present(huge_ptep_get(mm,addr,ptep)) to return true for non-zero
17	* ptes.	17	* ptes.
18	* (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes).	18	* (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes).
19	*/	19	*/
20	#define __HAVE_ARCH_HUGE_PTEP_GET	20	#define __HAVE_ARCH_HUGE_PTEP_GET
21	static inline pte_t huge_ptep_get(pte_t *ptep)	21	static inline pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
22	{	22	{
23	pte_t retval = *ptep;	23	pte_t retval = *ptep;
24	if (pte_val(retval))	24	if (pte_val(retval))

arch/arm64/include/asm/cacheflush.h+1−1

@@ -117,7 +117,7 @@ extern void copy_to_user_page(struct vm_area_struct , struct page , · 116 unmodified lines
117	* flush_dcache_folio is used when the kernel has written to the page	117	* flush_dcache_folio is used when the kernel has written to the page
118	* cache page at virtual address page->virtual.	118	* cache page at virtual address page->virtual.
119	*	119	*
120	* If this page isn't mapped (ie, page_mapping == NULL), or it might	120	* If this page isn't mapped (ie, folio_mapping == NULL), or it might
121	* have userspace mappings, then we _must_ always clean + invalidate	121	* have userspace mappings, then we _must_ always clean + invalidate
122	* the dcache entries associated with the kernel mapping.	122	* the dcache entries associated with the kernel mapping.
123	*	123	*

arch/arm64/include/asm/hugetlb.h+1−1

@@ -46,7 +46,7 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, · 45 unmodified lines
46	extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr,	46	extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr,
47	pte_t *ptep, unsigned long sz);	47	pte_t *ptep, unsigned long sz);
48	#define __HAVE_ARCH_HUGE_PTEP_GET	48	#define __HAVE_ARCH_HUGE_PTEP_GET
49	extern pte_t huge_ptep_get(pte_t *ptep);	49	extern pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep);
50		50
51	void __init arm64_hugetlb_cma_reserve(void);	51	void __init arm64_hugetlb_cma_reserve(void);
52		52

arch/arm64/mm/hugetlbpage.c+1−1

@@ -127,7 +127,7 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · 126 unmodified lines
127	return contig_ptes;	127	return contig_ptes;
128	}	128	}
129		129
130	pte_t huge_ptep_get(pte_t *ptep)	130	pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
131	{	131	{
132	int ncontig, i;	132	int ncontig, i;
133	size_t pgsize;	133	size_t pgsize;

arch/loongarch/include/asm/pgtable.h+2−2

@@ -467,8 +467,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 466 unmodified lines
467	#define update_mmu_cache(vma, addr, ptep) \	467	#define update_mmu_cache(vma, addr, ptep) \
468	update_mmu_cache_range(NULL, vma, addr, ptep, 1)	468	update_mmu_cache_range(NULL, vma, addr, ptep, 1)
469		469
470	#define __HAVE_ARCH_UPDATE_MMU_TLB	470	#define update_mmu_tlb_range(vma, addr, ptep, nr) \
471	#define update_mmu_tlb update_mmu_cache	471	update_mmu_cache_range(NULL, vma, addr, ptep, nr)
472		472
473	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,	473	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,
474	unsigned long address, pmd_t *pmdp)	474	unsigned long address, pmd_t *pmdp)

arch/mips/include/asm/pgtable.h+2−2

@@ -594,8 +594,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 593 unmodified lines
594	#define update_mmu_cache(vma, address, ptep) \	594	#define update_mmu_cache(vma, address, ptep) \
595	update_mmu_cache_range(NULL, vma, address, ptep, 1)	595	update_mmu_cache_range(NULL, vma, address, ptep, 1)
596		596
597	#define __HAVE_ARCH_UPDATE_MMU_TLB	597	#define update_mmu_tlb_range(vma, address, ptep, nr) \
598	#define update_mmu_tlb update_mmu_cache	598	update_mmu_cache_range(NULL, vma, address, ptep, nr)
599		599
600	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,	600	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,
601	unsigned long address, pmd_t *pmdp)	601	unsigned long address, pmd_t *pmdp)

arch/mips/mm/cache.c+1−1

@@ -112,7 +112,7 @@ void __flush_dcache_pages(struct page *page, unsigned int nr) · 111 unmodified lines
112	}	112	}
113		113
114	/*	114	/*
115	* We could delay the flush for the !page_mapping case too. But that	115	* We could delay the flush for the !folio_mapping case too. But that
116	* case is for exec env/arg pages and those are %99 certainly going to	116	* case is for exec env/arg pages and those are %99 certainly going to
117	* get faulted into the tlb (and thus flushed) anyways.	117	* get faulted into the tlb (and thus flushed) anyways.
118	*/	118	*/

arch/powerpc/Kconfig+0−1

@@ -135,7 +135,6 @@ config PPC · 134 unmodified lines
135	select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES	135	select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES
136	select ARCH_HAS_FORTIFY_SOURCE	136	select ARCH_HAS_FORTIFY_SOURCE
137	select ARCH_HAS_GCOV_PROFILE_ALL	137	select ARCH_HAS_GCOV_PROFILE_ALL
138	select ARCH_HAS_HUGEPD if HUGETLB_PAGE
139	select ARCH_HAS_KCOV	138	select ARCH_HAS_KCOV
140	select ARCH_HAS_KERNEL_FPU_SUPPORT if PPC64 && PPC_FPU	139	select ARCH_HAS_KERNEL_FPU_SUPPORT if PPC64 && PPC_FPU
141	select ARCH_HAS_MEMBARRIER_CALLBACKS	140	select ARCH_HAS_MEMBARRIER_CALLBACKS

arch/powerpc/include/asm/book3s/32/pgalloc.h+0−2

@@ -47,8 +47,6 @@ static inline void pgtable_free(void *table, unsigned index_size) · 46 unmodified lines
47	}	47	}
48	}	48	}
49		49
50	#define get_hugepd_cache_index(x) (x)
51
52	static inline void pgtable_free_tlb(struct mmu_gather *tlb,	50	static inline void pgtable_free_tlb(struct mmu_gather *tlb,
53	void *table, int shift)	51	void *table, int shift)
54	{	52	{

arch/powerpc/include/asm/book3s/64/hash-4k.h+0−15

@@ -74,21 +74,6 @@ · 73 unmodified lines
74	#define remap_4k_pfn(vma, addr, pfn, prot) \	74	#define remap_4k_pfn(vma, addr, pfn, prot) \
75	remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot))	75	remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot))
76		76
77	#ifdef CONFIG_HUGETLB_PAGE
78	static inline int hash__hugepd_ok(hugepd_t hpd)
79	{
80	unsigned long hpdval = hpd_val(hpd);
81	/*
82	* if it is not a pte and have hugepd shift mask
83	* set, then it is a hugepd directory pointer
84	*/
85	if (!(hpdval & _PAGE_PTE) && (hpdval & _PAGE_PRESENT) &&
86	((hpdval & HUGEPD_SHIFT_MASK) != 0))
87	return true;
88	return false;
89	}
90	#endif
91
92	/*	77	/*
93	* 4K PTE format is different from 64K PTE format. Saving the hash_slot is just	78	* 4K PTE format is different from 64K PTE format. Saving the hash_slot is just
94	* a matter of returning the PTE bits that need to be modified. On 64K PTE,	79	* a matter of returning the PTE bits that need to be modified. On 64K PTE,

arch/powerpc/include/asm/book3s/64/hash.h+33−7

 #ifdef __KERNEL__
 #include <asm/asm-const.h>
+#include <asm/book3s/64/slice.h>
 /*
  * Common bits between 4K and 64K pages in a linux-style PTE.
 			    pte_t *ptep, unsigned long pte, int huge);
 unsigned long htab_convert_pte_flags(unsigned long pteflags, unsigned long flags);
 /* Atomic PTE updates */
-static inline unsigned long hash__pte_update(struct mm_struct *mm,
+static inline unsigned long hash__pte_update_one(pte_t *ptep, unsigned long clr,
-					 unsigned long addr,
+						 unsigned long set)
-					 pte_t *ptep, unsigned long clr,
-					 unsigned long set,
-					 int huge)
 {
 	__be64 old_be, tmp_be;
-	unsigned long old;
 	__asm__ __volatile__(
 	"1:	ldarx	%0,0,%3		# pte_update\n\
 	: "r" (ptep), "r" (cpu_to_be64(clr)), "m" (*ptep),
 	  "r" (cpu_to_be64(H_PAGE_BUSY)), "r" (cpu_to_be64(set))
 	: "cc" );
+	return be64_to_cpu(old_be);
+}
+static inline unsigned long hash__pte_update(struct mm_struct *mm,
+					 unsigned long addr,
+					 pte_t *ptep, unsigned long clr,
+					 unsigned long set,
+					 int huge)
+{
+	unsigned long old;
+	old = hash__pte_update_one(ptep, clr, set);
+	if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && huge) {
+		unsigned int psize = get_slice_psize(mm, addr);
+		int nb, i;
+		if (psize == MMU_PAGE_16M)
+			nb = SZ_16M / PMD_SIZE;
+		else if (psize == MMU_PAGE_16G)
+			nb = SZ_16G / PUD_SIZE;
+		else
+			nb = 1;
+		WARN_ON_ONCE(nb == 1);	/* Should never happen */
+		for (i = 1; i < nb; i++)
+			hash__pte_update_one(ptep + i, clr, set);
+	}
 	/* huge pages use the old page table lock */
 	if (!huge)
 		assert_pte_locked(mm, addr);
-	old = be64_to_cpu(old_be);
 	if (old & H_PAGE_HASHPTE)
 		hpte_need_flush(mm, addr, ptep, old, huge);

arch/powerpc/include/asm/book3s/64/hugetlb.h+0−38

@@ -49,9 +49,6 @@ static inline bool gigantic_page_runtime_supported(void) · 48 unmodified lines
49	return true;	49	return true;
50	}	50	}
51		51
52	/* hugepd entry valid bit */
53	#define HUGEPD_VAL_BITS (0x8000000000000000UL)
54
55	#define huge_ptep_modify_prot_start huge_ptep_modify_prot_start	52	#define huge_ptep_modify_prot_start huge_ptep_modify_prot_start
56	extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,	53	extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,
57	unsigned long addr, pte_t *ptep);	54	unsigned long addr, pte_t *ptep);
@@ -60,29 +57,7 @@ extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, · 2 unmodified lines
60	extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma,	57	extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma,
61	unsigned long addr, pte_t *ptep,	58	unsigned long addr, pte_t *ptep,
62	pte_t old_pte, pte_t new_pte);	59	pte_t old_pte, pte_t new_pte);
63	/*
64	* This should work for other subarchs too. But right now we use the
65	* new format only for 64bit book3s
66	*/
67	static inline pte_t *hugepd_page(hugepd_t hpd)
68	{
69	BUG_ON(!hugepd_ok(hpd));
70	/*
71	* We have only four bits to encode, MMU page size
72	*/
73	BUILD_BUG_ON((MMU_PAGE_COUNT - 1) > 0xf);
74	return __va(hpd_val(hpd) & HUGEPD_ADDR_MASK);
75	}
76
77	static inline unsigned int hugepd_mmu_psize(hugepd_t hpd)
78	{
79	return (hpd_val(hpd) & HUGEPD_SHIFT_MASK) >> 2;
80	}
81		60
82	static inline unsigned int hugepd_shift(hugepd_t hpd)
83	{
84	return mmu_psize_to_shift(hugepd_mmu_psize(hpd));
85	}
86	static inline void flush_hugetlb_page(struct vm_area_struct *vma,	61	static inline void flush_hugetlb_page(struct vm_area_struct *vma,
87	unsigned long vmaddr)	62	unsigned long vmaddr)
88	{	63	{
@@ -90,19 +65,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, · 1 unmodified line
90	return radix__flush_hugetlb_page(vma, vmaddr);	65	return radix__flush_hugetlb_page(vma, vmaddr);
91	}	66	}
92		67
93	static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr,
94	unsigned int pdshift)
95	{
96	unsigned long idx = (addr & ((1UL << pdshift) - 1)) >> hugepd_shift(hpd);
97
98	return hugepd_page(hpd) + idx;
99	}
100
101	static inline void hugepd_populate(hugepd_t hpdp, pte_t new, unsigned int pshift)
102	{
103	*hpdp = __hugepd(__pa(new) \| HUGEPD_VAL_BITS \| (shift_to_mmu_psize(pshift) << 2));
104	}
105
106	void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr);	68	void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr);
107		69
108	static inline int check_and_get_huge_psize(int shift)	70	static inline int check_and_get_huge_psize(int shift)

arch/powerpc/include/asm/book3s/64/pgtable-4k.h+0−47

@@ -1,47 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 */
2	#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H
3	#define _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H
4	/*
5	* hash 4k can't share hugetlb and also doesn't support THP
6	*/
7	#ifndef __ASSEMBLY__
8	#ifdef CONFIG_HUGETLB_PAGE
9	/*
10	* With radix , we have hugepage ptes in the pud and pmd entries. We don't
11	* need to setup hugepage directory for them. Our pte and page directory format
12	* enable us to have this enabled.
13	*/
14	static inline int hugepd_ok(hugepd_t hpd)
15	{
16	if (radix_enabled())
17	return 0;
18	return hash__hugepd_ok(hpd);
19	}
20	#define is_hugepd(hpd) (hugepd_ok(hpd))
21
22	/*
23	* 16M and 16G huge page directory tables are allocated from slab cache
24	*
25	*/
26	#define H_16M_CACHE_INDEX (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE - 24)
27	#define H_16G_CACHE_INDEX \
28	(PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE + H_PUD_INDEX_SIZE - 34)
29
30	static inline int get_hugepd_cache_index(int index)
31	{
32	switch (index) {
33	case H_16M_CACHE_INDEX:
34	return HTLB_16M_INDEX;
35	case H_16G_CACHE_INDEX:
36	return HTLB_16G_INDEX;
37	default:
38	BUG();
39	}
40	/* should not reach */
41	}
42
43	#endif /* CONFIG_HUGETLB_PAGE */
44
45	#endif /* __ASSEMBLY__ */
46
47	#endif /_ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H /

arch/powerpc/include/asm/book3s/64/pgtable-64k.h+0−20

@@ -5,26 +5,6 @@ · 4 unmodified lines
5	#ifndef __ASSEMBLY__	5	#ifndef __ASSEMBLY__
6	#ifdef CONFIG_HUGETLB_PAGE	6	#ifdef CONFIG_HUGETLB_PAGE
7		7
8	/*
9	* With 64k page size, we have hugepage ptes in the pgd and pmd entries. We don't
10	* need to setup hugepage directory for them. Our pte and page directory format
11	* enable us to have this enabled.
12	*/
13	static inline int hugepd_ok(hugepd_t hpd)
14	{
15	return 0;
16	}
17
18	#define is_hugepd(pdep) 0
19
20	/*
21	* This should never get called
22	*/
23	static __always_inline int get_hugepd_cache_index(int index)
24	{
25	BUILD_BUG();
26	}
27
28	#endif /* CONFIG_HUGETLB_PAGE */	8	#endif /* CONFIG_HUGETLB_PAGE */
29		9
30	static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr,	10	static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr,

arch/powerpc/include/asm/book3s/64/pgtable.h+19−3

@@ -274,6 +274,24 @@ static inline bool pud_leaf(pud_t pud) · 273 unmodified lines
274	{	274	{
275	return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE));	275	return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE));
276	}	276	}
		277
		278	#define pmd_leaf_size pmd_leaf_size
		279	static inline unsigned long pmd_leaf_size(pmd_t pmd)
		280	{
		281	if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled())
		282	return SZ_16M;
		283	else
		284	return PMD_SIZE;
		285	}
		286
		287	#define pud_leaf_size pud_leaf_size
		288	static inline unsigned long pud_leaf_size(pud_t pud)
		289	{
		290	if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled())
		291	return SZ_16G;
		292	else
		293	return PUD_SIZE;
		294	}
277	#endif /* __ASSEMBLY__ */	295	#endif /* __ASSEMBLY__ */
278		296
279	#include <asm/book3s/64/hash.h>	297	#include <asm/book3s/64/hash.h>
@@ -285,11 +303,9 @@ static inline bool pud_leaf(pud_t pud) · 5 unmodified lines
285	#define MAX_PHYSMEM_BITS R_MAX_PHYSMEM_BITS	303	#define MAX_PHYSMEM_BITS R_MAX_PHYSMEM_BITS
286	#endif	304	#endif
287		305
288		306	/* hash 4k can't share hugetlb and also doesn't support THP */
289	#ifdef CONFIG_PPC_64K_PAGES	307	#ifdef CONFIG_PPC_64K_PAGES
290	#include <asm/book3s/64/pgtable-64k.h>	308	#include <asm/book3s/64/pgtable-64k.h>
291	#else
292	#include <asm/book3s/64/pgtable-4k.h>
293	#endif	309	#endif
294		310
295	#include <asm/barrier.h>	311	#include <asm/barrier.h>

arch/powerpc/include/asm/hugetlb.h+3−12

 }
 #define is_hugepage_only_range is_hugepage_only_range
-#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE
+#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT
-void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
+void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
-			    unsigned long end, unsigned long floor,
+		     pte_t pte, unsigned long sz);
-			    unsigned long ceiling);
 #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR
 static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
 {
 }
-#define hugepd_shift(x) 0
-static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr,
-				    unsigned pdshift)
-{
-	return NULL;
-}
 static inline void __init gigantic_hugetlb_cma_reserve(void)
 {
 }

arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h+8−30

@@ -4,42 +4,12 @@ · 3 unmodified lines
4		4
5	#define PAGE_SHIFT_8M 23	5	#define PAGE_SHIFT_8M 23
6		6
7	static inline pte_t *hugepd_page(hugepd_t hpd)
8	{
9	BUG_ON(!hugepd_ok(hpd));
10
11	return (pte_t *)__va(hpd_val(hpd) & ~HUGEPD_SHIFT_MASK);
12	}
13
14	static inline unsigned int hugepd_shift(hugepd_t hpd)
15	{
16	return PAGE_SHIFT_8M;
17	}
18
19	static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr,
20	unsigned int pdshift)
21	{
22	unsigned long idx = (addr & (SZ_4M - 1)) >> PAGE_SHIFT;
23
24	return hugepd_page(hpd) + idx;
25	}
26
27	static inline void flush_hugetlb_page(struct vm_area_struct *vma,	7	static inline void flush_hugetlb_page(struct vm_area_struct *vma,
28	unsigned long vmaddr)	8	unsigned long vmaddr)
29	{	9	{
30	flush_tlb_page(vma, vmaddr);	10	flush_tlb_page(vma, vmaddr);
31	}	11	}
32		12
33	static inline void hugepd_populate(hugepd_t hpdp, pte_t new, unsigned int pshift)
34	{
35	*hpdp = __hugepd(__pa(new) \| _PMD_USER \| _PMD_PRESENT \| _PMD_PAGE_8M);
36	}
37
38	static inline void hugepd_populate_kernel(hugepd_t hpdp, pte_t new, unsigned int pshift)
39	{
40	*hpdp = __hugepd(__pa(new) \| _PMD_PRESENT \| _PMD_PAGE_8M);
41	}
42
43	static inline int check_and_get_huge_psize(int shift)	13	static inline int check_and_get_huge_psize(int shift)
44	{	14	{
45	return shift_to_mmu_psize(shift);	15	return shift_to_mmu_psize(shift);
@@ -49,6 +19,14 @@ static inline int check_and_get_huge_psize(int shift) · 3 unmodified lines
49	void set_huge_pte_at(struct mm_struct mm, unsigned long addr, pte_t ptep,	19	void set_huge_pte_at(struct mm_struct mm, unsigned long addr, pte_t ptep,
50	pte_t pte, unsigned long sz);	20	pte_t pte, unsigned long sz);
51		21
		22	#define __HAVE_ARCH_HUGE_PTEP_GET
		23	static inline pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
		24	{
		25	if (ptep_is_8m_pmdp(mm, addr, ptep))
		26	ptep = pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M));
		27	return ptep_get(ptep);
		28	}
		29
52	#define __HAVE_ARCH_HUGE_PTE_CLEAR	30	#define __HAVE_ARCH_HUGE_PTE_CLEAR
53	static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr,	31	static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr,
54	pte_t *ptep, unsigned long sz)	32	pte_t *ptep, unsigned long sz)

arch/powerpc/include/asm/nohash/32/mmu-8xx.h+2−7

 #define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000)
-/* Page size definitions, common between 32 and 64-bit
+/*
+ * Page size definitions for 8xx
  *
  *    shift : is the "PAGE_SHIFT" value for that page size
- *    penc  : is the pte encoding mask
  *
  */
 struct mmu_psize_def {
 	unsigned int	shift;	/* number of bits */
-	unsigned int	enc;	/* PTE encoding */
-	unsigned int    ind;    /* Corresponding indirect page size shift */
-	unsigned int	flags;
-#define MMU_PAGE_SIZE_DIRECT	0x1	/* Supported as a direct size */
-#define MMU_PAGE_SIZE_INDIRECT	0x2	/* Supported as an indirect size */
 };
 extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT];

arch/powerpc/include/asm/nohash/32/pte-44x.h+0−3

@@ -75,9 +75,6 @@ · 74 unmodified lines
75	#define _PAGE_NO_CACHE 0x00000400 /* H: I bit */	75	#define _PAGE_NO_CACHE 0x00000400 /* H: I bit */
76	#define _PAGE_WRITETHRU 0x00000800 /* H: W bit */	76	#define _PAGE_WRITETHRU 0x00000800 /* H: W bit */
77		77
78	/* No page size encoding in the linux PTE */
79	#define _PAGE_PSIZE 0
80
81	/* TODO: Add large page lowmem mapping support */	78	/* TODO: Add large page lowmem mapping support */
82	#define _PMD_PRESENT 0	79	#define _PMD_PRESENT 0
83	#define _PMD_PRESENT_MASK (PAGE_MASK)	80	#define _PMD_PRESENT_MASK (PAGE_MASK)

arch/powerpc/include/asm/nohash/32/pte-85xx.h+0−3

@@ -31,9 +31,6 @@ · 30 unmodified lines
31	#define _PAGE_WRITETHRU 0x00400 /* H: W bit */	31	#define _PAGE_WRITETHRU 0x00400 /* H: W bit */
32	#define _PAGE_SPECIAL 0x00800 /* S: Special page */	32	#define _PAGE_SPECIAL 0x00800 /* S: Special page */
33		33
34	/* No page size encoding in the linux PTE */
35	#define _PAGE_PSIZE 0
36
37	#define _PMD_PRESENT 0	34	#define _PMD_PRESENT 0
38	#define _PMD_PRESENT_MASK (PAGE_MASK)	35	#define _PMD_PRESENT_MASK (PAGE_MASK)
39	#define _PMD_BAD (~PAGE_MASK)	36	#define _PMD_BAD (~PAGE_MASK)

arch/powerpc/include/asm/nohash/32/pte-8xx.h+36−22

 #define _PTE_NONE_MASK	0
 #ifdef CONFIG_PPC_16K_PAGES
-#define _PAGE_PSIZE	_PAGE_SPS
+#define _PAGE_BASE_NC	(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_SPS)
 #else
-#define _PAGE_PSIZE		0
+#define _PAGE_BASE_NC	(_PAGE_PRESENT | _PAGE_ACCESSED)
 #endif
-#define _PAGE_BASE_NC	(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE)
 #define _PAGE_BASE	(_PAGE_BASE_NC)
 #include <asm/pgtable-masks.h>
 #define pte_mkhuge pte_mkhuge
-static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p,
+static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
 				     unsigned long clr, unsigned long set, int huge);
 static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
 }
 #define __ptep_set_access_flags __ptep_set_access_flags
-static inline unsigned long pgd_leaf_size(pgd_t pgd)
+static inline unsigned long __pte_leaf_size(pmd_t pmd, pte_t pte)
-{
-	if (pgd_val(pgd) & _PMD_PAGE_8M)
-		return SZ_8M;
-	return SZ_4M;
-}
-#define pgd_leaf_size pgd_leaf_size
-static inline unsigned long pte_leaf_size(pte_t pte)
 {
 	pte_basic_t val = pte_val(pte);
+	if (pmd_val(pmd) & _PMD_PAGE_8M)
+		return SZ_8M;
 	if (val & _PAGE_HUGE)
 		return SZ_512K;
 	if (val & _PAGE_SPS)
 	return SZ_4K;
 }
-#define pte_leaf_size pte_leaf_size
+#define __pte_leaf_size __pte_leaf_size
 /*
  * On the 8xx, the page tables are a bit special. For 16k pages, we have
  * 4 identical entries. For 512k pages, we have 128 entries as if it was
  * 4k pages, but they are flagged as 512k pages for the hardware.
- * For other page sizes, we have a single entry in the table.
+ * For 8M pages, we have 1024 entries as if it was 4M pages (PMD_SIZE)
+ * but they are flagged as 8M pages for the hardware.
+ * For 4k pages, we have a single entry in the table.
  */
 static pmd_t *pmd_off(struct mm_struct *mm, unsigned long addr);
-static int hugepd_ok(hugepd_t hpd);
+static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address);
+static inline bool ptep_is_8m_pmdp(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
+{
+	return (pmd_t *)ptep == pmd_off(mm, ALIGN_DOWN(addr, SZ_8M));
+}
 static inline int number_of_cells_per_pte(pmd_t *pmd, pte_basic_t val, int huge)
 {
 	if (!huge)
 		return PAGE_SIZE / SZ_4K;
-	else if (hugepd_ok(*((hugepd_t *)pmd)))
+	else if ((pmd_val(*pmd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M)
-		return 1;
+		return SZ_4M / SZ_4K;
 	else if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !(val & _PAGE_HUGE))
 		return SZ_16K / SZ_4K;
 	else
 		return SZ_512K / SZ_4K;
 }
-static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p,
+static inline pte_basic_t __pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p,
-				     unsigned long clr, unsigned long set, int huge)
+				       unsigned long clr, unsigned long set, int huge)
 {
 	pte_basic_t *entry = (pte_basic_t *)p;
 	pte_basic_t old = pte_val(*p);
 	for (i = 0; i < num; i += PAGE_SIZE / SZ_4K, new += PAGE_SIZE) {
 		*entry++ = new;
-		if (IS_ENABLED(CONFIG_PPC_16K_PAGES) && num != 1) {
+		if (IS_ENABLED(CONFIG_PPC_16K_PAGES)) {
 			*entry++ = new;
 			*entry++ = new;
 			*entry++ = new;
 	return old;
 }
+static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
+				     unsigned long clr, unsigned long set, int huge)
+{
+	pte_basic_t old;
+	if (huge && ptep_is_8m_pmdp(mm, addr, ptep)) {
+		pmd_t *pmdp = (pmd_t *)ptep;
+		old = __pte_update(mm, addr, pte_offset_kernel(pmdp, 0), clr, set, huge);
+		__pte_update(mm, addr, pte_offset_kernel(pmdp + 1, 0), clr, set, huge);
+	} else {
+		old = __pte_update(mm, addr, ptep, clr, set, huge);
+	}
+	return old;
+}
 #define pte_update pte_update
 #ifdef CONFIG_PPC_16K_PAGES

arch/powerpc/include/asm/nohash/hugetlb-e500.h+9−30

@@ -2,38 +2,8 @@ · 1 unmodified line
2	#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H	2	#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H
3	#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H	3	#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H
4		4
5	static inline pte_t *hugepd_page(hugepd_t hpd)
6	{
7	if (WARN_ON(!hugepd_ok(hpd)))
8	return NULL;
9
10	return (pte_t *)((hpd_val(hpd) & ~HUGEPD_SHIFT_MASK) \| PD_HUGE);
11	}
12
13	static inline unsigned int hugepd_shift(hugepd_t hpd)
14	{
15	return hpd_val(hpd) & HUGEPD_SHIFT_MASK;
16	}
17
18	static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr,
19	unsigned int pdshift)
20	{
21	/*
22	* On FSL BookE, we have multiple higher-level table entries that
23	* point to the same hugepte. Just use the first one since they're all
24	* identical. So for that case, idx=0.
25	*/
26	return hugepd_page(hpd);
27	}
28
29	void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr);	5	void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr);
30		6
31	static inline void hugepd_populate(hugepd_t hpdp, pte_t new, unsigned int pshift)
32	{
33	/* We use the old format for PPC_E500 */
34	*hpdp = __hugepd(((unsigned long)new & ~PD_HUGE) \| pshift);
35	}
36
37	static inline int check_and_get_huge_psize(int shift)	7	static inline int check_and_get_huge_psize(int shift)
38	{	8	{
39	if (shift & 1) /* Not a power of 4 */	9	if (shift & 1) /* Not a power of 4 */
@@ -42,4 +12,13 @@ static inline int check_and_get_huge_psize(int shift) · 2 unmodified lines
42	return shift_to_mmu_psize(shift);	12	return shift_to_mmu_psize(shift);
43	}	13	}
44		14
		15	static inline pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags)
		16	{
		17	unsigned int tsize = shift - _PAGE_PSIZE_SHIFT_OFFSET;
		18	pte_basic_t val = (tsize << _PAGE_PSIZE_SHIFT) & _PAGE_PSIZE_MSK;
		19
		20	return __pte((pte_val(entry) & ~(pte_basic_t)_PAGE_PSIZE_MSK) \| val);
		21	}
		22	#define arch_make_huge_pte arch_make_huge_pte
		23
45	#endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */	24	#endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */

arch/powerpc/include/asm/nohash/mmu-e500.h+1−5

 /* Page size definitions, common between 32 and 64-bit
  *
  *    shift : is the "PAGE_SHIFT" value for that page size
- *    penc  : is the pte encoding mask
  *
  */
 struct mmu_psize_def
 {
 	unsigned int	shift;	/* number of bits */
-	unsigned int	enc;	/* PTE encoding */
-	unsigned int    ind;    /* Corresponding indirect page size shift */
 	unsigned int	flags;
 #define MMU_PAGE_SIZE_DIRECT	0x1	/* Supported as a direct size */
 #define MMU_PAGE_SIZE_INDIRECT	0x2	/* Supported as an indirect size */
 extern int book3e_htw_mode;
 #define PPC_HTW_NONE	0
-#define PPC_HTW_IBM	1
+#define PPC_HTW_E6500	1
-#define PPC_HTW_E6500	2
 /*
  * 64-bit booke platforms don't load the tlb in the tlb miss handler code.

arch/powerpc/include/asm/nohash/pgalloc.h+0−2

@@ -44,8 +44,6 @@ static inline void pgtable_free(void *table, int shift) · 43 unmodified lines
44	}	44	}
45	}	45	}
46		46
47	#define get_hugepd_cache_index(x) (x)
48
49	static inline void pgtable_free_tlb(struct mmu_gather tlb, void table, int shift)	47	static inline void pgtable_free_tlb(struct mmu_gather tlb, void table, int shift)
50	{	48	{
51	unsigned long pgf = (unsigned long)table;	49	unsigned long pgf = (unsigned long)table;

arch/powerpc/include/asm/nohash/pgtable.h+31−15

 extern int icache_44x_need_flush;
+#ifndef pte_huge_size
+static inline unsigned long pte_huge_size(pte_t pte)
+{
+	return PAGE_SIZE;
+}
+#endif
 /*
  * PTE updates. This function is called whenever an existing
  * valid PTE is updated. This does -not- include set_pte_at()
 {
 	pte_basic_t old = pte_val(*p);
 	pte_basic_t new = (old & ~(pte_basic_t)clr) | set;
+	unsigned long sz;
+	unsigned long pdsize;
+	int i;
 	if (new == old)
 		return old;
-	*p = __pte(new);
+	if (huge)
+		sz = pte_huge_size(__pte(old));
+	else
+		sz = PAGE_SIZE;
+	if (sz < PMD_SIZE)
+		pdsize = PAGE_SIZE;
+	else if (sz < PUD_SIZE)
+		pdsize = PMD_SIZE;
+	else if (sz < P4D_SIZE)
+		pdsize = PUD_SIZE;
+	else if (sz < PGDIR_SIZE)
+		pdsize = P4D_SIZE;
+	else
+		pdsize = PGDIR_SIZE;
+	for (i = 0; i < sz / pdsize; i++, p++) {
+		*p = __pte(new);
+		if (new)
+			new += (unsigned long long)(pdsize / PAGE_SIZE) << PTE_RPN_SHIFT;
+	}
 	if (IS_ENABLED(CONFIG_44x) && !is_kernel_addr(addr) && (old & _PAGE_EXEC))
 		icache_44x_need_flush = 1;
 #define pgprot_writecombine pgprot_noncached_wc
-#ifdef CONFIG_HUGETLB_PAGE
-static inline int hugepd_ok(hugepd_t hpd)
-{
-#ifdef CONFIG_PPC_8xx
-	return ((hpd_val(hpd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M);
-#else
-	/* We clear the top bit to indicate hugepd */
-	return (hpd_val(hpd) && (hpd_val(hpd) & PD_HUGE) == 0);
-#endif
-}
-#define is_hugepd(hpd)		(hugepd_ok(hpd))
-#endif
 int map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot);
 void unmap_kernel_page(unsigned long va);

arch/powerpc/include/asm/nohash/pte-e500.h+46−17

 #define _PAGE_BAP_SX	0x000040
 #define _PAGE_BAP_UX	0x000080
 #define _PAGE_PSIZE_MSK	0x000f00
-#define _PAGE_PSIZE_4K	0x000200
+#define _PAGE_TSIZE_4K	0x000100
-#define _PAGE_PSIZE_8K	0x000300
-#define _PAGE_PSIZE_16K	0x000400
-#define _PAGE_PSIZE_32K	0x000500
-#define _PAGE_PSIZE_64K	0x000600
-#define _PAGE_PSIZE_128K	0x000700
-#define _PAGE_PSIZE_256K	0x000800
-#define _PAGE_PSIZE_512K	0x000900
-#define _PAGE_PSIZE_1M	0x000a00
-#define _PAGE_PSIZE_2M	0x000b00
-#define _PAGE_PSIZE_4M	0x000c00
-#define _PAGE_PSIZE_8M	0x000d00
-#define _PAGE_PSIZE_16M	0x000e00
-#define _PAGE_PSIZE_32M	0x000f00
 #define _PAGE_DIRTY	0x001000 /* C: page changed */
 #define _PAGE_SW0	0x002000
 #define _PAGE_U3	0x004000
 #define _PAGE_NO_CACHE	0x400000 /* I: cache inhibit */
 #define _PAGE_WRITETHRU	0x800000 /* W: cache write-through */
+#define _PAGE_PSIZE_SHIFT		7
+#define _PAGE_PSIZE_SHIFT_OFFSET	10
 /* "Higher level" linux bit combinations */
 #define _PAGE_EXEC		(_PAGE_BAP_SX | _PAGE_BAP_UX) /* .. and was cache cleaned */
 #define _PAGE_READ		(_PAGE_BAP_SR | _PAGE_BAP_UR) /* User read permission */
 #define _PAGE_SPECIAL	_PAGE_SW0
-/* Base page size */
-#define _PAGE_PSIZE	_PAGE_PSIZE_4K
 #define	PTE_RPN_SHIFT	(24)
 #define PTE_WIMGE_SHIFT (19)
  * pages. We always set _PAGE_COHERENT when SMP is enabled or
  * the processor might need it for DMA coherency.
  */
-#define _PAGE_BASE_NC	(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE)
+#define _PAGE_BASE_NC	(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_TSIZE_4K)
 #if defined(CONFIG_SMP)
 #define _PAGE_BASE	(_PAGE_BASE_NC | _PAGE_COHERENT)
 #else
 }
 #define pte_mkexec pte_mkexec
+static inline unsigned long pte_huge_size(pte_t pte)
+{
+	pte_basic_t val = pte_val(pte);
+	return 1UL << (((val & _PAGE_PSIZE_MSK) >> _PAGE_PSIZE_SHIFT) + _PAGE_PSIZE_SHIFT_OFFSET);
+}
+#define pte_huge_size pte_huge_size
+static inline int pmd_leaf(pmd_t pmd)
+{
+	if (IS_ENABLED(CONFIG_PPC64))
+		return (long)pmd_val(pmd) > 0;
+	else
+		return pmd_val(pmd) & _PAGE_PSIZE_MSK;
+}
+#define pmd_leaf pmd_leaf
+static inline unsigned long pmd_leaf_size(pmd_t pmd)
+{
+	return pte_huge_size(__pte(pmd_val(pmd)));
+}
+#define pmd_leaf_size pmd_leaf_size
+#ifdef CONFIG_PPC64
+static inline int pud_leaf(pud_t pud)
+{
+	if (IS_ENABLED(CONFIG_PPC64))
+		return (long)pud_val(pud) > 0;
+	else
+		return pud_val(pud) & _PAGE_PSIZE_MSK;
+}
+#define pud_leaf pud_leaf
+static inline unsigned long pud_leaf_size(pud_t pud)
+{
+	return pte_huge_size(__pte(pud_val(pud)));
+}
+#define pud_leaf_size pud_leaf_size
+#endif
 #endif /* __ASSEMBLY__ */
 #endif /* __KERNEL__ */

arch/powerpc/include/asm/page.h+0−32

@@ -269,38 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) · 268 unmodified lines
269	#define is_kernel_addr(x) ((x) >= TASK_SIZE)	269	#define is_kernel_addr(x) ((x) >= TASK_SIZE)
270	#endif	270	#endif
271		271
272	#ifndef CONFIG_PPC_BOOK3S_64
273	/*
274	* Use the top bit of the higher-level page table entries to indicate whether
275	* the entries we point to contain hugepages. This works because we know that
276	* the page tables live in kernel space. If we ever decide to support having
277	* page tables at arbitrary addresses, this breaks and will have to change.
278	*/
279	#ifdef CONFIG_PPC64
280	#define PD_HUGE 0x8000000000000000UL
281	#else
282	#define PD_HUGE 0x80000000
283	#endif
284
285	#else /* CONFIG_PPC_BOOK3S_64 */
286	/*
287	* Book3S 64 stores real addresses in the hugepd entries to
288	* avoid overlaps with _PAGE_PRESENT and _PAGE_PTE.
289	*/
290	#define HUGEPD_ADDR_MASK (0x0ffffffffffffffful & ~HUGEPD_SHIFT_MASK)
291	#endif /* CONFIG_PPC_BOOK3S_64 */
292
293	/*
294	* Some number of bits at the level of the page table that points to
295	* a hugepte are used to encode the size. This masks those bits.
296	* On 8xx, HW assistance requires 4k alignment for the hugepte.
297	*/
298	#ifdef CONFIG_PPC_8xx
299	#define HUGEPD_SHIFT_MASK 0xfff
300	#else
301	#define HUGEPD_SHIFT_MASK 0x3f
302	#endif
303
304	#ifndef __ASSEMBLY__	272	#ifndef __ASSEMBLY__
305		273
306	#ifdef CONFIG_PPC_BOOK3S_64	274	#ifdef CONFIG_PPC_BOOK3S_64

arch/powerpc/include/asm/pgtable-be-types.h+0−10

@@ -101,14 +101,4 @@ static inline bool pmd_xchg(pmd_t *pmdp, pmd_t old, pmd_t new) · 100 unmodified lines
101	return pmd_raw(old) == prev;	101	return pmd_raw(old) == prev;
102	}	102	}
103		103
104	#ifdef CONFIG_ARCH_HAS_HUGEPD
105	typedef struct { __be64 pdbe; } hugepd_t;
106	#define __hugepd(x) ((hugepd_t) { cpu_to_be64(x) })
107
108	static inline unsigned long hpd_val(hugepd_t x)
109	{
110	return be64_to_cpu(x.pdbe);
111	}
112	#endif
113
114	#endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */	104	#endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */

arch/powerpc/include/asm/pgtable-types.h+4−9

 #endif /* CONFIG_PPC64 */
 /* PGD level */
+#if defined(CONFIG_PPC_E500) && defined(CONFIG_PTE_64BIT)
+typedef struct { unsigned long long pgd; } pgd_t;
+#else
 typedef struct { unsigned long pgd; } pgd_t;
+#endif
 #define __pgd(x)	((pgd_t) { (x) })
 static inline unsigned long pgd_val(pgd_t x)
 {
 }
 #endif
-#ifdef CONFIG_ARCH_HAS_HUGEPD
-typedef struct { unsigned long pd; } hugepd_t;
-#define __hugepd(x) ((hugepd_t) { (x) })
-static inline unsigned long hpd_val(hugepd_t x)
-{
-	return x.pd;
-}
-#endif
 #endif /* _ASM_POWERPC_PGTABLE_TYPES_H */

arch/powerpc/include/asm/pgtable.h+3−0

@@ -106,6 +106,9 @@ unsigned long vmalloc_to_phys(void *vmalloc_addr); · 105 unmodified lines
106		106
107	void pgtable_cache_add(unsigned int shift);	107	void pgtable_cache_add(unsigned int shift);
108		108
		109	#ifdef CONFIG_PPC32
		110	void __init *early_alloc_pgtable(unsigned long size);
		111	#endif
109	pte_t early_pte_alloc_kernel(pmd_t pmdp, unsigned long va);	112	pte_t early_pte_alloc_kernel(pmd_t pmdp, unsigned long va);
110		113
111	#if defined(CONFIG_STRICT_KERNEL_RWX) \|\| defined(CONFIG_PPC32)	114	#if defined(CONFIG_STRICT_KERNEL_RWX) \|\| defined(CONFIG_PPC32)

arch/powerpc/kernel/exceptions-64e.S+2−2

@@ -485,8 +485,8 @@ interrupt_base_book3e: /* fake trap */ · 484 unmodified lines
485	EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */	485	EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */
486	EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */	486	EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */
487	EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */	487	EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */
488	EXCEPTION_STUB(0x1c0, data_tlb_miss)	488	EXCEPTION_STUB(0x1c0, data_tlb_miss_bolted)
489	EXCEPTION_STUB(0x1e0, instruction_tlb_miss)	489	EXCEPTION_STUB(0x1e0, instruction_tlb_miss_bolted)
490	EXCEPTION_STUB(0x200, altivec_unavailable)	490	EXCEPTION_STUB(0x200, altivec_unavailable)
491	EXCEPTION_STUB(0x220, altivec_assist)	491	EXCEPTION_STUB(0x220, altivec_assist)
492	EXCEPTION_STUB(0x260, perfmon)	492	EXCEPTION_STUB(0x260, perfmon)

arch/powerpc/kernel/head_85xx.S+26−44

@@ -294,9 +294,10 @@ set_ivor: · 293 unmodified lines
294	/* Macros to hide the PTE size differences	294	/* Macros to hide the PTE size differences
295	*	295	*
296	* FIND_PTE -- walks the page tables given EA & pgdir pointer	296	* FIND_PTE -- walks the page tables given EA & pgdir pointer
297	* r10 -- EA of fault	297	* r10 -- free
298	* r11 -- PGDIR pointer	298	* r11 -- PGDIR pointer
299	* r12 -- free	299	* r12 -- free
		300	* r13 -- EA of fault
300	* label 2: is the bailout case	301	* label 2: is the bailout case
301	*	302	*
302	* if we find the pte (fall through):	303	* if we find the pte (fall through):
@@ -307,34 +308,34 @@ set_ivor: · 4 unmodified lines
307	#ifdef CONFIG_PTE_64BIT	308	#ifdef CONFIG_PTE_64BIT
308	#ifdef CONFIG_HUGETLB_PAGE	309	#ifdef CONFIG_HUGETLB_PAGE
309	#define FIND_PTE \	310	#define FIND_PTE \
310	rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \	311	rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \
311	lwzx r11, r12, r11; /* Get pgd/pmd entry */ \	312	add r12, r11, r12; \
		313	lwz r11, 4(r12); /* Get pgd/pmd entry */ \
		314	rlwinm. r10, r11, 32 - _PAGE_PSIZE_SHIFT, 0x1e; /* get tsize*/ \
		315	bne 1000f; /* Huge page (leaf entry) */ \
312	rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \	316	rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \
313	blt 1000f; /* Normal non-huge page */ \
314	beq 2f; /* Bail if no table */ \	317	beq 2f; /* Bail if no table */ \
315	oris r11, r11, PD_HUGE@h; /* Put back address bit */ \	318	rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \
316	andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \
317	xor r12, r10, r11; /* drop size bits from pointer */ \
318	b 1001f; \
319	1000: rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \
320	li r10, 0; /* clear r10 */ \	319	li r10, 0; /* clear r10 */ \
321	1001: lwz r11, 4(r12); /* Get pte entry */	320	lwz r11, 4(r12); /* Get pte entry */ \
		321	1000:
322	#else	322	#else
323	#define FIND_PTE \	323	#define FIND_PTE \
324	rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \	324	rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \
325	lwzx r11, r12, r11; /* Get pgd/pmd entry */ \	325	add r12, r11, r12; \
		326	lwz r11, 4(r12); /* Get pgd/pmd entry */ \
326	rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \	327	rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \
327	beq 2f; /* Bail if no table */ \	328	beq 2f; /* Bail if no table */ \
328	rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \	329	rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \
329	lwz r11, 4(r12); /* Get pte entry */	330	lwz r11, 4(r12); /* Get pte entry */
330	#endif /* HUGEPAGE */	331	#endif /* HUGEPAGE */
331	#else /* !PTE_64BIT */	332	#else /* !PTE_64BIT */
332	#define FIND_PTE \	333	#define FIND_PTE \
333	rlwimi r11, r10, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \	334	rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \
334	lwz r11, 0(r11); /* Get L1 entry */ \	335	lwz r11, 0(r11); /* Get L1 entry */ \
335	rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \	336	rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \
336	beq 2f; /* Bail if no table */ \	337	beq 2f; /* Bail if no table */ \
337	rlwimi r12, r10, 22, 20, 29; /* Compute PTE address */ \	338	rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \
338	lwz r11, 0(r12); /* Get Linux PTE */	339	lwz r11, 0(r12); /* Get Linux PTE */
339	#endif	340	#endif
340		341
@@ -441,13 +442,13 @@ START_BTB_FLUSH_SECTION · 100 unmodified lines
441	BTB_FLUSH(r10)	442	BTB_FLUSH(r10)
442	1:	443	1:
443	END_BTB_FLUSH_SECTION	444	END_BTB_FLUSH_SECTION
444	mfspr r10, SPRN_DEAR /* Get faulting address */	445	mfspr r13, SPRN_DEAR /* Get faulting address */
445		446
446	/* If we are faulting a kernel address, we have to use the	447	/* If we are faulting a kernel address, we have to use the
447	* kernel page tables.	448	* kernel page tables.
448	*/	449	*/
449	lis r11, PAGE_OFFSET@h	450	lis r11, PAGE_OFFSET@h
450	cmplw 5, r10, r11	451	cmplw 5, r13, r11
451	blt 5, 3f	452	blt 5, 3f
452	lis r11, swapper_pg_dir@h	453	lis r11, swapper_pg_dir@h
453	ori r11, r11, swapper_pg_dir@l	454	ori r11, r11, swapper_pg_dir@l
@@ -470,29 +471,14 @@ END_BTB_FLUSH_SECTION · 16 unmodified lines
470	#endif	471	#endif
471		472
472	4:	473	4:
473	/* Mask of required permission bits. Note that while we	474	FIND_PTE
474	* do copy ESR:ST to _PAGE_WRITE position as trying to write	475
475	* to an RO page is pretty common, we don't do it with
476	* _PAGE_DIRTY. We could do it, but it's a fairly rare
477	* event so I'd rather take the overhead when it happens
478	* rather than adding an instruction here. We should measure
479	* whether the whole thing is worth it in the first place
480	* as we could avoid loading SPRN_ESR completely in the first
481	* place...
482	*
483	* TODO: Is it worth doing that mfspr & rlwimi in the first
484	* place or can we save a couple of instructions here ?
485	*/
486	mfspr r12,SPRN_ESR
487	#ifdef CONFIG_PTE_64BIT	476	#ifdef CONFIG_PTE_64BIT
488	li r13,_PAGE_PRESENT\|_PAGE_BAP_SR	477	li r13,_PAGE_PRESENT\|_PAGE_BAP_SR
489	oris r13,r13,_PAGE_ACCESSED@h	478	oris r13,r13,_PAGE_ACCESSED@h
490	#else	479	#else
491	li r13,_PAGE_PRESENT\|_PAGE_READ\|_PAGE_ACCESSED	480	li r13,_PAGE_PRESENT\|_PAGE_READ\|_PAGE_ACCESSED
492	#endif	481	#endif
493	rlwimi r13,r12,11,29,29
494
495	FIND_PTE
496	andc. r13,r13,r11 /* Check permission */	482	andc. r13,r13,r11 /* Check permission */
497		483
498	#ifdef CONFIG_PTE_64BIT	484	#ifdef CONFIG_PTE_64BIT
@@ -549,13 +535,13 @@ START_BTB_FLUSH_SECTION · 50 unmodified lines
549	1:	535	1:
550	END_BTB_FLUSH_SECTION	536	END_BTB_FLUSH_SECTION
551		537
552	mfspr r10, SPRN_SRR0 /* Get faulting address */	538	mfspr r13, SPRN_SRR0 /* Get faulting address */
553		539
554	/* If we are faulting a kernel address, we have to use the	540	/* If we are faulting a kernel address, we have to use the
555	* kernel page tables.	541	* kernel page tables.
556	*/	542	*/
557	lis r11, PAGE_OFFSET@h	543	lis r11, PAGE_OFFSET@h
558	cmplw 5, r10, r11	544	cmplw 5, r13, r11
559	blt 5, 3f	545	blt 5, 3f
560	lis r11, swapper_pg_dir@h	546	lis r11, swapper_pg_dir@h
561	ori r11, r11, swapper_pg_dir@l	547	ori r11, r11, swapper_pg_dir@l
@@ -564,6 +550,7 @@ END_BTB_FLUSH_SECTION · 2 unmodified lines
564	rlwinm r12,r12,0,16,1	550	rlwinm r12,r12,0,16,1
565	mtspr SPRN_MAS1,r12	551	mtspr SPRN_MAS1,r12
566		552
		553	FIND_PTE
567	/* Make up the required permissions for kernel code */	554	/* Make up the required permissions for kernel code */
568	#ifdef CONFIG_PTE_64BIT	555	#ifdef CONFIG_PTE_64BIT
569	li r13,_PAGE_PRESENT \| _PAGE_BAP_SX	556	li r13,_PAGE_PRESENT \| _PAGE_BAP_SX
@@ -584,6 +571,7 @@ END_BTB_FLUSH_SECTION · 14 unmodified lines
584	beq 2f /* KUAP fault */	571	beq 2f /* KUAP fault */
585	#endif	572	#endif
586		573
		574	FIND_PTE
587	/* Make up the required permissions for user code */	575	/* Make up the required permissions for user code */
588	#ifdef CONFIG_PTE_64BIT	576	#ifdef CONFIG_PTE_64BIT
589	li r13,_PAGE_PRESENT \| _PAGE_BAP_UX	577	li r13,_PAGE_PRESENT \| _PAGE_BAP_UX
@@ -593,7 +581,6 @@ END_BTB_FLUSH_SECTION · 3 unmodified lines
593	#endif	581	#endif
594		582
595	4:	583	4:
596	FIND_PTE
597	andc. r13,r13,r11 /* Check permission */	584	andc. r13,r13,r11 /* Check permission */
598		585
599	#ifdef CONFIG_PTE_64BIT	586	#ifdef CONFIG_PTE_64BIT
@@ -746,17 +733,12 @@ finish_tlb_load: · 146 unmodified lines
746	lwz r15, 0(r14)	733	lwz r15, 0(r14)
747	100: stw r15, 0(r17)	734	100: stw r15, 0(r17)
748		735
749	/*
750	* Calc MAS1_TSIZE from r10 (which has pshift encoded)
751	* tlb_enc = (pshift - 10).
752	*/
753	subi r15, r10, 10
754	mfspr r16, SPRN_MAS1	736	mfspr r16, SPRN_MAS1
755	rlwimi r16, r15, 7, 20, 24	737	rlwimi r16, r10, MAS1_TSIZE_SHIFT, MAS1_TSIZE_MASK
756	mtspr SPRN_MAS1, r16	738	mtspr SPRN_MAS1, r16
757		739
758	/* copy the pshift for use later */	740	/* copy the pshift for use later */
759	mr r14, r10	741	addi r14, r10, _PAGE_PSIZE_SHIFT_OFFSET
760		742
761	/* fall through */	743	/* fall through */
762		744

arch/powerpc/kernel/head_8xx.S+2−8

 	oris	r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha
 3:
 	lwz	r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11)	/* Get the level 1 entry */
+	rlwinm	r11, r11, 0, ~_PMD_PAGE_8M
 	mtspr	SPRN_MD_TWC, r11
-	mtcrf	0x01, r11
 	mfspr	r11, SPRN_MD_TWC
 	lwz	r11, 0(r11)	/* Get the pte */
-	bt	28,200f		/* bit 28 = Large page (8M) */
 	/* concat physical page address(r11) and page offset(r10) */
 	rlwimi	r11, r10, 0, 32 - PAGE_SHIFT, 31
-201:	lwz	r11,0(r11)
+	lwz	r11,0(r11)
 /* Check if it really is a dcbx instruction. */
 /* dcbt and dcbtst does not generate DTLB Misses/Errors,
  * no need to include them here */
 141:	mfspr	r10,SPRN_M_TW
 	b	DARFixed	/* Nope, go back to normal TLB processing */
-200:
-	/* concat physical page address(r11) and page offset(r10) */
-	rlwimi	r11, r10, 0, 32 - PAGE_SHIFT_8M, 31
-	b	201b
 144:	mfspr	r10, SPRN_DSISR
 	rlwinm	r10, r10,0,7,5	/* Clear store bit for buggy dcbst insn */
 	mtspr	SPRN_DSISR, r10

arch/powerpc/kernel/setup_64.c+1−5

 {
 #ifdef CONFIG_PPC_BOOK3E_64
 	/* Freescale BookE bolts the entire linear mapping */
-	/* XXX: BookE ppc64_rma_limit setup seems to disagree? */
+	return linear_map_top;
-	if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E))
-		return linear_map_top;
-	/* Other BookE, we assume the first GB is bolted */
-	return 1ul << 30;
 #else
 	/* BookS radix, does not take faults on linear mapping */
 	if (early_radix_enabled())

arch/powerpc/mm/book3s64/hash_utils.c+7−4

 	__pmd_table_size = H_PMD_TABLE_SIZE;
 	__pud_table_size = H_PUD_TABLE_SIZE;
 	__pgd_table_size = H_PGD_TABLE_SIZE;
-	/*
-	 * 4k use hugepd format, so for hash set then to
-	 * zero
-	 */
 	__pmd_val_bits = HASH_PMD_VAL_BITS;
 	__pud_val_bits = HASH_PUD_VAL_BITS;
 	__pgd_val_bits = HASH_PGD_VAL_BITS;
 		goto bail;
 	}
+	if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) {
+		if (hugeshift == PMD_SHIFT && psize == MMU_PAGE_16M)
+			hugeshift = mmu_psize_defs[MMU_PAGE_16M].shift;
+		if (hugeshift == PUD_SHIFT && psize == MMU_PAGE_16G)
+			hugeshift = mmu_psize_defs[MMU_PAGE_16G].shift;
+	}
 	/*
 	 * Add _PAGE_PRESENT to the required access perm. If there are parallel
 	 * updates to the pte that can possibly clear _PAGE_PTE, catch that too.

arch/powerpc/mm/book3s64/hugetlbpage.c+10−0

@@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 52 unmodified lines
53	/* If PTE permissions don't match, take page fault */	53	/* If PTE permissions don't match, take page fault */
54	if (unlikely(!check_pte_access(access, old_pte)))	54	if (unlikely(!check_pte_access(access, old_pte)))
55	return 1;	55	return 1;
		56	/*
		57	* If hash-4k, hugepages use seeral contiguous PxD entries
		58	* so bail out and let mm make the page young or dirty
		59	*/
		60	if (IS_ENABLED(CONFIG_PPC_4K_PAGES)) {
		61	if (!(old_pte & _PAGE_ACCESSED))
		62	return 1;
		63	if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY))
		64	return 1;
		65	}
56		66
57	/*	67	/*
58	* Try to lock the PTE, add ACCESSED and DIRTY if it was	68	* Try to lock the PTE, add ACCESSED and DIRTY if it was

arch/powerpc/mm/book3s64/pgtable.c+0−12

@@ -461,18 +461,6 @@ static inline void pgtable_free(void *table, int index) · 460 unmodified lines
461	case PUD_INDEX:	461	case PUD_INDEX:
462	__pud_free(table);	462	__pud_free(table);
463	break;	463	break;
464	#if defined(CONFIG_PPC_4K_PAGES) && defined(CONFIG_HUGETLB_PAGE)
465	/* 16M hugepd directory at pud level */
466	case HTLB_16M_INDEX:
467	BUILD_BUG_ON(H_16M_CACHE_INDEX <= 0);
468	kmem_cache_free(PGT_CACHE(H_16M_CACHE_INDEX), table);
469	break;
470	/* 16G hugepd directory at the pgd level */
471	case HTLB_16G_INDEX:
472	BUILD_BUG_ON(H_16G_CACHE_INDEX <= 0);
473	kmem_cache_free(PGT_CACHE(H_16G_CACHE_INDEX), table);
474	break;
475	#endif
476	/* We don't free pgd table via RCU callback */	464	/* We don't free pgd table via RCU callback */
477	default:	465	default:
478	BUG();	466	BUG();

arch/powerpc/mm/hugetlbpage.c+26−429

@@ -28,8 +28,6 @@ · 27 unmodified lines
28		28
29	bool hugetlb_disabled = false;	29	bool hugetlb_disabled = false;
30		30
31	#define hugepd_none(hpd) (hpd_val(hpd) == 0)
32
33	#define PTE_T_ORDER (__builtin_ffs(sizeof(pte_basic_t)) - \	31	#define PTE_T_ORDER (__builtin_ffs(sizeof(pte_basic_t)) - \
34	__builtin_ffs(sizeof(void *)))	32	__builtin_ffs(sizeof(void *)))
35		33
@@ -42,156 +40,43 @@ pte_t huge_pte_offset(struct mm_struct mm, unsigned long addr, unsigned long s · 6 unmodified lines
42	return __find_linux_pte(mm->pgd, addr, NULL, NULL);	40	return __find_linux_pte(mm->pgd, addr, NULL, NULL);
43	}	41	}
44		42
45	static int __hugepte_alloc(struct mm_struct mm, hugepd_t hpdp,	43	pte_t huge_pte_alloc(struct mm_struct mm, struct vm_area_struct *vma,
46	unsigned long address, unsigned int pdshift,	44	unsigned long addr, unsigned long sz)
47	unsigned int pshift, spinlock_t *ptl)
48	{	45	{
49	struct kmem_cache *cachep;	46	p4d_t *p4d;
50	pte_t *new;	47	pud_t *pud;
51	int i;	48	pmd_t *pmd;
52	int num_hugepd;
53
54	if (pshift >= pdshift) {
55	cachep = PGT_CACHE(PTE_T_ORDER);
56	num_hugepd = 1 << (pshift - pdshift);
57	} else {
58	cachep = PGT_CACHE(pdshift - pshift);
59	num_hugepd = 1;
60	}
61
62	if (!cachep) {
63	WARN_ONCE(1, "No page table cache created for hugetlb tables");
64	return -ENOMEM;
65	}
66
67	new = kmem_cache_alloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL));
68		49
69	BUG_ON(pshift > HUGEPD_SHIFT_MASK);	50	addr &= ~(sz - 1);
70	BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
71		51
72	if (!new)	52	p4d = p4d_offset(pgd_offset(mm, addr), addr);
73	return -ENOMEM;	53	if (!mm_pud_folded(mm) && sz >= P4D_SIZE)
		54	return (pte_t *)p4d;
74		55
75	/*	56	pud = pud_alloc(mm, p4d, addr);
76	* Make sure other cpus find the hugepd set only after a	57	if (!pud)
77	* properly initialized page table is visible to them.	58	return NULL;
78	* For more details look for comment in __pte_alloc().	59	if (!mm_pmd_folded(mm) && sz >= PUD_SIZE)
79	*/	60	return (pte_t *)pud;
80	smp_wmb();
81		61
82	spin_lock(ptl);	62	pmd = pmd_alloc(mm, pud, addr);
83	/*	63	if (!pmd)
84	* We have multiple higher-level entries that point to the same	64	return NULL;
85	* actual pte location. Fill in each as we go and backtrack on error.
86	* We need all of these so the DTLB pgtable walk code can find the
87	* right higher-level entry without knowing if it's a hugepage or not.
88	*/
89	for (i = 0; i < num_hugepd; i++, hpdp++) {
90	if (unlikely(!hugepd_none(*hpdp)))
91	break;
92	hugepd_populate(hpdp, new, pshift);
93	}
94	/* If we bailed from the for loop early, an error occurred, clean up */
95	if (i < num_hugepd) {
96	for (i = i - 1 ; i >= 0; i--, hpdp--)
97	*hpdp = __hugepd(0);
98	kmem_cache_free(cachep, new);
99	} else {
100	kmemleak_ignore(new);
101	}
102	spin_unlock(ptl);
103	return 0;
104	}
105		65
106	/*	66	if (sz >= PMD_SIZE) {
107	* At this point we do the placement change only for BOOK3S 64. This would	67	/* On 8xx, all hugepages are handled as contiguous PTEs */
108	* possibly work on other subarchs.	68	if (IS_ENABLED(CONFIG_PPC_8xx)) {
109	*/	69	int i;
110	pte_t huge_pte_alloc(struct mm_struct mm, struct vm_area_struct *vma,
111	unsigned long addr, unsigned long sz)
112	{
113	pgd_t *pg;
114	p4d_t *p4;
115	pud_t *pu;
116	pmd_t *pm;
117	hugepd_t *hpdp = NULL;
118	unsigned pshift = __ffs(sz);
119	unsigned pdshift = PGDIR_SHIFT;
120	spinlock_t *ptl;
121
122	addr &= ~(sz-1);
123	pg = pgd_offset(mm, addr);
124	p4 = p4d_offset(pg, addr);
125		70
126	#ifdef CONFIG_PPC_BOOK3S_64	71	for (i = 0; i < sz / PMD_SIZE; i++) {
127	if (pshift == PGDIR_SHIFT)	72	if (!pte_alloc_huge(mm, pmd + i, addr))
128	/* 16GB huge page */	73	return NULL;
129	return (pte_t *) p4;
130	else if (pshift > PUD_SHIFT) {
131	/*
132	* We need to use hugepd table
133	*/
134	ptl = &mm->page_table_lock;
135	hpdp = (hugepd_t *)p4;
136	} else {
137	pdshift = PUD_SHIFT;
138	pu = pud_alloc(mm, p4, addr);
139	if (!pu)
140	return NULL;
141	if (pshift == PUD_SHIFT)
142	return (pte_t *)pu;
143	else if (pshift > PMD_SHIFT) {
144	ptl = pud_lockptr(mm, pu);
145	hpdp = (hugepd_t *)pu;
146	} else {
147	pdshift = PMD_SHIFT;
148	pm = pmd_alloc(mm, pu, addr);
149	if (!pm)
150	return NULL;
151	if (pshift == PMD_SHIFT)
152	/* 16MB hugepage */
153	return (pte_t *)pm;
154	else {
155	ptl = pmd_lockptr(mm, pm);
156	hpdp = (hugepd_t *)pm;
157	}	74	}
158	}	75	}
		76	return (pte_t *)pmd;
159	}	77	}
160	#else
161	if (pshift >= PGDIR_SHIFT) {
162	ptl = &mm->page_table_lock;
163	hpdp = (hugepd_t *)p4;
164	} else {
165	pdshift = PUD_SHIFT;
166	pu = pud_alloc(mm, p4, addr);
167	if (!pu)
168	return NULL;
169	if (pshift >= PUD_SHIFT) {
170	ptl = pud_lockptr(mm, pu);
171	hpdp = (hugepd_t *)pu;
172	} else {
173	pdshift = PMD_SHIFT;
174	pm = pmd_alloc(mm, pu, addr);
175	if (!pm)
176	return NULL;
177	ptl = pmd_lockptr(mm, pm);
178	hpdp = (hugepd_t *)pm;
179	}
180	}
181	#endif
182	if (!hpdp)
183	return NULL;
184
185	if (IS_ENABLED(CONFIG_PPC_8xx) && pshift < PMD_SHIFT)
186	return pte_alloc_huge(mm, (pmd_t *)hpdp, addr);
187
188	BUG_ON(!hugepd_none(hpdp) && !hugepd_ok(hpdp));
189		78
190	if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr,	79	return pte_alloc_huge(mm, pmd, addr);
191	pdshift, pshift, ptl))
192	return NULL;
193
194	return hugepte_offset(*hpdp, addr, pdshift);
195	}	80	}
196		81
197	#ifdef CONFIG_PPC_BOOK3S_64	82	#ifdef CONFIG_PPC_BOOK3S_64
@@ -248,264 +133,6 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid) · 50 unmodified lines
248	return __alloc_bootmem_huge_page(h, nid);	133	return __alloc_bootmem_huge_page(h, nid);
249	}	134	}
250		135
251	#ifndef CONFIG_PPC_BOOK3S_64
252	#define HUGEPD_FREELIST_SIZE \
253	((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
254
255	struct hugepd_freelist {
256	struct rcu_head rcu;
257	unsigned int index;
258	void *ptes[];
259	};
260
261	static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur);
262
263	static void hugepd_free_rcu_callback(struct rcu_head *head)
264	{
265	struct hugepd_freelist *batch =
266	container_of(head, struct hugepd_freelist, rcu);
267	unsigned int i;
268
269	for (i = 0; i < batch->index; i++)
270	kmem_cache_free(PGT_CACHE(PTE_T_ORDER), batch->ptes[i]);
271
272	free_page((unsigned long)batch);
273	}
274
275	static void hugepd_free(struct mmu_gather tlb, void hugepte)
276	{
277	struct hugepd_freelist **batchp;
278
279	batchp = &get_cpu_var(hugepd_freelist_cur);
280
281	if (atomic_read(&tlb->mm->mm_users) < 2 \|\|
282	mm_is_thread_local(tlb->mm)) {
283	kmem_cache_free(PGT_CACHE(PTE_T_ORDER), hugepte);
284	put_cpu_var(hugepd_freelist_cur);
285	return;
286	}
287
288	if (*batchp == NULL) {
289	batchp = (struct hugepd_freelist )__get_free_page(GFP_ATOMIC);
290	(*batchp)->index = 0;
291	}
292
293	(batchp)->ptes[(batchp)->index++] = hugepte;
294	if ((*batchp)->index == HUGEPD_FREELIST_SIZE) {
295	call_rcu(&(*batchp)->rcu, hugepd_free_rcu_callback);
296	*batchp = NULL;
297	}
298	put_cpu_var(hugepd_freelist_cur);
299	}
300	#else
301	static inline void hugepd_free(struct mmu_gather tlb, void hugepte) {}
302	#endif
303
304	/* Return true when the entry to be freed maps more than the area being freed */
305	static bool range_is_outside_limits(unsigned long start, unsigned long end,
306	unsigned long floor, unsigned long ceiling,
307	unsigned long mask)
308	{
309	if ((start & mask) < floor)
310	return true;
311	if (ceiling) {
312	ceiling &= mask;
313	if (!ceiling)
314	return true;
315	}
316	return end - 1 > ceiling - 1;
317	}
318
319	static void free_hugepd_range(struct mmu_gather tlb, hugepd_t hpdp, int pdshift,
320	unsigned long start, unsigned long end,
321	unsigned long floor, unsigned long ceiling)
322	{
323	pte_t hugepte = hugepd_page(hpdp);
324	int i;
325
326	unsigned long pdmask = ~((1UL << pdshift) - 1);
327	unsigned int num_hugepd = 1;
328	unsigned int shift = hugepd_shift(*hpdp);
329
330	/* Note: On fsl the hpdp may be the first of several */
331	if (shift > pdshift)
332	num_hugepd = 1 << (shift - pdshift);
333
334	if (range_is_outside_limits(start, end, floor, ceiling, pdmask))
335	return;
336
337	for (i = 0; i < num_hugepd; i++, hpdp++)
338	*hpdp = __hugepd(0);
339
340	if (shift >= pdshift)
341	hugepd_free(tlb, hugepte);
342	else
343	pgtable_free_tlb(tlb, hugepte,
344	get_hugepd_cache_index(pdshift - shift));
345	}
346
347	static void hugetlb_free_pte_range(struct mmu_gather tlb, pmd_t pmd,
348	unsigned long addr, unsigned long end,
349	unsigned long floor, unsigned long ceiling)
350	{
351	pgtable_t token = pmd_pgtable(*pmd);
352
353	if (range_is_outside_limits(addr, end, floor, ceiling, PMD_MASK))
354	return;
355
356	pmd_clear(pmd);
357	pte_free_tlb(tlb, token, addr);
358	mm_dec_nr_ptes(tlb->mm);
359	}
360
361	static void hugetlb_free_pmd_range(struct mmu_gather tlb, pud_t pud,
362	unsigned long addr, unsigned long end,
363	unsigned long floor, unsigned long ceiling)
364	{
365	pmd_t *pmd;
366	unsigned long next;
367	unsigned long start;
368
369	start = addr;
370	do {
371	unsigned long more;
372
373	pmd = pmd_offset(pud, addr);
374	next = pmd_addr_end(addr, end);
375	if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
376	if (pmd_none_or_clear_bad(pmd))
377	continue;
378
379	/*
380	* if it is not hugepd pointer, we should already find
381	* it cleared.
382	*/
383	WARN_ON(!IS_ENABLED(CONFIG_PPC_8xx));
384
385	hugetlb_free_pte_range(tlb, pmd, addr, end, floor, ceiling);
386
387	continue;
388	}
389	/*
390	* Increment next by the size of the huge mapping since
391	* there may be more than one entry at this level for a
392	* single hugepage, but all of them point to
393	* the same kmem cache that holds the hugepte.
394	*/
395	more = addr + (1UL << hugepd_shift((hugepd_t )pmd));
396	if (more > next)
397	next = more;
398
399	free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT,
400	addr, next, floor, ceiling);
401	} while (addr = next, addr != end);
402
403	if (range_is_outside_limits(start, end, floor, ceiling, PUD_MASK))
404	return;
405
406	pmd = pmd_offset(pud, start & PUD_MASK);
407	pud_clear(pud);
408	pmd_free_tlb(tlb, pmd, start & PUD_MASK);
409	mm_dec_nr_pmds(tlb->mm);
410	}
411
412	static void hugetlb_free_pud_range(struct mmu_gather tlb, p4d_t p4d,
413	unsigned long addr, unsigned long end,
414	unsigned long floor, unsigned long ceiling)
415	{
416	pud_t *pud;
417	unsigned long next;
418	unsigned long start;
419
420	start = addr;
421	do {
422	pud = pud_offset(p4d, addr);
423	next = pud_addr_end(addr, end);
424	if (!is_hugepd(__hugepd(pud_val(*pud)))) {
425	if (pud_none_or_clear_bad(pud))
426	continue;
427	hugetlb_free_pmd_range(tlb, pud, addr, next, floor,
428	ceiling);
429	} else {
430	unsigned long more;
431	/*
432	* Increment next by the size of the huge mapping since
433	* there may be more than one entry at this level for a
434	* single hugepage, but all of them point to
435	* the same kmem cache that holds the hugepte.
436	*/
437	more = addr + (1UL << hugepd_shift((hugepd_t )pud));
438	if (more > next)
439	next = more;
440
441	free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT,
442	addr, next, floor, ceiling);
443	}
444	} while (addr = next, addr != end);
445
446	if (range_is_outside_limits(start, end, floor, ceiling, PGDIR_MASK))
447	return;
448
449	pud = pud_offset(p4d, start & PGDIR_MASK);
450	p4d_clear(p4d);
451	pud_free_tlb(tlb, pud, start & PGDIR_MASK);
452	mm_dec_nr_puds(tlb->mm);
453	}
454
455	/*
456	* This function frees user-level page tables of a process.
457	*/
458	void hugetlb_free_pgd_range(struct mmu_gather *tlb,
459	unsigned long addr, unsigned long end,
460	unsigned long floor, unsigned long ceiling)
461	{
462	pgd_t *pgd;
463	p4d_t *p4d;
464	unsigned long next;
465
466	/*
467	* Because there are a number of different possible pagetable
468	* layouts for hugepage ranges, we limit knowledge of how
469	* things should be laid out to the allocation path
470	* (huge_pte_alloc(), above). Everything else works out the
471	* structure as it goes from information in the hugepd
472	* pointers. That means that we can't here use the
473	* optimization used in the normal page free_pgd_range(), of
474	* checking whether we're actually covering a large enough
475	* range to have to do anything at the top level of the walk
476	* instead of at the bottom.
477	*
478	* To make sense of this, you should probably go read the big
479	* block comment at the top of the normal free_pgd_range(),
480	* too.
481	*/
482
483	do {
484	next = pgd_addr_end(addr, end);
485	pgd = pgd_offset(tlb->mm, addr);
486	p4d = p4d_offset(pgd, addr);
487	if (!is_hugepd(__hugepd(pgd_val(*pgd)))) {
488	if (p4d_none_or_clear_bad(p4d))
489	continue;
490	hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling);
491	} else {
492	unsigned long more;
493	/*
494	* Increment next by the size of the huge mapping since
495	* there may be more than one entry at the pgd level
496	* for a single hugepage, but all of them point to the
497	* same kmem cache that holds the hugepte.
498	*/
499	more = addr + (1UL << hugepd_shift((hugepd_t )pgd));
500	if (more > next)
501	next = more;
502
503	free_hugepd_range(tlb, (hugepd_t *)p4d, PGDIR_SHIFT,
504	addr, next, floor, ceiling);
505	}
506	} while (addr = next, addr != end);
507	}
508
509	bool __init arch_hugetlb_valid_size(unsigned long size)	136	bool __init arch_hugetlb_valid_size(unsigned long size)
510	{	137	{
511	int shift = __ffs(size);	138	int shift = __ffs(size);
@@ -552,44 +179,14 @@ static int __init hugetlbpage_init(void) · 40 unmodified lines
552		179
553	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {	180	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
554	unsigned shift;	181	unsigned shift;
555	unsigned pdshift;
556		182
557	if (!mmu_psize_defs[psize].shift)	183	if (!mmu_psize_defs[psize].shift)
558	continue;	184	continue;
559		185
560	shift = mmu_psize_to_shift(psize);	186	shift = mmu_psize_to_shift(psize);
561		187
562	#ifdef CONFIG_PPC_BOOK3S_64
563	if (shift > PGDIR_SHIFT)
564	continue;
565	else if (shift > PUD_SHIFT)
566	pdshift = PGDIR_SHIFT;
567	else if (shift > PMD_SHIFT)
568	pdshift = PUD_SHIFT;
569	else
570	pdshift = PMD_SHIFT;
571	#else
572	if (shift < PUD_SHIFT)
573	pdshift = PMD_SHIFT;
574	else if (shift < PGDIR_SHIFT)
575	pdshift = PUD_SHIFT;
576	else
577	pdshift = PGDIR_SHIFT;
578	#endif
579
580	if (add_huge_page_size(1ULL << shift) < 0)	188	if (add_huge_page_size(1ULL << shift) < 0)
581	continue;	189	continue;
582	/*
583	* if we have pdshift and shift value same, we don't
584	* use pgt cache for hugepd.
585	*/
586	if (pdshift > shift) {
587	if (!IS_ENABLED(CONFIG_PPC_8xx))
588	pgtable_cache_add(pdshift - shift);
589	} else if (IS_ENABLED(CONFIG_PPC_E500) \|\|
590	IS_ENABLED(CONFIG_PPC_8xx)) {
591	pgtable_cache_add(PTE_T_ORDER);
592	}
593		190
594	configured = true;	191	configured = true;
595	}	192	}

arch/powerpc/mm/init-common.c+2−6

 	/* When batching pgtable pointers for RCU freeing, we store
 	 * the index size in the low bits.  Table alignment must be
 	 * big enough to fit it.
-	 *
+	 */
-	 * Likewise, hugeapge pagetable pointers contain a (different)
+	unsigned long minalign = MAX_PGTABLE_INDEX_SIZE + 1;
-	 * shift value in the low bits.  All tables must be aligned so
-	 * as to leave enough 0 bits in the address to contain it. */
-	unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
-				     HUGEPD_SHIFT_MASK + 1);
 	struct kmem_cache *new = NULL;
 	/* It would be nice if this was a BUILD_BUG_ON(), but at the

arch/powerpc/mm/kasan/8xx.c+13−8

 #include <linux/memblock.h>
 #include <linux/hugetlb.h>
+#include <asm/pgalloc.h>
 static int __init
 kasan_init_shadow_8M(unsigned long k_start, unsigned long k_end, void *block)
 {
 	pmd_t *pmd = pmd_off_k(k_start);
 	unsigned long k_cur, k_next;
-	for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd += 2, block += SZ_8M) {
+	for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++, block += SZ_4M) {
-		pte_basic_t *new;
+		pte_t *ptep;
+		int i;
 		k_next = pgd_addr_end(k_cur, k_end);
-		k_next = pgd_addr_end(k_next, k_end);
 		if ((void *)pmd_page_vaddr(*pmd) != kasan_early_shadow_pte)
 			continue;
-		new = memblock_alloc(sizeof(pte_basic_t), SZ_4K);
+		ptep = memblock_alloc(PTE_FRAG_SIZE, PTE_FRAG_SIZE);
-		if (!new)
+		if (!ptep)
 			return -ENOMEM;
-		*new = pte_val(pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block)), PAGE_KERNEL)));
+		for (i = 0; i < PTRS_PER_PTE; i++) {
+			pte_t pte = pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block + i * PAGE_SIZE)), PAGE_KERNEL));
-		hugepd_populate_kernel((hugepd_t *)pmd, (pte_t *)new, PAGE_SHIFT_8M);
+			__set_pte_at(&init_mm, k_cur, ptep + i, pte, 1);
-		hugepd_populate_kernel((hugepd_t *)pmd + 1, (pte_t *)new, PAGE_SHIFT_8M);
+		}
+		pmd_populate_kernel(&init_mm, pmd, ptep);
+		*pmd = __pmd(pmd_val(*pmd) | _PMD_PAGE_8M);
 	}
 	return 0;
 }

arch/powerpc/mm/nohash/8xx.c+20−23

 #include <linux/hugetlb.h>
 #include <asm/fixmap.h>
+#include <asm/pgalloc.h>
 #include <mm/mmu_decl.h>
 	return 0;
 }
-static pte_t __init *early_hugepd_alloc_kernel(hugepd_t *pmdp, unsigned long va)
-{
-	if (hpd_val(*pmdp) == 0) {
-		pte_t *ptep = memblock_alloc(sizeof(pte_basic_t), SZ_4K);
-		if (!ptep)
-			return NULL;
-		hugepd_populate_kernel((hugepd_t *)pmdp, ptep, PAGE_SHIFT_8M);
-		hugepd_populate_kernel((hugepd_t *)pmdp + 1, ptep, PAGE_SHIFT_8M);
-	}
-	return hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT);
-}
 static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,
 					     pgprot_t prot, int psize, bool new)
 {
 		if (WARN_ON(slab_is_available()))
 			return -EINVAL;
-		if (psize == MMU_PAGE_512K)
+		if (psize == MMU_PAGE_512K) {
 			ptep = early_pte_alloc_kernel(pmdp, va);
-		else
+			/* The PTE should never be already present */
-			ptep = early_hugepd_alloc_kernel((hugepd_t *)pmdp, va);
+			if (WARN_ON(pte_present(*ptep) && pgprot_val(prot)))
+				return -EINVAL;
+		} else {
+			if (WARN_ON(!pmd_none(*pmdp) || !pmd_none(*(pmdp + 1))))
+				return -EINVAL;
+			ptep = early_alloc_pgtable(PTE_FRAG_SIZE);
+			pmd_populate_kernel(&init_mm, pmdp, ptep);
+			ptep = early_alloc_pgtable(PTE_FRAG_SIZE);
+			pmd_populate_kernel(&init_mm, pmdp + 1, ptep);
+			ptep = (pte_t *)pmdp;
+		}
 	} else {
 		if (psize == MMU_PAGE_512K)
 			ptep = pte_offset_kernel(pmdp, va);
 		else
-			ptep = hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT);
+			ptep = (pte_t *)pmdp;
 	}
 	if (WARN_ON(!ptep))
 		return -ENOMEM;
-	/* The PTE should never be already present */
-	if (new && WARN_ON(pte_present(*ptep) && pgprot_val(prot)))
-		return -EINVAL;
 	set_huge_pte_at(&init_mm, va, ptep,
-			pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), psize);
+			pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)),
+			1UL << mmu_psize_to_shift(psize));
 	return 0;
 }

arch/powerpc/mm/nohash/Makefile+1−1

@@ -1,7 +1,7 @@
1	# SPDX-License-Identifier: GPL-2.0	1	# SPDX-License-Identifier: GPL-2.0
2		2
3	obj-y += mmu_context.o tlb.o tlb_low.o kup.o	3	obj-y += mmu_context.o tlb.o tlb_low.o kup.o
4	obj-$(CONFIG_PPC_BOOK3E_64) += tlb_low_64e.o book3e_pgtable.o	4	obj-$(CONFIG_PPC_BOOK3E_64) += tlb_64e.o tlb_low_64e.o book3e_pgtable.o
5	obj-$(CONFIG_44x) += 44x.o	5	obj-$(CONFIG_44x) += 44x.o
6	obj-$(CONFIG_PPC_8xx) += 8xx.o	6	obj-$(CONFIG_PPC_8xx) += 8xx.o
7	obj-$(CONFIG_PPC_E500) += e500.o	7	obj-$(CONFIG_PPC_E500) += e500.o

arch/powerpc/mm/nohash/book3e_pgtable.c+2−2

@@ -29,10 +29,10 @@ int __meminit vmemmap_create_mapping(unsigned long start, · 28 unmodified lines
29	_PAGE_KERNEL_RW;	29	_PAGE_KERNEL_RW;
30		30
31	/* PTEs only contain page size encodings up to 32M */	31	/* PTEs only contain page size encodings up to 32M */
32	BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].enc > 0xf);	32	BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].shift - 10 > 0xf);
33		33
34	/* Encode the size in the PTE */	34	/* Encode the size in the PTE */
35	flags \|= mmu_psize_defs[mmu_vmemmap_psize].enc << 8;	35	flags \|= (mmu_psize_defs[mmu_vmemmap_psize].shift - 10) << 8;
36		36
37	/* For each PTE for that area, map things. Note that we don't	37	/* For each PTE for that area, map things. Note that we don't
38	* increment phys because all PTEs are of the large size and	38	* increment phys because all PTEs are of the large size and

arch/powerpc/mm/nohash/tlb.c+2−405

@@ -53,37 +53,30 @@ · 52 unmodified lines
53	struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = {	53	struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = {
54	[MMU_PAGE_4K] = {	54	[MMU_PAGE_4K] = {
55	.shift = 12,	55	.shift = 12,
56	.enc = BOOK3E_PAGESZ_4K,
57	},	56	},
58	[MMU_PAGE_2M] = {	57	[MMU_PAGE_2M] = {
59	.shift = 21,	58	.shift = 21,
60	.enc = BOOK3E_PAGESZ_2M,
61	},	59	},
62	[MMU_PAGE_4M] = {	60	[MMU_PAGE_4M] = {
63	.shift = 22,	61	.shift = 22,
64	.enc = BOOK3E_PAGESZ_4M,
65	},	62	},
66	[MMU_PAGE_16M] = {	63	[MMU_PAGE_16M] = {
67	.shift = 24,	64	.shift = 24,
68	.enc = BOOK3E_PAGESZ_16M,
69	},	65	},
70	[MMU_PAGE_64M] = {	66	[MMU_PAGE_64M] = {
71	.shift = 26,	67	.shift = 26,
72	.enc = BOOK3E_PAGESZ_64M,
73	},	68	},
74	[MMU_PAGE_256M] = {	69	[MMU_PAGE_256M] = {
75	.shift = 28,	70	.shift = 28,
76	.enc = BOOK3E_PAGESZ_256M,
77	},	71	},
78	[MMU_PAGE_1G] = {	72	[MMU_PAGE_1G] = {
79	.shift = 30,	73	.shift = 30,
80	.enc = BOOK3E_PAGESZ_1GB,
81	},	74	},
82	};	75	};
83		76
84	static inline int mmu_get_tsize(int psize)	77	static inline int mmu_get_tsize(int psize)
85	{	78	{
86	return mmu_psize_defs[psize].enc;	79	return mmu_psize_defs[psize].shift - 10;
87	}	80	}
88	#else	81	#else
89	static inline int mmu_get_tsize(int psize)	82	static inline int mmu_get_tsize(int psize)
@@ -110,28 +103,6 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { · 20 unmodified lines
110	};	103	};
111	#endif	104	#endif
112		105
113	/* The variables below are currently only used on 64-bit Book3E
114	* though this will probably be made common with other nohash
115	* implementations at some point
116	*/
117	#ifdef CONFIG_PPC64
118
119	int mmu_pte_psize; /* Page size used for PTE pages */
120	int mmu_vmemmap_psize; /* Page size used for the virtual mem map */
121	int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */
122	unsigned long linear_map_top; /* Top of linear mapping */
123
124
125	/*
126	* Number of bytes to add to SPRN_SPRG_TLB_EXFRAME on crit/mcheck/debug
127	* exceptions. This is used for bolted and e6500 TLB miss handlers which
128	* do not modify this SPRG in the TLB miss code; for other TLB miss handlers,
129	* this is set to zero.
130	*/
131	int extlb_level_exc;
132
133	#endif /* CONFIG_PPC64 */
134
135	#ifdef CONFIG_PPC_E500	106	#ifdef CONFIG_PPC_E500
136	/* next_tlbcam_idx is used to round-robin tlbcam entry assignment */	107	/* next_tlbcam_idx is used to round-robin tlbcam entry assignment */
137	DEFINE_PER_CPU(int, next_tlbcam_idx);	108	DEFINE_PER_CPU(int, next_tlbcam_idx);
@@ -358,381 +329,7 @@ void tlb_flush(struct mmu_gather *tlb) · 220 unmodified lines
358	flush_tlb_mm(tlb->mm);	329	flush_tlb_mm(tlb->mm);
359	}	330	}
360		331
361	/*	332	#ifndef CONFIG_PPC64
362	* Below are functions specific to the 64-bit variant of Book3E though that
363	* may change in the future
364	*/
365
366	#ifdef CONFIG_PPC64
367
368	/*
369	* Handling of virtual linear page tables or indirect TLB entries
370	* flushing when PTE pages are freed
371	*/
372	void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address)
373	{
374	int tsize = mmu_psize_defs[mmu_pte_psize].enc;
375
376	if (book3e_htw_mode != PPC_HTW_NONE) {
377	unsigned long start = address & PMD_MASK;
378	unsigned long end = address + PMD_SIZE;
379	unsigned long size = 1UL << mmu_psize_defs[mmu_pte_psize].shift;
380
381	/* This isn't the most optimal, ideally we would factor out the
382	* while preempt & CPU mask mucking around, or even the IPI but
383	* it will do for now
384	*/
385	while (start < end) {
386	__flush_tlb_page(tlb->mm, start, tsize, 1);
387	start += size;
388	}
389	} else {
390	unsigned long rmask = 0xf000000000000000ul;
391	unsigned long rid = (address & rmask) \| 0x1000000000000000ul;
392	unsigned long vpte = address & ~rmask;
393
394	vpte = (vpte >> (PAGE_SHIFT - 3)) & ~0xffful;
395	vpte \|= rid;
396	__flush_tlb_page(tlb->mm, vpte, tsize, 0);
397	}
398	}
399
400	static void __init setup_page_sizes(void)
401	{
402	unsigned int tlb0cfg;
403	unsigned int tlb0ps;
404	unsigned int eptcfg;
405	int i, psize;
406
407	#ifdef CONFIG_PPC_E500
408	unsigned int mmucfg = mfspr(SPRN_MMUCFG);
409	int fsl_mmu = mmu_has_feature(MMU_FTR_TYPE_FSL_E);
410
411	if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) {
412	unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG);
413	unsigned int min_pg, max_pg;
414
415	min_pg = (tlb1cfg & TLBnCFG_MINSIZE) >> TLBnCFG_MINSIZE_SHIFT;
416	max_pg = (tlb1cfg & TLBnCFG_MAXSIZE) >> TLBnCFG_MAXSIZE_SHIFT;
417
418	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
419	struct mmu_psize_def *def;
420	unsigned int shift;
421
422	def = &mmu_psize_defs[psize];
423	shift = def->shift;
424
425	if (shift == 0 \|\| shift & 1)
426	continue;
427
428	/* adjust to be in terms of 4^shift Kb */
429	shift = (shift - 10) >> 1;
430
431	if ((shift >= min_pg) && (shift <= max_pg))
432	def->flags \|= MMU_PAGE_SIZE_DIRECT;
433	}
434
435	goto out;
436	}
437
438	if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) {
439	u32 tlb1cfg, tlb1ps;
440
441	tlb0cfg = mfspr(SPRN_TLB0CFG);
442	tlb1cfg = mfspr(SPRN_TLB1CFG);
443	tlb1ps = mfspr(SPRN_TLB1PS);
444	eptcfg = mfspr(SPRN_EPTCFG);
445
446	if ((tlb1cfg & TLBnCFG_IND) && (tlb0cfg & TLBnCFG_PT))
447	book3e_htw_mode = PPC_HTW_E6500;
448
449	/*
450	* We expect 4K subpage size and unrestricted indirect size.
451	* The lack of a restriction on indirect size is a Freescale
452	* extension, indicated by PSn = 0 but SPSn != 0.
453	*/
454	if (eptcfg != 2)
455	book3e_htw_mode = PPC_HTW_NONE;
456
457	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
458	struct mmu_psize_def *def = &mmu_psize_defs[psize];
459
460	if (!def->shift)
461	continue;
462
463	if (tlb1ps & (1U << (def->shift - 10))) {
464	def->flags \|= MMU_PAGE_SIZE_DIRECT;
465
466	if (book3e_htw_mode && psize == MMU_PAGE_2M)
467	def->flags \|= MMU_PAGE_SIZE_INDIRECT;
468	}
469	}
470
471	goto out;
472	}
473	#endif
474
475	tlb0cfg = mfspr(SPRN_TLB0CFG);
476	tlb0ps = mfspr(SPRN_TLB0PS);
477	eptcfg = mfspr(SPRN_EPTCFG);
478
479	/* Look for supported direct sizes */
480	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
481	struct mmu_psize_def *def = &mmu_psize_defs[psize];
482
483	if (tlb0ps & (1U << (def->shift - 10)))
484	def->flags \|= MMU_PAGE_SIZE_DIRECT;
485	}
486
487	/* Indirect page sizes supported ? */
488	if ((tlb0cfg & TLBnCFG_IND) == 0 \|\|
489	(tlb0cfg & TLBnCFG_PT) == 0)
490	goto out;
491
492	book3e_htw_mode = PPC_HTW_IBM;
493
494	/* Now, we only deal with one IND page size for each
495	* direct size. Hopefully all implementations today are
496	* unambiguous, but we might want to be careful in the
497	* future.
498	*/
499	for (i = 0; i < 3; i++) {
500	unsigned int ps, sps;
501
502	sps = eptcfg & 0x1f;
503	eptcfg >>= 5;
504	ps = eptcfg & 0x1f;
505	eptcfg >>= 5;
506	if (!ps \|\| !sps)
507	continue;
508	for (psize = 0; psize < MMU_PAGE_COUNT; psize++) {
509	struct mmu_psize_def *def = &mmu_psize_defs[psize];
510
511	if (ps == (def->shift - 10))
512	def->flags \|= MMU_PAGE_SIZE_INDIRECT;
513	if (sps == (def->shift - 10))
514	def->ind = ps + 10;
515	}
516	}
517
518	out:
519	/* Cleanup array and print summary */
520	pr_info("MMU: Supported page sizes\n");
521	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
522	struct mmu_psize_def *def = &mmu_psize_defs[psize];
523	const char *__page_type_names[] = {
524	"unsupported",
525	"direct",
526	"indirect",
527	"direct & indirect"
528	};
529	if (def->flags == 0) {
530	def->shift = 0;
531	continue;
532	}
533	pr_info(" %8ld KB as %s\n", 1ul << (def->shift - 10),
534	__page_type_names[def->flags & 0x3]);
535	}
536	}
537
538	static void __init setup_mmu_htw(void)
539	{
540	/*
541	* If we want to use HW tablewalk, enable it by patching the TLB miss
542	* handlers to branch to the one dedicated to it.
543	*/
544
545	switch (book3e_htw_mode) {
546	case PPC_HTW_IBM:
547	patch_exception(0x1c0, exc_data_tlb_miss_htw_book3e);
548	patch_exception(0x1e0, exc_instruction_tlb_miss_htw_book3e);
549	break;
550	#ifdef CONFIG_PPC_E500
551	case PPC_HTW_E6500:
552	extlb_level_exc = EX_TLB_SIZE;
553	patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e);
554	patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e);
555	break;
556	#endif
557	}
558	pr_info("MMU: Book3E HW tablewalk %s\n",
559	book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported");
560	}
561
562	/*
563	* Early initialization of the MMU TLB code
564	*/
565	static void early_init_this_mmu(void)
566	{
567	unsigned int mas4;
568
569	/* Set MAS4 based on page table setting */
570
571	mas4 = 0x4 << MAS4_WIMGED_SHIFT;
572	switch (book3e_htw_mode) {
573	case PPC_HTW_E6500:
574	mas4 \|= MAS4_INDD;
575	mas4 \|= BOOK3E_PAGESZ_2M << MAS4_TSIZED_SHIFT;
576	mas4 \|= MAS4_TLBSELD(1);
577	mmu_pte_psize = MMU_PAGE_2M;
578	break;
579
580	case PPC_HTW_IBM:
581	mas4 \|= MAS4_INDD;
582	mas4 \|= BOOK3E_PAGESZ_1M << MAS4_TSIZED_SHIFT;
583	mmu_pte_psize = MMU_PAGE_1M;
584	break;
585
586	case PPC_HTW_NONE:
587	mas4 \|= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT;
588	mmu_pte_psize = mmu_virtual_psize;
589	break;
590	}
591	mtspr(SPRN_MAS4, mas4);
592
593	#ifdef CONFIG_PPC_E500
594	if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) {
595	unsigned int num_cams;
596	bool map = true;
597
598	/* use a quarter of the TLBCAM for bolted linear map */
599	num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4;
600
601	/*
602	* Only do the mapping once per core, or else the
603	* transient mapping would cause problems.
604	*/
605	#ifdef CONFIG_SMP
606	if (hweight32(get_tensr()) > 1)
607	map = false;
608	#endif
609
610	if (map)
611	linear_map_top = map_mem_in_cams(linear_map_top,
612	num_cams, false, true);
613	}
614	#endif
615
616	/* A sync won't hurt us after mucking around with
617	* the MMU configuration
618	*/
619	mb();
620	}
621
622	static void __init early_init_mmu_global(void)
623	{
624	/* XXX This should be decided at runtime based on supported
625	* page sizes in the TLB, but for now let's assume 16M is
626	* always there and a good fit (which it probably is)
627	*
628	* Freescale booke only supports 4K pages in TLB0, so use that.
629	*/
630	if (mmu_has_feature(MMU_FTR_TYPE_FSL_E))
631	mmu_vmemmap_psize = MMU_PAGE_4K;
632	else
633	mmu_vmemmap_psize = MMU_PAGE_16M;
634
635	/* XXX This code only checks for TLB 0 capabilities and doesn't
636	* check what page size combos are supported by the HW. It
637	* also doesn't handle the case where a separate array holds
638	* the IND entries from the array loaded by the PT.
639	*/
640	/* Look for supported page sizes */
641	setup_page_sizes();
642
643	/* Look for HW tablewalk support */
644	setup_mmu_htw();
645
646	#ifdef CONFIG_PPC_E500
647	if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) {
648	if (book3e_htw_mode == PPC_HTW_NONE) {
649	extlb_level_exc = EX_TLB_SIZE;
650	patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e);
651	patch_exception(0x1e0,
652	exc_instruction_tlb_miss_bolted_book3e);
653	}
654	}
655	#endif
656
657	/* Set the global containing the top of the linear mapping
658	* for use by the TLB miss code
659	*/
660	linear_map_top = memblock_end_of_DRAM();
661
662	ioremap_bot = IOREMAP_BASE;
663	}
664
665	static void __init early_mmu_set_memory_limit(void)
666	{
667	#ifdef CONFIG_PPC_E500
668	if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) {
669	/*
670	* Limit memory so we dont have linear faults.
671	* Unlike memblock_set_current_limit, which limits
672	* memory available during early boot, this permanently
673	* reduces the memory available to Linux. We need to
674	* do this because highmem is not supported on 64-bit.
675	*/
676	memblock_enforce_memory_limit(linear_map_top);
677	}
678	#endif
679
680	memblock_set_current_limit(linear_map_top);
681	}
682
683	/* boot cpu only */
684	void __init early_init_mmu(void)
685	{
686	early_init_mmu_global();
687	early_init_this_mmu();
688	early_mmu_set_memory_limit();
689	}
690
691	void early_init_mmu_secondary(void)
692	{
693	early_init_this_mmu();
694	}
695
696	void setup_initial_memory_limit(phys_addr_t first_memblock_base,
697	phys_addr_t first_memblock_size)
698	{
699	/* On non-FSL Embedded 64-bit, we adjust the RMA size to match
700	* the bolted TLB entry. We know for now that only 1G
701	* entries are supported though that may eventually
702	* change.
703	*
704	* on FSL Embedded 64-bit, usually all RAM is bolted, but with
705	* unusual memory sizes it's possible for some RAM to not be mapped
706	* (such RAM is not used at all by Linux, since we don't support
707	* highmem on 64-bit). We limit ppc64_rma_size to what would be
708	* mappable if this memblock is the only one. Additional memblocks
709	* can only increase, not decrease, the amount that ends up getting
710	* mapped. We still limit max to 1G even if we'll eventually map
711	* more. This is due to what the early init code is set up to do.
712	*
713	* We crop it to the size of the first MEMBLOCK to
714	* avoid going over total available memory just in case...
715	*/
716	#ifdef CONFIG_PPC_E500
717	if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) {
718	unsigned long linear_sz;
719	unsigned int num_cams;
720
721	/* use a quarter of the TLBCAM for bolted linear map */
722	num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4;
723
724	linear_sz = map_mem_in_cams(first_memblock_size, num_cams,
725	true, true);
726
727	ppc64_rma_size = min_t(u64, linear_sz, 0x40000000);
728	} else
729	#endif
730	ppc64_rma_size = min_t(u64, first_memblock_size, 0x40000000);
731
732	/* Finally limit subsequent allocations */
733	memblock_set_current_limit(first_memblock_base + ppc64_rma_size);
734	}
735	#else /* ! CONFIG_PPC64 */
736	void __init early_init_mmu(void)	333	void __init early_init_mmu(void)
737	{	334	{
738	unsigned long root = of_get_flat_dt_root();	335	unsigned long root = of_get_flat_dt_root();

arch/powerpc/mm/nohash/tlb_64e.c+314−0

@@ -0,0 +1,314 @@
	1	// SPDX-License-Identifier: GPL-2.0-or-later
	2	/*
	3	* Copyright 2008,2009 Ben Herrenschmidt <benh@kernel.crashing.org>
	4	* IBM Corp.
	5	*
	6	* Derived from arch/ppc/mm/init.c:
	7	* Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org)
	8	*
	9	* Modifications by Paul Mackerras (PowerMac) (paulus@cs.anu.edu.au)
	10	* and Cort Dougan (PReP) (cort@cs.nmt.edu)
	11	* Copyright (C) 1996 Paul Mackerras
	12	*
	13	* Derived from "arch/i386/mm/init.c"
	14	* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
	15	*/
	16
	17	#include <linux/kernel.h>
	18	#include <linux/export.h>
	19	#include <linux/mm.h>
	20	#include <linux/init.h>
	21	#include <linux/pagemap.h>
	22	#include <linux/memblock.h>
	23
	24	#include <asm/pgalloc.h>
	25	#include <asm/tlbflush.h>
	26	#include <asm/tlb.h>
	27	#include <asm/code-patching.h>
	28	#include <asm/cputhreads.h>
	29
	30	#include <mm/mmu_decl.h>
	31
	32	/* The variables below are currently only used on 64-bit Book3E
	33	* though this will probably be made common with other nohash
	34	* implementations at some point
	35	*/
	36	int mmu_pte_psize; /* Page size used for PTE pages */
	37	int mmu_vmemmap_psize; /* Page size used for the virtual mem map */
	38	int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */
	39	unsigned long linear_map_top; /* Top of linear mapping */
	40
	41
	42	/*
	43	* Number of bytes to add to SPRN_SPRG_TLB_EXFRAME on crit/mcheck/debug
	44	* exceptions. This is used for bolted and e6500 TLB miss handlers which
	45	* do not modify this SPRG in the TLB miss code; for other TLB miss handlers,
	46	* this is set to zero.
	47	*/
	48	int extlb_level_exc;
	49
	50	/*
	51	* Handling of virtual linear page tables or indirect TLB entries
	52	* flushing when PTE pages are freed
	53	*/
	54	void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address)
	55	{
	56	int tsize = mmu_psize_defs[mmu_pte_psize].shift - 10;
	57
	58	if (book3e_htw_mode != PPC_HTW_NONE) {
	59	unsigned long start = address & PMD_MASK;
	60	unsigned long end = address + PMD_SIZE;
	61	unsigned long size = 1UL << mmu_psize_defs[mmu_pte_psize].shift;
	62
	63	/* This isn't the most optimal, ideally we would factor out the
	64	* while preempt & CPU mask mucking around, or even the IPI but
	65	* it will do for now
	66	*/
	67	while (start < end) {
	68	__flush_tlb_page(tlb->mm, start, tsize, 1);
	69	start += size;
	70	}
	71	} else {
	72	unsigned long rmask = 0xf000000000000000ul;
	73	unsigned long rid = (address & rmask) \| 0x1000000000000000ul;
	74	unsigned long vpte = address & ~rmask;
	75
	76	vpte = (vpte >> (PAGE_SHIFT - 3)) & ~0xffful;
	77	vpte \|= rid;
	78	__flush_tlb_page(tlb->mm, vpte, tsize, 0);
	79	}
	80	}
	81
	82	static void __init setup_page_sizes(void)
	83	{
	84	unsigned int tlb0cfg;
	85	unsigned int eptcfg;
	86	int psize;
	87
	88	unsigned int mmucfg = mfspr(SPRN_MMUCFG);
	89
	90	if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) {
	91	unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG);
	92	unsigned int min_pg, max_pg;
	93
	94	min_pg = (tlb1cfg & TLBnCFG_MINSIZE) >> TLBnCFG_MINSIZE_SHIFT;
	95	max_pg = (tlb1cfg & TLBnCFG_MAXSIZE) >> TLBnCFG_MAXSIZE_SHIFT;
	96
	97	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
	98	struct mmu_psize_def *def;
	99	unsigned int shift;
	100
	101	def = &mmu_psize_defs[psize];
	102	shift = def->shift;
	103
	104	if (shift == 0 \|\| shift & 1)
	105	continue;
	106
	107	/* adjust to be in terms of 4^shift Kb */
	108	shift = (shift - 10) >> 1;
	109
	110	if ((shift >= min_pg) && (shift <= max_pg))
	111	def->flags \|= MMU_PAGE_SIZE_DIRECT;
	112	}
	113
	114	goto out;
	115	}
	116
	117	if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) {
	118	u32 tlb1cfg, tlb1ps;
	119
	120	tlb0cfg = mfspr(SPRN_TLB0CFG);
	121	tlb1cfg = mfspr(SPRN_TLB1CFG);
	122	tlb1ps = mfspr(SPRN_TLB1PS);
	123	eptcfg = mfspr(SPRN_EPTCFG);
	124
	125	if ((tlb1cfg & TLBnCFG_IND) && (tlb0cfg & TLBnCFG_PT))
	126	book3e_htw_mode = PPC_HTW_E6500;
	127
	128	/*
	129	* We expect 4K subpage size and unrestricted indirect size.
	130	* The lack of a restriction on indirect size is a Freescale
	131	* extension, indicated by PSn = 0 but SPSn != 0.
	132	*/
	133	if (eptcfg != 2)
	134	book3e_htw_mode = PPC_HTW_NONE;
	135
	136	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
	137	struct mmu_psize_def *def = &mmu_psize_defs[psize];
	138
	139	if (!def->shift)
	140	continue;
	141
	142	if (tlb1ps & (1U << (def->shift - 10))) {
	143	def->flags \|= MMU_PAGE_SIZE_DIRECT;
	144
	145	if (book3e_htw_mode && psize == MMU_PAGE_2M)
	146	def->flags \|= MMU_PAGE_SIZE_INDIRECT;
	147	}
	148	}
	149
	150	goto out;
	151	}
	152	out:
	153	/* Cleanup array and print summary */
	154	pr_info("MMU: Supported page sizes\n");
	155	for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
	156	struct mmu_psize_def *def = &mmu_psize_defs[psize];
	157	const char *__page_type_names[] = {
	158	"unsupported",
	159	"direct",
	160	"indirect",
	161	"direct & indirect"
	162	};
	163	if (def->flags == 0) {
	164	def->shift = 0;
	165	continue;
	166	}
	167	pr_info(" %8ld KB as %s\n", 1ul << (def->shift - 10),
	168	__page_type_names[def->flags & 0x3]);
	169	}
	170	}
	171
	172	/*
	173	* Early initialization of the MMU TLB code
	174	*/
	175	static void early_init_this_mmu(void)
	176	{
	177	unsigned int mas4;
	178
	179	/* Set MAS4 based on page table setting */
	180
	181	mas4 = 0x4 << MAS4_WIMGED_SHIFT;
	182	switch (book3e_htw_mode) {
	183	case PPC_HTW_E6500:
	184	mas4 \|= MAS4_INDD;
	185	mas4 \|= BOOK3E_PAGESZ_2M << MAS4_TSIZED_SHIFT;
	186	mas4 \|= MAS4_TLBSELD(1);
	187	mmu_pte_psize = MMU_PAGE_2M;
	188	break;
	189
	190	case PPC_HTW_NONE:
	191	mas4 \|= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT;
	192	mmu_pte_psize = mmu_virtual_psize;
	193	break;
	194	}
	195	mtspr(SPRN_MAS4, mas4);
	196
	197	unsigned int num_cams;
	198	bool map = true;
	199
	200	/* use a quarter of the TLBCAM for bolted linear map */
	201	num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4;
	202
	203	/*
	204	* Only do the mapping once per core, or else the
	205	* transient mapping would cause problems.
	206	*/
	207	#ifdef CONFIG_SMP
	208	if (hweight32(get_tensr()) > 1)
	209	map = false;
	210	#endif
	211
	212	if (map)
	213	linear_map_top = map_mem_in_cams(linear_map_top,
	214	num_cams, false, true);
	215
	216	/* A sync won't hurt us after mucking around with
	217	* the MMU configuration
	218	*/
	219	mb();
	220	}
	221
	222	static void __init early_init_mmu_global(void)
	223	{
	224	/*
	225	* Freescale booke only supports 4K pages in TLB0, so use that.
	226	*/
	227	mmu_vmemmap_psize = MMU_PAGE_4K;
	228
	229	/* XXX This code only checks for TLB 0 capabilities and doesn't
	230	* check what page size combos are supported by the HW. It
	231	* also doesn't handle the case where a separate array holds
	232	* the IND entries from the array loaded by the PT.
	233	*/
	234	/* Look for supported page sizes */
	235	setup_page_sizes();
	236
	237	/*
	238	* If we want to use HW tablewalk, enable it by patching the TLB miss
	239	* handlers to branch to the one dedicated to it.
	240	*/
	241	extlb_level_exc = EX_TLB_SIZE;
	242	switch (book3e_htw_mode) {
	243	case PPC_HTW_E6500:
	244	patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e);
	245	patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e);
	246	break;
	247	}
	248
	249	pr_info("MMU: Book3E HW tablewalk %s\n",
	250	book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported");
	251
	252	/* Set the global containing the top of the linear mapping
	253	* for use by the TLB miss code
	254	*/
	255	linear_map_top = memblock_end_of_DRAM();
	256
	257	ioremap_bot = IOREMAP_BASE;
	258	}
	259
	260	static void __init early_mmu_set_memory_limit(void)
	261	{
	262	/*
	263	* Limit memory so we dont have linear faults.
	264	* Unlike memblock_set_current_limit, which limits
	265	* memory available during early boot, this permanently
	266	* reduces the memory available to Linux. We need to
	267	* do this because highmem is not supported on 64-bit.
	268	*/
	269	memblock_enforce_memory_limit(linear_map_top);
	270
	271	memblock_set_current_limit(linear_map_top);
	272	}
	273
	274	/* boot cpu only */
	275	void __init early_init_mmu(void)
	276	{
	277	early_init_mmu_global();
	278	early_init_this_mmu();
	279	early_mmu_set_memory_limit();
	280	}
	281
	282	void early_init_mmu_secondary(void)
	283	{
	284	early_init_this_mmu();
	285	}
	286
	287	void setup_initial_memory_limit(phys_addr_t first_memblock_base,
	288	phys_addr_t first_memblock_size)
	289	{
	290	/*
	291	* On FSL Embedded 64-bit, usually all RAM is bolted, but with
	292	* unusual memory sizes it's possible for some RAM to not be mapped
	293	* (such RAM is not used at all by Linux, since we don't support
	294	* highmem on 64-bit). We limit ppc64_rma_size to what would be
	295	* mappable if this memblock is the only one. Additional memblocks
	296	* can only increase, not decrease, the amount that ends up getting
	297	* mapped. We still limit max to 1G even if we'll eventually map
	298	* more. This is due to what the early init code is set up to do.
	299	*
	300	* We crop it to the size of the first MEMBLOCK to
	301	* avoid going over total available memory just in case...
	302	*/
	303	unsigned long linear_sz;
	304	unsigned int num_cams;
	305
	306	/* use a quarter of the TLBCAM for bolted linear map */
	307	num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4;
	308
	309	linear_sz = map_mem_in_cams(first_memblock_size, num_cams, true, true);
	310	ppc64_rma_size = min_t(u64, linear_sz, 0x40000000);
	311
	312	/* Finally limit subsequent allocations */
	313	memblock_set_current_limit(first_memblock_base + ppc64_rma_size);
	314	}

arch/powerpc/mm/nohash/tlb_low_64e.S+1−427

@@ -450,11 +450,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_SMT) · 449 unmodified lines
450		450
451	tlb_miss_huge_e6500:	451	tlb_miss_huge_e6500:
452	beq tlb_miss_fault_e6500	452	beq tlb_miss_fault_e6500
453	li r10,1	453	rlwinm r15,r14,32-_PAGE_PSIZE_SHIFT,0x1e
454	andi. r15,r14,HUGEPD_SHIFT_MASK@l /* r15 = psize */
455	rldimi r14,r10,63,0 /* Set PD_HUGE */
456	xor r14,r14,r15 /* Clear size bits */
457	ldx r14,0,r14
458		454
459	/*	455	/*
460	* Now we build the MAS for a huge page.	456	* Now we build the MAS for a huge page.
@@ -465,7 +461,6 @@ tlb_miss_huge_e6500: · 4 unmodified lines
465	* MAS 2,3+7: Needs to be redone similar to non-tablewalk handler	461	* MAS 2,3+7: Needs to be redone similar to non-tablewalk handler
466	*/	462	*/
467		463
468	subi r15,r15,10 /* Convert psize to tsize */
469	mfspr r10,SPRN_MAS1	464	mfspr r10,SPRN_MAS1
470	rlwinm r10,r10,0,~MAS1_IND	465	rlwinm r10,r10,0,~MAS1_IND
471	rlwimi r10,r15,MAS1_TSIZE_SHIFT,MAS1_TSIZE_MASK	466	rlwimi r10,r15,MAS1_TSIZE_SHIFT,MAS1_TSIZE_MASK
@@ -511,232 +506,6 @@ itlb_miss_fault_e6500: · 39 unmodified lines
511	tlb_epilog_bolted	506	tlb_epilog_bolted
512	b exc_instruction_storage_book3e	507	b exc_instruction_storage_book3e
513		508
514	/**********************************************************************
515	* *
516	* TLB miss handling for Book3E with TLB reservation and HES support *
517	* *
518	**********************************************************************/
519
520
521	/* Data TLB miss */
522	START_EXCEPTION(data_tlb_miss)
523	TLB_MISS_PROLOG
524
525	/* Now we handle the fault proper. We only save DEAR in normal
526	* fault case since that's the only interesting values here.
527	* We could probably also optimize by not saving SRR0/1 in the
528	* linear mapping case but I'll leave that for later
529	*/
530	mfspr r14,SPRN_ESR
531	mfspr r16,SPRN_DEAR /* get faulting address */
532	srdi r15,r16,44 /* get region */
533	xoris r15,r15,0xc
534	cmpldi cr0,r15,0 /* linear mapping ? */
535	beq tlb_load_linear /* yes -> go to linear map load */
536	cmpldi cr1,r15,1 /* vmalloc mapping ? */
537
538	/* The page tables are mapped virtually linear. At this point, though,
539	* we don't know whether we are trying to fault in a first level
540	* virtual address or a virtual page table address. We can get that
541	* from bit 0x1 of the region ID which we have set for a page table
542	*/
543	andis. r10,r15,0x1
544	bne- virt_page_table_tlb_miss
545
546	std r14,EX_TLB_ESR(r12); /* save ESR */
547	std r16,EX_TLB_DEAR(r12); /* save DEAR */
548
549	/* We need _PAGE_PRESENT and _PAGE_ACCESSED set */
550	li r11,_PAGE_PRESENT
551	oris r11,r11,_PAGE_ACCESSED@h
552
553	/* We do the user/kernel test for the PID here along with the RW test
554	*/
555	srdi. r15,r16,60 /* Check for user region */
556
557	/* We pre-test some combination of permissions to avoid double
558	* faults:
559	*
560	* We move the ESR:ST bit into the position of _PAGE_BAP_SW in the PTE
561	* ESR_ST is 0x00800000
562	* _PAGE_BAP_SW is 0x00000010
563	* So the shift is >> 19. This tests for supervisor writeability.
564	* If the page happens to be supervisor writeable and not user
565	* writeable, we will take a new fault later, but that should be
566	* a rare enough case.
567	*
568	* We also move ESR_ST in _PAGE_DIRTY position
569	* _PAGE_DIRTY is 0x00001000 so the shift is >> 11
570	*
571	* MAS1 is preset for all we need except for TID that needs to
572	* be cleared for kernel translations
573	*/
574	rlwimi r11,r14,32-19,27,27
575	rlwimi r11,r14,32-16,19,19
576	beq normal_tlb_miss_user
577	/* XXX replace the RMW cycles with immediate loads + writes */
578	1: mfspr r10,SPRN_MAS1
579	rlwinm r10,r10,0,16,1 /* Clear TID */
580	mtspr SPRN_MAS1,r10
581	beq+ cr1,normal_tlb_miss
582
583	/* We got a crappy address, just fault with whatever DEAR and ESR
584	* are here
585	*/
586	TLB_MISS_EPILOG_ERROR
587	b exc_data_storage_book3e
588
589	/* Instruction TLB miss */
590	START_EXCEPTION(instruction_tlb_miss)
591	TLB_MISS_PROLOG
592
593	/* If we take a recursive fault, the second level handler may need
594	* to know whether we are handling a data or instruction fault in
595	* order to get to the right store fault handler. We provide that
596	* info by writing a crazy value in ESR in our exception frame
597	*/
598	li r14,-1 /* store to exception frame is done later */
599
600	/* Now we handle the fault proper. We only save DEAR in the non
601	* linear mapping case since we know the linear mapping case will
602	* not re-enter. We could indeed optimize and also not save SRR0/1
603	* in the linear mapping case but I'll leave that for later
604	*
605	* Faulting address is SRR0 which is already in r16
606	*/
607	srdi r15,r16,44 /* get region */
608	xoris r15,r15,0xc
609	cmpldi cr0,r15,0 /* linear mapping ? */
610	beq tlb_load_linear /* yes -> go to linear map load */
611	cmpldi cr1,r15,1 /* vmalloc mapping ? */
612
613	/* We do the user/kernel test for the PID here along with the RW test
614	*/
615	li r11,_PAGE_PRESENT\|_PAGE_BAP_UX /* Base perm */
616	oris r11,r11,_PAGE_ACCESSED@h
617
618	srdi. r15,r16,60 /* Check for user region */
619	std r14,EX_TLB_ESR(r12) /* write crazy -1 to frame */
620	beq normal_tlb_miss_user
621
622	li r11,_PAGE_PRESENT\|_PAGE_BAP_SX /* Base perm */
623	oris r11,r11,_PAGE_ACCESSED@h
624	/* XXX replace the RMW cycles with immediate loads + writes */
625	mfspr r10,SPRN_MAS1
626	rlwinm r10,r10,0,16,1 /* Clear TID */
627	mtspr SPRN_MAS1,r10
628	beq+ cr1,normal_tlb_miss
629
630	/* We got a crappy address, just fault */
631	TLB_MISS_EPILOG_ERROR
632	b exc_instruction_storage_book3e
633
634	/*
635	* This is the guts of the first-level TLB miss handler for direct
636	* misses. We are entered with:
637	*
638	* r16 = faulting address
639	* r15 = region ID
640	* r14 = crap (free to use)
641	* r13 = PACA
642	* r12 = TLB exception frame in PACA
643	* r11 = PTE permission mask
644	* r10 = crap (free to use)
645	*/
646	normal_tlb_miss_user:
647	#ifdef CONFIG_PPC_KUAP
648	mfspr r14,SPRN_MAS1
649	rlwinm. r14,r14,0,0x3fff0000
650	beq- normal_tlb_miss_access_fault /* KUAP fault */
651	#endif
652	normal_tlb_miss:
653	/* So we first construct the page table address. We do that by
654	* shifting the bottom of the address (not the region ID) by
655	* PAGE_SHIFT-3, clearing the bottom 3 bits (get a PTE ptr) and
656	* or'ing the fourth high bit.
657	*
658	* NOTE: For 64K pages, we do things slightly differently in
659	* order to handle the weird page table format used by linux
660	*/
661	srdi r15,r16,44
662	oris r10,r15,0x1
663	rldicl r14,r16,64-(PAGE_SHIFT-3),PAGE_SHIFT-3+4
664	sldi r15,r10,44
665	clrrdi r14,r14,19
666	or r10,r15,r14
667
668	ld r14,0(r10)
669
670	finish_normal_tlb_miss:
671	/* Check if required permissions are met */
672	andc. r15,r11,r14
673	bne- normal_tlb_miss_access_fault
674
675	/* Now we build the MAS:
676	*
677	* MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG
678	* MAS 1 : Almost fully setup
679	* - PID already updated by caller if necessary
680	* - TSIZE need change if !base page size, not
681	* yet implemented for now
682	* MAS 2 : Defaults not useful, need to be redone
683	* MAS 3+7 : Needs to be done
684	*
685	* TODO: mix up code below for better scheduling
686	*/
687	clrrdi r10,r16,12 /* Clear low crap in EA */
688	rlwimi r10,r14,32-19,27,31 /* Insert WIMGE */
689	mtspr SPRN_MAS2,r10
690
691	/* Check page size, if not standard, update MAS1 */
692	rldicl r10,r14,64-8,64-8
693	cmpldi cr0,r10,BOOK3E_PAGESZ_4K
694	beq- 1f
695	mfspr r11,SPRN_MAS1
696	rlwimi r11,r14,31,21,24
697	rlwinm r11,r11,0,21,19
698	mtspr SPRN_MAS1,r11
699	1:
700	/* Move RPN in position */
701	rldicr r11,r14,64-(PTE_RPN_SHIFT-PAGE_SHIFT),63-PAGE_SHIFT
702	clrldi r15,r11,12 /* Clear crap at the top */
703	rlwimi r15,r14,32-8,22,25 /* Move in U bits */
704	rlwimi r15,r14,32-2,26,31 /* Move in BAP bits */
705
706	/* Mask out SW and UW if !DIRTY (XXX optimize this !) */
707	andi. r11,r14,_PAGE_DIRTY
708	bne 1f
709	li r11,MAS3_SW\|MAS3_UW
710	andc r15,r15,r11
711	1:
712	srdi r16,r15,32
713	mtspr SPRN_MAS3,r15
714	mtspr SPRN_MAS7,r16
715
716	tlbwe
717
718	normal_tlb_miss_done:
719	/* We don't bother with restoring DEAR or ESR since we know we are
720	* level 0 and just going back to userland. They are only needed
721	* if you are going to take an access fault
722	*/
723	TLB_MISS_EPILOG_SUCCESS
724	rfi
725
726	normal_tlb_miss_access_fault:
727	/* We need to check if it was an instruction miss */
728	andi. r10,r11,_PAGE_BAP_UX
729	bne 1f
730	ld r14,EX_TLB_DEAR(r12)
731	ld r15,EX_TLB_ESR(r12)
732	mtspr SPRN_DEAR,r14
733	mtspr SPRN_ESR,r15
734	TLB_MISS_EPILOG_ERROR
735	b exc_data_storage_book3e
736	1: TLB_MISS_EPILOG_ERROR
737	b exc_instruction_storage_book3e
738
739
740	/*	509	/*
741	* This is the guts of the second-level TLB miss handler for direct	510	* This is the guts of the second-level TLB miss handler for direct
742	* misses. We are entered with:	511	* misses. We are entered with:
@@ -893,201 +662,6 @@ virt_page_table_tlb_miss_whacko_fault: · 150 unmodified lines
893	TLB_MISS_EPILOG_ERROR	662	TLB_MISS_EPILOG_ERROR
894	b exc_data_storage_book3e	663	b exc_data_storage_book3e
895		664
896
897	/**************************************************************
898	* *
899	* TLB miss handling for Book3E with hw page table support *
900	* *
901	**************************************************************/
902
903
904	/* Data TLB miss */
905	START_EXCEPTION(data_tlb_miss_htw)
906	TLB_MISS_PROLOG
907
908	/* Now we handle the fault proper. We only save DEAR in normal
909	* fault case since that's the only interesting values here.
910	* We could probably also optimize by not saving SRR0/1 in the
911	* linear mapping case but I'll leave that for later
912	*/
913	mfspr r14,SPRN_ESR
914	mfspr r16,SPRN_DEAR /* get faulting address */
915	srdi r11,r16,44 /* get region */
916	xoris r11,r11,0xc
917	cmpldi cr0,r11,0 /* linear mapping ? */
918	beq tlb_load_linear /* yes -> go to linear map load */
919	cmpldi cr1,r11,1 /* vmalloc mapping ? */
920
921	/* We do the user/kernel test for the PID here along with the RW test
922	*/
923	srdi. r11,r16,60 /* Check for user region */
924	ld r15,PACAPGD(r13) /* Load user pgdir */
925	beq htw_tlb_miss
926
927	/* XXX replace the RMW cycles with immediate loads + writes */
928	1: mfspr r10,SPRN_MAS1
929	rlwinm r10,r10,0,16,1 /* Clear TID */
930	mtspr SPRN_MAS1,r10
931	ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */
932	beq+ cr1,htw_tlb_miss
933
934	/* We got a crappy address, just fault with whatever DEAR and ESR
935	* are here
936	*/
937	TLB_MISS_EPILOG_ERROR
938	b exc_data_storage_book3e
939
940	/* Instruction TLB miss */
941	START_EXCEPTION(instruction_tlb_miss_htw)
942	TLB_MISS_PROLOG
943
944	/* If we take a recursive fault, the second level handler may need
945	* to know whether we are handling a data or instruction fault in
946	* order to get to the right store fault handler. We provide that
947	* info by keeping a crazy value for ESR in r14
948	*/
949	li r14,-1 /* store to exception frame is done later */
950
951	/* Now we handle the fault proper. We only save DEAR in the non
952	* linear mapping case since we know the linear mapping case will
953	* not re-enter. We could indeed optimize and also not save SRR0/1
954	* in the linear mapping case but I'll leave that for later
955	*
956	* Faulting address is SRR0 which is already in r16
957	*/
958	srdi r11,r16,44 /* get region */
959	xoris r11,r11,0xc
960	cmpldi cr0,r11,0 /* linear mapping ? */
961	beq tlb_load_linear /* yes -> go to linear map load */
962	cmpldi cr1,r11,1 /* vmalloc mapping ? */
963
964	/* We do the user/kernel test for the PID here along with the RW test
965	*/
966	srdi. r11,r16,60 /* Check for user region */
967	ld r15,PACAPGD(r13) /* Load user pgdir */
968	beq htw_tlb_miss
969
970	/* XXX replace the RMW cycles with immediate loads + writes */
971	1: mfspr r10,SPRN_MAS1
972	rlwinm r10,r10,0,16,1 /* Clear TID */
973	mtspr SPRN_MAS1,r10
974	ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */
975	beq+ htw_tlb_miss
976
977	/* We got a crappy address, just fault */
978	TLB_MISS_EPILOG_ERROR
979	b exc_instruction_storage_book3e
980
981
982	/*
983	* This is the guts of the second-level TLB miss handler for direct
984	* misses. We are entered with:
985	*
986	* r16 = virtual page table faulting address
987	* r15 = PGD pointer
988	* r14 = ESR
989	* r13 = PACA
990	* r12 = TLB exception frame in PACA
991	* r11 = crap (free to use)
992	* r10 = crap (free to use)
993	*
994	* It can be re-entered by the linear mapping miss handler. However, to
995	* avoid too much complication, it will save/restore things for us
996	*/
997	htw_tlb_miss:
998	#ifdef CONFIG_PPC_KUAP
999	mfspr r10,SPRN_MAS1
1000	rlwinm. r10,r10,0,0x3fff0000
1001	beq- htw_tlb_miss_fault /* KUAP fault */
1002	#endif
1003	/* Search if we already have a TLB entry for that virtual address, and
1004	* if we do, bail out.
1005	*
1006	* MAS1:IND should be already set based on MAS4
1007	*/
1008	PPC_TLBSRX_DOT(0,R16)
1009	beq htw_tlb_miss_done
1010
1011	/* Now, we need to walk the page tables. First check if we are in
1012	* range.
1013	*/
1014	rldicl. r10,r16,64-PGTABLE_EADDR_SIZE,PGTABLE_EADDR_SIZE+4
1015	bne- htw_tlb_miss_fault
1016
1017	/* Get the PGD pointer */
1018	cmpldi cr0,r15,0
1019	beq- htw_tlb_miss_fault
1020
1021	/* Get to PGD entry */
1022	rldicl r11,r16,64-(PGDIR_SHIFT-3),64-PGD_INDEX_SIZE-3
1023	clrrdi r10,r11,3
1024	ldx r15,r10,r15
1025	cmpdi cr0,r15,0
1026	bge htw_tlb_miss_fault
1027
1028	/* Get to PUD entry */
1029	rldicl r11,r16,64-(PUD_SHIFT-3),64-PUD_INDEX_SIZE-3
1030	clrrdi r10,r11,3
1031	ldx r15,r10,r15
1032	cmpdi cr0,r15,0
1033	bge htw_tlb_miss_fault
1034
1035	/* Get to PMD entry */
1036	rldicl r11,r16,64-(PMD_SHIFT-3),64-PMD_INDEX_SIZE-3
1037	clrrdi r10,r11,3
1038	ldx r15,r10,r15
1039	cmpdi cr0,r15,0
1040	bge htw_tlb_miss_fault
1041
1042	/* Ok, we're all right, we can now create an indirect entry for
1043	* a 1M or 256M page.
1044	*
1045	* The last trick is now that because we use "half" pages for
1046	* the HTW (1M IND is 2K and 256M IND is 32K) we need to account
1047	* for an added LSB bit to the RPN. For 64K pages, there is no
1048	* problem as we already use 32K arrays (half PTE pages), but for
1049	* 4K page we need to extract a bit from the virtual address and
1050	* insert it into the "PA52" bit of the RPN.
1051	*/
1052	rlwimi r15,r16,32-9,20,20
1053	/* Now we build the MAS:
1054	*
1055	* MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG
1056	* MAS 1 : Almost fully setup
1057	* - PID already updated by caller if necessary
1058	* - TSIZE for now is base ind page size always
1059	* MAS 2 : Use defaults
1060	* MAS 3+7 : Needs to be done
1061	*/
1062	ori r10,r15,(BOOK3E_PAGESZ_4K << MAS3_SPSIZE_SHIFT)
1063
1064	srdi r16,r10,32
1065	mtspr SPRN_MAS3,r10
1066	mtspr SPRN_MAS7,r16
1067
1068	tlbwe
1069
1070	htw_tlb_miss_done:
1071	/* We don't bother with restoring DEAR or ESR since we know we are
1072	* level 0 and just going back to userland. They are only needed
1073	* if you are going to take an access fault
1074	*/
1075	TLB_MISS_EPILOG_SUCCESS
1076	rfi
1077
1078	htw_tlb_miss_fault:
1079	/* We need to check if it was an instruction miss. We know this
1080	* though because r14 would contain -1
1081	*/
1082	cmpdi cr0,r14,-1
1083	beq 1f
1084	mtspr SPRN_DEAR,r16
1085	mtspr SPRN_ESR,r14
1086	TLB_MISS_EPILOG_ERROR
1087	b exc_data_storage_book3e
1088	1: TLB_MISS_EPILOG_ERROR
1089	b exc_instruction_storage_book3e
1090
1091	/*	665	/*
1092	* This is the guts of "any" level TLB miss handler for kernel linear	666	* This is the guts of "any" level TLB miss handler for kernel linear
1093	* mapping misses. We are entered with:	667	* mapping misses. We are entered with:

arch/powerpc/mm/pgtable.c+61−33

 }
 #if defined(CONFIG_PPC_8xx)
-void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
+static void __set_huge_pte_at(pmd_t *pmd, pte_t *ptep, pte_basic_t val)
-		     pte_t pte, unsigned long sz)
 {
-	pmd_t *pmd = pmd_off(mm, addr);
-	pte_basic_t val;
 	pte_basic_t *entry = (pte_basic_t *)ptep;
 	int num, i;
 	 */
 	VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep));
-	pte = set_pte_filter(pte, addr);
-	val = pte_val(pte);
 	num = number_of_cells_per_pte(pmd, val, 1);
 	for (i = 0; i < num; i++, entry++, val += SZ_4K)
 		*entry = val;
 }
+void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
+		     pte_t pte, unsigned long sz)
+{
+	pmd_t *pmdp = pmd_off(mm, addr);
+	pte = set_pte_filter(pte, addr);
+	if (sz == SZ_8M) { /* Flag both PMD entries as 8M and fill both page tables */
+		*pmdp = __pmd(pmd_val(*pmdp) | _PMD_PAGE_8M);
+		*(pmdp + 1) = __pmd(pmd_val(*(pmdp + 1)) | _PMD_PAGE_8M);
+		__set_huge_pte_at(pmdp, pte_offset_kernel(pmdp, 0), pte_val(pte));
+		__set_huge_pte_at(pmdp, pte_offset_kernel(pmdp + 1, 0), pte_val(pte) + SZ_4M);
+	} else {
+		__set_huge_pte_at(pmdp, ptep, pte_val(pte));
+	}
+}
+#else
+void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
+		     pte_t pte, unsigned long sz)
+{
+	unsigned long pdsize;
+	int i;
+	pte = set_pte_filter(pte, addr);
+	/*
+	 * Make sure hardware valid bit is not set. We don't do
+	 * tlb flush for this update.
+	 */
+	VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep));
+	if (sz < PMD_SIZE)
+		pdsize = PAGE_SIZE;
+	else if (sz < PUD_SIZE)
+		pdsize = PMD_SIZE;
+	else if (sz < P4D_SIZE)
+		pdsize = PUD_SIZE;
+	else if (sz < PGDIR_SIZE)
+		pdsize = P4D_SIZE;
+	else
+		pdsize = PGDIR_SIZE;
+	for (i = 0; i < sz / pdsize; i++, ptep++, addr += pdsize) {
+		__set_pte_at(mm, addr, ptep, pte, 0);
+		pte = __pte(pte_val(pte) + ((unsigned long long)pdsize / PAGE_SIZE << PFN_PTE_SHIFT));
+	}
+}
 #endif
 #endif /* CONFIG_HUGETLB_PAGE */
 EXPORT_SYMBOL_GPL(vmalloc_to_phys);
 /*
- * We have 4 cases for pgds and pmds:
+ * We have 3 cases for pgds and pmds:
  * (1) invalid (all zeroes)
  * (2) pointer to next table, as normal; bottom 6 bits == 0
  * (3) leaf pte for huge page _PAGE_PTE set
- * (4) hugepd pointer, _PAGE_PTE = 0 and bits [2..6] indicate size of table
  *
  * So long as we atomically load page table pointers we are safe against teardown,
  * we can follow the address down to the page and take a ref on it.
 			bool *is_thp, unsigned *hpage_shift)
 {
 	pgd_t *pgdp;
+#ifdef CONFIG_PPC64
 	p4d_t p4d, *p4dp;
 	pud_t pud, *pudp;
+#endif
 	pmd_t pmd, *pmdp;
 	pte_t *ret_pte;
-	hugepd_t *hpdp = NULL;
 	unsigned pdshift;
 	if (hpage_shift)
 	 * page fault or a page unmap. The return pte_t * is still not
 	 * stable. So should be checked there for above conditions.
 	 * Top level is an exception because it is folded into p4d.
+	 *
+	 * On PPC32, P4D/PUD/PMD are folded into PGD so go straight to
+	 * PMD level.
 	 */
 	pgdp = pgdir + pgd_index(ea);
+#ifdef CONFIG_PPC64
 	p4dp = p4d_offset(pgdp, ea);
 	p4d  = READ_ONCE(*p4dp);
 	pdshift = P4D_SHIFT;
 		goto out;
 	}
-	if (is_hugepd(__hugepd(p4d_val(p4d)))) {
-		hpdp = (hugepd_t *)&p4d;
-		goto out_huge;
-	}
 	/*
 	 * Even if we end up with an unmap, the pgtable will not
 	 * be freed, because we do an rcu free and here we are
 		goto out;
 	}
-	if (is_hugepd(__hugepd(pud_val(pud)))) {
-		hpdp = (hugepd_t *)&pud;
-		goto out_huge;
-	}
-	pdshift = PMD_SHIFT;
 	pmdp = pmd_offset(&pud, ea);
+#else
+	pmdp = pmd_offset(pud_offset(p4d_offset(pgdp, ea), ea), ea);
+#endif
+	pdshift = PMD_SHIFT;
 	pmd  = READ_ONCE(*pmdp);
 	/*
 		goto out;
 	}
-	if (is_hugepd(__hugepd(pmd_val(pmd)))) {
-		hpdp = (hugepd_t *)&pmd;
-		goto out_huge;
-	}
 	return pte_offset_kernel(&pmd, ea);
-out_huge:
-	if (!hpdp)
-		return NULL;
-	ret_pte = hugepte_offset(*hpdp, ea, pdshift);
-	pdshift = hugepd_shift(*hpdp);
 out:
 	if (hpage_shift)
 		*hpage_shift = pdshift;

arch/powerpc/mm/pgtable_32.c+1−1

@@ -48,7 +48,7 @@ notrace void __init early_ioremap_init(void) · 47 unmodified lines
48	early_ioremap_setup();	48	early_ioremap_setup();
49	}	49	}
50		50
51	static void __init *early_alloc_pgtable(unsigned long size)	51	void __init *early_alloc_pgtable(unsigned long size)
52	{	52	{
53	void *ptr = memblock_alloc(size, size);	53	void *ptr = memblock_alloc(size, size);
54		54

arch/riscv/include/asm/hugetlb.h+1−1

@@ -44,7 +44,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 43 unmodified lines
44	pte_t pte, int dirty);	44	pte_t pte, int dirty);
45		45
46	#define __HAVE_ARCH_HUGE_PTEP_GET	46	#define __HAVE_ARCH_HUGE_PTEP_GET
47	pte_t huge_ptep_get(pte_t *ptep);	47	pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep);
48		48
49	pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags);	49	pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags);
50	#define arch_make_huge_pte arch_make_huge_pte	50	#define arch_make_huge_pte arch_make_huge_pte

arch/riscv/include/asm/pgtable.h+2−2

@@ -514,8 +514,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 513 unmodified lines
514	#define update_mmu_cache(vma, addr, ptep) \	514	#define update_mmu_cache(vma, addr, ptep) \
515	update_mmu_cache_range(NULL, vma, addr, ptep, 1)	515	update_mmu_cache_range(NULL, vma, addr, ptep, 1)
516		516
517	#define __HAVE_ARCH_UPDATE_MMU_TLB	517	#define update_mmu_tlb_range(vma, addr, ptep, nr) \
518	#define update_mmu_tlb update_mmu_cache	518	update_mmu_cache_range(NULL, vma, addr, ptep, nr)
519		519
520	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,	520	static inline void update_mmu_cache_pmd(struct vm_area_struct *vma,
521	unsigned long address, pmd_t *pmdp)	521	unsigned long address, pmd_t *pmdp)

arch/riscv/mm/hugetlbpage.c+1−1

@@ -3,7 +3,7 @@ · 2 unmodified lines
3	#include <linux/err.h>	3	#include <linux/err.h>
4		4
5	#ifdef CONFIG_RISCV_ISA_SVNAPOT	5	#ifdef CONFIG_RISCV_ISA_SVNAPOT
6	pte_t huge_ptep_get(pte_t *ptep)	6	pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
7	{	7	{
8	unsigned long pte_num;	8	unsigned long pte_num;
9	int i;	9	int i;

arch/s390/Kconfig+1−0

@@ -159,6 +159,7 @@ config S390 · 158 unmodified lines
159	select HAVE_ARCH_KASAN	159	select HAVE_ARCH_KASAN
160	select HAVE_ARCH_KASAN_VMALLOC	160	select HAVE_ARCH_KASAN_VMALLOC
161	select HAVE_ARCH_KCSAN	161	select HAVE_ARCH_KCSAN
		162	select HAVE_ARCH_KMSAN
162	select HAVE_ARCH_KFENCE	163	select HAVE_ARCH_KFENCE
163	select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET	164	select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET
164	select HAVE_ARCH_SECCOMP_FILTER	165	select HAVE_ARCH_SECCOMP_FILTER

arch/s390/Makefile+1−1

@@ -36,7 +36,7 @@ KBUILD_CFLAGS_DECOMPRESSOR += $(if $(CONFIG_DEBUG_INFO_DWARF4), $(call cc-option · 35 unmodified lines
36	KBUILD_CFLAGS_DECOMPRESSOR += $(if $(CONFIG_CC_NO_ARRAY_BOUNDS),-Wno-array-bounds)	36	KBUILD_CFLAGS_DECOMPRESSOR += $(if $(CONFIG_CC_NO_ARRAY_BOUNDS),-Wno-array-bounds)
37		37
38	UTS_MACHINE := s390x	38	UTS_MACHINE := s390x
39	STACK_SIZE := $(if $(CONFIG_KASAN),65536,16384)	39	STACK_SIZE := $(if $(CONFIG_KASAN),65536,$(if $(CONFIG_KMSAN),65536,16384))
40	CHECKFLAGS += -D__s390__ -D__s390x__	40	CHECKFLAGS += -D__s390__ -D__s390x__
41		41
42	export LD_BFD	42	export LD_BFD

arch/s390/boot/Makefile+3−0

 # Makefile for the linux s390-specific parts of the memory manager.
 #
+# Tooling runtimes are unavailable and cannot be linked for early boot code
 KCOV_INSTRUMENT := n
 GCOV_PROFILE := n
 UBSAN_SANITIZE := n
 KASAN_SANITIZE := n
 KCSAN_SANITIZE := n
+KMSAN_SANITIZE := n
 KBUILD_AFLAGS := $(KBUILD_AFLAGS_DECOMPRESSOR)
 KBUILD_CFLAGS := $(KBUILD_CFLAGS_DECOMPRESSOR)
 obj-$(CONFIG_RANDOMIZE_BASE)	+= kaslr.o
 obj-y	+= $(if $(CONFIG_KERNEL_UNCOMPRESSED),,decompressor.o) info.o
 obj-$(CONFIG_KERNEL_ZSTD) += clz_ctz.o
+obj-$(CONFIG_KMSAN) += kmsan.o
 obj-all := $(obj-y) piggy.o syms.o
 targets	:= bzImage section_cmp.boot.data section_cmp.boot.preserved.data $(obj-y)

arch/s390/boot/kmsan.c+6−0

@@ -0,0 +1,6 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	#include <linux/kmsan-checks.h>
	3
	4	void kmsan_unpoison_memory(const void *address, size_t size)
	5	{
	6	}

arch/s390/boot/startup.c+7−0

 	MODULES_END = round_down(kernel_start, _SEGMENT_SIZE);
 	MODULES_VADDR = MODULES_END - MODULES_LEN;
 	VMALLOC_END = MODULES_VADDR;
+	if (IS_ENABLED(CONFIG_KMSAN))
+		VMALLOC_END -= MODULES_LEN * 2;
 	/* allow vmalloc area to occupy up to about 1/2 of the rest virtual space left */
 	vsize = (VMALLOC_END - FIXMAP_SIZE) / 2;
 	vsize = round_down(vsize, _SEGMENT_SIZE);
 	vmalloc_size = min(vmalloc_size, vsize);
+	if (IS_ENABLED(CONFIG_KMSAN)) {
+		/* take 2/3 of vmalloc area for KMSAN shadow and origins */
+		vmalloc_size = round_down(vmalloc_size / 3, _SEGMENT_SIZE);
+		VMALLOC_END -= vmalloc_size * 2;
+	}
 	VMALLOC_START = VMALLOC_END - vmalloc_size;
 	__memcpy_real_area = round_down(VMALLOC_START - MEMCPY_REAL_SIZE, PAGE_SIZE);

arch/s390/boot/string.c+16−0

 // SPDX-License-Identifier: GPL-2.0
+#define IN_BOOT_STRING_C 1
 #include <linux/ctype.h>
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #undef CONFIG_KASAN
 #undef CONFIG_KASAN_GENERIC
+#undef CONFIG_KMSAN
 #include "../lib/string.c"
+/*
+ * Duplicate some functions from the common lib/string.c
+ * instead of fully including it.
+ */
 int strncmp(const char *cs, const char *ct, size_t count)
 {
 	unsigned char c1, c2;
 	return 0;
 }
+void *memset64(uint64_t *s, uint64_t v, size_t count)
+{
+	uint64_t *xs = s;
+	while (count--)
+		*xs++ = v;
+	return s;
+}
 char *skip_spaces(const char *str)
 {
 	while (isspace(*str))

arch/s390/include/asm/checksum.h+2−0

 #define _S390_CHECKSUM_H
 #include <linux/instrumented.h>
+#include <linux/kmsan-checks.h>
 #include <linux/in6.h>
 static inline __wsum cksm(const void *buff, int len, __wsum sum)
 	};
 	instrument_read(buff, len);
+	kmsan_check_memory(buff, len);
 	asm volatile("\n"
 		"0:	cksm	%[sum],%[rp]\n"
 		"	jo	0b\n"

arch/s390/include/asm/cpacf.h+3−0

 #define _ASM_S390_CPACF_H
 #include <asm/facility.h>
+#include <linux/kmsan-checks.h>
 /*
  * Instruction opcodes for the CPACF instructions
 		: [ucbuf] "+&d" (u.pair), [cbuf] "+&d" (c.pair)
 		: [fc] "K" (CPACF_PRNO_TRNG), [opc] "i" (CPACF_PRNO)
 		: "cc", "memory", "0");
+	kmsan_unpoison_memory(ucbuf, ucbuf_len);
+	kmsan_unpoison_memory(cbuf, cbuf_len);
 }
 /**

arch/s390/include/asm/cpu_mf.h+6−0

 #define _ASM_S390_CPU_MF_H
 #include <linux/errno.h>
+#include <linux/kmsan-checks.h>
 #include <asm/asm-extable.h>
 #include <asm/facility.h>
 		: "=d" (cc)
 		: "Q" (*dest), "d" (range), "i" (set)
 		: "cc", "memory");
+	/*
+	 * If cc == 2, less than RANGE counters are stored, but it's not easy
+	 * to tell how many. Always unpoison the whole range for simplicity.
+	 */
+	kmsan_unpoison_memory(dest, range * sizeof(u64));
 	return cc;
 }

arch/s390/include/asm/hugetlb.h+2−2

@@ -19,7 +19,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 18 unmodified lines
19	pte_t *ptep, pte_t pte, unsigned long sz);	19	pte_t *ptep, pte_t pte, unsigned long sz);
20	void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr,	20	void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr,
21	pte_t *ptep, pte_t pte);	21	pte_t *ptep, pte_t pte);
22	pte_t huge_ptep_get(pte_t *ptep);	22	pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep);
23	pte_t huge_ptep_get_and_clear(struct mm_struct *mm,	23	pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
24	unsigned long addr, pte_t *ptep);	24	unsigned long addr, pte_t *ptep);
25		25
@@ -64,7 +64,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 38 unmodified lines
64	unsigned long addr, pte_t *ptep,	64	unsigned long addr, pte_t *ptep,
65	pte_t pte, int dirty)	65	pte_t pte, int dirty)
66	{	66	{
67	int changed = !pte_same(huge_ptep_get(ptep), pte);	67	int changed = !pte_same(huge_ptep_get(vma->vm_mm, addr, ptep), pte);
68	if (changed) {	68	if (changed) {
69	huge_ptep_get_and_clear(vma->vm_mm, addr, ptep);	69	huge_ptep_get_and_clear(vma->vm_mm, addr, ptep);
70	__set_huge_pte_at(vma->vm_mm, addr, ptep, pte);	70	__set_huge_pte_at(vma->vm_mm, addr, ptep, pte);

arch/s390/include/asm/irqflags.h+14−3

 	asm volatile("ssm   %0" : : "Q" (flags) : "memory");
 }
-static __always_inline unsigned long arch_local_save_flags(void)
+#ifdef CONFIG_KMSAN
+#define arch_local_irq_attributes noinline notrace __no_sanitize_memory __maybe_unused
+#else
+#define arch_local_irq_attributes __always_inline
+#endif
+static arch_local_irq_attributes unsigned long arch_local_save_flags(void)
 {
 	return __arch_local_irq_stnsm(0xff);
 }
-static __always_inline unsigned long arch_local_irq_save(void)
+static arch_local_irq_attributes unsigned long arch_local_irq_save(void)
 {
 	return __arch_local_irq_stnsm(0xfc);
 }
 	arch_local_irq_save();
 }
-static __always_inline void arch_local_irq_enable(void)
+static arch_local_irq_attributes void arch_local_irq_enable_external(void)
+{
+	__arch_local_irq_stosm(0x01);
+}
+static arch_local_irq_attributes void arch_local_irq_enable(void)
 {
 	__arch_local_irq_stosm(0x03);
 }

arch/s390/include/asm/kmsan.h+59−0

@@ -0,0 +1,59 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _ASM_S390_KMSAN_H
	3	#define _ASM_S390_KMSAN_H
	4
	5	#include <asm/lowcore.h>
	6	#include <asm/page.h>
	7	#include <linux/kmsan.h>
	8	#include <linux/mmzone.h>
	9	#include <linux/stddef.h>
	10
	11	#ifndef MODULE
	12
	13	static inline bool is_lowcore_addr(void *addr)
	14	{
	15	return addr >= (void *)&S390_lowcore &&
	16	addr < (void *)(&S390_lowcore + 1);
	17	}
	18
	19	static inline void arch_kmsan_get_meta_or_null(void addr, bool is_origin)
	20	{
	21	if (is_lowcore_addr(addr)) {
	22	/*
	23	* Different lowcores accessed via S390_lowcore are described
	24	* by the same struct page. Resolve the prefix manually in
	25	* order to get a distinct struct page.
	26	*/
	27	addr += (void *)lowcore_ptr[raw_smp_processor_id()] -
	28	(void *)&S390_lowcore;
	29	if (KMSAN_WARN_ON(is_lowcore_addr(addr)))
	30	return NULL;
	31	return kmsan_get_metadata(addr, is_origin);
	32	}
	33	return NULL;
	34	}
	35
	36	static inline bool kmsan_virt_addr_valid(void *addr)
	37	{
	38	bool ret;
	39
	40	/*
	41	* pfn_valid() relies on RCU, and may call into the scheduler on exiting
	42	* the critical section. However, this would result in recursion with
	43	* KMSAN. Therefore, disable preemption here, and re-enable preemption
	44	* below while suppressing reschedules to avoid recursion.
	45	*
	46	* Note, this sacrifices occasionally breaking scheduling guarantees.
	47	* Although, a kernel compiled with KMSAN has already given up on any
	48	* performance guarantees due to being heavily instrumented.
	49	*/
	50	preempt_disable();
	51	ret = virt_addr_valid(addr);
	52	preempt_enable_no_resched();
	53
	54	return ret;
	55	}
	56
	57	#endif /* !MODULE */
	58
	59	#endif /* _ASM_S390_KMSAN_H */

arch/s390/include/asm/pgtable.h+12−0

@@ -107,6 +107,18 @@ static inline int is_module_addr(void *addr) · 106 unmodified lines
107	return 1;	107	return 1;
108	}	108	}
109		109
		110	#ifdef CONFIG_KMSAN
		111	#define KMSAN_VMALLOC_SIZE (VMALLOC_END - VMALLOC_START)
		112	#define KMSAN_VMALLOC_SHADOW_START VMALLOC_END
		113	#define KMSAN_VMALLOC_SHADOW_END (KMSAN_VMALLOC_SHADOW_START + KMSAN_VMALLOC_SIZE)
		114	#define KMSAN_VMALLOC_ORIGIN_START KMSAN_VMALLOC_SHADOW_END
		115	#define KMSAN_VMALLOC_ORIGIN_END (KMSAN_VMALLOC_ORIGIN_START + KMSAN_VMALLOC_SIZE)
		116	#define KMSAN_MODULES_SHADOW_START KMSAN_VMALLOC_ORIGIN_END
		117	#define KMSAN_MODULES_SHADOW_END (KMSAN_MODULES_SHADOW_START + MODULES_LEN)
		118	#define KMSAN_MODULES_ORIGIN_START KMSAN_MODULES_SHADOW_END
		119	#define KMSAN_MODULES_ORIGIN_END (KMSAN_MODULES_ORIGIN_START + MODULES_LEN)
		120	#endif
		121
110	#ifdef CONFIG_RANDOMIZE_BASE	122	#ifdef CONFIG_RANDOMIZE_BASE
111	#define KASLR_LEN (1UL << 31)	123	#define KASLR_LEN (1UL << 31)
112	#else	124	#else

arch/s390/include/asm/string.h+15−5

 #define __HAVE_ARCH_MEMCPY	/* gcc builtin & arch function */
 #define __HAVE_ARCH_MEMMOVE	/* gcc builtin & arch function */
 #define __HAVE_ARCH_MEMSET	/* gcc builtin & arch function */
-#define __HAVE_ARCH_MEMSET16	/* arch function */
-#define __HAVE_ARCH_MEMSET32	/* arch function */
-#define __HAVE_ARCH_MEMSET64	/* arch function */
 void *memcpy(void *dest, const void *src, size_t n);
 void *memset(void *s, int c, size_t n);
 void *memmove(void *dest, const void *src, size_t n);
-#ifndef CONFIG_KASAN
+#if !defined(CONFIG_KASAN) && !defined(CONFIG_KMSAN)
 #define __HAVE_ARCH_MEMCHR	/* inline & arch function */
 #define __HAVE_ARCH_MEMCMP	/* arch function */
 #define __HAVE_ARCH_MEMSCAN	/* inline & arch function */
 #define __HAVE_ARCH_STRNCPY	/* arch function */
 #define __HAVE_ARCH_STRNLEN	/* inline & arch function */
 #define __HAVE_ARCH_STRSTR	/* arch function */
+#define __HAVE_ARCH_MEMSET16	/* arch function */
+#define __HAVE_ARCH_MEMSET32	/* arch function */
+#define __HAVE_ARCH_MEMSET64	/* arch function */
 /* Prototypes for non-inlined arch strings functions. */
 int memcmp(const void *s1, const void *s2, size_t n);
 char *strncat(char *dest, const char *src, size_t n);
 char *strncpy(char *dest, const char *src, size_t n);
 char *strstr(const char *s1, const char *s2);
-#endif /* !CONFIG_KASAN */
+#endif /* !defined(CONFIG_KASAN) && !defined(CONFIG_KMSAN) */
 #undef __HAVE_ARCH_STRCHR
 #undef __HAVE_ARCH_STRNCHR
 void *__memset32(uint32_t *s, uint32_t v, size_t count);
 void *__memset64(uint64_t *s, uint64_t v, size_t count);
+#ifdef __HAVE_ARCH_MEMSET16
 static inline void *memset16(uint16_t *s, uint16_t v, size_t count)
 {
 	return __memset16(s, v, count * sizeof(v));
 }
+#endif
+#ifdef __HAVE_ARCH_MEMSET32
 static inline void *memset32(uint32_t *s, uint32_t v, size_t count)
 {
 	return __memset32(s, v, count * sizeof(v));
 }
+#endif
+#ifdef __HAVE_ARCH_MEMSET64
+#ifdef IN_BOOT_STRING_C
+void *memset64(uint64_t *s, uint64_t v, size_t count);
+#else
 static inline void *memset64(uint64_t *s, uint64_t v, size_t count)
 {
 	return __memset64(s, v, count * sizeof(v));
 }
+#endif
+#endif
 #if !defined(IN_ARCH_STRING_C) && (!defined(CONFIG_FORTIFY_SOURCE) || defined(__NO_FORTIFY))

arch/s390/include/asm/thread_info.h+1−1

@@ -16,7 +16,7 @@ · 15 unmodified lines
16	/*	16	/*
17	* General size of kernel stacks	17	* General size of kernel stacks
18	*/	18	*/
19	#ifdef CONFIG_KASAN	19	#if defined(CONFIG_KASAN) \|\| defined(CONFIG_KMSAN)
20	#define THREAD_SIZE_ORDER 4	20	#define THREAD_SIZE_ORDER 4
21	#else	21	#else
22	#define THREAD_SIZE_ORDER 2	22	#define THREAD_SIZE_ORDER 2

arch/s390/include/asm/uaccess.h+80−32

 #include <asm/extable.h>
 #include <asm/facility.h>
 #include <asm-generic/access_ok.h>
+#include <linux/instrumented.h>
 void debug_user_asce(int exit);
 int __noreturn __put_user_bad(void);
-#define __put_user_asm(to, from, size)					\
+#ifdef CONFIG_KMSAN
-({									\
+#define get_put_user_noinstr_attributes \
+	noinline __maybe_unused __no_sanitize_memory
+#else
+#define get_put_user_noinstr_attributes __always_inline
+#endif
+#define DEFINE_PUT_USER(type)						\
+static get_put_user_noinstr_attributes int				\
+__put_user_##type##_noinstr(unsigned type __user *to,			\
+			    unsigned type *from,			\
+			    unsigned long size)				\
+{									\
 	union oac __oac_spec = {					\
 		.oac1.as = PSW_BITS_AS_SECONDARY,			\
 		.oac1.a = 1,						\
 	};								\
-	int __rc;							\
+	int rc;								\
 									\
 	asm volatile(							\
 		"	lr	0,%[spec]\n"				\
 		"2:\n"							\
 		EX_TABLE_UA_STORE(0b, 2b, %[rc])			\
 		EX_TABLE_UA_STORE(1b, 2b, %[rc])			\
-		: [rc] "=&d" (__rc), [_to] "+Q" (*(to))			\
+		: [rc] "=&d" (rc), [_to] "+Q" (*(to))			\
 		: [_size] "d" (size), [_from] "Q" (*(from)),		\
 		  [spec] "d" (__oac_spec.val)				\
 		: "cc", "0");						\
-	__rc;								\
+	return rc;							\
-})
+}									\
+									\
+static __always_inline int						\
+__put_user_##type(unsigned type __user *to, unsigned type *from,	\
+		  unsigned long size)					\
+{									\
+	int rc;								\
+									\
+	rc = __put_user_##type##_noinstr(to, from, size);		\
+	instrument_put_user(*from, to, size);				\
+	return rc;							\
+}
+DEFINE_PUT_USER(char);
+DEFINE_PUT_USER(short);
+DEFINE_PUT_USER(int);
+DEFINE_PUT_USER(long);
 static __always_inline int __put_user_fn(void *x, void __user *ptr, unsigned long size)
 {
 	switch (size) {
 	case 1:
-		rc = __put_user_asm((unsigned char __user *)ptr,
+		rc = __put_user_char((unsigned char __user *)ptr,
-				    (unsigned char *)x,
+				     (unsigned char *)x,
-				    size);
+				     size);
 		break;
 	case 2:
-		rc = __put_user_asm((unsigned short __user *)ptr,
+		rc = __put_user_short((unsigned short __user *)ptr,
-				    (unsigned short *)x,
+				      (unsigned short *)x,
-				    size);
+				      size);
 		break;
 	case 4:
-		rc = __put_user_asm((unsigned int __user *)ptr,
+		rc = __put_user_int((unsigned int __user *)ptr,
 				    (unsigned int *)x,
 				    size);
 		break;
 	case 8:
-		rc = __put_user_asm((unsigned long __user *)ptr,
+		rc = __put_user_long((unsigned long __user *)ptr,
-				    (unsigned long *)x,
+				     (unsigned long *)x,
-				    size);
+				     size);
 		break;
 	default:
 		__put_user_bad();
 int __noreturn __get_user_bad(void);
-#define __get_user_asm(to, from, size)					\
+#define DEFINE_GET_USER(type)						\
-({									\
+static get_put_user_noinstr_attributes int				\
+__get_user_##type##_noinstr(unsigned type *to,				\
+			    unsigned type __user *from,			\
+			    unsigned long size)				\
+{									\
 	union oac __oac_spec = {					\
 		.oac2.as = PSW_BITS_AS_SECONDARY,			\
 		.oac2.a = 1,						\
 	};								\
-	int __rc;							\
+	int rc;								\
 									\
 	asm volatile(							\
 		"	lr	0,%[spec]\n"				\
 		"2:\n"							\
 		EX_TABLE_UA_LOAD_MEM(0b, 2b, %[rc], %[_to], %[_ksize])	\
 		EX_TABLE_UA_LOAD_MEM(1b, 2b, %[rc], %[_to], %[_ksize])	\
-		: [rc] "=&d" (__rc), "=Q" (*(to))			\
+		: [rc] "=&d" (rc), "=Q" (*(to))				\
 		: [_size] "d" (size), [_from] "Q" (*(from)),		\
 		  [spec] "d" (__oac_spec.val), [_to] "a" (to),		\
 		  [_ksize] "K" (size)					\
 		: "cc", "0");						\
-	__rc;								\
+	return rc;							\
-})
+}									\
+									\
+static __always_inline int						\
+__get_user_##type(unsigned type *to, unsigned type __user *from,	\
+		  unsigned long size)					\
+{									\
+	int rc;								\
+									\
+	rc = __get_user_##type##_noinstr(to, from, size);		\
+	instrument_get_user(*to);					\
+	return rc;							\
+}
+DEFINE_GET_USER(char);
+DEFINE_GET_USER(short);
+DEFINE_GET_USER(int);
+DEFINE_GET_USER(long);
 static __always_inline int __get_user_fn(void *x, const void __user *ptr, unsigned long size)
 {
 	switch (size) {
 	case 1:
-		rc = __get_user_asm((unsigned char *)x,
+		rc = __get_user_char((unsigned char *)x,
-				    (unsigned char __user *)ptr,
+				     (unsigned char __user *)ptr,
-				    size);
+				     size);
 		break;
 	case 2:
-		rc = __get_user_asm((unsigned short *)x,
+		rc = __get_user_short((unsigned short *)x,
-				    (unsigned short __user *)ptr,
+				      (unsigned short __user *)ptr,
-				    size);
+				      size);
 		break;
 	case 4:
-		rc = __get_user_asm((unsigned int *)x,
+		rc = __get_user_int((unsigned int *)x,
 				    (unsigned int __user *)ptr,
 				    size);
 		break;
 	case 8:
-		rc = __get_user_asm((unsigned long *)x,
+		rc = __get_user_long((unsigned long *)x,
-				    (unsigned long __user *)ptr,
+				     (unsigned long __user *)ptr,
-				    size);
+				     size);
 		break;
 	default:
 		__get_user_bad();

arch/s390/kernel/diag.c+6−4

 	int rc = -EOPNOTSUPP;
 	diag_stat_inc(DIAG_STAT_X224);
-	asm volatile(
+	asm volatile("\n"
-		"	diag	%1,%2,0x224\n"
+		"	diag	%[type],%[addr],0x224\n"
-		"0:	lhi	%0,0x0\n"
+		"0:	lhi	%[rc],0\n"
 		"1:\n"
 		EX_TABLE(0b,1b)
-		: "+d" (rc) :"d" (0), "d" (addr) : "memory");
+		: [rc] "+d" (rc)
+		, "=m" (*(struct { char buf[PAGE_SIZE]; } *)ptr)
+		: [type] "d" (0), [addr] "d" (addr));
 	return rc;
 }
 EXPORT_SYMBOL(diag224);

arch/s390/kernel/ftrace.c+2−0

 #include <linux/ftrace.h>
 #include <linux/kernel.h>
 #include <linux/types.h>
+#include <linux/kmsan-checks.h>
 #include <linux/kprobes.h>
 #include <linux/execmem.h>
 #include <trace/syscall.h>
 	if (bit < 0)
 		return;
+	kmsan_unpoison_memory(fregs, sizeof(*fregs));
 	regs = ftrace_get_regs(fregs);
 	p = get_kprobe((kprobe_opcode_t *)ip);
 	if (!regs || unlikely(!p) || kprobe_disabled(p))

arch/s390/kernel/traps.c+6−0

 #include <linux/uaccess.h>
 #include <linux/cpu.h>
 #include <linux/entry-common.h>
+#include <linux/kmsan.h>
 #include <asm/asm-extable.h>
 #include <asm/vtime.h>
 #include <asm/fpu.h>
 void kernel_stack_overflow(struct pt_regs *regs)
 {
+	/*
+	 * Normally regs are unpoisoned by the generic entry code, but
+	 * kernel_stack_overflow() is a rare case that is called bypassing it.
+	 */
+	kmsan_unpoison_entry_regs(regs);
 	bust_spinlocks(1);
 	printk("Kernel stack overflow.\n");
 	show_regs(regs);

arch/s390/kernel/unwind_bc.c+4−0

 	       READ_ONCE_NOCHECK(regs->psw.mask) & PSW_MASK_PSTATE;
 }
+/* Avoid KMSAN false positives from touching uninitialized frames. */
+__no_kmsan_checks
 bool unwind_next_frame(struct unwind_state *state)
 {
 	struct stack_info *info = &state->stack_info;
 }
 EXPORT_SYMBOL_GPL(unwind_next_frame);
+/* Avoid KMSAN false positives from touching uninitialized frames. */
+__no_kmsan_checks
 void __unwind_start(struct unwind_state *state, struct task_struct *task,
 		    struct pt_regs *regs, unsigned long first_frame)
 {

arch/s390/mm/hugetlbpage.c+2−2

@@ -169,7 +169,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 168 unmodified lines
169	__set_huge_pte_at(mm, addr, ptep, pte);	169	__set_huge_pte_at(mm, addr, ptep, pte);
170	}	170	}
171		171
172	pte_t huge_ptep_get(pte_t *ptep)	172	pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
173	{	173	{
174	return __rste_to_pte(pte_val(*ptep));	174	return __rste_to_pte(pte_val(*ptep));
175	}	175	}
@@ -177,7 +177,7 @@ pte_t huge_ptep_get(pte_t *ptep) · 1 unmodified line
177	pte_t huge_ptep_get_and_clear(struct mm_struct *mm,	177	pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
178	unsigned long addr, pte_t *ptep)	178	unsigned long addr, pte_t *ptep)
179	{	179	{
180	pte_t pte = huge_ptep_get(ptep);	180	pte_t pte = huge_ptep_get(mm, addr, ptep);
181	pmd_t pmdp = (pmd_t ) ptep;	181	pmd_t pmdp = (pmd_t ) ptep;
182	pud_t pudp = (pud_t ) ptep;	182	pud_t pudp = (pud_t ) ptep;
183		183

arch/sparc/mm/init_64.c+1−1

@@ -490,7 +490,7 @@ void flush_dcache_folio(struct folio *folio) · 489 unmodified lines
490	}	490	}
491	set_dcache_dirty(folio, this_cpu);	491	set_dcache_dirty(folio, this_cpu);
492	} else {	492	} else {
493	/* We could delay the flush for the !page_mapping	493	/* We could delay the flush for the !folio_mapping
494	* case too. But that case is for exec env/arg	494	* case too. But that case is for exec env/arg
495	* pages and those are %99 certainly going to get	495	* pages and those are %99 certainly going to get
496	* faulted into the tlb (and thus flushed) anyways.	496	* faulted into the tlb (and thus flushed) anyways.

arch/x86/mm/init_64.c+0−14

@@ -988,8 +988,6 @@ static void __meminit free_pagetable(struct page *page, int order) · 987 unmodified lines
988		988
989	/* bootmem page has reserved flag */	989	/* bootmem page has reserved flag */
990	if (PageReserved(page)) {	990	if (PageReserved(page)) {
991	__ClearPageReserved(page);
992
993	magic = page->index;	991	magic = page->index;
994	if (magic == SECTION_INFO \|\| magic == MIX_SECTION_INFO) {	992	if (magic == SECTION_INFO \|\| magic == MIX_SECTION_INFO) {
995	while (nr_pages--)	993	while (nr_pages--)
@@ -1362,18 +1360,6 @@ void __init mem_init(void) · 366 unmodified lines
1362	preallocate_vmalloc_pages();	1360	preallocate_vmalloc_pages();
1363	}	1361	}
1364		1362
1365	#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
1366	int __init deferred_page_init_max_threads(const struct cpumask *node_cpumask)
1367	{
1368	/*
1369	* More CPUs always led to greater speedups on tested systems, up to
1370	* all the nodes' CPUs. Use all since the system is otherwise idle
1371	* now.
1372	*/
1373	return max_t(int, cpumask_weight(node_cpumask), 1);
1374	}
1375	#endif
1376
1377	int kernel_set_to_readonly;	1363	int kernel_set_to_readonly;
1378		1364
1379	void mark_rodata_ro(void)	1365	void mark_rodata_ro(void)

arch/x86/mm/pat/set_memory.c+2−2

@@ -1119,8 +1119,8 @@ __split_large_page(struct cpa_data cpa, pte_t kpte, unsigned long address, · 1118 unmodified lines
1119	lpinc = PMD_SIZE;	1119	lpinc = PMD_SIZE;
1120	/*	1120	/*
1121	* Clear the PSE flags if the PRESENT flag is not set	1121	* Clear the PSE flags if the PRESENT flag is not set
1122	* otherwise pmd_present/pmd_huge will return true	1122	* otherwise pmd_present() will return true even on a non
1123	* even on a non present pmd.	1123	* present pmd.
1124	*/	1124	*/
1125	if (!(pgprot_val(ref_prot) & _PAGE_PRESENT))	1125	if (!(pgprot_val(ref_prot) & _PAGE_PRESENT))
1126	pgprot_val(ref_prot) &= ~_PAGE_PSE;	1126	pgprot_val(ref_prot) &= ~_PAGE_PSE;

arch/xtensa/include/asm/pgtable.h+3−3

@@ -410,9 +410,9 @@ void update_mmu_cache_range(struct vm_fault vmf, struct vm_area_struct vma, · 409 unmodified lines
410		410
411	typedef pte_t *pte_addr_t;	411	typedef pte_t *pte_addr_t;
412		412
413	void update_mmu_tlb(struct vm_area_struct *vma,	413	void update_mmu_tlb_range(struct vm_area_struct *vma,
414	unsigned long address, pte_t *ptep);	414	unsigned long address, pte_t *ptep, unsigned int nr);
415	#define __HAVE_ARCH_UPDATE_MMU_TLB	415	#define update_mmu_tlb_range update_mmu_tlb_range
416		416
417	#endif /* !defined (__ASSEMBLY__) */	417	#endif /* !defined (__ASSEMBLY__) */
418		418

arch/xtensa/mm/tlb.c+3−3

@@ -163,10 +163,10 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) · 162 unmodified lines
163	}	163	}
164	}	164	}
165		165
166	void update_mmu_tlb(struct vm_area_struct *vma,	166	void update_mmu_tlb_range(struct vm_area_struct *vma,
167	unsigned long address, pte_t *ptep)	167	unsigned long address, pte_t *ptep, unsigned int nr)
168	{	168	{
169	local_flush_tlb_page(vma, address);	169	local_flush_tlb_range(vma, address, address + PAGE_SIZE * nr);
170	}	170	}
171		171
172	#ifdef CONFIG_DEBUG_TLB_SANITY	172	#ifdef CONFIG_DEBUG_TLB_SANITY

drivers/acpi/numa/hmat.c+2−5

 	return NOTIFY_OK;
 }
-static int hmat_set_default_dram_perf(void)
+static int __init hmat_set_default_dram_perf(void)
 {
 	int rc;
 	int nid, pxm;
 	struct memory_target *target;
 	struct access_coordinate *attrs;
-	if (!default_dram_type)
+	for_each_node_mask(nid, default_dram_nodes) {
-		return -EIO;
-	for_each_node_mask(nid, default_dram_type->nodes) {
 		pxm = node_to_pxm(nid);
 		target = find_mem_target(pxm);
 		if (!target)

drivers/block/zram/Kconfig+1−0

@@ -2,6 +2,7 @@ · 1 unmodified line
2	config ZRAM	2	config ZRAM
3	tristate "Compressed RAM block device support"	3	tristate "Compressed RAM block device support"
4	depends on BLOCK && SYSFS && MMU	4	depends on BLOCK && SYSFS && MMU
		5	depends on HAVE_ZSMALLOC
5	depends on CRYPTO_LZO \|\| CRYPTO_ZSTD \|\| CRYPTO_LZ4 \|\| CRYPTO_LZ4HC \|\| CRYPTO_842	6	depends on CRYPTO_LZO \|\| CRYPTO_ZSTD \|\| CRYPTO_LZ4 \|\| CRYPTO_LZ4HC \|\| CRYPTO_842
6	select ZSMALLOC	7	select ZSMALLOC
7	help	8	help

drivers/dma-buf/Kconfig+1−0

@@ -35,6 +35,7 @@ config UDMABUF · 34 unmodified lines
35	default n	35	default n
36	depends on DMA_SHARED_BUFFER	36	depends on DMA_SHARED_BUFFER
37	depends on MEMFD_CREATE \|\| COMPILE_TEST	37	depends on MEMFD_CREATE \|\| COMPILE_TEST
		38	depends on MMU
38	help	39	help
39	A driver to let userspace turn memfd regions into dma-bufs.	40	A driver to let userspace turn memfd regions into dma-bufs.
40	Qemu can use this to create host dmabufs for guest framebuffers.	41	Qemu can use this to create host dmabufs for guest framebuffers.

drivers/dma-buf/udmabuf.c+173−59

@@ -10,6 +10,7 @@ · 9 unmodified lines
10	#include <linux/miscdevice.h>	10	#include <linux/miscdevice.h>
11	#include <linux/module.h>	11	#include <linux/module.h>
12	#include <linux/shmem_fs.h>	12	#include <linux/shmem_fs.h>
		13	#include <linux/hugetlb.h>
13	#include <linux/slab.h>	14	#include <linux/slab.h>
14	#include <linux/udmabuf.h>	15	#include <linux/udmabuf.h>
15	#include <linux/vmalloc.h>	16	#include <linux/vmalloc.h>
@@ -25,9 +26,16 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a dmabuf, in megabytes. Default is · 9 unmodified lines
25		26
26	struct udmabuf {	27	struct udmabuf {
27	pgoff_t pagecount;	28	pgoff_t pagecount;
28	struct page **pages;	29	struct folio **folios;
29	struct sg_table *sg;	30	struct sg_table *sg;
30	struct miscdevice *device;	31	struct miscdevice *device;
		32	pgoff_t *offsets;
		33	struct list_head unpin_list;
		34	};
		35
		36	struct udmabuf_folio {
		37	struct folio *folio;
		38	struct list_head list;
31	};	39	};
32		40
33	static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)	41	static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf)
@@ -35,12 +43,15 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) · 1 unmodified line
35	struct vm_area_struct *vma = vmf->vma;	43	struct vm_area_struct *vma = vmf->vma;
36	struct udmabuf *ubuf = vma->vm_private_data;	44	struct udmabuf *ubuf = vma->vm_private_data;
37	pgoff_t pgoff = vmf->pgoff;	45	pgoff_t pgoff = vmf->pgoff;
		46	unsigned long pfn;
38		47
39	if (pgoff >= ubuf->pagecount)	48	if (pgoff >= ubuf->pagecount)
40	return VM_FAULT_SIGBUS;	49	return VM_FAULT_SIGBUS;
41	vmf->page = ubuf->pages[pgoff];	50
42	get_page(vmf->page);	51	pfn = folio_pfn(ubuf->folios[pgoff]);
43	return 0;	52	pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT;
		53
		54	return vmf_insert_pfn(vma, vmf->address, pfn);
44	}	55	}
45		56
46	static const struct vm_operations_struct udmabuf_vm_ops = {	57	static const struct vm_operations_struct udmabuf_vm_ops = {
@@ -56,17 +67,28 @@ static int mmap_udmabuf(struct dma_buf buf, struct vm_area_struct vma) · 9 unmodified lines
56		67
57	vma->vm_ops = &udmabuf_vm_ops;	68	vma->vm_ops = &udmabuf_vm_ops;
58	vma->vm_private_data = ubuf;	69	vma->vm_private_data = ubuf;
		70	vm_flags_set(vma, VM_PFNMAP \| VM_DONTEXPAND \| VM_DONTDUMP);
59	return 0;	71	return 0;
60	}	72	}
61		73
62	static int vmap_udmabuf(struct dma_buf buf, struct iosys_map map)	74	static int vmap_udmabuf(struct dma_buf buf, struct iosys_map map)
63	{	75	{
64	struct udmabuf *ubuf = buf->priv;	76	struct udmabuf *ubuf = buf->priv;
		77	struct page **pages;
65	void *vaddr;	78	void *vaddr;
		79	pgoff_t pg;
66		80
67	dma_resv_assert_held(buf->resv);	81	dma_resv_assert_held(buf->resv);
68		82
69	vaddr = vm_map_ram(ubuf->pages, ubuf->pagecount, -1);	83	pages = kmalloc_array(ubuf->pagecount, sizeof(*pages), GFP_KERNEL);
		84	if (!pages)
		85	return -ENOMEM;
		86
		87	for (pg = 0; pg < ubuf->pagecount; pg++)
		88	pages[pg] = &ubuf->folios[pg]->page;
		89
		90	vaddr = vm_map_ram(pages, ubuf->pagecount, -1);
		91	kfree(pages);
70	if (!vaddr)	92	if (!vaddr)
71	return -EINVAL;	93	return -EINVAL;
72		94
@@ -88,23 +110,30 @@ static struct sg_table get_sg_table(struct device dev, struct dma_buf *buf, · 15 unmodified lines
88	{	110	{
89	struct udmabuf *ubuf = buf->priv;	111	struct udmabuf *ubuf = buf->priv;
90	struct sg_table *sg;	112	struct sg_table *sg;
		113	struct scatterlist *sgl;
		114	unsigned int i = 0;
91	int ret;	115	int ret;
92		116
93	sg = kzalloc(sizeof(*sg), GFP_KERNEL);	117	sg = kzalloc(sizeof(*sg), GFP_KERNEL);
94	if (!sg)	118	if (!sg)
95	return ERR_PTR(-ENOMEM);	119	return ERR_PTR(-ENOMEM);
96	ret = sg_alloc_table_from_pages(sg, ubuf->pages, ubuf->pagecount,	120
97	0, ubuf->pagecount << PAGE_SHIFT,	121	ret = sg_alloc_table(sg, ubuf->pagecount, GFP_KERNEL);
98	GFP_KERNEL);
99	if (ret < 0)	122	if (ret < 0)
100	goto err;	123	goto err_alloc;
		124
		125	for_each_sg(sg->sgl, sgl, ubuf->pagecount, i)
		126	sg_set_folio(sgl, ubuf->folios[i], PAGE_SIZE,
		127	ubuf->offsets[i]);
		128
101	ret = dma_map_sgtable(dev, sg, direction, 0);	129	ret = dma_map_sgtable(dev, sg, direction, 0);
102	if (ret < 0)	130	if (ret < 0)
103	goto err;	131	goto err_map;
104	return sg;	132	return sg;
105		133
106	err:	134	err_map:
107	sg_free_table(sg);	135	sg_free_table(sg);
		136	err_alloc:
108	kfree(sg);	137	kfree(sg);
109	return ERR_PTR(ret);	138	return ERR_PTR(ret);
110	}	139	}
@@ -130,18 +159,45 @@ static void unmap_udmabuf(struct dma_buf_attachment *at, · 19 unmodified lines
130	return put_sg_table(at->dev, sg, direction);	159	return put_sg_table(at->dev, sg, direction);
131	}	160	}
132		161
		162	static void unpin_all_folios(struct list_head *unpin_list)
		163	{
		164	struct udmabuf_folio *ubuf_folio;
		165
		166	while (!list_empty(unpin_list)) {
		167	ubuf_folio = list_first_entry(unpin_list,
		168	struct udmabuf_folio, list);
		169	unpin_folio(ubuf_folio->folio);
		170
		171	list_del(&ubuf_folio->list);
		172	kfree(ubuf_folio);
		173	}
		174	}
		175
		176	static int add_to_unpin_list(struct list_head *unpin_list,
		177	struct folio *folio)
		178	{
		179	struct udmabuf_folio *ubuf_folio;
		180
		181	ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL);
		182	if (!ubuf_folio)
		183	return -ENOMEM;
		184
		185	ubuf_folio->folio = folio;
		186	list_add_tail(&ubuf_folio->list, unpin_list);
		187	return 0;
		188	}
		189
133	static void release_udmabuf(struct dma_buf *buf)	190	static void release_udmabuf(struct dma_buf *buf)
134	{	191	{
135	struct udmabuf *ubuf = buf->priv;	192	struct udmabuf *ubuf = buf->priv;
136	struct device *dev = ubuf->device->this_device;	193	struct device *dev = ubuf->device->this_device;
137	pgoff_t pg;
138		194
139	if (ubuf->sg)	195	if (ubuf->sg)
140	put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);	196	put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL);
141		197
142	for (pg = 0; pg < ubuf->pagecount; pg++)	198	unpin_all_folios(&ubuf->unpin_list);
143	put_page(ubuf->pages[pg]);	199	kfree(ubuf->offsets);
144	kfree(ubuf->pages);	200	kfree(ubuf->folios);
145	kfree(ubuf);	201	kfree(ubuf);
146	}	202	}
147		203
@@ -194,24 +250,64 @@ static const struct dma_buf_ops udmabuf_ops = { · 46 unmodified lines
194	#define SEALS_WANTED (F_SEAL_SHRINK)	250	#define SEALS_WANTED (F_SEAL_SHRINK)
195	#define SEALS_DENIED (F_SEAL_WRITE)	251	#define SEALS_DENIED (F_SEAL_WRITE)
196		252
		253	static int check_memfd_seals(struct file *memfd)
		254	{
		255	int seals;
		256
		257	if (!memfd)
		258	return -EBADFD;
		259
		260	if (!shmem_file(memfd) && !is_file_hugepages(memfd))
		261	return -EBADFD;
		262
		263	seals = memfd_fcntl(memfd, F_GET_SEALS, 0);
		264	if (seals == -EINVAL)
		265	return -EBADFD;
		266
		267	if ((seals & SEALS_WANTED) != SEALS_WANTED \|\|
		268	(seals & SEALS_DENIED) != 0)
		269	return -EINVAL;
		270
		271	return 0;
		272	}
		273
		274	static int export_udmabuf(struct udmabuf *ubuf,
		275	struct miscdevice *device,
		276	u32 flags)
		277	{
		278	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
		279	struct dma_buf *buf;
		280
		281	ubuf->device = device;
		282	exp_info.ops = &udmabuf_ops;
		283	exp_info.size = ubuf->pagecount << PAGE_SHIFT;
		284	exp_info.priv = ubuf;
		285	exp_info.flags = O_RDWR;
		286
		287	buf = dma_buf_export(&exp_info);
		288	if (IS_ERR(buf))
		289	return PTR_ERR(buf);
		290
		291	return dma_buf_fd(buf, flags);
		292	}
		293
197	static long udmabuf_create(struct miscdevice *device,	294	static long udmabuf_create(struct miscdevice *device,
198	struct udmabuf_create_list *head,	295	struct udmabuf_create_list *head,
199	struct udmabuf_create_item *list)	296	struct udmabuf_create_item *list)
200	{	297	{
201	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);	298	pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0;
		299	long nr_folios, ret = -EINVAL;
202	struct file *memfd = NULL;	300	struct file *memfd = NULL;
203	struct address_space *mapping = NULL;	301	struct folio **folios;
204	struct udmabuf *ubuf;	302	struct udmabuf *ubuf;
205	struct dma_buf *buf;	303	u32 i, j, k, flags;
206	pgoff_t pgoff, pgcnt, pgidx, pgbuf = 0, pglimit;	304	loff_t end;
207	struct page *page;
208	int seals, ret = -EINVAL;
209	u32 i, flags;
210		305
211	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);	306	ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL);
212	if (!ubuf)	307	if (!ubuf)
213	return -ENOMEM;	308	return -ENOMEM;
214		309
		310	INIT_LIST_HEAD(&ubuf->unpin_list);
215	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;	311	pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT;
216	for (i = 0; i < head->count; i++) {	312	for (i = 0; i < head->count; i++) {
217	if (!IS_ALIGNED(list[i].offset, PAGE_SIZE))	313	if (!IS_ALIGNED(list[i].offset, PAGE_SIZE))
@@ -226,66 +322,84 @@ static long udmabuf_create(struct miscdevice *device, · 8 unmodified lines
226	if (!ubuf->pagecount)	322	if (!ubuf->pagecount)
227	goto err;	323	goto err;
228		324
229	ubuf->pages = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->pages),	325	ubuf->folios = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->folios),
230	GFP_KERNEL);	326	GFP_KERNEL);
231	if (!ubuf->pages) {	327	if (!ubuf->folios) {
		328	ret = -ENOMEM;
		329	goto err;
		330	}
		331	ubuf->offsets = kcalloc(ubuf->pagecount, sizeof(*ubuf->offsets),
		332	GFP_KERNEL);
		333	if (!ubuf->offsets) {
232	ret = -ENOMEM;	334	ret = -ENOMEM;
233	goto err;	335	goto err;
234	}	336	}
235		337
236	pgbuf = 0;	338	pgbuf = 0;
237	for (i = 0; i < head->count; i++) {	339	for (i = 0; i < head->count; i++) {
238	ret = -EBADFD;
239	memfd = fget(list[i].memfd);	340	memfd = fget(list[i].memfd);
240	if (!memfd)	341	ret = check_memfd_seals(memfd);
		342	if (ret < 0)
241	goto err;	343	goto err;
242	mapping = memfd->f_mapping;	344
243	if (!shmem_mapping(mapping))	345	pgcnt = list[i].size >> PAGE_SHIFT;
244	goto err;	346	folios = kmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL);
245	seals = memfd_fcntl(memfd, F_GET_SEALS, 0);	347	if (!folios) {
246	if (seals == -EINVAL)	348	ret = -ENOMEM;
247	goto err;	349	goto err;
248	ret = -EINVAL;	350	}
249	if ((seals & SEALS_WANTED) != SEALS_WANTED \|\|	351
250	(seals & SEALS_DENIED) != 0)	352	end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1;
		353	ret = memfd_pin_folios(memfd, list[i].offset, end,
		354	folios, pgcnt, &pgoff);
		355	if (ret <= 0) {
		356	kfree(folios);
		357	if (!ret)
		358	ret = -EINVAL;
251	goto err;	359	goto err;
252	pgoff = list[i].offset >> PAGE_SHIFT;	360	}
253	pgcnt = list[i].size >> PAGE_SHIFT;	361
254	for (pgidx = 0; pgidx < pgcnt; pgidx++) {	362	nr_folios = ret;
255	page = shmem_read_mapping_page(mapping, pgoff + pgidx);	363	pgoff >>= PAGE_SHIFT;
256	if (IS_ERR(page)) {	364	for (j = 0, k = 0; j < pgcnt; j++) {
257	ret = PTR_ERR(page);	365	ubuf->folios[pgbuf] = folios[k];
258	goto err;	366	ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT;
		367
		368	if (j == 0 \|\| ubuf->folios[pgbuf-1] != folios[k]) {
		369	ret = add_to_unpin_list(&ubuf->unpin_list,
		370	folios[k]);
		371	if (ret < 0) {
		372	kfree(folios);
		373	goto err;
		374	}
		375	}
		376
		377	pgbuf++;
		378	if (++pgoff == folio_nr_pages(folios[k])) {
		379	pgoff = 0;
		380	if (++k == nr_folios)
		381	break;
259	}	382	}
260	ubuf->pages[pgbuf++] = page;
261	}	383	}
		384
		385	kfree(folios);
262	fput(memfd);	386	fput(memfd);
263	memfd = NULL;	387	memfd = NULL;
264	}	388	}
265		389
266	exp_info.ops = &udmabuf_ops;	390	flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0;
267	exp_info.size = ubuf->pagecount << PAGE_SHIFT;	391	ret = export_udmabuf(ubuf, device, flags);
268	exp_info.priv = ubuf;	392	if (ret < 0)
269	exp_info.flags = O_RDWR;
270
271	ubuf->device = device;
272	buf = dma_buf_export(&exp_info);
273	if (IS_ERR(buf)) {
274	ret = PTR_ERR(buf);
275	goto err;	393	goto err;
276	}
277		394
278	flags = 0;	395	return ret;
279	if (head->flags & UDMABUF_FLAGS_CLOEXEC)
280	flags \|= O_CLOEXEC;
281	return dma_buf_fd(buf, flags);
282		396
283	err:	397	err:
284	while (pgbuf > 0)
285	put_page(ubuf->pages[--pgbuf]);
286	if (memfd)	398	if (memfd)
287	fput(memfd);	399	fput(memfd);
288	kfree(ubuf->pages);	400	unpin_all_folios(&ubuf->unpin_list);
		401	kfree(ubuf->offsets);
		402	kfree(ubuf->folios);
289	kfree(ubuf);	403	kfree(ubuf);
290	return ret;	404	return ret;
291	}	405	}

drivers/hv/hv_balloon.c+2−3

 		if (!PageOffline(pg))
 			__SetPageOffline(pg);
 		return;
-	}
+	} else if (!PageOffline(pg))
-	if (PageOffline(pg))
+		return;
-		__ClearPageOffline(pg);
 	/* This frame is currently backed; online the page. */
 	generic_online_page(pg, 0);

drivers/s390/char/sclp.c+1−1

@@ -736,7 +736,7 @@ sclp_sync_wait(void) · 735 unmodified lines
736	cr0_sync.val = cr0.val & ~CR0_IRQ_SUBCLASS_MASK;	736	cr0_sync.val = cr0.val & ~CR0_IRQ_SUBCLASS_MASK;
737	cr0_sync.val \|= 1UL << (63 - 54);	737	cr0_sync.val \|= 1UL << (63 - 54);
738	local_ctl_load(0, &cr0_sync);	738	local_ctl_load(0, &cr0_sync);
739	__arch_local_irq_stosm(0x01);	739	arch_local_irq_enable_external();
740	/* Loop until driver state indicates finished request */	740	/* Loop until driver state indicates finished request */
741	while (sclp_running_state != sclp_running_state_idle) {	741	while (sclp_running_state != sclp_running_state_idle) {
742	/* Check for expired request timer */	742	/* Check for expired request timer */

drivers/video/fbdev/core/fb_defio.c+7−6

@@ -146,7 +146,7 @@ static vm_fault_t fb_deferred_io_fault(struct vm_fault *vmf) · 145 unmodified lines
146	printk(KERN_ERR "no mapping available\n");	146	printk(KERN_ERR "no mapping available\n");
147		147
148	BUG_ON(!page->mapping);	148	BUG_ON(!page->mapping);
149	page->index = vmf->pgoff; /* for page_mkclean() */	149	page->index = vmf->pgoff; /* for folio_mkclean() */
150		150
151	vmf->page = page;	151	vmf->page = page;
152	return 0;	152	return 0;
@@ -194,7 +194,7 @@ static vm_fault_t fb_deferred_io_track_page(struct fb_info *info, unsigned long · 41 unmodified lines
194		194
195	/*	195	/*
196	* We want the page to remain locked from ->page_mkwrite until	196	* We want the page to remain locked from ->page_mkwrite until
197	* the PTE is marked dirty to avoid page_mkclean() being called	197	* the PTE is marked dirty to avoid folio_mkclean() being called
198	* before the PTE is updated, which would leave the page ignored	198	* before the PTE is updated, which would leave the page ignored
199	* by defio.	199	* by defio.
200	* Do this by locking the page here and informing the caller	200	* Do this by locking the page here and informing the caller
@@ -277,10 +277,11 @@ static void fb_deferred_io_work(struct work_struct *work) · 76 unmodified lines
277	/* here we mkclean the pages, then do all deferred IO */	277	/* here we mkclean the pages, then do all deferred IO */
278	mutex_lock(&fbdefio->lock);	278	mutex_lock(&fbdefio->lock);
279	list_for_each_entry(pageref, &fbdefio->pagereflist, list) {	279	list_for_each_entry(pageref, &fbdefio->pagereflist, list) {
280	struct page *cur = pageref->page;	280	struct folio *folio = page_folio(pageref->page);
281	lock_page(cur);	281
282	page_mkclean(cur);	282	folio_lock(folio);
283	unlock_page(cur);	283	folio_mkclean(folio);
		284	folio_unlock(folio);
284	}	285	}
285		286
286	/* driver's callback with pagereflist */	287	/* driver's callback with pagereflist */

drivers/virtio/virtio_mem.c+14−15

 	for (; nr_pages--; pfn++) {
 		struct page *page = pfn_to_page(pfn);
-		__SetPageOffline(page);
+		if (!onlined)
-		if (!onlined) {
+			/*
+			 * Pages that have not been onlined yet were initialized
+			 * to PageOffline(). Remember that we have to route them
+			 * through generic_online_page().
+			 */
 			SetPageDirty(page);
-			/* FIXME: remove after cleanups */
+		else
-			ClearPageReserved(page);
+			__SetPageOffline(page);
-		}
+		VM_WARN_ON_ONCE(!PageOffline(page));
 	}
 	page_offline_end();
 }
 	for (; nr_pages--; pfn++) {
 		struct page *page = pfn_to_page(pfn);
-		__ClearPageOffline(page);
 		if (!onlined)
+			/* generic_online_page() will clear PageOffline(). */
 			ClearPageDirty(page);
+		else
+			__ClearPageOffline(page);
 	}
 }
 	struct page *page;
 	unsigned long i;
-	/*
-	 * Drop our reference to the pages so the memory can get offlined
-	 * and add the unplugged pages to the managed page counters (so
-	 * offlining code can correctly subtract them again).
-	 */
-	adjust_managed_page_count(pfn_to_page(pfn), nr_pages);
 	/* Drop our reference to the pages so the memory can get offlined. */
 	for (i = 0; i < nr_pages; i++) {
 		page = pfn_to_page(pfn + i);
 	unsigned long i;
 	/*
-	 * Get the reference we dropped when going offline and subtract the
+	 * Get the reference again that we dropped via page_ref_dec_and_test()
-	 * unplugged pages from the managed page counters.
+	 * when going offline.
 	 */
-	adjust_managed_page_count(pfn_to_page(pfn), -nr_pages);
 	for (i = 0; i < nr_pages; i++)
 		page_ref_inc(pfn_to_page(pfn + i));
 }

drivers/xen/balloon.c+7−2

 /* balloon_append: add the given page to the balloon. */
 static void balloon_append(struct page *page)
 {
-	__SetPageOffline(page);
+	if (!PageOffline(page))
+		__SetPageOffline(page);
 	/* Lowmem is re-populated first, so highmem pages go at list tail. */
 	if (PageHighMem(page)) {
 		xenmem_reservation_va_mapping_update(1, &page, &frame_list[i]);
-		/* Relinquish the page back to the allocator. */
+		/*
+		 * Relinquish the page back to the allocator. Note that
+		 * some pages, including ones added via xen_online_page(), might
+		 * not be marked reserved; free_reserved_page() will handle that.
+		 */
 		free_reserved_page(page);
 	}

fs/afs/dir.c+3−3

@@ -533,14 +533,14 @@ static int afs_dir_iterate(struct inode dir, struct dir_context ctx, · 532 unmodified lines
533	break;	533	break;
534	}	534	}
535		535
536	offset = round_down(ctx->pos, sizeof(*dblock)) - folio_file_pos(folio);	536	offset = round_down(ctx->pos, sizeof(*dblock)) - folio_pos(folio);
537	size = min_t(loff_t, folio_size(folio),	537	size = min_t(loff_t, folio_size(folio),
538	req->actual_len - folio_file_pos(folio));	538	req->actual_len - folio_pos(folio));
539		539
540	do {	540	do {
541	dblock = kmap_local_folio(folio, offset);	541	dblock = kmap_local_folio(folio, offset);
542	ret = afs_dir_iterate_block(dvnode, ctx, dblock,	542	ret = afs_dir_iterate_block(dvnode, ctx, dblock,
543	folio_file_pos(folio) + offset);	543	folio_pos(folio) + offset);
544	kunmap_local(dblock);	544	kunmap_local(dblock);
545	if (ret != 1)	545	if (ret != 1)
546	goto out;	546	goto out;

fs/afs/dir_edit.c+2−2

@@ -256,7 +256,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode, · 255 unmodified lines
256	folio = folio0;	256	folio = folio0;
257	}	257	}
258		258
259	block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio));	259	block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_pos(folio));
260		260
261	/* Abandon the edit if we got a callback break. */	261	/* Abandon the edit if we got a callback break. */
262	if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))	262	if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))
@@ -417,7 +417,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode, · 154 unmodified lines
417	folio = folio0;	417	folio = folio0;
418	}	418	}
419		419
420	block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio));	420	block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_pos(folio));
421		421
422	/* Abandon the edit if we got a callback break. */	422	/* Abandon the edit if we got a callback break. */
423	if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))	423	if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))

fs/aio.c+3−12

 	struct kioctx *ctx;
 	unsigned long flags;
 	pgoff_t idx;
-	int rc;
+	int rc = 0;
-	/*
-	 * We cannot support the _NO_COPY case here, because copy needs to
-	 * happen under the ctx->completion_lock. That does not work with the
-	 * migration workflow of MIGRATE_SYNC_NO_COPY.
-	 */
-	if (mode == MIGRATE_SYNC_NO_COPY)
-		return -EINVAL;
-	rc = 0;
 	/* mapping->i_private_lock here protects against the kioctx teardown.  */
 	spin_lock(&mapping->i_private_lock);
 	 * events from being lost.
 	 */
 	spin_lock_irqsave(&ctx->completion_lock, flags);
-	folio_migrate_copy(dst, src);
+	folio_copy(dst, src);
+	folio_migrate_flags(dst, src);
 	BUG_ON(ctx->ring_folios[idx] != src);
 	ctx->ring_folios[idx] = dst;
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);

fs/btrfs/relocation.c+1−2

@@ -2981,8 +2981,7 @@ static int relocate_one_folio(struct reloc_control *rc, · 2980 unmodified lines
2981		2981
2982	if (folio_test_readahead(folio))	2982	if (folio_test_readahead(folio))
2983	page_cache_async_readahead(inode->i_mapping, ra, NULL,	2983	page_cache_async_readahead(inode->i_mapping, ra, NULL,
2984	folio, index,	2984	folio, last_index + 1 - index);
2985	last_index + 1 - index);
2986		2985
2987	if (!folio_test_uptodate(folio)) {	2986	if (!folio_test_uptodate(folio)) {
2988	btrfs_read_folio(NULL, folio);	2987	btrfs_read_folio(NULL, folio);

fs/btrfs/send.c+1−1

@@ -5306,7 +5306,7 @@ static int put_file_data(struct send_ctx *sctx, u64 offset, u32 len) · 5305 unmodified lines
5306		5306
5307	if (folio_test_readahead(folio))	5307	if (folio_test_readahead(folio))
5308	page_cache_async_readahead(mapping, &sctx->ra, NULL, folio,	5308	page_cache_async_readahead(mapping, &sctx->ra, NULL, folio,
5309	index, last_index + 1 - index);	5309	last_index + 1 - index);
5310		5310
5311	if (!folio_test_uptodate(folio)) {	5311	if (!folio_test_uptodate(folio)) {
5312	btrfs_read_folio(NULL, folio);	5312	btrfs_read_folio(NULL, folio);

fs/ceph/dir.c+1−1

@@ -141,7 +141,7 @@ __dcache_find_get_entry(struct dentry *parent, u64 idx, · 140 unmodified lines
141	if (ptr_pos >= i_size_read(dir))	141	if (ptr_pos >= i_size_read(dir))
142	return NULL;	142	return NULL;
143		143
144	if (!cache_ctl->page \|\| ptr_pgoff != page_index(cache_ctl->page)) {	144	if (!cache_ctl->page \|\| ptr_pgoff != cache_ctl->page->index) {
145	ceph_readdir_cache_release(cache_ctl);	145	ceph_readdir_cache_release(cache_ctl);
146	cache_ctl->page = find_lock_page(&dir->i_data, ptr_pgoff);	146	cache_ctl->page = find_lock_page(&dir->i_data, ptr_pgoff);
147	if (!cache_ctl->page) {	147	if (!cache_ctl->page) {

fs/ceph/inode.c+1−1

@@ -1863,7 +1863,7 @@ static int fill_readdir_cache(struct inode dir, struct dentry dn, · 1862 unmodified lines
1863	unsigned idx = ctl->index % nsize;	1863	unsigned idx = ctl->index % nsize;
1864	pgoff_t pgoff = ctl->index / nsize;	1864	pgoff_t pgoff = ctl->index / nsize;
1865		1865
1866	if (!ctl->page \|\| pgoff != page_index(ctl->page)) {	1866	if (!ctl->page \|\| pgoff != ctl->page->index) {
1867	ceph_readdir_cache_release(ctl);	1867	ceph_readdir_cache_release(ctl);
1868	if (idx == 0)	1868	if (idx == 0)
1869	ctl->page = grab_cache_page(&dir->i_data, pgoff);	1869	ctl->page = grab_cache_page(&dir->i_data, pgoff);

fs/hugetlbfs/inode.c+9−11

 				  unsigned long flags)
 {
 	struct mm_struct *mm = current->mm;
-	struct vm_area_struct *vma;
+	struct vm_area_struct *vma, *prev;
 	struct hstate *h = hstate_file(file);
 	const unsigned long mmap_end = arch_get_mmap_end(addr, len, flags);
 	if (len & ~huge_page_mask(h))
 		return -EINVAL;
-	if (len > TASK_SIZE)
+	if (len > mmap_end - mmap_min_addr)
 		return -ENOMEM;
 	if (flags & MAP_FIXED) {
 	if (addr) {
 		addr = ALIGN(addr, huge_page_size(h));
-		vma = find_vma(mm, addr);
+		vma = find_vma_prev(mm, addr, &prev);
-		if (mmap_end - len >= addr &&
+		if (mmap_end - len >= addr && addr >= mmap_min_addr &&
-		    (!vma || addr + len <= vm_start_gap(vma)))
+		    (!vma || addr + len <= vm_start_gap(vma)) &&
+		    (!prev || addr >= vm_end_gap(prev)))
 			return addr;
 	}
 	if (!ptep)
 		return false;
-	pte = huge_ptep_get(ptep);
+	pte = huge_ptep_get(vma->vm_mm, addr, ptep);
 	if (huge_pte_none(pte) || !pte_present(pte))
 		return false;
 			error = PTR_ERR(folio);
 			goto out;
 		}
-		clear_huge_page(&folio->page, addr, pages_per_huge_page(h));
+		folio_zero_user(folio, ALIGN_DOWN(addr, hpage_size));
 		__folio_mark_uptodate(folio);
 		error = hugetlb_add_to_page_cache(folio, mapping, index);
 		if (unlikely(error)) {
 		hugetlb_set_folio_subpool(src, NULL);
 	}
-	if (mode != MIGRATE_SYNC_NO_COPY)
+	folio_migrate_flags(dst, src);
-		folio_migrate_copy(dst, src);
-	else
-		folio_migrate_flags(dst, src);
 	return MIGRATEPAGE_SUCCESS;
 }

fs/netfs/buffered_read.c+2−2

@@ -271,7 +271,7 @@ int netfs_read_folio(struct file file, struct folio folio) · 270 unmodified lines
271	kenter("%lx", folio->index);	271	kenter("%lx", folio->index);
272		272
273	rreq = netfs_alloc_request(mapping, file,	273	rreq = netfs_alloc_request(mapping, file,
274	folio_file_pos(folio), folio_size(folio),	274	folio_pos(folio), folio_size(folio),
275	NETFS_READPAGE);	275	NETFS_READPAGE);
276	if (IS_ERR(rreq)) {	276	if (IS_ERR(rreq)) {
277	ret = PTR_ERR(rreq);	277	ret = PTR_ERR(rreq);
@@ -470,7 +470,7 @@ retry: · 192 unmodified lines
470	}	470	}
471		471
472	rreq = netfs_alloc_request(mapping, file,	472	rreq = netfs_alloc_request(mapping, file,
473	folio_file_pos(folio), folio_size(folio),	473	folio_pos(folio), folio_size(folio),
474	NETFS_READ_FOR_WRITE);	474	NETFS_READ_FOR_WRITE);
475	if (IS_ERR(rreq)) {	475	if (IS_ERR(rreq)) {
476	ret = PTR_ERR(rreq);	476	ret = PTR_ERR(rreq);

fs/netfs/buffered_write.c+1−1

@@ -54,7 +54,7 @@ static enum netfs_how_to_modify netfs_how_to_modify(struct netfs_inode *ctx, · 53 unmodified lines
54	{	54	{
55	struct netfs_folio *finfo = netfs_folio_info(folio);	55	struct netfs_folio *finfo = netfs_folio_info(folio);
56	struct netfs_group *group = netfs_folio_group(folio);	56	struct netfs_group *group = netfs_folio_group(folio);
57	loff_t pos = folio_file_pos(folio);	57	loff_t pos = folio_pos(folio);
58		58
59	kenter("");	59	kenter("");
60		60

fs/nfs/file.c+1−1

@@ -591,7 +591,7 @@ static vm_fault_t nfs_vm_page_mkwrite(struct vm_fault *vmf) · 590 unmodified lines
591		591
592	dfprintk(PAGECACHE, "NFS: vm_page_mkwrite(%pD2(%lu), offset %lld)\n",	592	dfprintk(PAGECACHE, "NFS: vm_page_mkwrite(%pD2(%lu), offset %lld)\n",
593	filp, filp->f_mapping->host->i_ino,	593	filp, filp->f_mapping->host->i_ino,
594	(long long)folio_file_pos(folio));	594	(long long)folio_pos(folio));
595		595
596	sb_start_pagefault(inode->i_sb);	596	sb_start_pagefault(inode->i_sb);
597		597

fs/nfs/iostat.h+4−0

 	nfs_add_server_stats(NFS_SERVER(inode), stat, addend);
 }
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag).
+ */
 #define nfs_alloc_iostats()	alloc_percpu(struct nfs_iostats)
 static inline void nfs_free_iostats(struct nfs_iostats __percpu *stats)

fs/nfs/write.c+2−2

@@ -182,7 +182,7 @@ static void nfs_grow_file(struct folio *folio, unsigned int offset, · 181 unmodified lines
182	end_index = ((i_size - 1) >> folio_shift(folio)) << folio_order(folio);	182	end_index = ((i_size - 1) >> folio_shift(folio)) << folio_order(folio);
183	if (i_size > 0 && folio->index < end_index)	183	if (i_size > 0 && folio->index < end_index)
184	goto out;	184	goto out;
185	end = folio_file_pos(folio) + (loff_t)offset + (loff_t)count;	185	end = folio_pos(folio) + (loff_t)offset + (loff_t)count;
186	if (i_size >= end)	186	if (i_size >= end)
187	goto out;	187	goto out;
188	trace_nfs_size_grow(inode, end);	188	trace_nfs_size_grow(inode, end);
@@ -1344,7 +1344,7 @@ int nfs_update_folio(struct file file, struct folio folio, · 1155 unmodified lines
1344	nfs_inc_stats(inode, NFSIOS_VFSUPDATEPAGE);	1344	nfs_inc_stats(inode, NFSIOS_VFSUPDATEPAGE);
1345		1345
1346	dprintk("NFS: nfs_update_folio(%pD2 %d@%lld)\n", file, count,	1346	dprintk("NFS: nfs_update_folio(%pD2 %d@%lld)\n", file, count,
1347	(long long)(folio_file_pos(folio) + offset));	1347	(long long)(folio_pos(folio) + offset));
1348		1348
1349	if (!count)	1349	if (!count)
1350	goto out;	1350	goto out;

fs/nilfs2/bmap.c+2−8

 __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap,
 			      const struct buffer_head *bh)
 {
-	struct buffer_head *pbh;
+	loff_t pos = folio_pos(bh->b_folio) + bh_offset(bh);
-	__u64 key;
-	key = page_index(bh->b_page) << (PAGE_SHIFT -
+	return pos >> bmap->b_inode->i_blkbits;
-					 bmap->b_inode->i_blkbits);
-	for (pbh = page_buffers(bh->b_page); pbh != bh; pbh = pbh->b_this_page)
-		key++;
-	return key;
 }
 __u64 nilfs_bmap_find_target_seq(const struct nilfs_bmap *bmap, __u64 key)

fs/proc/internal.h+33−0

@@ -13,6 +13,7 @@ · 12 unmodified lines
13	#include <linux/binfmts.h>	13	#include <linux/binfmts.h>
14	#include <linux/sched/coredump.h>	14	#include <linux/sched/coredump.h>
15	#include <linux/sched/task.h>	15	#include <linux/sched/task.h>
		16	#include <linux/mm.h>
16		17
17	struct ctl_table_header;	18	struct ctl_table_header;
18	struct mempolicy;	19	struct mempolicy;
@@ -142,6 +143,38 @@ unsigned name_to_int(const struct qstr *qstr); · 123 unmodified lines
142	/* Worst case buffer size needed for holding an integer. */	143	/* Worst case buffer size needed for holding an integer. */
143	#define PROC_NUMBUF 13	144	#define PROC_NUMBUF 13
144		145
		146	/**
		147	* folio_precise_page_mapcount() - Number of mappings of this folio page.
		148	* @folio: The folio.
		149	* @page: The page.
		150	*
		151	* The number of present user page table entries that reference this page
		152	* as tracked via the RMAP: either referenced directly (PTE) or as part of
		153	* a larger area that covers this page (e.g., PMD).
		154	*
		155	* Use this function only for the calculation of existing statistics
		156	* (USS, PSS, mapcount_max) and for debugging purposes (/proc/kpagecount).
		157	*
		158	* Do not add new users.
		159	*
		160	* Returns: The number of mappings of this folio page. 0 for
		161	* folios that are not mapped to user space or are not tracked via the RMAP
		162	* (e.g., shared zeropage).
		163	*/
		164	static inline int folio_precise_page_mapcount(struct folio *folio,
		165	struct page *page)
		166	{
		167	int mapcount = atomic_read(&page->_mapcount) + 1;
		168
		169	/* Handle page_has_type() pages */
		170	if (mapcount < PAGE_MAPCOUNT_RESERVE + 1)
		171	mapcount = 0;
		172	if (folio_test_large(folio))
		173	mapcount += folio_entire_mapcount(folio);
		174
		175	return mapcount;
		176	}
		177
145	/*	178	/*
146	* array.c	179	* array.c
147	*/	180	*/

fs/proc/page.c+19−23

@@ -37,21 +37,19 @@ static inline unsigned long get_max_dump_pfn(void) · 36 unmodified lines
37	#endif	37	#endif
38	}	38	}
39		39
40	/* /proc/kpagecount - an array exposing page counts	40	/* /proc/kpagecount - an array exposing page mapcounts
41	*	41	*
42	* Each entry is a u64 representing the corresponding	42	* Each entry is a u64 representing the corresponding
43	* physical page count.	43	* physical page mapcount.
44	*/	44	*/
45	static ssize_t kpagecount_read(struct file file, char __user buf,	45	static ssize_t kpagecount_read(struct file file, char __user buf,
46	size_t count, loff_t *ppos)	46	size_t count, loff_t *ppos)
47	{	47	{
48	const unsigned long max_dump_pfn = get_max_dump_pfn();	48	const unsigned long max_dump_pfn = get_max_dump_pfn();
49	u64 __user out = (u64 __user )buf;	49	u64 __user out = (u64 __user )buf;
50	struct page *ppage;
51	unsigned long src = *ppos;	50	unsigned long src = *ppos;
52	unsigned long pfn;	51	unsigned long pfn;
53	ssize_t ret = 0;	52	ssize_t ret = 0;
54	u64 pcount;
55		53
56	pfn = src / KPMSIZE;	54	pfn = src / KPMSIZE;
57	if (src & KPMMASK \|\| count & KPMMASK)	55	if (src & KPMMASK \|\| count & KPMMASK)
@@ -61,18 +59,19 @@ static ssize_t kpagecount_read(struct file file, char __user buf, · 3 unmodified lines
61	count = min_t(unsigned long, count, (max_dump_pfn * KPMSIZE) - src);	59	count = min_t(unsigned long, count, (max_dump_pfn * KPMSIZE) - src);
62		60
63	while (count > 0) {	61	while (count > 0) {
		62	struct page *page;
		63	u64 mapcount = 0;
		64
64	/*	65	/*
65	* TODO: ZONE_DEVICE support requires to identify	66	* TODO: ZONE_DEVICE support requires to identify
66	* memmaps that were actually initialized.	67	* memmaps that were actually initialized.
67	*/	68	*/
68	ppage = pfn_to_online_page(pfn);	69	page = pfn_to_online_page(pfn);
69		70	if (page)
70	if (!ppage)	71	mapcount = folio_precise_page_mapcount(page_folio(page),
71	pcount = 0;	72	page);
72	else
73	pcount = page_mapcount(ppage);
74		73
75	if (put_user(pcount, out)) {	74	if (put_user(mapcount, out)) {
76	ret = -EFAULT;	75	ret = -EFAULT;
77	break;	76	break;
78	}	77	}
@@ -148,19 +147,16 @@ u64 stable_page_flags(const struct page *page) · 69 unmodified lines
148	u \|= 1 << KPF_COMPOUND_TAIL;	147	u \|= 1 << KPF_COMPOUND_TAIL;
149	if (folio_test_hugetlb(folio))	148	if (folio_test_hugetlb(folio))
150	u \|= 1 << KPF_HUGE;	149	u \|= 1 << KPF_HUGE;
151	/*	150	else if (folio_test_large(folio) &&
152	* We need to check PageLRU/PageAnon	151	folio_test_large_rmappable(folio)) {
153	* to make sure a given page is a thp, not a non-huge compound page.	152	/* Note: we indicate any THPs here, not just PMD-sized ones */
154	*/	153	u \|= 1 << KPF_THP;
155	else if (folio_test_large(folio)) {	154	} else if (is_huge_zero_folio(folio)) {
156	if ((k & (1 << PG_lru)) \|\| is_anon)
157	u \|= 1 << KPF_THP;
158	else if (is_huge_zero_folio(folio)) {
159	u \|= 1 << KPF_ZERO_PAGE;
160	u \|= 1 << KPF_THP;
161	}
162	} else if (is_zero_pfn(page_to_pfn(page)))
163	u \|= 1 << KPF_ZERO_PAGE;	155	u \|= 1 << KPF_ZERO_PAGE;
		156	u \|= 1 << KPF_THP;
		157	} else if (is_zero_folio(folio)) {
		158	u \|= 1 << KPF_ZERO_PAGE;
		159	}
164		160
165	/*	161	/*
166	* Caveats on high order pages: PG_buddy and PG_slab will only be set	162	* Caveats on high order pages: PG_buddy and PG_slab will only be set

fs/proc/task_mmu.c+398−105

@@ -22,6 +22,7 @@ · 21 unmodified lines
22	#include <linux/pkeys.h>	22	#include <linux/pkeys.h>
23	#include <linux/minmax.h>	23	#include <linux/minmax.h>
24	#include <linux/overflow.h>	24	#include <linux/overflow.h>
		25	#include <linux/buildid.h>
25		26
26	#include <asm/elf.h>	27	#include <asm/elf.h>
27	#include <asm/tlb.h>	28	#include <asm/tlb.h>
@@ -239,6 +240,67 @@ static int do_maps_open(struct inode inode, struct file file, · 211 unmodified lines
239	sizeof(struct proc_maps_private));	240	sizeof(struct proc_maps_private));
240	}	241	}
241		242
		243	static void get_vma_name(struct vm_area_struct *vma,
		244	const struct path **path,
		245	const char **name,
		246	const char **name_fmt)
		247	{
		248	struct anon_vma_name *anon_name = vma->vm_mm ? anon_vma_name(vma) : NULL;
		249
		250	*name = NULL;
		251	*path = NULL;
		252	*name_fmt = NULL;
		253
		254	/*
		255	* Print the dentry name for named mappings, and a
		256	* special [heap] marker for the heap:
		257	*/
		258	if (vma->vm_file) {
		259	/*
		260	* If user named this anon shared memory via
		261	* prctl(PR_SET_VMA ..., use the provided name.
		262	*/
		263	if (anon_name) {
		264	*name_fmt = "[anon_shmem:%s]";
		265	*name = anon_name->name;
		266	} else {
		267	*path = file_user_path(vma->vm_file);
		268	}
		269	return;
		270	}
		271
		272	if (vma->vm_ops && vma->vm_ops->name) {
		273	*name = vma->vm_ops->name(vma);
		274	if (*name)
		275	return;
		276	}
		277
		278	*name = arch_vma_name(vma);
		279	if (*name)
		280	return;
		281
		282	if (!vma->vm_mm) {
		283	*name = "[vdso]";
		284	return;
		285	}
		286
		287	if (vma_is_initial_heap(vma)) {
		288	*name = "[heap]";
		289	return;
		290	}
		291
		292	if (vma_is_initial_stack(vma)) {
		293	*name = "[stack]";
		294	return;
		295	}
		296
		297	if (anon_name) {
		298	*name_fmt = "[anon:%s]";
		299	*name = anon_name->name;
		300	return;
		301	}
		302	}
		303
242	static void show_vma_header_prefix(struct seq_file *m,	304	static void show_vma_header_prefix(struct seq_file *m,
243	unsigned long start, unsigned long end,	305	unsigned long start, unsigned long end,
244	vm_flags_t flags, unsigned long long pgoff,	306	vm_flags_t flags, unsigned long long pgoff,
@@ -262,17 +324,15 @@ static void show_vma_header_prefix(struct seq_file *m, · 17 unmodified lines
262	static void	324	static void
263	show_map_vma(struct seq_file m, struct vm_area_struct vma)	325	show_map_vma(struct seq_file m, struct vm_area_struct vma)
264	{	326	{
265	struct anon_vma_name *anon_name = NULL;	327	const struct path *path;
266	struct mm_struct *mm = vma->vm_mm;	328	const char name_fmt, name;
267	struct file *file = vma->vm_file;
268	vm_flags_t flags = vma->vm_flags;	329	vm_flags_t flags = vma->vm_flags;
269	unsigned long ino = 0;	330	unsigned long ino = 0;
270	unsigned long long pgoff = 0;	331	unsigned long long pgoff = 0;
271	unsigned long start, end;	332	unsigned long start, end;
272	dev_t dev = 0;	333	dev_t dev = 0;
273	const char *name = NULL;
274		334
275	if (file) {	335	if (vma->vm_file) {
276	const struct inode *inode = file_user_inode(vma->vm_file);	336	const struct inode *inode = file_user_inode(vma->vm_file);
277		337
278	dev = inode->i_sb->s_dev;	338	dev = inode->i_sb->s_dev;
@@ -283,57 +343,15 @@ show_map_vma(struct seq_file m, struct vm_area_struct vma) · 4 unmodified lines
283	start = vma->vm_start;	343	start = vma->vm_start;
284	end = vma->vm_end;	344	end = vma->vm_end;
285	show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino);	345	show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino);
286	if (mm)
287	anon_name = anon_vma_name(vma);
288		346
289	/*	347	get_vma_name(vma, &path, &name, &name_fmt);
290	* Print the dentry name for named mappings, and a	348	if (path) {
291	* special [heap] marker for the heap:
292	*/
293	if (file) {
294	seq_pad(m, ' ');	349	seq_pad(m, ' ');
295	/*	350	seq_path(m, path, "\n");
296	* If user named this anon shared memory via	351	} else if (name_fmt) {
297	* prctl(PR_SET_VMA ..., use the provided name.	352	seq_pad(m, ' ');
298	*/	353	seq_printf(m, name_fmt, name);
299	if (anon_name)	354	} else if (name) {
300	seq_printf(m, "[anon_shmem:%s]", anon_name->name);
301	else
302	seq_path(m, file_user_path(file), "\n");
303	goto done;
304	}
305
306	if (vma->vm_ops && vma->vm_ops->name) {
307	name = vma->vm_ops->name(vma);
308	if (name)
309	goto done;
310	}
311
312	name = arch_vma_name(vma);
313	if (!name) {
314	if (!mm) {
315	name = "[vdso]";
316	goto done;
317	}
318
319	if (vma_is_initial_heap(vma)) {
320	name = "[heap]";
321	goto done;
322	}
323
324	if (vma_is_initial_stack(vma)) {
325	name = "[stack]";
326	goto done;
327	}
328
329	if (anon_name) {
330	seq_pad(m, ' ');
331	seq_printf(m, "[anon:%s]", anon_name->name);
332	}
333	}
334
335	done:
336	if (name) {
337	seq_pad(m, ' ');	355	seq_pad(m, ' ');
338	seq_puts(m, name);	356	seq_puts(m, name);
339	}	357	}
@@ -358,11 +376,268 @@ static int pid_maps_open(struct inode inode, struct file file) · 18 unmodified lines
358	return do_maps_open(inode, file, &proc_pid_maps_op);	376	return do_maps_open(inode, file, &proc_pid_maps_op);
359	}	377	}
360		378
		379	#define PROCMAP_QUERY_VMA_FLAGS ( \
		380	PROCMAP_QUERY_VMA_READABLE \| \
		381	PROCMAP_QUERY_VMA_WRITABLE \| \
		382	PROCMAP_QUERY_VMA_EXECUTABLE \| \
		383	PROCMAP_QUERY_VMA_SHARED \
		384	)
		385
		386	#define PROCMAP_QUERY_VALID_FLAGS_MASK ( \
		387	PROCMAP_QUERY_COVERING_OR_NEXT_VMA \| \
		388	PROCMAP_QUERY_FILE_BACKED_VMA \| \
		389	PROCMAP_QUERY_VMA_FLAGS \
		390	)
		391
		392	static int query_vma_setup(struct mm_struct *mm)
		393	{
		394	return mmap_read_lock_killable(mm);
		395	}
		396
		397	static void query_vma_teardown(struct mm_struct mm, struct vm_area_struct vma)
		398	{
		399	mmap_read_unlock(mm);
		400	}
		401
		402	static struct vm_area_struct query_vma_find_by_addr(struct mm_struct mm, unsigned long addr)
		403	{
		404	return find_vma(mm, addr);
		405	}
		406
		407	static struct vm_area_struct query_matching_vma(struct mm_struct mm,
		408	unsigned long addr, u32 flags)
		409	{
		410	struct vm_area_struct *vma;
		411
		412	next_vma:
		413	vma = query_vma_find_by_addr(mm, addr);
		414	if (!vma)
		415	goto no_vma;
		416
		417	/* user requested only file-backed VMA, keep iterating */
		418	if ((flags & PROCMAP_QUERY_FILE_BACKED_VMA) && !vma->vm_file)
		419	goto skip_vma;
		420
		421	/* VMA permissions should satisfy query flags */
		422	if (flags & PROCMAP_QUERY_VMA_FLAGS) {
		423	u32 perm = 0;
		424
		425	if (flags & PROCMAP_QUERY_VMA_READABLE)
		426	perm \|= VM_READ;
		427	if (flags & PROCMAP_QUERY_VMA_WRITABLE)
		428	perm \|= VM_WRITE;
		429	if (flags & PROCMAP_QUERY_VMA_EXECUTABLE)
		430	perm \|= VM_EXEC;
		431	if (flags & PROCMAP_QUERY_VMA_SHARED)
		432	perm \|= VM_MAYSHARE;
		433
		434	if ((vma->vm_flags & perm) != perm)
		435	goto skip_vma;
		436	}
		437
		438	/* found covering VMA or user is OK with the matching next VMA */
		439	if ((flags & PROCMAP_QUERY_COVERING_OR_NEXT_VMA) \|\| vma->vm_start <= addr)
		440	return vma;
		441
		442	skip_vma:
		443	/*
		444	* If the user needs closest matching VMA, keep iterating.
		445	*/
		446	addr = vma->vm_end;
		447	if (flags & PROCMAP_QUERY_COVERING_OR_NEXT_VMA)
		448	goto next_vma;
		449
		450	no_vma:
		451	return ERR_PTR(-ENOENT);
		452	}
		453
		454	static int do_procmap_query(struct proc_maps_private priv, void __user uarg)
		455	{
		456	struct procmap_query karg;
		457	struct vm_area_struct *vma;
		458	struct mm_struct *mm;
		459	const char *name = NULL;
		460	char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL;
		461	__u64 usize;
		462	int err;
		463
		464	if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize)))
		465	return -EFAULT;
		466	/* argument struct can never be that large, reject abuse */
		467	if (usize > PAGE_SIZE)
		468	return -E2BIG;
		469	/* argument struct should have at least query_flags and query_addr fields */
		470	if (usize < offsetofend(struct procmap_query, query_addr))
		471	return -EINVAL;
		472	err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);
		473	if (err)
		474	return err;
		475
		476	/* reject unknown flags */
		477	if (karg.query_flags & ~PROCMAP_QUERY_VALID_FLAGS_MASK)
		478	return -EINVAL;
		479	/* either both buffer address and size are set, or both should be zero */
		480	if (!!karg.vma_name_size != !!karg.vma_name_addr)
		481	return -EINVAL;
		482	if (!!karg.build_id_size != !!karg.build_id_addr)
		483	return -EINVAL;
		484
		485	mm = priv->mm;
		486	if (!mm \|\| !mmget_not_zero(mm))
		487	return -ESRCH;
		488
		489	err = query_vma_setup(mm);
		490	if (err) {
		491	mmput(mm);
		492	return err;
		493	}
		494
		495	vma = query_matching_vma(mm, karg.query_addr, karg.query_flags);
		496	if (IS_ERR(vma)) {
		497	err = PTR_ERR(vma);
		498	vma = NULL;
		499	goto out;
		500	}
		501
		502	karg.vma_start = vma->vm_start;
		503	karg.vma_end = vma->vm_end;
		504
		505	karg.vma_flags = 0;
		506	if (vma->vm_flags & VM_READ)
		507	karg.vma_flags \|= PROCMAP_QUERY_VMA_READABLE;
		508	if (vma->vm_flags & VM_WRITE)
		509	karg.vma_flags \|= PROCMAP_QUERY_VMA_WRITABLE;
		510	if (vma->vm_flags & VM_EXEC)
		511	karg.vma_flags \|= PROCMAP_QUERY_VMA_EXECUTABLE;
		512	if (vma->vm_flags & VM_MAYSHARE)
		513	karg.vma_flags \|= PROCMAP_QUERY_VMA_SHARED;
		514
		515	karg.vma_page_size = vma_kernel_pagesize(vma);
		516
		517	if (vma->vm_file) {
		518	const struct inode *inode = file_user_inode(vma->vm_file);
		519
		520	karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT;
		521	karg.dev_major = MAJOR(inode->i_sb->s_dev);
		522	karg.dev_minor = MINOR(inode->i_sb->s_dev);
		523	karg.inode = inode->i_ino;
		524	} else {
		525	karg.vma_offset = 0;
		526	karg.dev_major = 0;
		527	karg.dev_minor = 0;
		528	karg.inode = 0;
		529	}
		530
		531	if (karg.build_id_size) {
		532	__u32 build_id_sz;
		533
		534	err = build_id_parse(vma, build_id_buf, &build_id_sz);
		535	if (err) {
		536	karg.build_id_size = 0;
		537	} else {
		538	if (karg.build_id_size < build_id_sz) {
		539	err = -ENAMETOOLONG;
		540	goto out;
		541	}
		542	karg.build_id_size = build_id_sz;
		543	}
		544	}
		545
		546	if (karg.build_id_size) {
		547	__u32 build_id_sz;
		548
		549	err = build_id_parse(vma, build_id_buf, &build_id_sz);
		550	if (err) {
		551	karg.build_id_size = 0;
		552	} else {
		553	if (karg.build_id_size < build_id_sz) {
		554	err = -ENAMETOOLONG;
		555	goto out;
		556	}
		557	karg.build_id_size = build_id_sz;
		558	}
		559	}
		560
		561	if (karg.vma_name_size) {
		562	size_t name_buf_sz = min_t(size_t, PATH_MAX, karg.vma_name_size);
		563	const struct path *path;
		564	const char *name_fmt;
		565	size_t name_sz = 0;
		566
		567	get_vma_name(vma, &path, &name, &name_fmt);
		568
		569	if (path \|\| name_fmt \|\| name) {
		570	name_buf = kmalloc(name_buf_sz, GFP_KERNEL);
		571	if (!name_buf) {
		572	err = -ENOMEM;
		573	goto out;
		574	}
		575	}
		576	if (path) {
		577	name = d_path(path, name_buf, name_buf_sz);
		578	if (IS_ERR(name)) {
		579	err = PTR_ERR(name);
		580	goto out;
		581	}
		582	name_sz = name_buf + name_buf_sz - name;
		583	} else if (name \|\| name_fmt) {
		584	name_sz = 1 + snprintf(name_buf, name_buf_sz, name_fmt ?: "%s", name);
		585	name = name_buf;
		586	}
		587	if (name_sz > name_buf_sz) {
		588	err = -ENAMETOOLONG;
		589	goto out;
		590	}
		591	karg.vma_name_size = name_sz;
		592	}
		593
		594	/* unlock vma or mmap_lock, and put mm_struct before copying data to user */
		595	query_vma_teardown(mm, vma);
		596	mmput(mm);
		597
		598	if (karg.vma_name_size && copy_to_user(u64_to_user_ptr(karg.vma_name_addr),
		599	name, karg.vma_name_size)) {
		600	kfree(name_buf);
		601	return -EFAULT;
		602	}
		603	kfree(name_buf);
		604
		605	if (karg.build_id_size && copy_to_user(u64_to_user_ptr(karg.build_id_addr),
		606	build_id_buf, karg.build_id_size))
		607	return -EFAULT;
		608
		609	if (copy_to_user(uarg, &karg, min_t(size_t, sizeof(karg), usize)))
		610	return -EFAULT;
		611
		612	return 0;
		613
		614	out:
		615	query_vma_teardown(mm, vma);
		616	mmput(mm);
		617	kfree(name_buf);
		618	return err;
		619	}
		620
		621	static long procfs_procmap_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
		622	{
		623	struct seq_file *seq = file->private_data;
		624	struct proc_maps_private *priv = seq->private;
		625
		626	switch (cmd) {
		627	case PROCMAP_QUERY:
		628	return do_procmap_query(priv, (void __user *)arg);
		629	default:
		630	return -ENOIOCTLCMD;
		631	}
		632	}
		633
361	const struct file_operations proc_pid_maps_operations = {	634	const struct file_operations proc_pid_maps_operations = {
362	.open = pid_maps_open,	635	.open = pid_maps_open,
363	.read = seq_read,	636	.read = seq_read,
364	.llseek = seq_lseek,	637	.llseek = seq_lseek,
365	.release = proc_map_release,	638	.release = proc_map_release,
		639	.unlocked_ioctl = procfs_procmap_ioctl,
		640	.compat_ioctl = compat_ptr_ioctl,
366	};	641	};
367		642
368	/*	643	/*
@@ -442,7 +717,7 @@ static void smaps_page_accumulate(struct mem_size_stats *mss, · 73 unmodified lines
442		717
443	static void smaps_account(struct mem_size_stats mss, struct page page,	718	static void smaps_account(struct mem_size_stats mss, struct page page,
444	bool compound, bool young, bool dirty, bool locked,	719	bool compound, bool young, bool dirty, bool locked,
445	bool migration)	720	bool present)
446	{	721	{
447	struct folio *folio = page_folio(page);	722	struct folio *folio = page_folio(page);
448	int i, nr = compound ? compound_nr(page) : 1;	723	int i, nr = compound ? compound_nr(page) : 1;
@@ -471,24 +746,29 @@ static void smaps_account(struct mem_size_stats mss, struct page page, · 22 unmodified lines
471	* Then accumulate quantities that may depend on sharing, or that may	746	* Then accumulate quantities that may depend on sharing, or that may
472	* differ page-by-page.	747	* differ page-by-page.
473	*	748	*
474	* refcount == 1 guarantees the page is mapped exactly once.	749	* refcount == 1 for present entries guarantees that the folio is mapped
475	* If any subpage of the compound page mapped with PTE it would elevate	750	* exactly once. For large folios this implies that exactly one
476	* the refcount.	751	* PTE/PMD/... maps (a part of) this folio.
477	*	752	*
478	* The page_mapcount() is called to get a snapshot of the mapcount.	753	* Treat all non-present entries (where relying on the mapcount and
479	* Without holding the page lock this snapshot can be slightly wrong as	754	* refcount doesn't make sense) as "maybe shared, but not sure how
480	* we cannot always read the mapcount atomically. It is not safe to	755	* often". We treat device private entries as being fake-present.
481	* call page_mapcount() even with PTL held if the page is not mapped,	756	*
482	* especially for migration entries. Treat regular migration entries	757	* Note that it would not be safe to read the mapcount especially for
483	* as mapcount == 1.	758	* pages referenced by migration entries, even with the PTL held.
484	*/	759	*/
485	if ((folio_ref_count(folio) == 1) \|\| migration) {	760	if (folio_ref_count(folio) == 1 \|\| !present) {
486	smaps_page_accumulate(mss, folio, size, size << PSS_SHIFT,	761	smaps_page_accumulate(mss, folio, size, size << PSS_SHIFT,
487	dirty, locked, true);	762	dirty, locked, present);
488	return;	763	return;
489	}	764	}
		765	/*
		766	* We obtain a snapshot of the mapcount. Without holding the folio lock
		767	* this snapshot can be slightly wrong as we cannot always read the
		768	* mapcount atomically.
		769	*/
490	for (i = 0; i < nr; i++, page++) {	770	for (i = 0; i < nr; i++, page++) {
491	int mapcount = page_mapcount(page);	771	int mapcount = folio_precise_page_mapcount(folio, page);
492	unsigned long pss = PAGE_SIZE << PSS_SHIFT;	772	unsigned long pss = PAGE_SIZE << PSS_SHIFT;
493	if (mapcount >= 2)	773	if (mapcount >= 2)
494	pss /= mapcount;	774	pss /= mapcount;
@@ -531,13 +811,14 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 36 unmodified lines
531	struct vm_area_struct *vma = walk->vma;	811	struct vm_area_struct *vma = walk->vma;
532	bool locked = !!(vma->vm_flags & VM_LOCKED);	812	bool locked = !!(vma->vm_flags & VM_LOCKED);
533	struct page *page = NULL;	813	struct page *page = NULL;
534	bool migration = false, young = false, dirty = false;	814	bool present = false, young = false, dirty = false;
535	pte_t ptent = ptep_get(pte);	815	pte_t ptent = ptep_get(pte);
536		816
537	if (pte_present(ptent)) {	817	if (pte_present(ptent)) {
538	page = vm_normal_page(vma, addr, ptent);	818	page = vm_normal_page(vma, addr, ptent);
539	young = pte_young(ptent);	819	young = pte_young(ptent);
540	dirty = pte_dirty(ptent);	820	dirty = pte_dirty(ptent);
		821	present = true;
541	} else if (is_swap_pte(ptent)) {	822	} else if (is_swap_pte(ptent)) {
542	swp_entry_t swpent = pte_to_swp_entry(ptent);	823	swp_entry_t swpent = pte_to_swp_entry(ptent);
543		824
@@ -555,8 +836,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 11 unmodified lines
555	mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT;	836	mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT;
556	}	837	}
557	} else if (is_pfn_swap_entry(swpent)) {	838	} else if (is_pfn_swap_entry(swpent)) {
558	if (is_migration_entry(swpent))	839	if (is_device_private_entry(swpent))
559	migration = true;	840	present = true;
560	page = pfn_swap_entry_to_page(swpent);	841	page = pfn_swap_entry_to_page(swpent);
561	}	842	}
562	} else {	843	} else {
@@ -567,7 +848,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 4 unmodified lines
567	if (!page)	848	if (!page)
568	return;	849	return;
569		850
570	smaps_account(mss, page, false, young, dirty, locked, migration);	851	smaps_account(mss, page, false, young, dirty, locked, present);
571	}	852	}
572		853
573	#ifdef CONFIG_TRANSPARENT_HUGEPAGE	854	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -578,18 +859,17 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · 4 unmodified lines
578	struct vm_area_struct *vma = walk->vma;	859	struct vm_area_struct *vma = walk->vma;
579	bool locked = !!(vma->vm_flags & VM_LOCKED);	860	bool locked = !!(vma->vm_flags & VM_LOCKED);
580	struct page *page = NULL;	861	struct page *page = NULL;
		862	bool present = false;
581	struct folio *folio;	863	struct folio *folio;
582	bool migration = false;
583		864
584	if (pmd_present(*pmd)) {	865	if (pmd_present(*pmd)) {
585	page = vm_normal_page_pmd(vma, addr, *pmd);	866	page = vm_normal_page_pmd(vma, addr, *pmd);
		867	present = true;
586	} else if (unlikely(thp_migration_supported() && is_swap_pmd(*pmd))) {	868	} else if (unlikely(thp_migration_supported() && is_swap_pmd(*pmd))) {
587	swp_entry_t entry = pmd_to_swp_entry(*pmd);	869	swp_entry_t entry = pmd_to_swp_entry(*pmd);
588		870
589	if (is_migration_entry(entry)) {	871	if (is_pfn_swap_entry(entry))
590	migration = true;
591	page = pfn_swap_entry_to_page(entry);	872	page = pfn_swap_entry_to_page(entry);
592	}
593	}	873	}
594	if (IS_ERR_OR_NULL(page))	874	if (IS_ERR_OR_NULL(page))
595	return;	875	return;
@@ -604,7 +884,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · 8 unmodified lines
604	mss->file_thp += HPAGE_PMD_SIZE;	884	mss->file_thp += HPAGE_PMD_SIZE;
605		885
606	smaps_account(mss, page, true, pmd_young(pmd), pmd_dirty(pmd),	886	smaps_account(mss, page, true, pmd_young(pmd), pmd_dirty(pmd),
607	locked, migration);	887	locked, present);
608	}	888	}
609	#else	889	#else
610	static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,	890	static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
@@ -733,19 +1013,23 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · 122 unmodified lines
733	{	1013	{
734	struct mem_size_stats *mss = walk->private;	1014	struct mem_size_stats *mss = walk->private;
735	struct vm_area_struct *vma = walk->vma;	1015	struct vm_area_struct *vma = walk->vma;
736	pte_t ptent = huge_ptep_get(pte);	1016	pte_t ptent = huge_ptep_get(walk->mm, addr, pte);
737	struct folio *folio = NULL;	1017	struct folio *folio = NULL;
		1018	bool present = false;
738		1019
739	if (pte_present(ptent)) {	1020	if (pte_present(ptent)) {
740	folio = page_folio(pte_page(ptent));	1021	folio = page_folio(pte_page(ptent));
		1022	present = true;
741	} else if (is_swap_pte(ptent)) {	1023	} else if (is_swap_pte(ptent)) {
742	swp_entry_t swpent = pte_to_swp_entry(ptent);	1024	swp_entry_t swpent = pte_to_swp_entry(ptent);
743		1025
744	if (is_pfn_swap_entry(swpent))	1026	if (is_pfn_swap_entry(swpent))
745	folio = pfn_swap_entry_folio(swpent);	1027	folio = pfn_swap_entry_folio(swpent);
746	}	1028	}
		1029
747	if (folio) {	1030	if (folio) {
748	if (folio_likely_mapped_shared(folio) \|\|	1031	/* We treat non-present entries as "maybe shared". */
		1032	if (!present \|\| folio_likely_mapped_shared(folio) \|\|
749	hugetlb_pmd_shared(pte))	1033	hugetlb_pmd_shared(pte))
750	mss->shared_hugetlb += huge_page_size(hstate_vma(vma));	1034	mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
751	else	1035	else
@@ -1091,7 +1375,7 @@ struct clear_refs_private { · 339 unmodified lines
1091		1375
1092	static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, pte_t pte)	1376	static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, pte_t pte)
1093	{	1377	{
1094	struct page *page;	1378	struct folio *folio;
1095		1379
1096	if (!pte_write(pte))	1380	if (!pte_write(pte))
1097	return false;	1381	return false;
@@ -1099,10 +1383,10 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, · 1 unmodified line
1099	return false;	1383	return false;
1100	if (likely(!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)))	1384	if (likely(!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)))
1101	return false;	1385	return false;
1102	page = vm_normal_page(vma, addr, pte);	1386	folio = vm_normal_folio(vma, addr, pte);
1103	if (!page)	1387	if (!folio)
1104	return false;	1388	return false;
1105	return page_maybe_dma_pinned(page);	1389	return folio_maybe_dma_pinned(folio);
1106	}	1390	}
1107		1391
1108	static inline void clear_soft_dirty(struct vm_area_struct *vma,	1392	static inline void clear_soft_dirty(struct vm_area_struct *vma,
@@ -1418,7 +1702,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 309 unmodified lines
1418	{	1702	{
1419	u64 frame = 0, flags = 0;	1703	u64 frame = 0, flags = 0;
1420	struct page *page = NULL;	1704	struct page *page = NULL;
1421	bool migration = false;	1705	struct folio *folio;
1422		1706
1423	if (pte_present(pte)) {	1707	if (pte_present(pte)) {
1424	if (pm->show_pfn)	1708	if (pm->show_pfn)
@@ -1450,17 +1734,20 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 25 unmodified lines
1450	(offset << MAX_SWAPFILES_SHIFT);	1734	(offset << MAX_SWAPFILES_SHIFT);
1451	}	1735	}
1452	flags \|= PM_SWAP;	1736	flags \|= PM_SWAP;
1453	migration = is_migration_entry(entry);
1454	if (is_pfn_swap_entry(entry))	1737	if (is_pfn_swap_entry(entry))
1455	page = pfn_swap_entry_to_page(entry);	1738	page = pfn_swap_entry_to_page(entry);
1456	if (pte_marker_entry_uffd_wp(entry))	1739	if (pte_marker_entry_uffd_wp(entry))
1457	flags \|= PM_UFFD_WP;	1740	flags \|= PM_UFFD_WP;
1458	}	1741	}
1459		1742
1460	if (page && !PageAnon(page))	1743	if (page) {
1461	flags \|= PM_FILE;	1744	folio = page_folio(page);
1462	if (page && !migration && page_mapcount(page) == 1)	1745	if (!folio_test_anon(folio))
1463	flags \|= PM_MMAP_EXCLUSIVE;	1746	flags \|= PM_FILE;
		1747	if ((flags & PM_PRESENT) &&
		1748	folio_precise_page_mapcount(folio, page) == 1)
		1749	flags \|= PM_MMAP_EXCLUSIVE;
		1750	}
1464	if (vma->vm_flags & VM_SOFTDIRTY)	1751	if (vma->vm_flags & VM_SOFTDIRTY)
1465	flags \|= PM_SOFT_DIRTY;	1752	flags \|= PM_SOFT_DIRTY;
1466		1753
@@ -1476,13 +1763,14 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 9 unmodified lines
1476	pte_t pte, orig_pte;	1763	pte_t pte, orig_pte;
1477	int err = 0;	1764	int err = 0;
1478	#ifdef CONFIG_TRANSPARENT_HUGEPAGE	1765	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1479	bool migration = false;
1480		1766
1481	ptl = pmd_trans_huge_lock(pmdp, vma);	1767	ptl = pmd_trans_huge_lock(pmdp, vma);
1482	if (ptl) {	1768	if (ptl) {
		1769	unsigned int idx = (addr & ~PMD_MASK) >> PAGE_SHIFT;
1483	u64 flags = 0, frame = 0;	1770	u64 flags = 0, frame = 0;
1484	pmd_t pmd = *pmdp;	1771	pmd_t pmd = *pmdp;
1485	struct page *page = NULL;	1772	struct page *page = NULL;
		1773	struct folio *folio = NULL;
1486		1774
1487	if (vma->vm_flags & VM_SOFTDIRTY)	1775	if (vma->vm_flags & VM_SOFTDIRTY)
1488	flags \|= PM_SOFT_DIRTY;	1776	flags \|= PM_SOFT_DIRTY;
@@ -1496,8 +1784,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 7 unmodified lines
1496	if (pmd_uffd_wp(pmd))	1784	if (pmd_uffd_wp(pmd))
1497	flags \|= PM_UFFD_WP;	1785	flags \|= PM_UFFD_WP;
1498	if (pm->show_pfn)	1786	if (pm->show_pfn)
1499	frame = pmd_pfn(pmd) +	1787	frame = pmd_pfn(pmd) + idx;
1500	((addr & ~PMD_MASK) >> PAGE_SHIFT);
1501	}	1788	}
1502	#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION	1789	#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
1503	else if (is_swap_pmd(pmd)) {	1790	else if (is_swap_pmd(pmd)) {
@@ -1506,11 +1793,9 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 2 unmodified lines
1506		1793
1507	if (pm->show_pfn) {	1794	if (pm->show_pfn) {
1508	if (is_pfn_swap_entry(entry))	1795	if (is_pfn_swap_entry(entry))
1509	offset = swp_offset_pfn(entry);	1796	offset = swp_offset_pfn(entry) + idx;
1510	else	1797	else
1511	offset = swp_offset(entry);	1798	offset = swp_offset(entry) + idx;
1512	offset = offset +
1513	((addr & ~PMD_MASK) >> PAGE_SHIFT);
1514	frame = swp_type(entry) \|	1799	frame = swp_type(entry) \|
1515	(offset << MAX_SWAPFILES_SHIFT);	1800	(offset << MAX_SWAPFILES_SHIFT);
1516	}	1801	}
@@ -1520,17 +1805,25 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 3 unmodified lines
1520	if (pmd_swp_uffd_wp(pmd))	1805	if (pmd_swp_uffd_wp(pmd))
1521	flags \|= PM_UFFD_WP;	1806	flags \|= PM_UFFD_WP;
1522	VM_BUG_ON(!is_pmd_migration_entry(pmd));	1807	VM_BUG_ON(!is_pmd_migration_entry(pmd));
1523	migration = is_migration_entry(entry);
1524	page = pfn_swap_entry_to_page(entry);	1808	page = pfn_swap_entry_to_page(entry);
1525	}	1809	}
1526	#endif	1810	#endif
1527		1811
1528	if (page && !migration && page_mapcount(page) == 1)	1812	if (page) {
1529	flags \|= PM_MMAP_EXCLUSIVE;	1813	folio = page_folio(page);
		1814	if (!folio_test_anon(folio))
		1815	flags \|= PM_FILE;
		1816	}
		1817
		1818	for (; addr != end; addr += PAGE_SIZE, idx++) {
		1819	unsigned long cur_flags = flags;
		1820	pagemap_entry_t pme;
1530		1821
1531	for (; addr != end; addr += PAGE_SIZE) {	1822	if (folio && (flags & PM_PRESENT) &&
1532	pagemap_entry_t pme = make_pme(frame, flags);	1823	folio_precise_page_mapcount(folio, page + idx) == 1)
		1824	cur_flags \|= PM_MMAP_EXCLUSIVE;
1533		1825
		1826	pme = make_pme(frame, cur_flags);
1534	err = add_to_pagemap(&pme, pm);	1827	err = add_to_pagemap(&pme, pm);
1535	if (err)	1828	if (err)
1536	break;	1829	break;
@@ -1585,7 +1878,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, · 48 unmodified lines
1585	if (vma->vm_flags & VM_SOFTDIRTY)	1878	if (vma->vm_flags & VM_SOFTDIRTY)
1586	flags \|= PM_SOFT_DIRTY;	1879	flags \|= PM_SOFT_DIRTY;
1587		1880
1588	pte = huge_ptep_get(ptep);	1881	pte = huge_ptep_get(walk->mm, addr, ptep);
1589	if (pte_present(pte)) {	1882	if (pte_present(pte)) {
1590	struct folio *folio = page_folio(pte_page(pte));	1883	struct folio *folio = page_folio(pte_page(pte));
1591		1884
@@ -2274,7 +2567,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 682 unmodified lines
2274	if (~p->arg.flags & PM_SCAN_WP_MATCHING) {	2567	if (~p->arg.flags & PM_SCAN_WP_MATCHING) {
2275	/* Go the short route when not write-protecting pages. */	2568	/* Go the short route when not write-protecting pages. */
2276		2569
2277	pte = huge_ptep_get(ptep);	2570	pte = huge_ptep_get(walk->mm, start, ptep);
2278	categories = p->cur_vma_category \| pagemap_hugetlb_category(pte);	2571	categories = p->cur_vma_category \| pagemap_hugetlb_category(pte);
2279		2572
2280	if (!pagemap_scan_is_interesting_page(categories, p))	2573	if (!pagemap_scan_is_interesting_page(categories, p))
@@ -2286,7 +2579,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 5 unmodified lines
2286	i_mmap_lock_write(vma->vm_file->f_mapping);	2579	i_mmap_lock_write(vma->vm_file->f_mapping);
2287	ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);	2580	ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
2288		2581
2289	pte = huge_ptep_get(ptep);	2582	pte = huge_ptep_get(walk->mm, start, ptep);
2290	categories = p->cur_vma_category \| pagemap_hugetlb_category(pte);	2583	categories = p->cur_vma_category \| pagemap_hugetlb_category(pte);
2291		2584
2292	if (!pagemap_scan_is_interesting_page(categories, p))	2585	if (!pagemap_scan_is_interesting_page(categories, p))
@@ -2566,7 +2859,7 @@ static void gather_stats(struct page page, struct numa_maps md, int pte_dirty, · 273 unmodified lines
2566	unsigned long nr_pages)	2859	unsigned long nr_pages)
2567	{	2860	{
2568	struct folio *folio = page_folio(page);	2861	struct folio *folio = page_folio(page);
2569	int count = page_mapcount(page);	2862	int count = folio_precise_page_mapcount(folio, page);
2570		2863
2571	md->pages += nr_pages;	2864	md->pages += nr_pages;
2572	if (pte_dirty \|\| folio_test_dirty(folio))	2865	if (pte_dirty \|\| folio_test_dirty(folio))
@@ -2682,7 +2975,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, · 109 unmodified lines
2682	static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask,	2975	static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask,
2683	unsigned long addr, unsigned long end, struct mm_walk *walk)	2976	unsigned long addr, unsigned long end, struct mm_walk *walk)
2684	{	2977	{
2685	pte_t huge_pte = huge_ptep_get(pte);	2978	pte_t huge_pte = huge_ptep_get(walk->mm, addr, pte);
2686	struct numa_maps *md;	2979	struct numa_maps *md;
2687	struct page *page;	2980	struct page *page;
2688		2981

fs/userfaultfd.c+1−1

@@ -257,7 +257,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, · 256 unmodified lines
257	goto out;	257	goto out;
258		258
259	ret = false;	259	ret = false;
260	pte = huge_ptep_get(ptep);	260	pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep);
261		261
262	/*	262	/*
263	* Lockless access: we're in a wait_event so it's ok if it	263	* Lockless access: we're in a wait_event so it's ok if it

include/acpi/platform/aclinuxex.h+6−3

  * Interrupts are off during resume, just like they are for boot.
  * However, boot has  (system_state != SYSTEM_RUNNING)
  * to quiet __might_sleep() in kmalloc() and resume does not.
+ *
+ * These specialized allocators have to be macros for their allocations to be
+ * accounted separately (to have separate alloc_tag).
  */
 #define acpi_os_allocate(_size)	\
 		kmalloc(_size, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL)
 #define acpi_os_allocate_zeroed(_size)	\
 		kzalloc(_size, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL)
+#define acpi_os_acquire_object(_cache)	\
+		kmem_cache_zalloc(_cache, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL)
 static inline void acpi_os_free(void *memory)
 {
 	kfree(memory);
 }
-#define acpi_os_acquire_object(_cache)	\
-		kmem_cache_zalloc(_cache, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL)
 static inline acpi_thread_id acpi_os_get_thread_id(void)
 {
 	return (acpi_thread_id) (unsigned long)current;

include/asm-generic/hugetlb.h+1−1

@@ -144,7 +144,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 143 unmodified lines
144	#endif	144	#endif
145		145
146	#ifndef __HAVE_ARCH_HUGE_PTEP_GET	146	#ifndef __HAVE_ARCH_HUGE_PTEP_GET
147	static inline pte_t huge_ptep_get(pte_t *ptep)	147	static inline pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep)
148	{	148	{
149	return ptep_get(ptep);	149	return ptep_get(ptep);
150	}	150	}

include/linux/alloc_tag.h+1−1

@@ -122,7 +122,7 @@ static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag · 121 unmodified lines
122	"alloc_tag was not cleared (got tag for %s:%u)\n",	122	"alloc_tag was not cleared (got tag for %s:%u)\n",
123	ref->ct->filename, ref->ct->lineno);	123	ref->ct->filename, ref->ct->lineno);
124		124
125	WARN_ONCE(!tag, "current->alloc_tag not set");	125	WARN_ONCE(!tag, "current->alloc_tag not set\n");
126	}	126	}
127		127
128	static inline void alloc_tag_sub_check(union codetag_ref *ref)	128	static inline void alloc_tag_sub_check(union codetag_ref *ref)

include/linux/bpf.h+6−2

 	u32 btf_value_type_id;
 	u32 btf_vmlinux_value_type_id;
 	struct btf *btf;
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 	struct obj_cgroup *objcg;
 #endif
 	char name[BPF_OBJ_NAME_LEN];
 int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid,
 			unsigned long nr_pages, struct page **page_array);
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags,
 			   int node);
 void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags);
 void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size,
 				    size_t align, gfp_t flags);
 #else
+/*
+ * These specialized allocators have to be macros for their allocations to be
+ * accounted separately (to have separate alloc_tag).
+ */
 #define bpf_map_kmalloc_node(_map, _size, _flags, _node)	\
 		kmalloc_node(_size, _flags, _node)
 #define bpf_map_kzalloc(_map, _size, _flags)			\

include/linux/buffer_head.h+1−1

@@ -53,7 +53,7 @@ typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate); · 52 unmodified lines
53	* filesystem and block layers. Nowadays the basic I/O unit	53	* filesystem and block layers. Nowadays the basic I/O unit
54	* is the bio, and buffer_heads are used for extracting block	54	* is the bio, and buffer_heads are used for extracting block
55	* mappings (via a get_block_t call), for tracking state within	55	* mappings (via a get_block_t call), for tracking state within
56	* a page (via a page_mapping) and for wrapping bio submission	56	* a folio (via a folio_mapping) and for wrapping bio submission
57	* for backward compatibility reasons (e.g. submit_bh).	57	* for backward compatibility reasons (e.g. submit_bh).
58	*/	58	*/
59	struct buffer_head {	59	struct buffer_head {

include/linux/cgroup-defs.h+0−2

@@ -681,9 +681,7 @@ struct cftype { · 680 unmodified lines
681	__poll_t (poll)(struct kernfs_open_file of,	681	__poll_t (poll)(struct kernfs_open_file of,
682	struct poll_table_struct *pt);	682	struct poll_table_struct *pt);
683		683
684	#ifdef CONFIG_DEBUG_LOCK_ALLOC
685	struct lock_class_key lockdep_key;	684	struct lock_class_key lockdep_key;
686	#endif
687	};	685	};
688		686
689	/*	687	/*

include/linux/damon.h+16−1

  * @DAMOS_NOHUGEPAGE:	Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
  * @DAMOS_LRU_PRIO:	Prioritize the region on its LRU lists.
  * @DAMOS_LRU_DEPRIO:	Deprioritize the region on its LRU lists.
+ * @DAMOS_MIGRATE_HOT:  Migrate the regions prioritizing warmer regions.
+ * @DAMOS_MIGRATE_COLD:	Migrate the regions prioritizing colder regions.
  * @DAMOS_STAT:		Do nothing but count the stat.
  * @NR_DAMOS_ACTIONS:	Total number of DAMOS actions
  *
 	DAMOS_NOHUGEPAGE,
 	DAMOS_LRU_PRIO,
 	DAMOS_LRU_DEPRIO,
+	DAMOS_MIGRATE_HOT,
+	DAMOS_MIGRATE_COLD,
 	DAMOS_STAT,		/* Do nothing but only record the stat */
 	NR_DAMOS_ACTIONS,
 };
  * @apply_interval_us:	The time between applying the @action.
  * @quota:		Control the aggressiveness of this scheme.
  * @wmarks:		Watermarks for automated (in)activation of this scheme.
+ * @target_nid:		Destination node if @action is "migrate_{hot,cold}".
  * @filters:		Additional set of &struct damos_filter for &action.
  * @stat:		Statistics of this scheme.
  * @list:		List head for siblings.
  * monitoring context are inactive, DAMON stops monitoring either, and just
  * repeatedly checks the watermarks.
  *
+ * @target_nid is used to set the migration target node for migrate_hot or
+ * migrate_cold actions, which means it's only meaningful when @action is either
+ * "migrate_hot" or "migrate_cold".
+ *
  * Before applying the &action to a memory region, &struct damon_operations
  * implementation could check pages of the region and skip &action to respect
  * &filters
 /* public: */
 	struct damos_quota quota;
 	struct damos_watermarks wmarks;
+	union {
+		int target_nid;
+	};
 	struct list_head filters;
 	struct damos_stat stat;
 	struct list_head list;
 			enum damos_action action,
 			unsigned long apply_interval_us,
 			struct damos_quota *quota,
-			struct damos_watermarks *wmarks);
+			struct damos_watermarks *wmarks,
+			int target_nid);
 void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);
 void damon_destroy_scheme(struct damos *s);
+int damos_commit_quota_goals(struct damos_quota *dst, struct damos_quota *src);
 struct damon_target *damon_new_target(void);
 void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
 int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs);
 void damon_set_schemes(struct damon_ctx *ctx,
 			struct damos **schemes, ssize_t nr_schemes);
+int damon_commit_ctx(struct damon_ctx *old_ctx, struct damon_ctx *new_ctx);
 int damon_nr_running_ctxs(void);
 bool damon_is_registered_ops(enum damon_ops_id id);
 int damon_register_ops(struct damon_operations *ops);

include/linux/dma-fence-chain.h+4−0

  * dma_fence_chain_alloc
  *
  * Returns a new struct dma_fence_chain object or NULL on failure.
+ *
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag). The typecast is
+ * intentional to enforce typesafety.
  */
 #define dma_fence_chain_alloc()	\
 		((struct dma_fence_chain *)kmalloc(sizeof(struct dma_fence_chain), GFP_KERNEL))

include/linux/fault-inject.h+4−7

 struct kmem_cache;
-bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order);
 #ifdef CONFIG_FAIL_PAGE_ALLOC
-bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order);
+bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order);
 #else
-static inline bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
 {
 	return false;
 }
 #endif /* CONFIG_FAIL_PAGE_ALLOC */
-int should_failslab(struct kmem_cache *s, gfp_t gfpflags);
 #ifdef CONFIG_FAILSLAB
-extern bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags);
+int should_failslab(struct kmem_cache *s, gfp_t gfpflags);
 #else
-static inline bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags)
+static inline int should_failslab(struct kmem_cache *s, gfp_t gfpflags)
 {
 	return false;
 }

include/linux/gfp.h+8−0

 struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order,
 		struct mempolicy *mpol, pgoff_t ilx, int nid);
 struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order);
+struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
+		struct mempolicy *mpol, pgoff_t ilx, int nid);
 struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma,
 		unsigned long addr, bool hugepage);
 #else
 {
 	return __folio_alloc_node(gfp, order, numa_node_id());
 }
+static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
+		struct mempolicy *mpol, pgoff_t ilx, int nid)
+{
+	return folio_alloc_noprof(gfp, order);
+}
 #define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage)		\
 	folio_alloc_noprof(gfp, order)
 #endif
 #define alloc_pages(...)			alloc_hooks(alloc_pages_noprof(__VA_ARGS__))
 #define alloc_pages_mpol(...)			alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__))
 #define folio_alloc(...)			alloc_hooks(folio_alloc_noprof(__VA_ARGS__))
+#define folio_alloc_mpol(...)			alloc_hooks(folio_alloc_mpol_noprof(__VA_ARGS__))
 #define vma_alloc_folio(...)			alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__))
 #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0)

include/linux/hid_bpf.h+5−0

 static inline void hid_bpf_disconnect_device(struct hid_device *hdev) {}
 static inline void hid_bpf_destroy_device(struct hid_device *hid) {}
 static inline int hid_bpf_device_init(struct hid_device *hid) { return 0; }
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag). The typecast is
+ * intentional to enforce typesafety.
+ */
 #define call_hid_bpf_rdesc_fixup(_hdev, _rdesc, _size)	\
 		((u8 *)kmemdup(_rdesc, *(_size), GFP_KERNEL))

include/linux/highmem-internal.h+6−11

 		preempt_enable();
 }
-unsigned int __nr_free_highpages(void);
+unsigned long __nr_free_highpages(void);
-extern atomic_long_t _totalhigh_pages;
+unsigned long __totalhigh_pages(void);
-static inline unsigned int nr_free_highpages(void)
+static inline unsigned long nr_free_highpages(void)
 {
 	return __nr_free_highpages();
 }
 static inline unsigned long totalhigh_pages(void)
 {
-	return (unsigned long)atomic_long_read(&_totalhigh_pages);
+	return __totalhigh_pages();
-}
-static inline void totalhigh_pages_add(long count)
-{
-	atomic_long_add(count, &_totalhigh_pages);
 }
 static inline bool is_kmap_addr(const void *x)
 		preempt_enable();
 }
-static inline unsigned int nr_free_highpages(void) { return 0; }
+static inline unsigned long nr_free_highpages(void) { return 0; }
-static inline unsigned long totalhigh_pages(void) { return 0UL; }
+static inline unsigned long totalhigh_pages(void) { return 0; }
 static inline bool is_kmap_addr(const void *x)
 {

include/linux/highmem.h+7−1

 static inline void *kmap_atomic(struct page *page);
 /* Highmem related interfaces for management code */
-static inline unsigned int nr_free_highpages(void);
+static inline unsigned long nr_free_highpages(void);
 static inline unsigned long totalhigh_pages(void);
 #ifndef ARCH_HAS_FLUSH_ANON_PAGE
 	kunmap_local(vto);
 	kunmap_local(vfrom);
+	if (ret)
+		memory_failure_queue(page_to_pfn(from), 0);
 	return ret;
 }
 	kunmap_local(vto);
 	kunmap_local(vfrom);
+	if (ret)
+		memory_failure_queue(page_to_pfn(from), 0);
 	return ret;
 }
 #else

include/linux/huge_mm.h+41−12

 #include <linux/mm_types.h>
 #include <linux/fs.h> /* only for vma_is_dax() */
+#include <linux/kobject.h>
 vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf);
 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 				  struct kobj_attribute *attr, char *buf,
 				  enum transparent_hugepage_flag flag);
 extern struct kobj_attribute shmem_enabled_attr;
+extern struct kobj_attribute thpsize_shmem_enabled_attr;
 /*
  * Mask of all large folio orders supported for anonymous THP; all orders up to
 			(1<<TRANSPARENT_HUGEPAGE_FLAG);
 }
-static inline bool hugepage_flags_enabled(void)
-{
-	/*
-	 * We cover both the anon and the file-backed case here; we must return
-	 * true if globally enabled, even when all anon sizes are set to never.
-	 * So we don't need to look at huge_anon_orders_inherit.
-	 */
-	return hugepage_global_enabled() ||
-	       huge_anon_orders_always ||
-	       huge_anon_orders_madvise;
-}
 static inline int highest_order(unsigned long orders)
 {
 	return fls_long(orders) - 1;
 	return __thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
 }
+struct thpsize {
+	struct kobject kobj;
+	struct list_head node;
+	int order;
+};
+#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj)
 enum mthp_stat_item {
 	MTHP_STAT_ANON_FAULT_ALLOC,
 	MTHP_STAT_ANON_FAULT_FALLBACK,
 	MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE,
 	MTHP_STAT_SWPOUT,
 	MTHP_STAT_SWPOUT_FALLBACK,
+	MTHP_STAT_SHMEM_ALLOC,
+	MTHP_STAT_SHMEM_FALLBACK,
+	MTHP_STAT_SHMEM_FALLBACK_CHARGE,
+	MTHP_STAT_SPLIT,
+	MTHP_STAT_SPLIT_FAILED,
+	MTHP_STAT_SPLIT_DEFERRED,
 	__MTHP_STAT_COUNT
 };
 	return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION);
 }
+void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
+			   pmd_t *pmd, bool freeze, struct folio *folio);
+bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
+			   pmd_t *pmdp, struct folio *folio);
 #else /* CONFIG_TRANSPARENT_HUGEPAGE */
 static inline bool folio_test_pmd_mappable(struct folio *folio)
 		unsigned long address, bool freeze, struct folio *folio) {}
 static inline void split_huge_pmd_address(struct vm_area_struct *vma,
 		unsigned long address, bool freeze, struct folio *folio) {}
+static inline void split_huge_pmd_locked(struct vm_area_struct *vma,
+					 unsigned long address, pmd_t *pmd,
+					 bool freeze, struct folio *folio) {}
+static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma,
+					 unsigned long addr, pmd_t *pmdp,
+					 struct folio *folio)
+{
+	return false;
+}
 #define split_huge_pud(__vma, __pmd, __address)	\
 	do { } while (0)
 {
 	return false;
 }
+static inline int highest_order(unsigned long orders)
+{
+	return 0;
+}
+static inline int next_order(unsigned long *orders, int prev)
+{
+	return 0;
+}
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 static inline int split_folio_to_list_to_order(struct folio *folio,

include/linux/hugetlb.h+4−26

 struct mmu_gather;
 struct node;
-#ifndef CONFIG_ARCH_HAS_HUGEPD
-typedef struct { unsigned long pd; } hugepd_t;
-#define is_hugepd(hugepd) (0)
-#define __hugepd(x) ((hugepd_t) { (x) })
-#endif
 void free_huge_folio(struct folio *folio);
 #ifdef CONFIG_HUGETLB_PAGE
 bool folio_test_hugetlb_##flname(struct folio *folio)		\
 	{	void *private = &folio->private;		\
 		return test_bit(HPG_##flname, private);		\
-	}							\
+	}
-static inline int HPage##uname(struct page *page)		\
-	{ return test_bit(HPG_##flname, &(page->private)); }
 #define SETHPAGEFLAG(uname, flname)				\
 static __always_inline						\
 void folio_set_hugetlb_##flname(struct folio *folio)		\
 	{	void *private = &folio->private;		\
 		set_bit(HPG_##flname, private);			\
-	}							\
+	}
-static inline void SetHPage##uname(struct page *page)		\
-	{ set_bit(HPG_##flname, &(page->private)); }
 #define CLEARHPAGEFLAG(uname, flname)				\
 static __always_inline						\
 void folio_clear_hugetlb_##flname(struct folio *folio)		\
 	{	void *private = &folio->private;		\
 		clear_bit(HPG_##flname, private);		\
-	}							\
+	}
-static inline void ClearHPage##uname(struct page *page)		\
-	{ clear_bit(HPG_##flname, &(page->private)); }
 #else
 #define TESTHPAGEFLAG(uname, flname)				\
 static inline bool						\
 folio_test_hugetlb_##flname(struct folio *folio)		\
-	{ return 0; }						\
-static inline int HPage##uname(struct page *page)		\
 	{ return 0; }
 #define SETHPAGEFLAG(uname, flname)				\
 static inline void						\
 folio_set_hugetlb_##flname(struct folio *folio) 		\
-	{ }							\
-static inline void SetHPage##uname(struct page *page)		\
 	{ }
 #define CLEARHPAGEFLAG(uname, flname)				\
 static inline void						\
 folio_clear_hugetlb_##flname(struct folio *folio)		\
-	{ }							\
-static inline void ClearHPage##uname(struct page *page)		\
 	{ }
 #endif
 /* Defines one hugetlb page size */
 struct hstate {
 	struct mutex resize_lock;
+	struct lock_class_key resize_key;
 	int next_nid_to_alloc;
 	int next_nid_to_free;
 	unsigned int order;
 	unsigned int nr_huge_pages_node[MAX_NUMNODES];
 	unsigned int free_huge_pages_node[MAX_NUMNODES];
 	unsigned int surplus_huge_pages_node[MAX_NUMNODES];
-#ifdef CONFIG_CGROUP_HUGETLB
-	/* cgroup control files */
-	struct cftype cgroup_files_dfl[8];
-	struct cftype cgroup_files_legacy[10];
-#endif
 	char name[HSTATE_NAME_LEN];
 };

include/linux/jbd2.h+10−0

  */
 extern struct kmem_cache *jbd2_handle_cache;
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag). The typecast is
+ * intentional to enforce typesafety.
+ */
 #define jbd2_alloc_handle(_gfp_flags)	\
 		((handle_t *)kmem_cache_zalloc(jbd2_handle_cache, _gfp_flags))
  */
 extern struct kmem_cache *jbd2_inode_cache;
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag). The typecast is
+ * intentional to enforce typesafety.
+ */
 #define jbd2_alloc_inode(_gfp_flags)	\
 		((struct jbd2_inode *)kmem_cache_alloc(jbd2_inode_cache, _gfp_flags))

include/linux/kmsan.h+76−0

@@ -230,6 +230,67 @@ void kmsan_handle_urb(const struct urb *urb, bool is_out); · 229 unmodified lines
230	*/	230	*/
231	void kmsan_unpoison_entry_regs(const struct pt_regs *regs);	231	void kmsan_unpoison_entry_regs(const struct pt_regs *regs);
232		232
		233	/**
		234	* kmsan_get_metadata() - Return a pointer to KMSAN shadow or origins.
		235	* @addr: kernel address.
		236	* @is_origin: whether to return origins or shadow.
		237	*
		238	* Return NULL if metadata cannot be found.
		239	*/
		240	void kmsan_get_metadata(void addr, bool is_origin);
		241
		242	/**
		243	* kmsan_enable_current(): Enable KMSAN for the current task.
		244	*
		245	* Each kmsan_enable_current() current call must be preceded by a
		246	* kmsan_disable_current() call. These call pairs may be nested.
		247	*/
		248	void kmsan_enable_current(void);
		249
		250	/**
		251	* kmsan_disable_current(): Disable KMSAN for the current task.
		252	*
		253	* Each kmsan_disable_current() current call must be followed by a
		254	* kmsan_enable_current() call. These call pairs may be nested.
		255	*/
		256	void kmsan_disable_current(void);
		257
		258	/**
		259	* memset_no_sanitize_memory(): Fill memory without KMSAN instrumentation.
		260	* @s: address of kernel memory to fill.
		261	* @c: constant byte to fill the memory with.
		262	* @n: number of bytes to fill.
		263	*
		264	* This is like memset(), but without KMSAN instrumentation.
		265	*/
		266	static inline void memset_no_sanitize_memory(void s, int c, size_t n)
		267	{
		268	return __memset(s, c, n);
		269	}
		270
		271	extern bool kmsan_enabled;
		272	extern int panic_on_kmsan;
		273
		274	/*
		275	* KMSAN performs a lot of consistency checks that are currently enabled by
		276	* default. BUG_ON is normally discouraged in the kernel, unless used for
		277	* debugging, but KMSAN itself is a debugging tool, so it makes little sense to
		278	* recover if something goes wrong.
		279	*/
		280	#define KMSAN_WARN_ON(cond) \
		281	({ \
		282	const bool __cond = WARN_ON(cond); \
		283	if (unlikely(__cond)) { \
		284	WRITE_ONCE(kmsan_enabled, false); \
		285	if (panic_on_kmsan) { \
		286	/* Can't call panic() here because */ \
		287	/* of uaccess checks. */ \
		288	BUG(); \
		289	} \
		290	} \
		291	__cond; \
		292	})
		293
233	#else	294	#else
234		295
235	static inline void kmsan_init_shadow(void)	296	static inline void kmsan_init_shadow(void)
@@ -329,6 +390,21 @@ static inline void kmsan_unpoison_entry_regs(const struct pt_regs *regs) · 93 unmodified lines
329	{	390	{
330	}	391	}
331		392
		393	static inline void kmsan_enable_current(void)
		394	{
		395	}
		396
		397	static inline void kmsan_disable_current(void)
		398	{
		399	}
		400
		401	static inline void memset_no_sanitize_memory(void s, int c, size_t n)
		402	{
		403	return memset(s, c, n);
		404	}
		405
		406	#define KMSAN_WARN_ON WARN_ON
		407
332	#endif	408	#endif
333		409
334	#endif /* _LINUX_KMSAN_H */	410	#endif /* _LINUX_KMSAN_H */

include/linux/kmsan_types.h+1−1

@@ -31,7 +31,7 @@ struct kmsan_context_state { · 30 unmodified lines
31	struct kmsan_ctx {	31	struct kmsan_ctx {
32	struct kmsan_context_state cstate;	32	struct kmsan_context_state cstate;
33	int kmsan_in_runtime;	33	int kmsan_in_runtime;
34	bool allow_reporting;	34	unsigned int depth;
35	};	35	};
36		36
37	#endif /* _LINUX_KMSAN_TYPES_H */	37	#endif /* _LINUX_KMSAN_TYPES_H */

include/linux/list_lru.h+1−1

@@ -50,7 +50,7 @@ struct list_lru_node { · 49 unmodified lines
50		50
51	struct list_lru {	51	struct list_lru {
52	struct list_lru_node *node;	52	struct list_lru_node *node;
53	#ifdef CONFIG_MEMCG_KMEM	53	#ifdef CONFIG_MEMCG
54	struct list_head list;	54	struct list_head list;
55	int shrinker_id;	55	int shrinker_id;
56	bool memcg_aware;	56	bool memcg_aware;

include/linux/memblock.h+0−2

@@ -316,8 +316,6 @@ void __next_mem_pfn_range_in_zone(u64 idx, struct zone zone, · 315 unmodified lines
316	for (; i != U64_MAX; \	316	for (; i != U64_MAX; \
317	__next_mem_pfn_range_in_zone(&i, zone, p_start, p_end))	317	__next_mem_pfn_range_in_zone(&i, zone, p_start, p_end))
318		318
319	int __init deferred_page_init_max_threads(const struct cpumask *node_cpumask);
320
321	#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */	319	#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
322		320
323	/**	321	/**

include/linux/memcontrol.h+174−189

 	refcount_t ref;
 };
-/*
- * Per memcg event counter is incremented at every pagein/pageout. With THP,
- * it will be incremented by the number of pages. This counter is used
- * to trigger some periodic events. This is straightforward and better
- * than using jiffies etc. to handle periodic memcg event.
- */
-enum mem_cgroup_events_target {
-	MEM_CGROUP_TARGET_THRESH,
-	MEM_CGROUP_TARGET_SOFTLIMIT,
-	MEM_CGROUP_NTARGETS,
-};
 struct memcg_vmstats_percpu;
 struct memcg_vmstats;
 struct lruvec_stats_percpu;
  * per-node information in memory controller.
  */
 struct mem_cgroup_per_node {
-	struct lruvec		lruvec;
+	/* Keep the read-only fields at the start */
+	struct mem_cgroup	*memcg;		/* Back pointer, we cannot */
+						/* use container_of	   */
 	struct lruvec_stats_percpu __percpu	*lruvec_stats_percpu;
 	struct lruvec_stats			*lruvec_stats;
-	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
-	struct mem_cgroup_reclaim_iter	iter;
 	struct shrinker_info __rcu	*shrinker_info;
+#ifdef CONFIG_MEMCG_V1
+	/*
+	 * Memcg-v1 only stuff in middle as buffer between read mostly fields
+	 * and update often fields to avoid false sharing. If v1 stuff is
+	 * not present, an explicit padding is needed.
+	 */
 	struct rb_node		tree_node;	/* RB tree node */
 	unsigned long		usage_in_excess;/* Set to the value by which */
 						/* the soft limit is exceeded*/
 	bool			on_tree;
-	struct mem_cgroup	*memcg;		/* Back pointer, we cannot */
+#else
-						/* use container_of	   */
+	CACHELINE_PADDING(_pad1_);
+#endif
+	/* Fields which get updated often at the end. */
+	struct lruvec		lruvec;
+	unsigned long		lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS];
+	struct mem_cgroup_reclaim_iter	iter;
 };
 struct mem_cgroup_threshold {
 		struct page_counter memsw;	/* v1 only */
 	};
-	/* Legacy consumer-oriented counters */
-	struct page_counter kmem;		/* v1 only */
-	struct page_counter tcpmem;		/* v1 only */
 	/* Range enforcement for interrupt charges */
 	struct work_struct high_work;
-#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)
+#ifdef CONFIG_ZSWAP
 	unsigned long zswap_max;
 	/*
 	bool zswap_writeback;
 #endif
-	unsigned long soft_limit;
 	/* vmpressure notifications */
 	struct vmpressure vmpressure;
 	 */
 	bool oom_group;
-	/* protected by memcg_oom_lock */
+	int swappiness;
-	bool		oom_lock;
-	int		under_oom;
-	int	swappiness;
-	/* OOM-Killer disable */
-	int		oom_kill_disable;
 	/* memory.events and memory.events.local */
 	struct cgroup_file events_file;
 	/* handle for "memory.swap.events" */
 	struct cgroup_file swap_events_file;
-	/* protect arrays of thresholds */
-	struct mutex thresholds_lock;
-	/* thresholds for memory usage. RCU-protected */
-	struct mem_cgroup_thresholds thresholds;
-	/* thresholds for mem+swap usage. RCU-protected */
-	struct mem_cgroup_thresholds memsw_thresholds;
-	/* For oom notifier event fd */
-	struct list_head oom_notify;
-	/*
-	 * Should we move charges of a task when a task is moved into this
-	 * mem_cgroup ? And what type of charges should we move ?
-	 */
-	unsigned long move_charge_at_immigrate;
-	/* taken only while moving_account > 0 */
-	spinlock_t		move_lock;
-	unsigned long		move_lock_flags;
-	CACHELINE_PADDING(_pad1_);
 	/* memory.stat */
 	struct memcg_vmstats	*vmstats;
 	 */
 	unsigned long		socket_pressure;
-	/* Legacy tcp memory accounting */
-	bool			tcpmem_active;
-	int			tcpmem_pressure;
-#ifdef CONFIG_MEMCG_KMEM
 	int kmemcg_id;
 	/*
 	 * memcg->objcg is wiped out as a part of the objcg repaprenting
 	struct obj_cgroup	*orig_objcg;
 	/* list of inherited objcgs, protected by objcg_lock */
 	struct list_head objcg_list;
-#endif
-	CACHELINE_PADDING(_pad2_);
-	/*
-	 * set > 0 if pages under this cgroup are moving to other cgroup.
-	 */
-	atomic_t		moving_account;
-	struct task_struct	*move_lock_task;
 	struct memcg_vmstats_percpu __percpu *vmstats_percpu;
 	struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
 #endif
-	/* List of events which userspace want to receive */
-	struct list_head event_list;
-	spinlock_t event_list_lock;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	struct deferred_split deferred_split_queue;
 #endif
 	struct lru_gen_mm_list mm_list;
 #endif
+#ifdef CONFIG_MEMCG_V1
+	/* Legacy consumer-oriented counters */
+	struct page_counter kmem;		/* v1 only */
+	struct page_counter tcpmem;		/* v1 only */
+	unsigned long soft_limit;
+	/* protected by memcg_oom_lock */
+	bool oom_lock;
+	int under_oom;
+	/* OOM-Killer disable */
+	int oom_kill_disable;
+	/* protect arrays of thresholds */
+	struct mutex thresholds_lock;
+	/* thresholds for memory usage. RCU-protected */
+	struct mem_cgroup_thresholds thresholds;
+	/* thresholds for mem+swap usage. RCU-protected */
+	struct mem_cgroup_thresholds memsw_thresholds;
+	/* For oom notifier event fd */
+	struct list_head oom_notify;
+	/*
+	 * Should we move charges of a task when a task is moved into this
+	 * mem_cgroup ? And what type of charges should we move ?
+	 */
+	unsigned long move_charge_at_immigrate;
+	/* taken only while moving_account > 0 */
+	spinlock_t move_lock;
+	unsigned long move_lock_flags;
+	/* Legacy tcp memory accounting */
+	bool tcpmem_active;
+	int tcpmem_pressure;
+	/*
+	 * set > 0 if pages under this cgroup are moving to other cgroup.
+	 */
+	atomic_t moving_account;
+	struct task_struct *move_lock_task;
+	/* List of events which userspace want to receive */
+	struct list_head event_list;
+	spinlock_t event_list_lock;
+#endif /* CONFIG_MEMCG_V1 */
 	struct mem_cgroup_per_node *nodeinfo[];
 };
 	return __folio_memcg(folio);
 }
-static inline struct mem_cgroup *page_memcg(struct page *page)
-{
-	return folio_memcg(page_folio(page));
-}
 /**
  * folio_memcg_rcu - Locklessly get the memory cgroup associated with a folio.
  * @folio: Pointer to the folio.
 	return memcg;
 }
-#ifdef CONFIG_MEMCG_KMEM
 /*
  * folio_memcg_kmem - Check if the folio has the memcg_kmem flag set.
  * @folio: Pointer to the folio.
 	return folio->memcg_data & MEMCG_DATA_KMEM;
 }
-#else
-static inline bool folio_memcg_kmem(struct folio *folio)
-{
-	return false;
-}
-#endif
 static inline bool PageMemcgKmem(struct page *page)
 {
 	return folio_memcg_kmem(page_folio(page));
 void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg);
-static inline void mem_cgroup_enter_user_fault(void)
-{
-	WARN_ON(current->in_user_fault);
-	current->in_user_fault = 1;
-}
-static inline void mem_cgroup_exit_user_fault(void)
-{
-	WARN_ON(!current->in_user_fault);
-	current->in_user_fault = 0;
-}
-static inline bool task_in_memcg_oom(struct task_struct *p)
-{
-	return p->memcg_in_oom;
-}
-bool mem_cgroup_oom_synchronize(bool wait);
 struct mem_cgroup *mem_cgroup_get_oom_group(struct task_struct *victim,
 					    struct mem_cgroup *oom_domain);
 void mem_cgroup_print_oom_group(struct mem_cgroup *memcg);
-void folio_memcg_lock(struct folio *folio);
-void folio_memcg_unlock(struct folio *folio);
 void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
 		       int val);
-/* try to stablize folio_memcg() for all the pages in a memcg */
-static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
-{
-	rcu_read_lock();
-	if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account))
-		return true;
-	rcu_read_unlock();
-	return false;
-}
-static inline void mem_cgroup_unlock_pages(void)
-{
-	rcu_read_unlock();
-}
 /* idx can be of type enum memcg_stat_item or node_stat_item */
 static inline void mod_memcg_state(struct mem_cgroup *memcg,
 				   enum memcg_stat_item idx, int val)
 		return;
 	rcu_read_lock();
-	memcg = page_memcg(page);
+	memcg = folio_memcg(page_folio(page));
 	if (memcg)
 		mod_memcg_state(memcg, idx, val);
 	rcu_read_unlock();
 void split_page_memcg(struct page *head, int old_order, int new_order);
-unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,
-						gfp_t gfp_mask,
-						unsigned long *total_scanned);
 #else /* CONFIG_MEMCG */
 #define MEM_CGROUP_ID_SHIFT	0
 	return NULL;
 }
-static inline struct mem_cgroup *page_memcg(struct page *page)
-{
-	return NULL;
-}
 static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio)
 {
 	WARN_ON_ONCE(!rcu_read_lock_held());
 {
 }
-static inline void folio_memcg_lock(struct folio *folio)
-{
-}
-static inline void folio_memcg_unlock(struct folio *folio)
-{
-}
-static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
-{
-	/* to match folio_memcg_rcu() */
-	rcu_read_lock();
-	return true;
-}
-static inline void mem_cgroup_unlock_pages(void)
-{
-	rcu_read_unlock();
-}
 static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)
 {
 }
-static inline void mem_cgroup_enter_user_fault(void)
-{
-}
-static inline void mem_cgroup_exit_user_fault(void)
-{
-}
-static inline bool task_in_memcg_oom(struct task_struct *p)
-{
-	return false;
-}
-static inline bool mem_cgroup_oom_synchronize(bool wait)
-{
-	return false;
-}
 static inline struct mem_cgroup *mem_cgroup_get_oom_group(
 	struct task_struct *victim, struct mem_cgroup *oom_domain)
 {
 static inline void split_page_memcg(struct page *head, int old_order, int new_order)
 {
 }
-static inline
-unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,
-					    gfp_t gfp_mask,
-					    unsigned long *total_scanned)
-{
-	return 0;
-}
 #endif /* CONFIG_MEMCG */
 /*
  * if MEMCG_DATA_OBJEXTS is set.
  */
 struct slabobj_ext {
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 	struct obj_cgroup *objcg;
 #endif
 #ifdef CONFIG_MEM_ALLOC_PROFILING
 	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
 }
-/* Test requires a stable page->memcg binding, see page_memcg() */
+/* Test requires a stable folio->memcg binding, see folio_memcg() */
 static inline bool folio_matches_lruvec(struct folio *folio,
 		struct lruvec *lruvec)
 {
 void mem_cgroup_sk_free(struct sock *sk);
 static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg)
 {
+#ifdef CONFIG_MEMCG_V1
 	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		return !!memcg->tcpmem_pressure;
+#endif /* CONFIG_MEMCG_V1 */
 	do {
 		if (time_before(jiffies, READ_ONCE(memcg->socket_pressure)))
 			return true;
 }
 #endif
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 bool mem_cgroup_kmem_disabled(void);
 int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
 void __memcg_kmem_uncharge_page(struct page *page, int order);
 {
 }
-#endif /* CONFIG_MEMCG_KMEM */
+#endif /* CONFIG_MEMCG */
-#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)
+#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
 bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
 void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
 void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
 }
 #endif
+/* Cgroup v1-related declarations */
+#ifdef CONFIG_MEMCG_V1
+unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
+					gfp_t gfp_mask,
+					unsigned long *total_scanned);
+bool mem_cgroup_oom_synchronize(bool wait);
+static inline bool task_in_memcg_oom(struct task_struct *p)
+{
+	return p->memcg_in_oom;
+}
+void folio_memcg_lock(struct folio *folio);
+void folio_memcg_unlock(struct folio *folio);
+/* try to stablize folio_memcg() for all the pages in a memcg */
+static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
+{
+	rcu_read_lock();
+	if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account))
+		return true;
+	rcu_read_unlock();
+	return false;
+}
+static inline void mem_cgroup_unlock_pages(void)
+{
+	rcu_read_unlock();
+}
+static inline void mem_cgroup_enter_user_fault(void)
+{
+	WARN_ON(current->in_user_fault);
+	current->in_user_fault = 1;
+}
+static inline void mem_cgroup_exit_user_fault(void)
+{
+	WARN_ON(!current->in_user_fault);
+	current->in_user_fault = 0;
+}
+#else /* CONFIG_MEMCG_V1 */
+static inline
+unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
+					gfp_t gfp_mask,
+					unsigned long *total_scanned)
+{
+	return 0;
+}
+static inline void folio_memcg_lock(struct folio *folio)
+{
+}
+static inline void folio_memcg_unlock(struct folio *folio)
+{
+}
+static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
+{
+	/* to match folio_memcg_rcu() */
+	rcu_read_lock();
+	return true;
+}
+static inline void mem_cgroup_unlock_pages(void)
+{
+	rcu_read_unlock();
+}
+static inline bool task_in_memcg_oom(struct task_struct *p)
+{
+	return false;
+}
+static inline bool mem_cgroup_oom_synchronize(bool wait)
+{
+	return false;
+}
+static inline void mem_cgroup_enter_user_fault(void)
+{
+}
+static inline void mem_cgroup_exit_user_fault(void)
+{
+}
+#endif /* CONFIG_MEMCG_V1 */
 #endif /* _LINUX_MEMCONTROL_H */

include/linux/memfd.h+5−0

 #ifdef CONFIG_MEMFD_CREATE
 extern long memfd_fcntl(struct file *file, unsigned int cmd, unsigned int arg);
+struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx);
 #else
 static inline long memfd_fcntl(struct file *f, unsigned int c, unsigned int a)
 {
 	return -EINVAL;
 }
+static inline struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)
+{
+	return ERR_PTR(-EINVAL);
+}
 #endif
 #endif /* __LINUX_MEMFD_H */

include/linux/memory-tiers.h+2−0

 #ifdef CONFIG_NUMA
 extern bool numa_demotion_enabled;
 extern struct memory_dev_type *default_dram_type;
+extern nodemask_t default_dram_nodes;
 struct memory_dev_type *alloc_memory_type(int adistance);
 void put_memory_type(struct memory_dev_type *memtype);
 void init_node_memory_type(int node, struct memory_dev_type *default_type);
 #define numa_demotion_enabled	false
 #define default_dram_type	NULL
+#define default_dram_nodes	NODE_MASK_NONE
 /*
  * CONFIG_NUMA implementation returns non NULL error.
  */

include/linux/memory_hotplug.h+2−2

@@ -175,8 +175,8 @@ extern int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages, · 174 unmodified lines
175	extern void mhp_deinit_memmap_on_memory(unsigned long pfn, unsigned long nr_pages);	175	extern void mhp_deinit_memmap_on_memory(unsigned long pfn, unsigned long nr_pages);
176	extern int online_pages(unsigned long pfn, unsigned long nr_pages,	176	extern int online_pages(unsigned long pfn, unsigned long nr_pages,
177	struct zone zone, struct memory_group group);	177	struct zone zone, struct memory_group group);
178	extern void __offline_isolated_pages(unsigned long start_pfn,	178	extern unsigned long __offline_isolated_pages(unsigned long start_pfn,
179	unsigned long end_pfn);	179	unsigned long end_pfn);
180		180
181	typedef void (online_page_callback_t)(struct page page, unsigned int order);	181	typedef void (online_page_callback_t)(struct page page, unsigned int order);
182		182

include/linux/migrate.h+7−3

 #ifdef CONFIG_MIGRATION
 void putback_movable_pages(struct list_head *l);
-int migrate_folio_extra(struct address_space *mapping, struct folio *dst,
-		struct folio *src, enum migrate_mode mode, int extra_count);
 int migrate_folio(struct address_space *mapping, struct folio *dst,
 		struct folio *src, enum migrate_mode mode);
 int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free,
 void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl)
 		__releases(ptl);
 void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
-void folio_migrate_copy(struct folio *newfolio, struct folio *folio);
 int folio_migrate_mapping(struct address_space *mapping,
 		struct folio *newfolio, struct folio *folio, int extra_count);
 }
 #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
 			   int node);
 #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 static inline int migrate_misplaced_folio(struct folio *folio,
 					 struct vm_area_struct *vma, int node)
 {

include/linux/migrate_mode.h+1−5

  *	on most operations but not ->writepage as the potential stall time
  *	is too significant
  * MIGRATE_SYNC will block when migrating pages
- * MIGRATE_SYNC_NO_COPY will block when migrating pages but will not copy pages
- *	with the CPU. Instead, page copy happens outside the migratepage()
- *	callback and is likely using a DMA engine. See migrate_vma() and HMM
- *	(mm/hmm.c) for users of this mode.
  */
 enum migrate_mode {
 	MIGRATE_ASYNC,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
-	MIGRATE_SYNC_NO_COPY,
 };
 enum migrate_reason {
 	MR_CONTIG_RANGE,
 	MR_LONGTERM_PIN,
 	MR_DEMOTION,
+	MR_DAMON,
 	MR_TYPES
 };

include/linux/mm.h+13−68

 /*
  * How many times the entire folio is mapped as a single unit (eg by a
  * PMD or PUD entry).  This is probably not what you want, except for
- * debugging purposes - it does not include PTE-mapped sub-pages; look
+ * debugging purposes or implementation of other core folio_*() primitives.
- * at folio_mapcount() or page_mapcount() instead.
  */
 static inline int folio_entire_mapcount(const struct folio *folio)
 {
 	return atomic_read(&folio->_entire_mapcount) + 1;
 }
-/*
- * The atomic page->_mapcount, starts from -1: so that transitions
- * both from it and to it can be tracked, using atomic_inc_and_test
- * and atomic_add_negative(-1).
- */
-static inline void page_mapcount_reset(struct page *page)
-{
-	atomic_set(&(page)->_mapcount, -1);
-}
-/**
- * page_mapcount() - Number of times this precise page is mapped.
- * @page: The page.
- *
- * The number of times this page is mapped.  If this page is part of
- * a large folio, it includes the number of times this page is mapped
- * as part of that folio.
- *
- * Will report 0 for pages which cannot be mapped into userspace, eg
- * slab, page tables and similar.
- */
-static inline int page_mapcount(struct page *page)
-{
-	int mapcount = atomic_read(&page->_mapcount) + 1;
-	/* Handle page_has_type() pages */
-	if (mapcount < PAGE_MAPCOUNT_RESERVE + 1)
-		mapcount = 0;
-	if (unlikely(PageCompound(page)))
-		mapcount += folio_entire_mapcount(page_folio(page));
-	return mapcount;
-}
 static inline int folio_large_mapcount(const struct folio *folio)
 {
 	VM_WARN_ON_FOLIO(!folio_test_large(folio), folio);
 void split_page(struct page *page, unsigned int order);
 void folio_copy(struct folio *dst, struct folio *src);
+int folio_mc_copy(struct folio *dst, struct folio *src);
 unsigned long nr_free_buffer_pages(void);
  * issue.
  *
  * Locking: the lockless algorithm described in folio_try_get_rcu()
- * provides safe operation for get_user_pages(), page_mkclean() and
+ * provides safe operation for get_user_pages(), folio_mkclean() and
  * other calls that race to set up page table entries.
  */
 #define GUP_PIN_COUNTING_BIAS (1U << 10)
 void unpin_user_page(struct page *page);
+void unpin_folio(struct folio *folio);
 void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages,
 				 bool make_dirty);
 void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages,
 				      bool make_dirty);
 void unpin_user_pages(struct page **pages, unsigned long npages);
+void unpin_folios(struct folio **folios, unsigned long nfolios);
 static inline bool is_cow_mapping(vm_flags_t flags)
 {
  *
  * For more information, please see Documentation/core-api/pin_user_pages.rst.
  *
- * Return: True, if it is likely that the page has been "dma-pinned".
+ * Return: True, if it is likely that the folio has been "dma-pinned".
- * False, if the page is definitely not dma-pinned.
+ * False, if the folio is definitely not dma-pinned.
  */
 static inline bool folio_maybe_dma_pinned(struct folio *folio)
 {
 		GUP_PIN_COUNTING_BIAS;
 }
-static inline bool page_maybe_dma_pinned(struct page *page)
-{
-	return folio_maybe_dma_pinned(page_folio(page));
-}
 /*
  * This should most likely only be called during fork() to see whether we
  * should break the cow immediately for an anon page on the src mm.
 	return page_address(&folio->page);
 }
-extern pgoff_t __page_file_index(struct page *page);
-/*
- * Return the pagecache index of the passed page.  Regular pagecache pages
- * use ->index whereas swapcache pages use swp_offset(->private)
- */
-static inline pgoff_t page_index(struct page *page)
-{
-	if (unlikely(PageSwapCache(page)))
-		return __page_file_index(page);
-	return page->index;
-}
 /*
  * Return true only if the page has been allocated with
  * ALLOC_NO_WATERMARKS and the low watermark was not
 		    struct page **pages, unsigned int gup_flags);
 long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
 		    struct page **pages, unsigned int gup_flags);
+long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end,
+		      struct folio **folios, unsigned int max_folios,
+		      pgoff_t *offset);
 int get_user_pages_fast(unsigned long start, int nr_pages,
 			unsigned int gup_flags, struct page **pages);
 					bool *migratable_cleared);
 void num_poisoned_pages_inc(unsigned long pfn);
 void num_poisoned_pages_sub(unsigned long pfn, long i);
-struct task_struct *task_early_kill(struct task_struct *tsk, int force_early);
 #else
 static inline void memory_failure_queue(unsigned long pfn, int flags)
 {
 }
 #endif
-#if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_KSM)
-void add_to_kill_ksm(struct task_struct *tsk, struct page *p,
-		     struct vm_area_struct *vma, struct list_head *to_kill,
-		     unsigned long ksm_addr);
-#endif
 #if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_MEMORY_HOTPLUG)
 extern void memblk_nr_poison_inc(unsigned long pfn);
 extern void memblk_nr_poison_sub(unsigned long pfn, long i);
 enum mf_action_page_type {
 	MF_MSG_KERNEL,
 	MF_MSG_KERNEL_HIGH_ORDER,
-	MF_MSG_SLAB,
 	MF_MSG_DIFFERENT_COMPOUND,
 	MF_MSG_HUGE,
 	MF_MSG_FREE_HUGE,
+	MF_MSG_GET_HWPOISON,
 	MF_MSG_UNMAP_FAILED,
 	MF_MSG_DIRTY_SWAPCACHE,
 	MF_MSG_CLEAN_SWAPCACHE,
 	MF_MSG_BUDDY,
 	MF_MSG_DAX,
 	MF_MSG_UNSPLIT_THP,
+	MF_MSG_ALREADY_POISONED,
 	MF_MSG_UNKNOWN,
 };
 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
-extern void clear_huge_page(struct page *page,
+void folio_zero_user(struct folio *folio, unsigned long addr_hint);
-			    unsigned long addr_hint,
-			    unsigned int pages_per_huge_page);
 int copy_user_large_folio(struct folio *dst, struct folio *src,
 			  unsigned long addr_hint,
 			  struct vm_area_struct *vma);

include/linux/mm_types.h+22−11

  * which is guaranteed to be aligned.  If you use the same storage as
  * page->mapping, you must restore it to NULL before freeing the page.
  *
- * If your page will not be mapped to userspace, you can also use the four
+ * The mapcount field must not be used for own purposes.
- * bytes in the mapcount union, but you must call page_mapcount_reset()
- * before freeing it.
  *
  * If you want to use the refcount field, it must be used in such a way
  * that other CPUs temporarily incrementing and then decrementing the
 	union {		/* This union is 4 bytes in size. */
 		/*
-		 * If the page can be mapped to userspace, encodes the number
+		 * For head pages of typed folios, the value stored here
-		 * of times this page is referenced by a page table.
+		 * allows for determining what this page is used for. The
+		 * tail pages of typed folios will not store a type
+		 * (page_type == _mapcount == -1).
+		 *
+		 * See page-flags.h for a list of page types which are currently
+		 * stored here.
+		 *
+		 * Owners of typed folios may reuse the lower 16 bit of the
+		 * head page page_type field after setting the page type,
+		 * but must reset these 16 bit to -1 before clearing the
+		 * page type.
 		 */
-		atomic_t _mapcount;
+		unsigned int page_type;
 		/*
-		 * If the page is neither PageSlab nor mappable to userspace,
+		 * For pages that are part of non-typed folios for which mappings
-		 * the value stored here may help determine what this page
+		 * are tracked via the RMAP, encodes the number of times this page
-		 * is used for.  See page-flags.h for a list of page types
+		 * is directly referenced by a page table.
-		 * which are currently stored here.
+		 *
+		 * Note that the mapcount is always initialized to -1, so that
+		 * transitions both from it and to it can be tracked, using
+		 * atomic_inc_and_test() and atomic_add_negative(-1).
 		 */
-		unsigned int page_type;
+		atomic_t _mapcount;
 	};
 	/* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */

include/linux/mmzone.h+2−0

@@ -220,6 +220,8 @@ enum node_stat_item { · 219 unmodified lines
220	PGDEMOTE_KSWAPD,	220	PGDEMOTE_KSWAPD,
221	PGDEMOTE_DIRECT,	221	PGDEMOTE_DIRECT,
222	PGDEMOTE_KHUGEPAGED,	222	PGDEMOTE_KHUGEPAGED,
		223	NR_MEMMAP, /* page metadata allocated through buddy allocator */
		224	NR_MEMMAP_BOOT, /* page metadata allocated through boot allocator */
223	NR_VM_NODE_STAT_ITEMS	225	NR_VM_NODE_STAT_ITEMS
224	};	226	};
225		227

include/linux/page-flags.h+44−38

@@ -30,16 +30,11 @@ · 29 unmodified lines
30	* - Pages falling into physical memory gaps - not IORESOURCE_SYSRAM. Trying	30	* - Pages falling into physical memory gaps - not IORESOURCE_SYSRAM. Trying
31	* to read/write these pages might end badly. Don't touch!	31	* to read/write these pages might end badly. Don't touch!
32	* - The zero page(s)	32	* - The zero page(s)
33	* - Pages not added to the page allocator when onlining a section because
34	* they were excluded via the online_page_callback() or because they are
35	* PG_hwpoison.
36	* - Pages allocated in the context of kexec/kdump (loaded kernel image,	33	* - Pages allocated in the context of kexec/kdump (loaded kernel image,
37	* control pages, vmcoreinfo)	34	* control pages, vmcoreinfo)
38	* - MMIO/DMA pages. Some architectures don't allow to ioremap pages that are	35	* - MMIO/DMA pages. Some architectures don't allow to ioremap pages that are
39	* not marked PG_reserved (as they might be in use by somebody else who does	36	* not marked PG_reserved (as they might be in use by somebody else who does
40	* not respect the caching strategy).	37	* not respect the caching strategy).
41	* - Pages part of an offline section (struct pages of offline sections should
42	* not be trusted as they will be initialized when first onlined).
43	* - MCA pages on ia64	38	* - MCA pages on ia64
44	* - Pages holding CPU notes for POWER Firmware Assisted Dump	39	* - Pages holding CPU notes for POWER Firmware Assisted Dump
45	* - Device memory (e.g. PMEM, DAX, HMM)	40	* - Device memory (e.g. PMEM, DAX, HMM)
@@ -616,11 +611,6 @@ PAGEFLAG_FALSE(Uncached, uncached) · 570 unmodified lines
616	PAGEFLAG(HWPoison, hwpoison, PF_ANY)	611	PAGEFLAG(HWPoison, hwpoison, PF_ANY)
617	TESTSCFLAG(HWPoison, hwpoison, PF_ANY)	612	TESTSCFLAG(HWPoison, hwpoison, PF_ANY)
618	#define __PG_HWPOISON (1UL << PG_hwpoison)	613	#define __PG_HWPOISON (1UL << PG_hwpoison)
619	#define MAGIC_HWPOISON 0x48575053U /* HWPS */
620	extern void SetPageHWPoisonTakenOff(struct page *page);
621	extern void ClearPageHWPoisonTakenOff(struct page *page);
622	extern bool take_page_off_buddy(struct page *page);
623	extern bool put_page_back_buddy(struct page *page);
624	#else	614	#else
625	PAGEFLAG_FALSE(HWPoison, hwpoison)	615	PAGEFLAG_FALSE(HWPoison, hwpoison)
626	#define __PG_HWPOISON 0	616	#define __PG_HWPOISON 0
@@ -655,27 +645,28 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted) · 28 unmodified lines
655	#endif	645	#endif
656		646
657	/*	647	/*
658	* On an anonymous page mapped into a user virtual memory area,	648	* On an anonymous folio mapped into a user virtual memory area,
659	* page->mapping points to its anon_vma, not to a struct address_space;	649	* folio->mapping points to its anon_vma, not to a struct address_space;
660	* with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h.	650	* with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h.
661	*	651	*
662	* On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled,	652	* On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled,
663	* the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON	653	* the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON
664	* bit; and then page->mapping points, not to an anon_vma, but to a private	654	* bit; and then folio->mapping points, not to an anon_vma, but to a private
665	* structure which KSM associates with that merged page. See ksm.h.	655	* structure which KSM associates with that merged page. See ksm.h.
666	*	656	*
667	* PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable	657	* PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable
668	* page and then page->mapping points to a struct movable_operations.	658	* page and then folio->mapping points to a struct movable_operations.
669	*	659	*
670	* Please note that, confusingly, "page_mapping" refers to the inode	660	* Please note that, confusingly, "folio_mapping" refers to the inode
671	* address_space which maps the page from disk; whereas "page_mapped"	661	* address_space which maps the folio from disk; whereas "folio_mapped"
672	* refers to user virtual address space into which the page is mapped.	662	* refers to user virtual address space into which the folio is mapped.
673	*	663	*
674	* For slab pages, since slab reuses the bits in struct page to store its	664	* For slab pages, since slab reuses the bits in struct page to store its
675	* internal states, the page->mapping does not exist as such, nor do these	665	* internal states, the folio->mapping does not exist as such, nor do
676	* flags below. So in order to avoid testing non-existent bits, please	666	* these flags below. So in order to avoid testing non-existent bits,
677	* make sure that PageSlab(page) actually evaluates to false before calling	667	* please make sure that folio_test_slab(folio) actually evaluates to
678	* the following functions (e.g., PageAnon). See mm/slab.h.	668	* false before calling the following functions (e.g., folio_test_anon).
		669	* See mm/slab.h.
679	*/	670	*/
680	#define PAGE_MAPPING_ANON 0x1	671	#define PAGE_MAPPING_ANON 0x1
681	#define PAGE_MAPPING_MOVABLE 0x2	672	#define PAGE_MAPPING_MOVABLE 0x2
@@ -945,22 +936,28 @@ PAGEFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) · 263 unmodified lines
945	*/	936	*/
946		937
947	enum pagetype {	938	enum pagetype {
948	PG_buddy = 0x00000080,	939	PG_buddy = 0x40000000,
949	PG_offline = 0x00000100,	940	PG_offline = 0x20000000,
950	PG_table = 0x00000200,	941	PG_table = 0x10000000,
951	PG_guard = 0x00000400,	942	PG_guard = 0x08000000,
952	PG_hugetlb = 0x00000800,	943	PG_hugetlb = 0x04000000,
953	PG_slab = 0x00001000,	944	PG_slab = 0x02000000,
954		945	PG_zsmalloc = 0x01000000,
955	PAGE_TYPE_BASE = 0xf0000000,	946
956	/* Reserve 0x0000007f to catch underflows of _mapcount */	947	PAGE_TYPE_BASE = 0x80000000,
957	PAGE_MAPCOUNT_RESERVE = -128,	948
		949	/*
		950	* Reserve 0xffff0000 - 0xfffffffe to catch _mapcount underflows and
		951	* allow owners that set a type to reuse the lower 16 bit for their own
		952	* purposes.
		953	*/
		954	PAGE_MAPCOUNT_RESERVE = ~0x0000ffff,
958	};	955	};
959		956
960	#define PageType(page, flag) \	957	#define PageType(page, flag) \
961	((page->page_type & (PAGE_TYPE_BASE \| flag)) == PAGE_TYPE_BASE)	958	((READ_ONCE(page->page_type) & (PAGE_TYPE_BASE \| flag)) == PAGE_TYPE_BASE)
962	#define folio_test_type(folio, flag) \	959	#define folio_test_type(folio, flag) \
963	((folio->page.page_type & (PAGE_TYPE_BASE \| flag)) == PAGE_TYPE_BASE)	960	((READ_ONCE(folio->page.page_type) & (PAGE_TYPE_BASE \| flag)) == PAGE_TYPE_BASE)
964		961
965	static inline int page_type_has_type(unsigned int page_type)	962	static inline int page_type_has_type(unsigned int page_type)
966	{	963	{
@@ -969,7 +966,7 @@ static inline int page_type_has_type(unsigned int page_type) · 2 unmodified lines
969		966
970	static inline int page_has_type(const struct page *page)	967	static inline int page_has_type(const struct page *page)
971	{	968	{
972	return page_type_has_type(page->page_type);	969	return page_type_has_type(READ_ONCE(page->page_type));
973	}	970	}
974		971
975	#define FOLIO_TYPE_OPS(lname, fname) \	972	#define FOLIO_TYPE_OPS(lname, fname) \
@@ -1018,15 +1015,22 @@ PAGE_TYPE_OPS(Buddy, buddy, buddy) · 42 unmodified lines
1018	* The content of these pages is effectively stale. Such pages should not	1015	* The content of these pages is effectively stale. Such pages should not
1019	* be touched (read/write/dump/save) except by their owner.	1016	* be touched (read/write/dump/save) except by their owner.
1020	*	1017	*
		1018	* When a memory block gets onlined, all pages are initialized with a
		1019	* refcount of 1 and PageOffline(). generic_online_page() will
		1020	* take care of clearing PageOffline().
		1021	*
1021	* If a driver wants to allow to offline unmovable PageOffline() pages without	1022	* If a driver wants to allow to offline unmovable PageOffline() pages without
1022	* putting them back to the buddy, it can do so via the memory notifier by	1023	* putting them back to the buddy, it can do so via the memory notifier by
1023	* decrementing the reference count in MEM_GOING_OFFLINE and incrementing the	1024	* decrementing the reference count in MEM_GOING_OFFLINE and incrementing the
1024	* reference count in MEM_CANCEL_OFFLINE. When offlining, the PageOffline()	1025	* reference count in MEM_CANCEL_OFFLINE. When offlining, the PageOffline()
1025	* pages (now with a reference count of zero) are treated like free pages,	1026	* pages (now with a reference count of zero) are treated like free (unmanaged)
1026	* allowing the containing memory block to get offlined. A driver that	1027	* pages, allowing the containing memory block to get offlined. A driver that
1027	* relies on this feature is aware that re-onlining the memory block will	1028	* relies on this feature is aware that re-onlining the memory block will
1028	* require to re-set the pages PageOffline() and not giving them to the	1029	* require not giving them to the buddy via generic_online_page().
1029	* buddy via online_page_callback_t.	1030	*
		1031	* Memory offlining code will not adjust the managed page count for any
		1032	* PageOffline() pages, treating them like they were never exposed to the
		1033	* buddy using generic_online_page().
1030	*	1034	*
1031	* There are drivers that mark a page PageOffline() and expect there won't be	1035	* There are drivers that mark a page PageOffline() and expect there won't be
1032	* any further access to page content. PFN walkers that read content of random	1036	* any further access to page content. PFN walkers that read content of random
@@ -1070,6 +1074,8 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) · 37 unmodified lines
1070	FOLIO_TEST_FLAG_FALSE(hugetlb)	1074	FOLIO_TEST_FLAG_FALSE(hugetlb)
1071	#endif	1075	#endif
1072		1076
		1077	PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
		1078
1073	/**	1079	/**
1074	* PageHuge - Determine if the page belongs to hugetlbfs	1080	* PageHuge - Determine if the page belongs to hugetlbfs
1075	* @page: The page to test.	1081	* @page: The page to test.

include/linux/page_counter.h+4−0

@@ -81,4 +81,8 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) · 80 unmodified lines
81	counter->watermark = page_counter_read(counter);	81	counter->watermark = page_counter_read(counter);
82	}	82	}
83		83
		84	void page_counter_calculate_protection(struct page_counter *root,
		85	struct page_counter *counter,
		86	bool recursive_protection);
		87
84	#endif /* _LINUX_PAGE_COUNTER_H */	88	#endif /* _LINUX_PAGE_COUNTER_H */

include/linux/pagemap.h+7−26

 #endif
 }
-struct address_space *page_mapping(struct page *);
 struct address_space *folio_mapping(struct folio *);
 struct address_space *swapcache_mapping(struct folio *);
 			mapping_gfp_mask(mapping));
 }
-#define swapcache_index(folio)	__page_file_index(&(folio)->page)
+extern pgoff_t __folio_swap_cache_index(struct folio *folio);
 /**
  * folio_index - File index of a folio.
  */
 static inline pgoff_t folio_index(struct folio *folio)
 {
-        if (unlikely(folio_test_swapcache(folio)))
+	if (unlikely(folio_test_swapcache(folio)))
-                return swapcache_index(folio);
+		return __folio_swap_cache_index(folio);
-        return folio->index;
+	return folio->index;
 }
 /**
 	return ((loff_t)page->index) << PAGE_SHIFT;
 }
-static inline loff_t page_file_offset(struct page *page)
-{
-	return ((loff_t)page_index(page)) << PAGE_SHIFT;
-}
 /**
  * folio_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
 	return page_offset(&folio->page);
 }
-/**
- * folio_file_pos - Returns the byte position of this folio in its file.
- * @folio: The folio.
- *
- * This differs from folio_pos() for folios which belong to a swap file.
- * NFS is the only filesystem today which needs to use folio_file_pos().
- */
-static inline loff_t folio_file_pos(struct folio *folio)
-{
-	return page_file_offset(&folio->page);
-}
 /*
  * Get the offset in PAGE_SIZE (even for hugetlb folios).
  */
  * @mapping: address_space which holds the pagecache and I/O vectors
  * @ra: file_ra_state which holds the readahead state
  * @file: Used by the filesystem for authentication.
- * @folio: The folio at @index which triggered the readahead call.
+ * @folio: The folio which triggered the readahead call.
- * @index: Index of first page to be read.
  * @req_count: Total number of pages being read by the caller.
  *
  * page_cache_async_readahead() should be called when a page is used which
 static inline
 void page_cache_async_readahead(struct address_space *mapping,
 		struct file_ra_state *ra, struct file *file,
-		struct folio *folio, pgoff_t index, unsigned long req_count)
+		struct folio *folio, unsigned long req_count)
 {
-	DEFINE_READAHEAD(ractl, file, ra, mapping, index);
+	DEFINE_READAHEAD(ractl, file, ra, mapping, folio->index);
 	page_cache_async_ra(&ractl, folio, req_count);
 }

include/linux/percpu-defs.h+6−0

 	raw_cpu_cmpxchg(pcp, oval, nval);				\
 })
+#define __this_cpu_try_cmpxchg(pcp, ovalp, nval)			\
+({									\
+	__this_cpu_preempt_check("try_cmpxchg");			\
+	raw_cpu_try_cmpxchg(pcp, ovalp, nval);				\
+})
 #define __this_cpu_sub(pcp, val)	__this_cpu_add(pcp, -(typeof(pcp))(val))
 #define __this_cpu_inc(pcp)		__this_cpu_add(pcp, 1)
 #define __this_cpu_dec(pcp)		__this_cpu_sub(pcp, 1)

include/linux/pgalloc_tag.h+4−3

 static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext)
 {
-	return (void *)page_ext + page_alloc_tagging_ops.offset;
+	return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops);
 }
 static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref)
 static inline void pgalloc_tag_split(struct page *page, unsigned int nr)
 {
 	int i;
+	struct page_ext *first_page_ext;
 	struct page_ext *page_ext;
 	union codetag_ref *ref;
 	struct alloc_tag *tag;
 	if (!mem_alloc_profiling_enabled())
 		return;
-	page_ext = page_ext_get(page);
+	first_page_ext = page_ext = page_ext_get(page);
 	if (unlikely(!page_ext))
 		return;
 		page_ext = page_ext_next(page_ext);
 	}
 out:
-	page_ext_put(page_ext);
+	page_ext_put(first_page_ext);
 }
 static inline struct alloc_tag *pgalloc_tag_get(struct page *page)

include/linux/pgtable.h+31−9

@@ -729,13 +729,18 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, · 728 unmodified lines
729	* fault. This function updates TLB only, do nothing with cache or others.	729	* fault. This function updates TLB only, do nothing with cache or others.
730	* It is the difference with function update_mmu_cache.	730	* It is the difference with function update_mmu_cache.
731	*/	731	*/
732	#ifndef __HAVE_ARCH_UPDATE_MMU_TLB	732	#ifndef update_mmu_tlb_range
		733	static inline void update_mmu_tlb_range(struct vm_area_struct *vma,
		734	unsigned long address, pte_t *ptep, unsigned int nr)
		735	{
		736	}
		737	#endif
		738
733	static inline void update_mmu_tlb(struct vm_area_struct *vma,	739	static inline void update_mmu_tlb(struct vm_area_struct *vma,
734	unsigned long address, pte_t *ptep)	740	unsigned long address, pte_t *ptep)
735	{	741	{
		742	update_mmu_tlb_range(vma, address, ptep, 1);
736	}	743	}
737	#define __HAVE_ARCH_UPDATE_MMU_TLB
738	#endif
739		744
740	/*	745	/*
741	* Some architectures may be able to avoid expensive synchronization	746	* Some architectures may be able to avoid expensive synchronization
@@ -1084,6 +1089,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · 342 unmodified lines
1084	})	1089	})
1085		1090
1086	#ifndef __HAVE_ARCH_DO_SWAP_PAGE	1091	#ifndef __HAVE_ARCH_DO_SWAP_PAGE
		1092	static inline void arch_do_swap_page_nr(struct mm_struct *mm,
		1093	struct vm_area_struct *vma,
		1094	unsigned long addr,
		1095	pte_t pte, pte_t oldpte,
		1096	int nr)
		1097	{
		1098
		1099	}
		1100	#else
1087	/*	1101	/*
1088	* Some architectures support metadata associated with a page. When a	1102	* Some architectures support metadata associated with a page. When a
1089	* page is being swapped out, this metadata must be saved so it can be	1103	* page is being swapped out, this metadata must be saved so it can be
@@ -1092,12 +1106,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · 2 unmodified lines
1092	* page as metadata for the page. arch_do_swap_page() can restore this	1106	* page as metadata for the page. arch_do_swap_page() can restore this
1093	* metadata when a page is swapped back in.	1107	* metadata when a page is swapped back in.
1094	*/	1108	*/
1095	static inline void arch_do_swap_page(struct mm_struct *mm,	1109	static inline void arch_do_swap_page_nr(struct mm_struct *mm,
1096	struct vm_area_struct *vma,	1110	struct vm_area_struct *vma,
1097	unsigned long addr,	1111	unsigned long addr,
1098	pte_t pte, pte_t oldpte)	1112	pte_t pte, pte_t oldpte,
1099	{	1113	int nr)
1100		1114	{
		1115	for (int i = 0; i < nr; i++) {
		1116	arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE,
		1117	pte_advance_pfn(pte, i),
		1118	pte_advance_pfn(oldpte, i));
		1119	}
1101	}	1120	}
1102	#endif	1121	#endif
1103		1122
@@ -1888,9 +1907,12 @@ typedef unsigned int pgtbl_mod_mask; · 784 unmodified lines
1888	#ifndef pmd_leaf_size	1907	#ifndef pmd_leaf_size
1889	#define pmd_leaf_size(x) PMD_SIZE	1908	#define pmd_leaf_size(x) PMD_SIZE
1890	#endif	1909	#endif
		1910	#ifndef __pte_leaf_size
1891	#ifndef pte_leaf_size	1911	#ifndef pte_leaf_size
1892	#define pte_leaf_size(x) PAGE_SIZE	1912	#define pte_leaf_size(x) PAGE_SIZE
1893	#endif	1913	#endif
		1914	#define __pte_leaf_size(x,y) pte_leaf_size(y)
		1915	#endif
1894		1916
1895	/*	1917	/*
1896	* We always define pmd_pfn for all archs as it's used in lots of generic	1918	* We always define pmd_pfn for all archs as it's used in lots of generic

include/linux/rmap.h+28−17

 	/* hugetlb folios are handled separately. */
 	VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio);
+	/* When (un)mapping zeropages, we should never touch ref+mapcount. */
+	VM_WARN_ON_FOLIO(is_zero_folio(folio), folio);
 	/*
 	 * TODO: we get driver-allocated folios that have nothing to do with
 	 * the rmap using vm_insert_page(); therefore, we cannot assume that
 void folio_add_anon_rmap_pmd(struct folio *, struct page *,
 		struct vm_area_struct *, unsigned long address, rmap_t flags);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
-		unsigned long address);
+		unsigned long address, rmap_t flags);
 void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages,
 		struct vm_area_struct *);
 #define folio_add_file_rmap_pte(folio, page, vma) \
 	unsigned int flags;
 };
-#define DEFINE_PAGE_VMA_WALK(name, _page, _vma, _address, _flags)	\
-	struct page_vma_mapped_walk name = {				\
-		.pfn = page_to_pfn(_page),				\
-		.nr_pages = compound_nr(_page),				\
-		.pgoff = page_to_pgoff(_page),				\
-		.vma = _vma,						\
-		.address = _address,					\
-		.flags = _flags,					\
-	}
 #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags)	\
 	struct page_vma_mapped_walk name = {				\
 		.pfn = folio_pfn(_folio),				\
 		spin_unlock(pvmw->ptl);
 }
+/**
+ * page_vma_mapped_walk_restart - Restart the page table walk.
+ * @pvmw: Pointer to struct page_vma_mapped_walk.
+ *
+ * It restarts the page table walk when changes occur in the page
+ * table, such as splitting a PMD. Ensures that the PTL held during
+ * the previous walk is released and resets the state to allow for
+ * a new walk starting at the current address stored in pvmw->address.
+ */
+static inline void
+page_vma_mapped_walk_restart(struct page_vma_mapped_walk *pvmw)
+{
+	WARN_ON_ONCE(!pvmw->pmd && !pvmw->pte);
+	if (likely(pvmw->ptl))
+		spin_unlock(pvmw->ptl);
+	else
+		WARN_ON_ONCE(1);
+	pvmw->ptl = NULL;
+	pvmw->pmd = NULL;
+	pvmw->pte = NULL;
+}
 bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw);
 /*
 void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked);
-unsigned long page_mapped_in_vma(struct page *page, struct vm_area_struct *vma);
 /*
  * rmap_walk_control: To control rmap traversing for specific needs
  *
 }
 #endif	/* CONFIG_MMU */
-static inline int page_mkclean(struct page *page)
-{
-	return folio_mkclean(page_folio(page));
-}
 #endif	/* _LINUX_RMAP_H */

include/linux/sched.h+5−4

 #ifndef TIF_RESTORE_SIGMASK
 	unsigned			restore_sigmask:1;
 #endif
-#ifdef CONFIG_MEMCG
+#ifdef CONFIG_MEMCG_V1
 	unsigned			in_user_fault:1;
 #endif
 #ifdef CONFIG_LRU_GEN
 	unsigned int			kcov_softirq;
 #endif
-#ifdef CONFIG_MEMCG
+#ifdef CONFIG_MEMCG_V1
 	struct mem_cgroup		*memcg_in_oom;
+#endif
+#ifdef CONFIG_MEMCG
 	/* Number of pages to reclaim on returning to userland: */
 	unsigned int			memcg_nr_pages_over_high;
 	/* Used by memcontrol for targeted memcg charge: */
 	struct mem_cgroup		*active_memcg;
-#endif
-#ifdef CONFIG_MEMCG_KMEM
+	/* Cache for current->cgroups->memcg->objcg lookups: */
 	struct obj_cgroup		*objcg;
 #endif

include/linux/shmem_fs.h+9−0

 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 extern bool shmem_is_huge(struct inode *inode, pgoff_t index, bool shmem_huge_force,
 			  struct mm_struct *mm, unsigned long vm_flags);
+unsigned long shmem_allowable_huge_orders(struct inode *inode,
+				struct vm_area_struct *vma, pgoff_t index,
+				bool global_huge);
 #else
 static __always_inline bool shmem_is_huge(struct inode *inode, pgoff_t index, bool shmem_huge_force,
 					  struct mm_struct *mm, unsigned long vm_flags)
 {
 	return false;
 }
+static inline unsigned long shmem_allowable_huge_orders(struct inode *inode,
+				struct vm_area_struct *vma, pgoff_t index,
+				bool global_huge)
+{
+	return 0;
+}
 #endif
 #ifdef CONFIG_SHMEM

include/linux/skbuff.h+8−0

 }
 #define __dev_alloc_pages(...)	alloc_hooks(__dev_alloc_pages_noprof(__VA_ARGS__))
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag).
+ */
 #define dev_alloc_pages(_order) __dev_alloc_pages(GFP_ATOMIC | __GFP_NOWARN, _order)
 /**
 }
 #define __dev_alloc_page(...)	alloc_hooks(__dev_alloc_page_noprof(__VA_ARGS__))
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag).
+ */
 #define dev_alloc_page()	dev_alloc_pages(0)
 /**

include/linux/skmsg.h+5−0

 int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,
 			 struct sk_msg *msg);
+/*
+ * This specialized allocator has to be a macro for its allocations to be
+ * accounted separately (to have a separate alloc_tag). The typecast is
+ * intentional to enforce typesafety.
+ */
 #define sk_psock_init_link()	\
 		((struct sk_psock_link *)kzalloc(sizeof(struct sk_psock_link),	\
 						 GFP_ATOMIC | __GFP_NOWARN))

include/linux/slab.h+6−6

@@ -41,7 +41,7 @@ enum _slab_flag_bits { · 40 unmodified lines
41	#ifdef CONFIG_FAILSLAB	41	#ifdef CONFIG_FAILSLAB
42	_SLAB_FAILSLAB,	42	_SLAB_FAILSLAB,
43	#endif	43	#endif
44	#ifdef CONFIG_MEMCG_KMEM	44	#ifdef CONFIG_MEMCG
45	_SLAB_ACCOUNT,	45	_SLAB_ACCOUNT,
46	#endif	46	#endif
47	#ifdef CONFIG_KASAN_GENERIC	47	#ifdef CONFIG_KASAN_GENERIC
@@ -171,7 +171,7 @@ enum _slab_flag_bits { · 123 unmodified lines
171	# define SLAB_FAILSLAB __SLAB_FLAG_UNUSED	171	# define SLAB_FAILSLAB __SLAB_FLAG_UNUSED
172	#endif	172	#endif
173	/* Account to memcg */	173	/* Account to memcg */
174	#ifdef CONFIG_MEMCG_KMEM	174	#ifdef CONFIG_MEMCG
175	# define SLAB_ACCOUNT __SLAB_FLAG_BIT(_SLAB_ACCOUNT)	175	# define SLAB_ACCOUNT __SLAB_FLAG_BIT(_SLAB_ACCOUNT)
176	#else	176	#else
177	# define SLAB_ACCOUNT __SLAB_FLAG_UNUSED	177	# define SLAB_ACCOUNT __SLAB_FLAG_UNUSED
@@ -407,7 +407,7 @@ enum kmalloc_cache_type { · 229 unmodified lines
407	#ifndef CONFIG_ZONE_DMA	407	#ifndef CONFIG_ZONE_DMA
408	KMALLOC_DMA = KMALLOC_NORMAL,	408	KMALLOC_DMA = KMALLOC_NORMAL,
409	#endif	409	#endif
410	#ifndef CONFIG_MEMCG_KMEM	410	#ifndef CONFIG_MEMCG
411	KMALLOC_CGROUP = KMALLOC_NORMAL,	411	KMALLOC_CGROUP = KMALLOC_NORMAL,
412	#endif	412	#endif
413	KMALLOC_RANDOM_START = KMALLOC_NORMAL,	413	KMALLOC_RANDOM_START = KMALLOC_NORMAL,
@@ -420,7 +420,7 @@ enum kmalloc_cache_type { · 6 unmodified lines
420	#ifdef CONFIG_ZONE_DMA	420	#ifdef CONFIG_ZONE_DMA
421	KMALLOC_DMA,	421	KMALLOC_DMA,
422	#endif	422	#endif
423	#ifdef CONFIG_MEMCG_KMEM	423	#ifdef CONFIG_MEMCG
424	KMALLOC_CGROUP,	424	KMALLOC_CGROUP,
425	#endif	425	#endif
426	NR_KMALLOC_TYPES	426	NR_KMALLOC_TYPES
@@ -436,7 +436,7 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; · 9 unmodified lines
436	#define KMALLOC_NOT_NORMAL_BITS \	436	#define KMALLOC_NOT_NORMAL_BITS \
437	(__GFP_RECLAIMABLE \| \	437	(__GFP_RECLAIMABLE \| \
438	(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) \| \	438	(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) \| \
439	(IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0))	439	(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))
440		440
441	extern unsigned long random_kmalloc_seed;	441	extern unsigned long random_kmalloc_seed;
442		442
@@ -464,7 +464,7 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne · 21 unmodified lines
464	*/	464	*/
465	if (IS_ENABLED(CONFIG_ZONE_DMA) && (flags & __GFP_DMA))	465	if (IS_ENABLED(CONFIG_ZONE_DMA) && (flags & __GFP_DMA))
466	return KMALLOC_DMA;	466	return KMALLOC_DMA;
467	if (!IS_ENABLED(CONFIG_MEMCG_KMEM) \|\| (flags & __GFP_RECLAIMABLE))	467	if (!IS_ENABLED(CONFIG_MEMCG) \|\| (flags & __GFP_RECLAIMABLE))
468	return KMALLOC_RECLAIM;	468	return KMALLOC_RECLAIM;
469	else	469	else
470	return KMALLOC_CGROUP;	470	return KMALLOC_CGROUP;

include/linux/swap.h+11−3

 #define MEMCG_RECLAIM_MAY_SWAP (1 << 1)
 #define MEMCG_RECLAIM_PROACTIVE (1 << 2)
+#define MIN_SWAPPINESS 0
+#define MAX_SWAPPINESS 200
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 						  unsigned long nr_pages,
 						  gfp_t gfp_mask,
-						  unsigned int reclaim_options);
+						  unsigned int reclaim_options,
+						  int *swappiness);
 extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem,
 						gfp_t gfp_mask, bool noswap,
 						pg_data_t *pgdat,
 extern void swap_shmem_alloc(swp_entry_t);
 extern int swap_duplicate(swp_entry_t);
 extern int swapcache_prepare(swp_entry_t);
-extern void swap_free(swp_entry_t);
+extern void swap_free_nr(swp_entry_t entry, int nr_pages);
 extern void swapcache_free_entries(swp_entry_t *entries, int n);
 extern void free_swap_and_cache_nr(swp_entry_t entry, int nr);
 int swap_type_of(dev_t device, sector_t offset);
 	return 0;
 }
-static inline void swap_free(swp_entry_t swp)
+static inline void swap_free_nr(swp_entry_t entry, int nr_pages)
 {
 }
 	free_swap_and_cache_nr(entry, 1);
 }
+static inline void swap_free(swp_entry_t entry)
+{
+	swap_free_nr(entry, 1);
+}
 #ifdef CONFIG_MEMCG
 static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
 {

include/linux/swapops.h+2−2

@@ -334,7 +334,7 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry) · 333 unmodified lines
334		334
335	extern void migration_entry_wait(struct mm_struct mm, pmd_t pmd,	335	extern void migration_entry_wait(struct mm_struct mm, pmd_t pmd,
336	unsigned long address);	336	unsigned long address);
337	extern void migration_entry_wait_huge(struct vm_area_struct vma, pte_t pte);	337	extern void migration_entry_wait_huge(struct vm_area_struct vma, unsigned long addr, pte_t pte);
338	#else /* CONFIG_MIGRATION */	338	#else /* CONFIG_MIGRATION */
339	static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)	339	static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)
340	{	340	{
@@ -359,7 +359,7 @@ static inline int is_migration_entry(swp_entry_t swp) · 18 unmodified lines
359	static inline void migration_entry_wait(struct mm_struct mm, pmd_t pmd,	359	static inline void migration_entry_wait(struct mm_struct mm, pmd_t pmd,
360	unsigned long address) { }	360	unsigned long address) { }
361	static inline void migration_entry_wait_huge(struct vm_area_struct *vma,	361	static inline void migration_entry_wait_huge(struct vm_area_struct *vma,
362	pte_t *pte) { }	362	unsigned long addr, pte_t *pte) { }
363	static inline int is_writable_migration_entry(swp_entry_t entry)	363	static inline int is_writable_migration_entry(swp_entry_t entry)
364	{	364	{
365	return 0;	365	return 0;

include/linux/vmstat.h+4−0

@@ -624,4 +624,8 @@ static inline void lruvec_stat_sub_folio(struct folio *folio, · 623 unmodified lines
624	{	624	{
625	lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio));	625	lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio));
626	}	626	}
		627
		628	void __meminit mod_node_early_perpage_metadata(int nid, long delta);
		629	void __meminit store_early_perpage_metadata(void);
		630
627	#endif /* _LINUX_VMSTAT_H */	631	#endif /* _LINUX_VMSTAT_H */

include/linux/zswap.h+8−2

 void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
 void zswap_lruvec_state_init(struct lruvec *lruvec);
 void zswap_folio_swapin(struct folio *folio);
-bool is_zswap_enabled(void);
+bool zswap_is_enabled(void);
+bool zswap_never_enabled(void);
 #else
 struct zswap_lruvec_state {};
 static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {}
 static inline void zswap_folio_swapin(struct folio *folio) {}
-static inline bool is_zswap_enabled(void)
+static inline bool zswap_is_enabled(void)
 {
 	return false;
 }
+static inline bool zswap_never_enabled(void)
+{
+	return true;
+}
 #endif
 #endif /* _LINUX_ZSWAP_H */

include/ras/ras_event.h+2−2

 #define MF_PAGE_TYPE		\
 	EM ( MF_MSG_KERNEL, "reserved kernel page" )			\
 	EM ( MF_MSG_KERNEL_HIGH_ORDER, "high-order kernel page" )	\
-	EM ( MF_MSG_SLAB, "kernel slab page" )				\
-	EM ( MF_MSG_DIFFERENT_COMPOUND, "different compound page after locking" ) \
 	EM ( MF_MSG_HUGE, "huge page" )					\
 	EM ( MF_MSG_FREE_HUGE, "free huge page" )			\
+	EM ( MF_MSG_GET_HWPOISON, "get hwpoison page" )			\
 	EM ( MF_MSG_UNMAP_FAILED, "unmapping failed page" )		\
 	EM ( MF_MSG_DIRTY_SWAPCACHE, "dirty swapcache page" )		\
 	EM ( MF_MSG_CLEAN_SWAPCACHE, "clean swapcache page" )		\
 	EM ( MF_MSG_BUDDY, "free buddy page" )				\
 	EM ( MF_MSG_DAX, "dax page" )					\
 	EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" )			\
+	EM ( MF_MSG_ALREADY_POISONED, "already poisoned" )		\
 	EMe ( MF_MSG_UNKNOWN, "unknown page" )
 /*

include/trace/events/kmem.h+2−2

@@ -36,7 +36,7 @@ TRACE_EVENT(kmem_cache_alloc, · 35 unmodified lines
36	__entry->bytes_alloc = s->size;	36	__entry->bytes_alloc = s->size;
37	__entry->gfp_flags = (__force unsigned long)gfp_flags;	37	__entry->gfp_flags = (__force unsigned long)gfp_flags;
38	__entry->node = node;	38	__entry->node = node;
39	__entry->accounted = IS_ENABLED(CONFIG_MEMCG_KMEM) ?	39	__entry->accounted = IS_ENABLED(CONFIG_MEMCG) ?
40	((gfp_flags & __GFP_ACCOUNT) \|\|	40	((gfp_flags & __GFP_ACCOUNT) \|\|
41	(s->flags & SLAB_ACCOUNT)) : false;	41	(s->flags & SLAB_ACCOUNT)) : false;
42	),	42	),
@@ -87,7 +87,7 @@ TRACE_EVENT(kmalloc, · 44 unmodified lines
87	__entry->bytes_alloc,	87	__entry->bytes_alloc,
88	show_gfp_flags(__entry->gfp_flags),	88	show_gfp_flags(__entry->gfp_flags),
89	__entry->node,	89	__entry->node,
90	(IS_ENABLED(CONFIG_MEMCG_KMEM) &&	90	(IS_ENABLED(CONFIG_MEMCG) &&
91	(__entry->gfp_flags & (__force unsigned long)__GFP_ACCOUNT)) ? "true" : "false")	91	(__entry->gfp_flags & (__force unsigned long)__GFP_ACCOUNT)) ? "true" : "false")
92	);	92	);
93		93

include/trace/events/migrate.h+2−1

@@ -22,7 +22,8 @@ · 21 unmodified lines
22	EM( MR_NUMA_MISPLACED, "numa_misplaced") \	22	EM( MR_NUMA_MISPLACED, "numa_misplaced") \
23	EM( MR_CONTIG_RANGE, "contig_range") \	23	EM( MR_CONTIG_RANGE, "contig_range") \
24	EM( MR_LONGTERM_PIN, "longterm_pin") \	24	EM( MR_LONGTERM_PIN, "longterm_pin") \
25	EMe(MR_DEMOTION, "demotion")	25	EM( MR_DEMOTION, "demotion") \
		26	EMe(MR_DAMON, "damon")
26		27
27	/*	28	/*
28	* First define the enums in the above macros to be exported to userspace	29	* First define the enums in the above macros to be exported to userspace

include/uapi/linux/fs.h+157−1

@@ -336,8 +336,10 @@ typedef int __bitwise __kernel_rwf_t; · 335 unmodified lines
336	#define RWF_SUPPORTED (RWF_HIPRI \| RWF_DSYNC \| RWF_SYNC \| RWF_NOWAIT \|\	336	#define RWF_SUPPORTED (RWF_HIPRI \| RWF_DSYNC \| RWF_SYNC \| RWF_NOWAIT \|\
337	RWF_APPEND \| RWF_NOAPPEND \| RWF_ATOMIC)	337	RWF_APPEND \| RWF_NOAPPEND \| RWF_ATOMIC)
338		338
		339	#define PROCFS_IOCTL_MAGIC 'f'
		340
339	/* Pagemap ioctl */	341	/* Pagemap ioctl */
340	#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg)	342	#define PAGEMAP_SCAN _IOWR(PROCFS_IOCTL_MAGIC, 16, struct pm_scan_arg)
341		343
342	/* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */	344	/* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */
343	#define PAGE_IS_WPALLOWED (1 << 0)	345	#define PAGE_IS_WPALLOWED (1 << 0)
@@ -396,4 +398,158 @@ struct pm_scan_arg { · 52 unmodified lines
396	__u64 return_mask;	398	__u64 return_mask;
397	};	399	};
398		400
		401	/* /proc/<pid>/maps ioctl */
		402	#define PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 17, struct procmap_query)
		403
		404	enum procmap_query_flags {
		405	/*
		406	* VMA permission flags.
		407	*
		408	* Can be used as part of procmap_query.query_flags field to look up
		409	* only VMAs satisfying specified subset of permissions. E.g., specifying
		410	* PROCMAP_QUERY_VMA_READABLE only will return both readable and read/write VMAs,
		411	* while having PROCMAP_QUERY_VMA_READABLE \| PROCMAP_QUERY_VMA_WRITABLE will only
		412	* return read/write VMAs, though both executable/non-executable and
		413	* private/shared will be ignored.
		414	*
		415	* PROCMAP_QUERY_VMA_* flags are also returned in procmap_query.vma_flags
		416	* field to specify actual VMA permissions.
		417	*/
		418	PROCMAP_QUERY_VMA_READABLE = 0x01,
		419	PROCMAP_QUERY_VMA_WRITABLE = 0x02,
		420	PROCMAP_QUERY_VMA_EXECUTABLE = 0x04,
		421	PROCMAP_QUERY_VMA_SHARED = 0x08,
		422	/*
		423	* Query modifier flags.
		424	*
		425	* By default VMA that covers provided address is returned, or -ENOENT
		426	* is returned. With PROCMAP_QUERY_COVERING_OR_NEXT_VMA flag set, closest
		427	* VMA with vma_start > addr will be returned if no covering VMA is
		428	* found.
		429	*
		430	* PROCMAP_QUERY_FILE_BACKED_VMA instructs query to consider only VMAs that
		431	* have file backing. Can be combined with PROCMAP_QUERY_COVERING_OR_NEXT_VMA
		432	* to iterate all VMAs with file backing.
		433	*/
		434	PROCMAP_QUERY_COVERING_OR_NEXT_VMA = 0x10,
		435	PROCMAP_QUERY_FILE_BACKED_VMA = 0x20,
		436	};
		437
		438	/*
		439	* Input/output argument structured passed into ioctl() call. It can be used
		440	* to query a set of VMAs (Virtual Memory Areas) of a process.
		441	*
		442	* Each field can be one of three kinds, marked in a short comment to the
		443	* right of the field:
		444	* - "in", input argument, user has to provide this value, kernel doesn't modify it;
		445	* - "out", output argument, kernel sets this field with VMA data;
		446	* - "in/out", input and output argument; user provides initial value (used
		447	* to specify maximum allowable buffer size), and kernel sets it to actual
		448	* amount of data written (or zero, if there is no data).
		449	*
		450	* If matching VMA is found (according to criterias specified by
		451	* query_addr/query_flags, all the out fields are filled out, and ioctl()
		452	* returns 0. If there is no matching VMA, -ENOENT will be returned.
		453	* In case of any other error, negative error code other than -ENOENT is
		454	* returned.
		455	*
		456	* Most of the data is similar to the one returned as text in /proc/<pid>/maps
		457	* file, but procmap_query provides more querying flexibility. There are no
		458	* consistency guarantees between subsequent ioctl() calls, but data returned
		459	* for matched VMA is self-consistent.
		460	*/
		461	struct procmap_query {
		462	/* Query struct size, for backwards/forward compatibility */
		463	__u64 size;
		464	/*
		465	* Query flags, a combination of enum procmap_query_flags values.
		466	* Defines query filtering and behavior, see enum procmap_query_flags.
		467	*
		468	* Input argument, provided by user. Kernel doesn't modify it.
		469	*/
		470	__u64 query_flags; /* in */
		471	/*
		472	* Query address. By default, VMA that covers this address will
		473	* be looked up. PROCMAP_QUERY_* flags above modify this default
		474	* behavior further.
		475	*
		476	* Input argument, provided by user. Kernel doesn't modify it.
		477	*/
		478	__u64 query_addr; /* in */
		479	/* VMA starting (inclusive) and ending (exclusive) address, if VMA is found. */
		480	__u64 vma_start; /* out */
		481	__u64 vma_end; /* out */
		482	/* VMA permissions flags. A combination of PROCMAP_QUERY_VMA_* flags. */
		483	__u64 vma_flags; /* out */
		484	/* VMA backing page size granularity. */
		485	__u64 vma_page_size; /* out */
		486	/*
		487	* VMA file offset. If VMA has file backing, this specifies offset
		488	* within the file that VMA's start address corresponds to.
		489	* Is set to zero if VMA has no backing file.
		490	*/
		491	__u64 vma_offset; /* out */
		492	/* Backing file's inode number, or zero, if VMA has no backing file. */
		493	__u64 inode; /* out */
		494	/* Backing file's device major/minor number, or zero, if VMA has no backing file. */
		495	__u32 dev_major; /* out */
		496	__u32 dev_minor; /* out */
		497	/*
		498	* If set to non-zero value, signals the request to return VMA name
		499	* (i.e., VMA's backing file's absolute path, with " (deleted)" suffix
		500	* appended, if file was unlinked from FS) for matched VMA. VMA name
		501	* can also be some special name (e.g., "[heap]", "[stack]") or could
		502	* be even user-supplied with prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME).
		503	*
		504	* Kernel will set this field to zero, if VMA has no associated name.
		505	* Otherwise kernel will return actual amount of bytes filled in
		506	* user-supplied buffer (see vma_name_addr field below), including the
		507	* terminating zero.
		508	*
		509	* If VMA name is longer that user-supplied maximum buffer size,
		510	* -E2BIG error is returned.
		511	*
		512	* If this field is set to non-zero value, vma_name_addr should point
		513	* to valid user space memory buffer of at least vma_name_size bytes.
		514	* If set to zero, vma_name_addr should be set to zero as well
		515	*/
		516	__u32 vma_name_size; /* in/out */
		517	/*
		518	* If set to non-zero value, signals the request to extract and return
		519	* VMA's backing file's build ID, if the backing file is an ELF file
		520	* and it contains embedded build ID.
		521	*
		522	* Kernel will set this field to zero, if VMA has no backing file,
		523	* backing file is not an ELF file, or ELF file has no build ID
		524	* embedded.
		525	*
		526	* Build ID is a binary value (not a string). Kernel will set
		527	* build_id_size field to exact number of bytes used for build ID.
		528	* If build ID is requested and present, but needs more bytes than
		529	* user-supplied maximum buffer size (see build_id_addr field below),
		530	* -E2BIG error will be returned.
		531	*
		532	* If this field is set to non-zero value, build_id_addr should point
		533	* to valid user space memory buffer of at least build_id_size bytes.
		534	* If set to zero, build_id_addr should be set to zero as well
		535	*/
		536	__u32 build_id_size; /* in/out */
		537	/*
		538	* User-supplied address of a buffer of at least vma_name_size bytes
		539	* for kernel to fill with matched VMA's name (see vma_name_size field
		540	* description above for details).
		541	*
		542	* Should be set to zero if VMA name should not be returned.
		543	*/
		544	__u64 vma_name_addr; /* in */
		545	/*
		546	* User-supplied address of a buffer of at least build_id_size bytes
		547	* for kernel to fill with matched VMA's ELF build ID, if available
		548	* (see build_id_size field description above for details).
		549	*
		550	* Should be set to zero if build ID should not be returned.
		551	*/
		552	__u64 build_id_addr; /* in */
		553	};
		554
399	#endif /* _UAPI_LINUX_FS_H */	555	#endif /* _UAPI_LINUX_FS_H */

init/Kconfig+15−3

@@ -971,10 +971,22 @@ config MEMCG · 970 unmodified lines
971	help	971	help
972	Provides control over the memory footprint of tasks in a cgroup.	972	Provides control over the memory footprint of tasks in a cgroup.
973		973
974	config MEMCG_KMEM	974	config MEMCG_V1
975	bool	975	bool "Legacy cgroup v1 memory controller"
976	depends on MEMCG	976	depends on MEMCG
977	default y	977	default n
		978	help
		979	Legacy cgroup v1 memory controller which has been deprecated by
		980	cgroup v2 implementation. The v1 is there for legacy applications
		981	which haven't migrated to the new cgroup v2 interface yet. If you
		982	do not have any such application then you are completely fine leaving
		983	this option disabled.
		984
		985	Please note that feature set of the legacy memory controller is likely
		986	going to shrink due to deprecation process. New deployments with v1
		987	controller are highly discouraged.
		988
		989	San N is unsure.
978		990
979	config BLK_CGROUP	991	config BLK_CGROUP
980	bool "IO controller"	992	bool "IO controller"

kernel/bpf/memalloc.c+3−6

 static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c)
 {
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 	if (c->objcg)
 		return get_mem_cgroup_from_objcg(c->objcg);
-#endif
-#ifdef CONFIG_MEMCG
 	return root_mem_cgroup;
 #else
 	return NULL;
 			size += LLIST_NODE_SZ; /* room for llist_node */
 		unit_size = size;
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 		if (memcg_bpf_enabled())
 			objcg = get_obj_cgroup_from_current();
 #endif
 	pcc = __alloc_percpu_gfp(sizeof(*cc), 8, GFP_KERNEL);
 	if (!pcc)
 		return -ENOMEM;
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 	objcg = get_obj_cgroup_from_current();
 #endif
 	ma->objcg = objcg;

kernel/bpf/syscall.c+3−3

@@ -385,7 +385,7 @@ void bpf_map_free_id(struct bpf_map *map) · 384 unmodified lines
385	spin_unlock_irqrestore(&map_idr_lock, flags);	385	spin_unlock_irqrestore(&map_idr_lock, flags);
386	}	386	}
387		387
388	#ifdef CONFIG_MEMCG_KMEM	388	#ifdef CONFIG_MEMCG
389	static void bpf_map_save_memcg(struct bpf_map *map)	389	static void bpf_map_save_memcg(struct bpf_map *map)
390	{	390	{
391	/* Currently if a map is created by a process belonging to the root	391	/* Currently if a map is created by a process belonging to the root
@@ -486,7 +486,7 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, · 94 unmodified lines
486	unsigned long i, j;	486	unsigned long i, j;
487	struct page *pg;	487	struct page *pg;
488	int ret = 0;	488	int ret = 0;
489	#ifdef CONFIG_MEMCG_KMEM	489	#ifdef CONFIG_MEMCG
490	struct mem_cgroup memcg, old_memcg;	490	struct mem_cgroup memcg, old_memcg;
491		491
492	memcg = bpf_map_get_memcg(map);	492	memcg = bpf_map_get_memcg(map);
@@ -505,7 +505,7 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, · 12 unmodified lines
505	break;	505	break;
506	}	506	}
507		507
508	#ifdef CONFIG_MEMCG_KMEM	508	#ifdef CONFIG_MEMCG
509	set_active_memcg(old_memcg);	509	set_active_memcg(old_memcg);
510	mem_cgroup_put(memcg);	510	mem_cgroup_put(memcg);
511	#endif	511	#endif

kernel/bpf/verifier.c+4−0

  * Assume non-sleepable from bpf safety point of view.
  */
 BTF_ID(func, __filemap_add_folio)
+#ifdef CONFIG_FAIL_PAGE_ALLOC
 BTF_ID(func, should_fail_alloc_page)
+#endif
+#ifdef CONFIG_FAILSLAB
 BTF_ID(func, should_failslab)
+#endif
 BTF_SET_END(btf_non_sleepable_error_inject)
 static int check_non_sleepable_error_inject(u32 btf_id)

kernel/events/core.c+1−1

@@ -7634,7 +7634,7 @@ again: · 7633 unmodified lines
7634		7634
7635	pte = ptep_get_lockless(ptep);	7635	pte = ptep_get_lockless(ptep);
7636	if (pte_present(pte))	7636	if (pte_present(pte))
7637	size = pte_leaf_size(pte);	7637	size = __pte_leaf_size(pmd, pte);
7638	pte_unmap(ptep);	7638	pte_unmap(ptep);
7639	#endif /* CONFIG_HAVE_GUP_FAST */	7639	#endif /* CONFIG_HAVE_GUP_FAST */
7640		7640

kernel/events/uprobes.c+1−1

@@ -181,7 +181,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, · 180 unmodified lines
181		181
182	if (new_page) {	182	if (new_page) {
183	folio_get(new_folio);	183	folio_get(new_folio);
184	folio_add_new_anon_rmap(new_folio, vma, addr);	184	folio_add_new_anon_rmap(new_folio, vma, addr, RMAP_EXCLUSIVE);
185	folio_add_lru_vma(new_folio, vma);	185	folio_add_lru_vma(new_folio, vma);
186	} else	186	} else
187	/* no new page, just dec_mm_counter for old_page */	187	/* no new page, just dec_mm_counter for old_page */

kernel/exit.c+43−38

 }
 #ifdef CONFIG_MEMCG
+/* drops tasklist_lock if succeeds */
+static bool __try_to_set_owner(struct task_struct *tsk, struct mm_struct *mm)
+{
+	bool ret = false;
+	task_lock(tsk);
+	if (likely(tsk->mm == mm)) {
+		/* tsk can't pass exit_mm/exec_mmap and exit */
+		read_unlock(&tasklist_lock);
+		WRITE_ONCE(mm->owner, tsk);
+		lru_gen_migrate_mm(mm);
+		ret = true;
+	}
+	task_unlock(tsk);
+	return ret;
+}
+static bool try_to_set_owner(struct task_struct *g, struct mm_struct *mm)
+{
+	struct task_struct *t;
+	for_each_thread(g, t) {
+		struct mm_struct *t_mm = READ_ONCE(t->mm);
+		if (t_mm == mm) {
+			if (__try_to_set_owner(t, mm))
+				return true;
+		} else if (t_mm)
+			break;
+	}
+	return false;
+}
 /*
  * A task is exiting.   If it owned this mm, find a new owner for the mm.
  */
 void mm_update_next_owner(struct mm_struct *mm)
 {
-	struct task_struct *c, *g, *p = current;
+	struct task_struct *g, *p = current;
-retry:
 	/*
 	 * If the exiting or execing task is not the owner, it's
 	 * someone else's problem.
 	/*
 	 * Search in the children
 	 */
-	list_for_each_entry(c, &p->children, sibling) {
+	list_for_each_entry(g, &p->children, sibling) {
-		if (c->mm == mm)
+		if (try_to_set_owner(g, mm))
-			goto assign_new_owner;
+			goto ret;
 	}
 	/*
 	 * Search in the siblings
 	 */
-	list_for_each_entry(c, &p->real_parent->children, sibling) {
+	list_for_each_entry(g, &p->real_parent->children, sibling) {
-		if (c->mm == mm)
+		if (try_to_set_owner(g, mm))
-			goto assign_new_owner;
+			goto ret;
 	}
 	/*
 	 * Search through everything else, we should not get here often.
 	 */
 			break;
 		if (g->flags & PF_KTHREAD)
 			continue;
-		for_each_thread(g, c) {
+		if (try_to_set_owner(g, mm))
-			if (c->mm == mm)
+			goto ret;
-				goto assign_new_owner;
-			if (c->mm)
-				break;
-		}
 	}
 	read_unlock(&tasklist_lock);
 	/*
 	 * ptrace or page migration (get_task_mm()).  Mark owner as NULL.
 	 */
 	WRITE_ONCE(mm->owner, NULL);
+ ret:
 	return;
-assign_new_owner:
-	BUG_ON(c == p);
-	get_task_struct(c);
-	/*
-	 * The task_lock protects c->mm from changing.
-	 * We always want mm->owner->mm == mm
-	 */
-	task_lock(c);
-	/*
-	 * Delay read_unlock() till we have the task_lock()
-	 * to ensure that c does not slip away underneath us
-	 */
-	read_unlock(&tasklist_lock);
-	if (c->mm != mm) {
-		task_unlock(c);
-		put_task_struct(c);
-		goto retry;
-	}
-	WRITE_ONCE(mm->owner, c);
-	lru_gen_migrate_mm(mm);
-	task_unlock(c);
-	put_task_struct(c);
 }
 #endif /* CONFIG_MEMCG */

kernel/fork.c+9−9

 #include <linux/fs.h>
 #include <linux/mm.h>
 #include <linux/mm_inline.h>
+#include <linux/memblock.h>
 #include <linux/nsproxy.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
 /*
  * set_max_threads
  */
-static void set_max_threads(unsigned int max_threads_suggested)
+static void __init set_max_threads(unsigned int max_threads_suggested)
 {
 	u64 threads;
-	unsigned long nr_pages = totalram_pages();
+	unsigned long nr_pages = PHYS_PFN(memblock_phys_mem_size() - memblock_reserved_size());
 	/*
 	 * The number of threads shall be limited such that the thread
 int arch_task_struct_size __read_mostly;
 #endif
-static void task_struct_whitelist(unsigned long *offset, unsigned long *size)
+static void __init task_struct_whitelist(unsigned long *offset, unsigned long *size)
 {
 	/* Fetch thread_struct whitelist for the architecture. */
 	arch_thread_struct_whitelist(offset, size);
 {
 	struct mm_struct *mm;
+	if (task->flags & PF_KTHREAD)
+		return NULL;
 	task_lock(task);
 	mm = task->mm;
-	if (mm) {
+	if (mm)
-		if (task->flags & PF_KTHREAD)
+		mmget(mm);
-			mm = NULL;
-		else
-			mmget(mm);
-	}
 	task_unlock(task);
 	return mm;
 }

kernel/power/swap.c+2−3

 	while ((node = swsusp_extents.rb_node)) {
 		struct swsusp_extent *ext;
-		unsigned long offset;
 		ext = rb_entry(node, struct swsusp_extent, node);
 		rb_erase(node, &swsusp_extents);
-		for (offset = ext->start; offset <= ext->end; offset++)
+		swap_free_nr(swp_entry(swap, ext->start),
-			swap_free(swp_entry(swap, offset));
+			     ext->end - ext->start + 1);
 		kfree(ext);
 	}

kernel/trace/ftrace.c+1−0

@@ -7920,6 +7920,7 @@ out: · 7919 unmodified lines
7920	void arch_ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip,	7920	void arch_ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip,
7921	struct ftrace_ops op, struct ftrace_regs fregs)	7921	struct ftrace_ops op, struct ftrace_regs fregs)
7922	{	7922	{
		7923	kmsan_unpoison_memory(fregs, sizeof(*fregs));
7923	__ftrace_ops_list_func(ip, parent_ip, NULL, fregs);	7924	__ftrace_ops_list_func(ip, parent_ip, NULL, fregs);
7924	}	7925	}
7925	#else	7926	#else

lib/maple_tree.c+6−9

  *
  * Return: The contents that was stored at the index.
  */
-static inline void *mas_wr_store_entry(struct ma_wr_state *wr_mas)
+static inline void mas_wr_store_entry(struct ma_wr_state *wr_mas)
 {
 	struct ma_state *mas = wr_mas->mas;
 	wr_mas->content = mas_start(mas);
 	if (mas_is_none(mas) || mas_is_ptr(mas)) {
 		mas_store_root(mas, wr_mas->entry);
-		return wr_mas->content;
+		return;
 	}
 	if (unlikely(!mas_wr_walk(wr_mas))) {
 		mas_wr_spanning_store(wr_mas);
-		return wr_mas->content;
+		return;
 	}
 	/* At this point, we are at the leaf node that needs to be altered. */
 	mas_wr_end_piv(wr_mas);
 	/* New root for a single pointer */
-	if (unlikely(!mas->index && mas->last == ULONG_MAX)) {
+	if (unlikely(!mas->index && mas->last == ULONG_MAX))
 		mas_new_root(mas, wr_mas->entry);
-		return wr_mas->content;
+	else
-	}
+		mas_wr_modify(wr_mas);
-	mas_wr_modify(wr_mas);
-	return wr_mas->content;
 }
 /**

lib/test_hmm.c+1−0

@@ -1550,4 +1550,5 @@ static void __exit hmm_dmirror_exit(void) · 1549 unmodified lines
1550		1550
1551	module_init(hmm_dmirror_init);	1551	module_init(hmm_dmirror_init);
1552	module_exit(hmm_dmirror_exit);	1552	module_exit(hmm_dmirror_exit);
		1553	MODULE_DESCRIPTION("HMM (Heterogeneous Memory Management) test module");
1553	MODULE_LICENSE("GPL");	1554	MODULE_LICENSE("GPL");

lib/test_maple_tree.c+1−0

@@ -3946,4 +3946,5 @@ static void __exit maple_tree_harvest(void) · 3945 unmodified lines
3946	module_init(maple_tree_seed);	3946	module_init(maple_tree_seed);
3947	module_exit(maple_tree_harvest);	3947	module_exit(maple_tree_harvest);
3948	MODULE_AUTHOR("Liam R. Howlett <Liam.Howlett@Oracle.com>");	3948	MODULE_AUTHOR("Liam R. Howlett <Liam.Howlett@Oracle.com>");
		3949	MODULE_DESCRIPTION("maple tree API test module");
3949	MODULE_LICENSE("GPL");	3950	MODULE_LICENSE("GPL");

lib/test_ubsan.c+1−0

@@ -156,4 +156,5 @@ static void __exit test_ubsan_exit(void) · 155 unmodified lines
156	module_exit(test_ubsan_exit);	156	module_exit(test_ubsan_exit);
157		157
158	MODULE_AUTHOR("Jinbum Park <jinb.park7@gmail.com>");	158	MODULE_AUTHOR("Jinbum Park <jinb.park7@gmail.com>");
		159	MODULE_DESCRIPTION("UBSAN unit test");
159	MODULE_LICENSE("GPL v2");	160	MODULE_LICENSE("GPL v2");

lib/test_xarray.c+1−0

@@ -2173,4 +2173,5 @@ static void xarray_exit(void) · 2172 unmodified lines
2173	module_init(xarray_checks);	2173	module_init(xarray_checks);
2174	module_exit(xarray_exit);	2174	module_exit(xarray_exit);
2175	MODULE_AUTHOR("Matthew Wilcox <willy@infradead.org>");	2175	MODULE_AUTHOR("Matthew Wilcox <willy@infradead.org>");
		2176	MODULE_DESCRIPTION("XArray API test module");
2176	MODULE_LICENSE("GPL");	2177	MODULE_LICENSE("GPL");

lib/zlib_dfltcc/dfltcc.h+1−0

@@ -80,6 +80,7 @@ struct dfltcc_param_v0 { · 79 unmodified lines
80	uint8_t csb[1152];	80	uint8_t csb[1152];
81	};	81	};
82		82
		83	static_assert(offsetof(struct dfltcc_param_v0, csb) == 384);
83	static_assert(sizeof(struct dfltcc_param_v0) == 1536);	84	static_assert(sizeof(struct dfltcc_param_v0) == 1536);
84		85
85	#define CVT_CRC32 0	86	#define CVT_CRC32 0

lib/zlib_dfltcc/dfltcc_util.h+28−0

@@ -2,6 +2,8 @@ · 1 unmodified line
2	#ifndef DFLTCC_UTIL_H	2	#ifndef DFLTCC_UTIL_H
3	#define DFLTCC_UTIL_H	3	#define DFLTCC_UTIL_H
4		4
		5	#include "dfltcc.h"
		6	#include <linux/kmsan-checks.h>
5	#include <linux/zutil.h>	7	#include <linux/zutil.h>
6		8
7	/*	9	/*
@@ -20,6 +22,7 @@ typedef enum { · 12 unmodified lines
20	#define DFLTCC_CMPR 2	22	#define DFLTCC_CMPR 2
21	#define DFLTCC_XPND 4	23	#define DFLTCC_XPND 4
22	#define HBT_CIRCULAR (1 << 7)	24	#define HBT_CIRCULAR (1 << 7)
		25	#define DFLTCC_FN_MASK ((1 << 7) - 1)
23	#define HB_BITS 15	26	#define HB_BITS 15
24	#define HB_SIZE (1 << HB_BITS)	27	#define HB_SIZE (1 << HB_BITS)
25		28
@@ -34,6 +37,7 @@ static inline dfltcc_cc dfltcc( · 8 unmodified lines
34	)	37	)
35	{	38	{
36	Byte t2 = op1 ? op1 : NULL;	39	Byte t2 = op1 ? op1 : NULL;
		40	unsigned char *orig_t2 = t2;
37	size_t t3 = len1 ? *len1 : 0;	41	size_t t3 = len1 ? *len1 : 0;
38	const Byte t4 = op2 ? op2 : NULL;	42	const Byte t4 = op2 ? op2 : NULL;
39	size_t t5 = len2 ? *len2 : 0;	43	size_t t5 = len2 ? *len2 : 0;
@@ -59,6 +63,30 @@ static inline dfltcc_cc dfltcc( · 19 unmodified lines
59	: "cc", "memory");	63	: "cc", "memory");
60	t2 = r2; t3 = r3; t4 = r4; t5 = r5;	64	t2 = r2; t3 = r3; t4 = r4; t5 = r5;
61		65
		66	/*
		67	* Unpoison the parameter block and the output buffer.
		68	* This is a no-op in non-KMSAN builds.
		69	*/
		70	switch (fn & DFLTCC_FN_MASK) {
		71	case DFLTCC_QAF:
		72	kmsan_unpoison_memory(param, sizeof(struct dfltcc_qaf_param));
		73	break;
		74	case DFLTCC_GDHT:
		75	kmsan_unpoison_memory(param, offsetof(struct dfltcc_param_v0, csb));
		76	break;
		77	case DFLTCC_CMPR:
		78	kmsan_unpoison_memory(param, sizeof(struct dfltcc_param_v0));
		79	kmsan_unpoison_memory(
		80	orig_t2,
		81	t2 - orig_t2 +
		82	(((struct dfltcc_param_v0 *)param)->sbb == 0 ? 0 : 1));
		83	break;
		84	case DFLTCC_XPND:
		85	kmsan_unpoison_memory(param, sizeof(struct dfltcc_param_v0));
		86	kmsan_unpoison_memory(orig_t2, t2 - orig_t2);
		87	break;
		88	}
		89
62	if (op1)	90	if (op1)
63	*op1 = t2;	91	*op1 = t2;
64	if (len1)	92	if (len1)

mm/Kconfig+10−13

 choice
 	prompt "Default allocator"
 	depends on ZSWAP
-	default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if MMU
+	default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if HAVE_ZSMALLOC
 	default ZSWAP_ZPOOL_DEFAULT_ZBUD
 	help
 	  Selects the default allocator for the compressed cache for
 config ZSWAP_ZPOOL_DEFAULT_ZSMALLOC
 	bool "zsmalloc"
+	depends on HAVE_ZSMALLOC
 	select ZSMALLOC
 	help
 	  Use the zsmalloc allocator as the default allocator.
 	  page. It is a ZBUD derivative so the simplicity and determinism are
 	  still there.
+config HAVE_ZSMALLOC
+	def_bool y
+	depends on MMU
+	depends on PAGE_SIZE_LESS_THAN_256KB # we want <= 64 KiB
 config ZSMALLOC
 	tristate
 	prompt "N:1 compression allocator (zsmalloc)" if ZSWAP
-	depends on MMU
+	depends on HAVE_ZSMALLOC
 	help
 	  zsmalloc is a slab-based memory allocator designed to store
 	  pages of various compression levels efficiently. It achieves
 	  from userspace allocation.  Keeping a user from writing to low pages
 	  can help reduce the impact of kernel NULL pointer bugs.
-	  For most ppc64 and x86 users with lots of address space
+	  For most arm64, ppc64 and x86 users with lots of address space
 	  a value of 65536 is reasonable and should cause no problems.
 	  On arm and other archs it should not be higher than 32768.
 	  Programs which use vm86 functionality or have some need to map
 	depends on SPARSEMEM
 	depends on !NEED_PER_CPU_KM
 	depends on 64BIT
+	depends on !KMSAN
 	select PADATA
 	help
 	  Ordinarily all struct pages are initialised during early boot in a
 config ARCH_HAS_PTE_SPECIAL
 	bool
-#
-# Some architectures require a special hugepage directory format that is
-# required to support multiple hugepage sizes. For example a4fe3ce76
-# "powerpc/mm: Allow more flexible layouts for hugepage pagetables"
-# introduced it on powerpc.  This allows for a more flexible hugepage
-# pagetable layouts.
-#
-config ARCH_HAS_HUGEPD
-	bool
 config MAPPING_DIRTY_HELPERS
         bool

mm/Makefile+2−0

 KCOV_INSTRUMENT_debug-pagealloc.o := n
 KCOV_INSTRUMENT_kmemleak.o := n
 KCOV_INSTRUMENT_memcontrol.o := n
+KCOV_INSTRUMENT_memcontrol-v1.o := n
 KCOV_INSTRUMENT_mmzone.o := n
 KCOV_INSTRUMENT_vmstat.o := n
 KCOV_INSTRUMENT_failslab.o := n
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
 obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
+obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
 obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
 ifdef CONFIG_SWAP
 obj-$(CONFIG_MEMCG) += swap_cgroup.o

mm/balloon_compaction.c+0−8

@@ -234,14 +234,6 @@ static int balloon_page_migrate(struct page newpage, struct page page, · 233 unmodified lines
234	{	234	{
235	struct balloon_dev_info *balloon = balloon_page_device(page);	235	struct balloon_dev_info *balloon = balloon_page_device(page);
236		236
237	/*
238	* We can not easily support the no copy case here so ignore it as it
239	* is unlikely to be used with balloon pages. See include/linux/hmm.h
240	* for a user of the MIGRATE_SYNC_NO_COPY mode.
241	*/
242	if (mode == MIGRATE_SYNC_NO_COPY)
243	return -EINVAL;
244
245	VM_BUG_ON_PAGE(!PageLocked(page), page);	237	VM_BUG_ON_PAGE(!PageLocked(page), page);
246	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);	238	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);
247		239

mm/damon/core.c+337−1

@@ -354,7 +354,8 @@ struct damos damon_new_scheme(struct damos_access_pattern pattern, · 353 unmodified lines
354	enum damos_action action,	354	enum damos_action action,
355	unsigned long apply_interval_us,	355	unsigned long apply_interval_us,
356	struct damos_quota *quota,	356	struct damos_quota *quota,
357	struct damos_watermarks *wmarks)	357	struct damos_watermarks *wmarks,
		358	int target_nid)
358	{	359	{
359	struct damos *scheme;	360	struct damos *scheme;
360		361
@@ -381,6 +382,8 @@ struct damos damon_new_scheme(struct damos_access_pattern pattern, · 20 unmodified lines
381	scheme->wmarks = *wmarks;	382	scheme->wmarks = *wmarks;
382	scheme->wmarks.activated = true;	383	scheme->wmarks.activated = true;
383		384
		385	scheme->target_nid = target_nid;
		386
384	return scheme;	387	return scheme;
385	}	388	}
386		389
@@ -663,6 +666,339 @@ void damon_set_schemes(struct damon_ctx ctx, struct damos *schemes, · 276 unmodified lines
663	damon_add_scheme(ctx, schemes[i]);	666	damon_add_scheme(ctx, schemes[i]);
664	}	667	}
665		668
		669	static struct damos_quota_goal *damos_nth_quota_goal(
		670	int n, struct damos_quota *q)
		671	{
		672	struct damos_quota_goal *goal;
		673	int i = 0;
		674
		675	damos_for_each_quota_goal(goal, q) {
		676	if (i++ == n)
		677	return goal;
		678	}
		679	return NULL;
		680	}
		681
		682	static void damos_commit_quota_goal(
		683	struct damos_quota_goal dst, struct damos_quota_goal src)
		684	{
		685	dst->metric = src->metric;
		686	dst->target_value = src->target_value;
		687	if (dst->metric == DAMOS_QUOTA_USER_INPUT)
		688	dst->current_value = src->current_value;
		689	/* keep last_psi_total as is, since it will be updated in next cycle */
		690	}
		691
		692	/**
		693	* damos_commit_quota_goals() - Commit DAMOS quota goals to another quota.
		694	* @dst: The commit destination DAMOS quota.
		695	* @src: The commit source DAMOS quota.
		696	*
		697	* Copies user-specified parameters for quota goals from @src to @dst. Users
		698	* should use this function for quota goals-level parameters update of running
		699	* DAMON contexts, instead of manual in-place updates.
		700	*
		701	* This function should be called from parameters-update safe context, like
		702	* DAMON callbacks.
		703	*/
		704	int damos_commit_quota_goals(struct damos_quota dst, struct damos_quota src)
		705	{
		706	struct damos_quota_goal dst_goal, next, src_goal, new_goal;
		707	int i = 0, j = 0;
		708
		709	damos_for_each_quota_goal_safe(dst_goal, next, dst) {
		710	src_goal = damos_nth_quota_goal(i++, src);
		711	if (src_goal)
		712	damos_commit_quota_goal(dst_goal, src_goal);
		713	else
		714	damos_destroy_quota_goal(dst_goal);
		715	}
		716	damos_for_each_quota_goal_safe(src_goal, next, src) {
		717	if (j++ < i)
		718	continue;
		719	new_goal = damos_new_quota_goal(
		720	src_goal->metric, src_goal->target_value);
		721	if (!new_goal)
		722	return -ENOMEM;
		723	damos_add_quota_goal(dst, new_goal);
		724	}
		725	return 0;
		726	}
		727
		728	static int damos_commit_quota(struct damos_quota dst, struct damos_quota src)
		729	{
		730	int err;
		731
		732	dst->reset_interval = src->reset_interval;
		733	dst->ms = src->ms;
		734	dst->sz = src->sz;
		735	err = damos_commit_quota_goals(dst, src);
		736	if (err)
		737	return err;
		738	dst->weight_sz = src->weight_sz;
		739	dst->weight_nr_accesses = src->weight_nr_accesses;
		740	dst->weight_age = src->weight_age;
		741	return 0;
		742	}
		743
		744	static struct damos_filter damos_nth_filter(int n, struct damos s)
		745	{
		746	struct damos_filter *filter;
		747	int i = 0;
		748
		749	damos_for_each_filter(filter, s) {
		750	if (i++ == n)
		751	return filter;
		752	}
		753	return NULL;
		754	}
		755
		756	static void damos_commit_filter_arg(
		757	struct damos_filter dst, struct damos_filter src)
		758	{
		759	switch (dst->type) {
		760	case DAMOS_FILTER_TYPE_MEMCG:
		761	dst->memcg_id = src->memcg_id;
		762	break;
		763	case DAMOS_FILTER_TYPE_ADDR:
		764	dst->addr_range = src->addr_range;
		765	break;
		766	case DAMOS_FILTER_TYPE_TARGET:
		767	dst->target_idx = src->target_idx;
		768	break;
		769	default:
		770	break;
		771	}
		772	}
		773
		774	static void damos_commit_filter(
		775	struct damos_filter dst, struct damos_filter src)
		776	{
		777	dst->type = src->type;
		778	dst->matching = src->matching;
		779	damos_commit_filter_arg(dst, src);
		780	}
		781
		782	static int damos_commit_filters(struct damos dst, struct damos src)
		783	{
		784	struct damos_filter dst_filter, next, src_filter, new_filter;
		785	int i = 0, j = 0;
		786
		787	damos_for_each_filter_safe(dst_filter, next, dst) {
		788	src_filter = damos_nth_filter(i++, src);
		789	if (src_filter)
		790	damos_commit_filter(dst_filter, src_filter);
		791	else
		792	damos_destroy_filter(dst_filter);
		793	}
		794
		795	damos_for_each_filter_safe(src_filter, next, src) {
		796	if (j++ < i)
		797	continue;
		798
		799	new_filter = damos_new_filter(
		800	src_filter->type, src_filter->matching);
		801	if (!new_filter)
		802	return -ENOMEM;
		803	damos_commit_filter_arg(new_filter, src_filter);
		804	damos_add_filter(dst, new_filter);
		805	}
		806	return 0;
		807	}
		808
		809	static struct damos damon_nth_scheme(int n, struct damon_ctx ctx)
		810	{
		811	struct damos *s;
		812	int i = 0;
		813
		814	damon_for_each_scheme(s, ctx) {
		815	if (i++ == n)
		816	return s;
		817	}
		818	return NULL;
		819	}
		820
		821	static int damos_commit(struct damos dst, struct damos src)
		822	{
		823	int err;
		824
		825	dst->pattern = src->pattern;
		826	dst->action = src->action;
		827	dst->apply_interval_us = src->apply_interval_us;
		828
		829	err = damos_commit_quota(&dst->quota, &src->quota);
		830	if (err)
		831	return err;
		832
		833	dst->wmarks = src->wmarks;
		834
		835	err = damos_commit_filters(dst, src);
		836	return err;
		837	}
		838
		839	static int damon_commit_schemes(struct damon_ctx dst, struct damon_ctx src)
		840	{
		841	struct damos dst_scheme, next, src_scheme, new_scheme;
		842	int i = 0, j = 0, err;
		843
		844	damon_for_each_scheme_safe(dst_scheme, next, dst) {
		845	src_scheme = damon_nth_scheme(i++, src);
		846	if (src_scheme) {
		847	err = damos_commit(dst_scheme, src_scheme);
		848	if (err)
		849	return err;
		850	} else {
		851	damon_destroy_scheme(dst_scheme);
		852	}
		853	}
		854
		855	damon_for_each_scheme_safe(src_scheme, next, src) {
		856	if (j++ < i)
		857	continue;
		858	new_scheme = damon_new_scheme(&src_scheme->pattern,
		859	src_scheme->action,
		860	src_scheme->apply_interval_us,
		861	&src_scheme->quota, &src_scheme->wmarks,
		862	NUMA_NO_NODE);
		863	if (!new_scheme)
		864	return -ENOMEM;
		865	damon_add_scheme(dst, new_scheme);
		866	}
		867	return 0;
		868	}
		869
		870	static struct damon_target damon_nth_target(int n, struct damon_ctx ctx)
		871	{
		872	struct damon_target *t;
		873	int i = 0;
		874
		875	damon_for_each_target(t, ctx) {
		876	if (i++ == n)
		877	return t;
		878	}
		879	return NULL;
		880	}
		881
		882	/*
		883	* The caller should ensure the regions of @src are
		884	* 1. valid (end >= src) and
		885	* 2. sorted by starting address.
		886	*
		887	* If @src has no region, @dst keeps current regions.
		888	*/
		889	static int damon_commit_target_regions(
		890	struct damon_target dst, struct damon_target src)
		891	{
		892	struct damon_region *src_region;
		893	struct damon_addr_range *ranges;
		894	int i = 0, err;
		895
		896	damon_for_each_region(src_region, src)
		897	i++;
		898	if (!i)
		899	return 0;
		900
		901	ranges = kmalloc_array(i, sizeof(*ranges), GFP_KERNEL \| __GFP_NOWARN);
		902	if (!ranges)
		903	return -ENOMEM;
		904	i = 0;
		905	damon_for_each_region(src_region, src)
		906	ranges[i++] = src_region->ar;
		907	err = damon_set_regions(dst, ranges, i);
		908	kfree(ranges);
		909	return err;
		910	}
		911
		912	static int damon_commit_target(
		913	struct damon_target *dst, bool dst_has_pid,
		914	struct damon_target *src, bool src_has_pid)
		915	{
		916	int err;
		917
		918	err = damon_commit_target_regions(dst, src);
		919	if (err)
		920	return err;
		921	if (dst_has_pid)
		922	put_pid(dst->pid);
		923	if (src_has_pid)
		924	get_pid(src->pid);
		925	dst->pid = src->pid;
		926	return 0;
		927	}
		928
		929	static int damon_commit_targets(
		930	struct damon_ctx dst, struct damon_ctx src)
		931	{
		932	struct damon_target dst_target, next, src_target, new_target;
		933	int i = 0, j = 0, err;
		934
		935	damon_for_each_target_safe(dst_target, next, dst) {
		936	src_target = damon_nth_target(i++, src);
		937	if (src_target) {
		938	err = damon_commit_target(
		939	dst_target, damon_target_has_pid(dst),
		940	src_target, damon_target_has_pid(src));
		941	if (err)
		942	return err;
		943	} else {
		944	if (damon_target_has_pid(dst))
		945	put_pid(dst_target->pid);
		946	damon_destroy_target(dst_target);
		947	}
		948	}
		949
		950	damon_for_each_target_safe(src_target, next, src) {
		951	if (j++ < i)
		952	continue;
		953	new_target = damon_new_target();
		954	if (!new_target)
		955	return -ENOMEM;
		956	err = damon_commit_target(new_target, false,
		957	src_target, damon_target_has_pid(src));
		958	if (err)
		959	return err;
		960	}
		961	return 0;
		962	}
		963
		964	/**
		965	* damon_commit_ctx() - Commit parameters of a DAMON context to another.
		966	* @dst: The commit destination DAMON context.
		967	* @src: The commit source DAMON context.
		968	*
		969	* This function copies user-specified parameters from @src to @dst and update
		970	* the internal status and results accordingly. Users should use this function
		971	* for context-level parameters update of running context, instead of manual
		972	* in-place updates.
		973	*
		974	* This function should be called from parameters-update safe context, like
		975	* DAMON callbacks.
		976	*/
		977	int damon_commit_ctx(struct damon_ctx dst, struct damon_ctx src)
		978	{
		979	int err;
		980
		981	err = damon_commit_schemes(dst, src);
		982	if (err)
		983	return err;
		984	err = damon_commit_targets(dst, src);
		985	if (err)
		986	return err;
		987	/*
		988	* schemes and targets should be updated first, since
		989	* 1. damon_set_attrs() updates monitoring results of targets and
		990	* next_apply_sis of schemes, and
		991	* 2. ops update should be done after pid handling is done (target
		992	* committing require putting pids).
		993	*/
		994	err = damon_set_attrs(dst, &src->attrs);
		995	if (err)
		996	return err;
		997	dst->ops = src->ops;
		998
		999	return 0;
		1000	}
		1001
666	/**	1002	/**
667	* damon_nr_running_ctxs() - Return number of currently running contexts.	1003	* damon_nr_running_ctxs() - Return number of currently running contexts.
668	*/	1004	*/

mm/damon/dbgfs.c+1−1

@@ -281,7 +281,7 @@ static struct damos *str_to_schemes(const char str, ssize_t len, · 280 unmodified lines
281		281
282	pos += parsed;	282	pos += parsed;
283	scheme = damon_new_scheme(&pattern, action, 0, &quota,	283	scheme = damon_new_scheme(&pattern, action, 0, &quota,
284	&wmarks);	284	&wmarks, NUMA_NO_NODE);
285	if (!scheme)	285	if (!scheme)
286	goto fail;	286	goto fail;
287		287

mm/damon/lru_sort.c+22−34

 			/* under the quota. */
 			&quota,
 			/* (De)activate this according to the watermarks. */
-			&damon_lru_sort_wmarks);
+			&damon_lru_sort_wmarks,
+			NUMA_NO_NODE);
 }
 /* Create a DAMON-based operation scheme for hot memory regions */
 	return damon_lru_sort_new_scheme(&pattern, DAMOS_LRU_DEPRIO);
 }
-static void damon_lru_sort_copy_quota_status(struct damos_quota *dst,
-		struct damos_quota *src)
-{
-	dst->total_charged_sz = src->total_charged_sz;
-	dst->total_charged_ns = src->total_charged_ns;
-	dst->charged_sz = src->charged_sz;
-	dst->charged_from = src->charged_from;
-	dst->charge_target_from = src->charge_target_from;
-	dst->charge_addr_from = src->charge_addr_from;
-}
 static int damon_lru_sort_apply_parameters(void)
 {
-	struct damos *scheme, *hot_scheme, *cold_scheme;
+	struct damon_ctx *param_ctx;
-	struct damos *old_hot_scheme = NULL, *old_cold_scheme = NULL;
+	struct damon_target *param_target;
+	struct damos *hot_scheme, *cold_scheme;
 	unsigned int hot_thres, cold_thres;
-	int err = 0;
+	int err;
-	err = damon_set_attrs(ctx, &damon_lru_sort_mon_attrs);
+	err = damon_modules_new_paddr_ctx_target(&param_ctx, &param_target);
 	if (err)
 		return err;
-	damon_for_each_scheme(scheme, ctx) {
+	err = damon_set_attrs(ctx, &damon_lru_sort_mon_attrs);
-		if (!old_hot_scheme) {
+	if (err)
-			old_hot_scheme = scheme;
+		goto out;
-			continue;
-		}
-		old_cold_scheme = scheme;
-	}
+	err = -ENOMEM;
 	hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) *
 		hot_thres_access_freq / 1000;
 	hot_scheme = damon_lru_sort_new_hot_scheme(hot_thres);
 	if (!hot_scheme)
-		return -ENOMEM;
+		goto out;
-	if (old_hot_scheme)
-		damon_lru_sort_copy_quota_status(&hot_scheme->quota,
-				&old_hot_scheme->quota);
 	cold_thres = cold_min_age / damon_lru_sort_mon_attrs.aggr_interval;
 	cold_scheme = damon_lru_sort_new_cold_scheme(cold_thres);
 	if (!cold_scheme) {
 		damon_destroy_scheme(hot_scheme);
-		return -ENOMEM;
+		goto out;
 	}
-	if (old_cold_scheme)
-		damon_lru_sort_copy_quota_status(&cold_scheme->quota,
-				&old_cold_scheme->quota);
-	damon_set_schemes(ctx, &hot_scheme, 1);
+	damon_set_schemes(param_ctx, &hot_scheme, 1);
-	damon_add_scheme(ctx, cold_scheme);
+	damon_add_scheme(param_ctx, cold_scheme);
-	return damon_set_region_biggest_system_ram_default(target,
+	err = damon_set_region_biggest_system_ram_default(param_target,
 					&monitor_region_start,
 					&monitor_region_end);
+	if (err)
+		goto out;
+	err = damon_commit_ctx(ctx, param_ctx);
+out:
+	damon_destroy_ctx(param_ctx);
+	return err;
 }
 static int damon_lru_sort_turn(bool on)

mm/damon/paddr.c+157−0

@@ -12,6 +12,9 @@ · 11 unmodified lines
12	#include <linux/pagemap.h>	12	#include <linux/pagemap.h>
13	#include <linux/rmap.h>	13	#include <linux/rmap.h>
14	#include <linux/swap.h>	14	#include <linux/swap.h>
		15	#include <linux/memory-tiers.h>
		16	#include <linux/migrate.h>
		17	#include <linux/mm_inline.h>
15		18
16	#include "../internal.h"	19	#include "../internal.h"
17	#include "ops-common.h"	20	#include "ops-common.h"
@@ -325,6 +328,153 @@ static unsigned long damon_pa_deactivate_pages(struct damon_region *r, · 307 unmodified lines
325	return damon_pa_mark_accessed_or_deactivate(r, s, false);	328	return damon_pa_mark_accessed_or_deactivate(r, s, false);
326	}	329	}
327		330
		331	static unsigned int __damon_pa_migrate_folio_list(
		332	struct list_head migrate_folios, struct pglist_data pgdat,
		333	int target_nid)
		334	{
		335	unsigned int nr_succeeded = 0;
		336	nodemask_t allowed_mask = NODE_MASK_NONE;
		337	struct migration_target_control mtc = {
		338	/*
		339	* Allocate from 'node', or fail quickly and quietly.
		340	* When this happens, 'page' will likely just be discarded
		341	* instead of migrated.
		342	*/
		343	.gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) \|
		344	__GFP_NOWARN \| __GFP_NOMEMALLOC \| GFP_NOWAIT,
		345	.nid = target_nid,
		346	.nmask = &allowed_mask
		347	};
		348
		349	if (pgdat->node_id == target_nid \|\| target_nid == NUMA_NO_NODE)
		350	return 0;
		351
		352	if (list_empty(migrate_folios))
		353	return 0;
		354
		355	/* Migration ignores all cpuset and mempolicy settings */
		356	migrate_pages(migrate_folios, alloc_migrate_folio, NULL,
		357	(unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON,
		358	&nr_succeeded);
		359
		360	return nr_succeeded;
		361	}
		362
		363	static unsigned int damon_pa_migrate_folio_list(struct list_head *folio_list,
		364	struct pglist_data *pgdat,
		365	int target_nid)
		366	{
		367	unsigned int nr_migrated = 0;
		368	struct folio *folio;
		369	LIST_HEAD(ret_folios);
		370	LIST_HEAD(migrate_folios);
		371
		372	while (!list_empty(folio_list)) {
		373	struct folio *folio;
		374
		375	cond_resched();
		376
		377	folio = lru_to_folio(folio_list);
		378	list_del(&folio->lru);
		379
		380	if (!folio_trylock(folio))
		381	goto keep;
		382
		383	/* Relocate its contents to another node. */
		384	list_add(&folio->lru, &migrate_folios);
		385	folio_unlock(folio);
		386	continue;
		387	keep:
		388	list_add(&folio->lru, &ret_folios);
		389	}
		390	/* 'folio_list' is always empty here */
		391
		392	/* Migrate folios selected for migration */
		393	nr_migrated += __damon_pa_migrate_folio_list(
		394	&migrate_folios, pgdat, target_nid);
		395	/*
		396	* Folios that could not be migrated are still in @migrate_folios. Add
		397	* those back on @folio_list
		398	*/
		399	if (!list_empty(&migrate_folios))
		400	list_splice_init(&migrate_folios, folio_list);
		401
		402	try_to_unmap_flush();
		403
		404	list_splice(&ret_folios, folio_list);
		405
		406	while (!list_empty(folio_list)) {
		407	folio = lru_to_folio(folio_list);
		408	list_del(&folio->lru);
		409	folio_putback_lru(folio);
		410	}
		411
		412	return nr_migrated;
		413	}
		414
		415	static unsigned long damon_pa_migrate_pages(struct list_head *folio_list,
		416	int target_nid)
		417	{
		418	int nid;
		419	unsigned long nr_migrated = 0;
		420	LIST_HEAD(node_folio_list);
		421	unsigned int noreclaim_flag;
		422
		423	if (list_empty(folio_list))
		424	return nr_migrated;
		425
		426	noreclaim_flag = memalloc_noreclaim_save();
		427
		428	nid = folio_nid(lru_to_folio(folio_list));
		429	do {
		430	struct folio *folio = lru_to_folio(folio_list);
		431
		432	if (nid == folio_nid(folio)) {
		433	list_move(&folio->lru, &node_folio_list);
		434	continue;
		435	}
		436
		437	nr_migrated += damon_pa_migrate_folio_list(&node_folio_list,
		438	NODE_DATA(nid),
		439	target_nid);
		440	nid = folio_nid(lru_to_folio(folio_list));
		441	} while (!list_empty(folio_list));
		442
		443	nr_migrated += damon_pa_migrate_folio_list(&node_folio_list,
		444	NODE_DATA(nid),
		445	target_nid);
		446
		447	memalloc_noreclaim_restore(noreclaim_flag);
		448
		449	return nr_migrated;
		450	}
		451
		452	static unsigned long damon_pa_migrate(struct damon_region r, struct damos s)
		453	{
		454	unsigned long addr, applied;
		455	LIST_HEAD(folio_list);
		456
		457	for (addr = r->ar.start; addr < r->ar.end; addr += PAGE_SIZE) {
		458	struct folio *folio = damon_get_folio(PHYS_PFN(addr));
		459
		460	if (!folio)
		461	continue;
		462
		463	if (damos_pa_filter_out(s, folio))
		464	goto put_folio;
		465
		466	if (!folio_isolate_lru(folio))
		467	goto put_folio;
		468	list_add(&folio->lru, &folio_list);
		469	put_folio:
		470	folio_put(folio);
		471	}
		472	applied = damon_pa_migrate_pages(&folio_list, s->target_nid);
		473	cond_resched();
		474	return applied * PAGE_SIZE;
		475	}
		476
		477
328	static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,	478	static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx,
329	struct damon_target t, struct damon_region r,	479	struct damon_target t, struct damon_region r,
330	struct damos *scheme)	480	struct damos *scheme)
@@ -336,6 +486,9 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, · 5 unmodified lines
336	return damon_pa_mark_accessed(r, scheme);	486	return damon_pa_mark_accessed(r, scheme);
337	case DAMOS_LRU_DEPRIO:	487	case DAMOS_LRU_DEPRIO:
338	return damon_pa_deactivate_pages(r, scheme);	488	return damon_pa_deactivate_pages(r, scheme);
		489	case DAMOS_MIGRATE_HOT:
		490	case DAMOS_MIGRATE_COLD:
		491	return damon_pa_migrate(r, scheme);
339	case DAMOS_STAT:	492	case DAMOS_STAT:
340	break;	493	break;
341	default:	494	default:
@@ -356,6 +509,10 @@ static int damon_pa_scheme_score(struct damon_ctx *context, · 14 unmodified lines
356	return damon_hot_score(context, r, scheme);	509	return damon_hot_score(context, r, scheme);
357	case DAMOS_LRU_DEPRIO:	510	case DAMOS_LRU_DEPRIO:
358	return damon_cold_score(context, r, scheme);	511	return damon_cold_score(context, r, scheme);
		512	case DAMOS_MIGRATE_HOT:
		513	return damon_hot_score(context, r, scheme);
		514	case DAMOS_MIGRATE_COLD:
		515	return damon_cold_score(context, r, scheme);
359	default:	516	default:
360	break;	517	break;
361	}	518	}

mm/damon/reclaim.c+27−38

 			/* under the quota. */
 			&damon_reclaim_quota,
 			/* (De)activate this according to the watermarks. */
-			&damon_reclaim_wmarks);
+			&damon_reclaim_wmarks,
-}
+			NUMA_NO_NODE);
-static void damon_reclaim_copy_quota_status(struct damos_quota *dst,
-		struct damos_quota *src)
-{
-	dst->total_charged_sz = src->total_charged_sz;
-	dst->total_charged_ns = src->total_charged_ns;
-	dst->charged_sz = src->charged_sz;
-	dst->charged_from = src->charged_from;
-	dst->charge_target_from = src->charge_target_from;
-	dst->charge_addr_from = src->charge_addr_from;
-	dst->esz_bp = src->esz_bp;
 }
 static int damon_reclaim_apply_parameters(void)
 {
-	struct damos *scheme, *old_scheme;
+	struct damon_ctx *param_ctx;
+	struct damon_target *param_target;
+	struct damos *scheme;
 	struct damos_quota_goal *goal;
 	struct damos_filter *filter;
-	int err = 0;
+	int err;
-	err = damon_set_attrs(ctx, &damon_reclaim_mon_attrs);
+	err = damon_modules_new_paddr_ctx_target(&param_ctx, &param_target);
 	if (err)
 		return err;
-	/* Will be freed by next 'damon_set_schemes()' below */
+	err = damon_set_attrs(ctx, &damon_reclaim_mon_attrs);
+	if (err)
+		goto out;
+	err = -ENOMEM;
 	scheme = damon_reclaim_new_scheme();
 	if (!scheme)
-		return -ENOMEM;
+		goto out;
-	if (!list_empty(&ctx->schemes)) {
+	damon_set_schemes(ctx, &scheme, 1);
-		damon_for_each_scheme(old_scheme, ctx)
-			damon_reclaim_copy_quota_status(&scheme->quota,
-					&old_scheme->quota);
-	}
 	if (quota_mem_pressure_us) {
 		goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US,
 				quota_mem_pressure_us);
-		if (!goal) {
+		if (!goal)
-			damon_destroy_scheme(scheme);
+			goto out;
-			return -ENOMEM;
-		}
 		damos_add_quota_goal(&scheme->quota, goal);
 	}
 	if (quota_autotune_feedback) {
 		goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000);
-		if (!goal) {
+		if (!goal)
-			damon_destroy_scheme(scheme);
+			goto out;
-			return -ENOMEM;
-		}
 		goal->current_value = quota_autotune_feedback;
 		damos_add_quota_goal(&scheme->quota, goal);
 	}
 	if (skip_anon) {
 		filter = damos_new_filter(DAMOS_FILTER_TYPE_ANON, true);
-		if (!filter) {
+		if (!filter)
-			/* Will be freed by next 'damon_set_schemes()' below */
+			goto out;
-			damon_destroy_scheme(scheme);
-			return -ENOMEM;
-		}
 		damos_add_filter(scheme, filter);
 	}
-	damon_set_schemes(ctx, &scheme, 1);
-	return damon_set_region_biggest_system_ram_default(target,
+	err = damon_set_region_biggest_system_ram_default(param_target,
 					&monitor_region_start,
 					&monitor_region_end);
+	if (err)
+		goto out;
+	err = damon_commit_ctx(ctx, param_ctx);
+out:
+	damon_destroy_ctx(param_ctx);
+	return err;
 }
 static int damon_reclaim_turn(bool on)

mm/damon/sysfs-common.h+1−1

@@ -38,7 +38,7 @@ void damon_sysfs_schemes_rm_dirs(struct damon_sysfs_schemes *schemes); · 37 unmodified lines
38		38
39	extern const struct kobj_type damon_sysfs_schemes_ktype;	39	extern const struct kobj_type damon_sysfs_schemes_ktype;
40		40
41	int damon_sysfs_set_schemes(struct damon_ctx *ctx,	41	int damon_sysfs_add_schemes(struct damon_ctx *ctx,
42	struct damon_sysfs_schemes *sysfs_schemes);	42	struct damon_sysfs_schemes *sysfs_schemes);
43		43
44	void damon_sysfs_schemes_update_stats(	44	void damon_sysfs_schemes_update_stats(

mm/damon/sysfs-schemes.c+54−73

  */
 #include <linux/slab.h>
+#include <linux/numa.h>
 #include "sysfs-common.h"
 	struct damon_sysfs_scheme_filters *filters;
 	struct damon_sysfs_stats *stats;
 	struct damon_sysfs_scheme_regions *tried_regions;
+	int target_nid;
 };
 /* This should match with enum damos_action */
 	"nohugepage",
 	"lru_prio",
 	"lru_deprio",
+	"migrate_hot",
+	"migrate_cold",
 	"stat",
 };
 	scheme->kobj = (struct kobject){};
 	scheme->action = action;
 	scheme->apply_interval_us = apply_interval_us;
+	scheme->target_nid = NUMA_NO_NODE;
 	return scheme;
 }
 	return err ? err : count;
 }
+static ssize_t target_nid_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	struct damon_sysfs_scheme *scheme = container_of(kobj,
+			struct damon_sysfs_scheme, kobj);
+	return sysfs_emit(buf, "%d\n", scheme->target_nid);
+}
+static ssize_t target_nid_store(struct kobject *kobj,
+		struct kobj_attribute *attr, const char *buf, size_t count)
+{
+	struct damon_sysfs_scheme *scheme = container_of(kobj,
+			struct damon_sysfs_scheme, kobj);
+	int err = 0;
+	/* TODO: error handling for target_nid range. */
+	err = kstrtoint(buf, 0, &scheme->target_nid);
+	return err ? err : count;
+}
 static void damon_sysfs_scheme_release(struct kobject *kobj)
 {
 	kfree(container_of(kobj, struct damon_sysfs_scheme, kobj));
 static struct kobj_attribute damon_sysfs_scheme_apply_interval_us_attr =
 		__ATTR_RW_MODE(apply_interval_us, 0600);
+static struct kobj_attribute damon_sysfs_scheme_target_nid_attr =
+		__ATTR_RW_MODE(target_nid, 0600);
 static struct attribute *damon_sysfs_scheme_attrs[] = {
 	&damon_sysfs_scheme_action_attr.attr,
 	&damon_sysfs_scheme_apply_interval_us_attr.attr,
+	&damon_sysfs_scheme_target_nid_attr.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(damon_sysfs_scheme);
 	return found ? 0 : -EINVAL;
 }
-static int damon_sysfs_set_scheme_filters(struct damos *scheme,
+static int damon_sysfs_add_scheme_filters(struct damos *scheme,
 		struct damon_sysfs_scheme_filters *sysfs_filters)
 {
 	int i;
-	struct damos_filter *filter, *next;
-	damos_for_each_filter_safe(filter, next, scheme)
-		damos_destroy_filter(filter);
 	for (i = 0; i < sysfs_filters->nr; i++) {
 		struct damon_sysfs_scheme_filter *sysfs_filter =
 	return 0;
 }
-static int damos_sysfs_set_quota_score(
+static int damos_sysfs_add_quota_score(
 		struct damos_sysfs_quota_goals *sysfs_goals,
 		struct damos_quota *quota)
 {
-	struct damos_quota_goal *goal, *next;
+	struct damos_quota_goal *goal;
 	int i;
-	damos_for_each_quota_goal_safe(goal, next, quota)
-		damos_destroy_quota_goal(goal);
 	for (i = 0; i < sysfs_goals->nr; i++) {
 		struct damos_sysfs_quota_goal *sysfs_goal =
 			sysfs_goals->goals_arr[i];
 		struct damon_ctx *ctx)
 {
 	struct damos *scheme;
+	struct damos_quota quota = {};
 	int i = 0;
+	INIT_LIST_HEAD(&quota.goals);
 	damon_for_each_scheme(scheme, ctx) {
 		struct damon_sysfs_scheme *sysfs_scheme;
+		struct damos_quota_goal *g, *g_next;
 		int err;
 		/* user could have removed the scheme sysfs dir */
 			break;
 		sysfs_scheme = sysfs_schemes->schemes_arr[i];
-		err = damos_sysfs_set_quota_score(sysfs_scheme->quotas->goals,
+		err = damos_sysfs_add_quota_score(sysfs_scheme->quotas->goals,
-				&scheme->quota);
+				&quota);
+		if (err) {
+			damos_for_each_quota_goal_safe(g, g_next, &quota)
+				damos_destroy_quota_goal(g);
+			return err;
+		}
+		err = damos_commit_quota_goals(&scheme->quota, &quota);
+		damos_for_each_quota_goal_safe(g, g_next, &quota)
+			damos_destroy_quota_goal(g);
 		if (err)
-			/* kdamond will clean up schemes and terminated */
 			return err;
 		i++;
 	}
 	};
 	scheme = damon_new_scheme(&pattern, sysfs_scheme->action,
-			sysfs_scheme->apply_interval_us, &quota, &wmarks);
+			sysfs_scheme->apply_interval_us, &quota, &wmarks,
+			sysfs_scheme->target_nid);
 	if (!scheme)
 		return NULL;
-	err = damos_sysfs_set_quota_score(sysfs_quotas->goals, &scheme->quota);
+	err = damos_sysfs_add_quota_score(sysfs_quotas->goals, &scheme->quota);
 	if (err) {
 		damon_destroy_scheme(scheme);
 		return NULL;
 	}
-	err = damon_sysfs_set_scheme_filters(scheme, sysfs_filters);
+	err = damon_sysfs_add_scheme_filters(scheme, sysfs_filters);
 	if (err) {
 		damon_destroy_scheme(scheme);
 		return NULL;
 	return scheme;
 }
-static void damon_sysfs_update_scheme(struct damos *scheme,
+int damon_sysfs_add_schemes(struct damon_ctx *ctx,
-		struct damon_sysfs_scheme *sysfs_scheme)
-{
-	struct damon_sysfs_access_pattern *access_pattern =
-		sysfs_scheme->access_pattern;
-	struct damon_sysfs_quotas *sysfs_quotas = sysfs_scheme->quotas;
-	struct damon_sysfs_weights *sysfs_weights = sysfs_quotas->weights;
-	struct damon_sysfs_watermarks *sysfs_wmarks = sysfs_scheme->watermarks;
-	int err;
-	scheme->pattern.min_sz_region = access_pattern->sz->min;
-	scheme->pattern.max_sz_region = access_pattern->sz->max;
-	scheme->pattern.min_nr_accesses = access_pattern->nr_accesses->min;
-	scheme->pattern.max_nr_accesses = access_pattern->nr_accesses->max;
-	scheme->pattern.min_age_region = access_pattern->age->min;
-	scheme->pattern.max_age_region = access_pattern->age->max;
-	scheme->action = sysfs_scheme->action;
-	scheme->apply_interval_us = sysfs_scheme->apply_interval_us;
-	scheme->quota.ms = sysfs_quotas->ms;
-	scheme->quota.sz = sysfs_quotas->sz;
-	scheme->quota.reset_interval = sysfs_quotas->reset_interval_ms;
-	scheme->quota.weight_sz = sysfs_weights->sz;
-	scheme->quota.weight_nr_accesses = sysfs_weights->nr_accesses;
-	scheme->quota.weight_age = sysfs_weights->age;
-	err = damos_sysfs_set_quota_score(sysfs_quotas->goals, &scheme->quota);
-	if (err) {
-		damon_destroy_scheme(scheme);
-		return;
-	}
-	scheme->wmarks.metric = sysfs_wmarks->metric;
-	scheme->wmarks.interval = sysfs_wmarks->interval_us;
-	scheme->wmarks.high = sysfs_wmarks->high;
-	scheme->wmarks.mid = sysfs_wmarks->mid;
-	scheme->wmarks.low = sysfs_wmarks->low;
-	err = damon_sysfs_set_scheme_filters(scheme, sysfs_scheme->filters);
-	if (err)
-		damon_destroy_scheme(scheme);
-}
-int damon_sysfs_set_schemes(struct damon_ctx *ctx,
 		struct damon_sysfs_schemes *sysfs_schemes)
 {
-	struct damos *scheme, *next;
+	int i;
-	int i = 0;
-	damon_for_each_scheme_safe(scheme, next, ctx) {
-		if (i < sysfs_schemes->nr)
-			damon_sysfs_update_scheme(scheme,
-					sysfs_schemes->schemes_arr[i]);
-		else
-			damon_destroy_scheme(scheme);
-		i++;
-	}
-	for (; i < sysfs_schemes->nr; i++) {
+	for (i = 0; i < sysfs_schemes->nr; i++) {
 		struct damos *scheme, *next;
 		scheme = damon_sysfs_mk_scheme(sysfs_schemes->schemes_arr[i]);

mm/damon/sysfs-test.h+5−5

@@ -38,7 +38,7 @@ static int __damon_sysfs_test_get_any_pid(int min, int max) · 37 unmodified lines
38	return -1;	38	return -1;
39	}	39	}
40		40
41	static void damon_sysfs_test_set_targets(struct kunit *test)	41	static void damon_sysfs_test_add_targets(struct kunit *test)
42	{	42	{
43	struct damon_sysfs_targets *sysfs_targets;	43	struct damon_sysfs_targets *sysfs_targets;
44	struct damon_sysfs_target *sysfs_target;	44	struct damon_sysfs_target *sysfs_target;
@@ -56,13 +56,13 @@ static void damon_sysfs_test_set_targets(struct kunit *test) · 11 unmodified lines
56		56
57	ctx = damon_new_ctx();	57	ctx = damon_new_ctx();
58		58
59	damon_sysfs_set_targets(ctx, sysfs_targets);	59	damon_sysfs_add_targets(ctx, sysfs_targets);
60	KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));	60	KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));
61		61
62	sysfs_target->pid = __damon_sysfs_test_get_any_pid(	62	sysfs_target->pid = __damon_sysfs_test_get_any_pid(
63	sysfs_target->pid + 1, 200);	63	sysfs_target->pid + 1, 200);
64	damon_sysfs_set_targets(ctx, sysfs_targets);	64	damon_sysfs_add_targets(ctx, sysfs_targets);
65	KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx));	65	KUNIT_EXPECT_EQ(test, 2u, nr_damon_targets(ctx));
66		66
67	damon_destroy_ctx(ctx);	67	damon_destroy_ctx(ctx);
68	kfree(sysfs_targets->targets_arr);	68	kfree(sysfs_targets->targets_arr);
@@ -71,7 +71,7 @@ static void damon_sysfs_test_set_targets(struct kunit *test) · 2 unmodified lines
71	}	71	}
72		72
73	static struct kunit_case damon_sysfs_test_cases[] = {	73	static struct kunit_case damon_sysfs_test_cases[] = {
74	KUNIT_CASE(damon_sysfs_test_set_targets),	74	KUNIT_CASE(damon_sysfs_test_add_targets),
75	{},	75	{},
76	};	76	};
77		77

mm/damon/sysfs.c+18−63

@@ -1162,72 +1162,16 @@ destroy_targets_out: · 1161 unmodified lines
1162	return err;	1162	return err;
1163	}	1163	}
1164		1164
1165	static int damon_sysfs_update_target_pid(struct damon_target *target, int pid)	1165	static int damon_sysfs_add_targets(struct damon_ctx *ctx,
1166	{
1167	struct pid *pid_new;
1168
1169	pid_new = find_get_pid(pid);
1170	if (!pid_new)
1171	return -EINVAL;
1172
1173	if (pid_new == target->pid) {
1174	put_pid(pid_new);
1175	return 0;
1176	}
1177
1178	put_pid(target->pid);
1179	target->pid = pid_new;
1180	return 0;
1181	}
1182
1183	static int damon_sysfs_update_target(struct damon_target *target,
1184	struct damon_ctx *ctx,
1185	struct damon_sysfs_target *sys_target)
1186	{
1187	int err = 0;
1188
1189	if (damon_target_has_pid(ctx)) {
1190	err = damon_sysfs_update_target_pid(target, sys_target->pid);
1191	if (err)
1192	return err;
1193	}
1194
1195	/*
1196	* Do monitoring target region boundary update only if one or more
1197	* regions are set by the user. This is for keeping current monitoring
1198	* target results and range easier, especially for dynamic monitoring
1199	* target regions update ops like 'vaddr'.
1200	*/
1201	if (sys_target->regions->nr)
1202	err = damon_sysfs_set_regions(target, sys_target->regions);
1203	return err;
1204	}
1205
1206	static int damon_sysfs_set_targets(struct damon_ctx *ctx,
1207	struct damon_sysfs_targets *sysfs_targets)	1166	struct damon_sysfs_targets *sysfs_targets)
1208	{	1167	{
1209	struct damon_target t, next;	1168	int i, err;
1210	int i = 0, err;
1211		1169
1212	/* Multiple physical address space monitoring targets makes no sense */	1170	/* Multiple physical address space monitoring targets makes no sense */
1213	if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1)	1171	if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1)
1214	return -EINVAL;	1172	return -EINVAL;
1215		1173
1216	damon_for_each_target_safe(t, next, ctx) {	1174	for (i = 0; i < sysfs_targets->nr; i++) {
1217	if (i < sysfs_targets->nr) {
1218	err = damon_sysfs_update_target(t, ctx,
1219	sysfs_targets->targets_arr[i]);
1220	if (err)
1221	return err;
1222	} else {
1223	if (damon_target_has_pid(ctx))
1224	put_pid(t->pid);
1225	damon_destroy_target(t);
1226	}
1227	i++;
1228	}
1229
1230	for (; i < sysfs_targets->nr; i++) {
1231	struct damon_sysfs_target *st = sysfs_targets->targets_arr[i];	1175	struct damon_sysfs_target *st = sysfs_targets->targets_arr[i];
1232		1176
1233	err = damon_sysfs_add_target(st, ctx);	1177	err = damon_sysfs_add_target(st, ctx);
@@ -1339,12 +1283,15 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 105 unmodified lines
1339	err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);	1283	err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
1340	if (err)	1284	if (err)
1341	return err;	1285	return err;
1342	err = damon_sysfs_set_targets(ctx, sys_ctx->targets);	1286	err = damon_sysfs_add_targets(ctx, sys_ctx->targets);
1343	if (err)	1287	if (err)
1344	return err;	1288	return err;
1345	return damon_sysfs_set_schemes(ctx, sys_ctx->schemes);	1289	return damon_sysfs_add_schemes(ctx, sys_ctx->schemes);
1346	}	1290	}
1347		1291
		1292	static struct damon_ctx *damon_sysfs_build_ctx(
		1293	struct damon_sysfs_context *sys_ctx);
		1294
1348	/*	1295	/*
1349	* damon_sysfs_commit_input() - Commit user inputs to a running kdamond.	1296	* damon_sysfs_commit_input() - Commit user inputs to a running kdamond.
1350	* @kdamond: The kobject wrapper for the associated kdamond.	1297	* @kdamond: The kobject wrapper for the associated kdamond.
@@ -1353,14 +1300,22 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 2 unmodified lines
1353	*/	1300	*/
1354	static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond)	1301	static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond)
1355	{	1302	{
		1303	struct damon_ctx *param_ctx;
		1304	int err;
		1305
1356	if (!damon_sysfs_kdamond_running(kdamond))	1306	if (!damon_sysfs_kdamond_running(kdamond))
1357	return -EINVAL;	1307	return -EINVAL;
1358	/* TODO: Support multiple contexts per kdamond */	1308	/* TODO: Support multiple contexts per kdamond */
1359	if (kdamond->contexts->nr != 1)	1309	if (kdamond->contexts->nr != 1)
1360	return -EINVAL;	1310	return -EINVAL;
1361		1311
1362	return damon_sysfs_apply_inputs(kdamond->damon_ctx,	1312	param_ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0]);
1363	kdamond->contexts->contexts_arr[0]);	1313	if (IS_ERR(param_ctx))
		1314	return PTR_ERR(param_ctx);
		1315	err = damon_commit_ctx(kdamond->damon_ctx, param_ctx);
		1316	damon_sysfs_destroy_targets(param_ctx);
		1317	damon_destroy_ctx(param_ctx);
		1318	return err;
1364	}	1319	}
1365		1320
1366	static int damon_sysfs_commit_schemes_quota_goals(	1321	static int damon_sysfs_commit_schemes_quota_goals(

mm/damon/vaddr.c+3−3

@@ -339,7 +339,7 @@ static void damon_hugetlb_mkold(pte_t pte, struct mm_struct mm, · 338 unmodified lines
339	struct vm_area_struct *vma, unsigned long addr)	339	struct vm_area_struct *vma, unsigned long addr)
340	{	340	{
341	bool referenced = false;	341	bool referenced = false;
342	pte_t entry = huge_ptep_get(pte);	342	pte_t entry = huge_ptep_get(mm, addr, pte);
343	struct folio *folio = pfn_folio(pte_pfn(entry));	343	struct folio *folio = pfn_folio(pte_pfn(entry));
344	unsigned long psize = huge_page_size(hstate_vma(vma));	344	unsigned long psize = huge_page_size(hstate_vma(vma));
345		345
@@ -373,7 +373,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, · 27 unmodified lines
373	pte_t entry;	373	pte_t entry;
374		374
375	ptl = huge_pte_lock(h, walk->mm, pte);	375	ptl = huge_pte_lock(h, walk->mm, pte);
376	entry = huge_ptep_get(pte);	376	entry = huge_ptep_get(walk->mm, addr, pte);
377	if (!pte_present(entry))	377	if (!pte_present(entry))
378	goto out;	378	goto out;
379		379
@@ -509,7 +509,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, · 129 unmodified lines
509	pte_t entry;	509	pte_t entry;
510		510
511	ptl = huge_pte_lock(h, walk->mm, pte);	511	ptl = huge_pte_lock(h, walk->mm, pte);
512	entry = huge_ptep_get(pte);	512	entry = huge_ptep_get(walk->mm, addr, pte);
513	if (!pte_present(entry))	513	if (!pte_present(entry))
514	goto out;	514	goto out;
515		515

mm/dmapool_test.c+1−0

@@ -144,4 +144,5 @@ static void dmapool_exit(void) · 143 unmodified lines
144		144
145	module_init(dmapool_checks);	145	module_init(dmapool_checks);
146	module_exit(dmapool_exit);	146	module_exit(dmapool_exit);
		147	MODULE_DESCRIPTION("dma_pool timing test");
147	MODULE_LICENSE("GPL");	148	MODULE_LICENSE("GPL");

mm/fail_page_alloc.c+3−1

 // SPDX-License-Identifier: GPL-2.0
 #include <linux/fault-inject.h>
+#include <linux/error-injection.h>
 #include <linux/mm.h>
 static struct {
 }
 __setup("fail_page_alloc=", setup_fail_page_alloc);
-bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
 {
 	int flags = 0;
 	return should_fail_ex(&fail_page_alloc.attr, 1 << order, flags);
 }
+ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE);
 #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS

mm/failslab.c+8−6

 // SPDX-License-Identifier: GPL-2.0
 #include <linux/fault-inject.h>
+#include <linux/error-injection.h>
 #include <linux/slab.h>
 #include <linux/mm.h>
 #include "slab.h"
 	.cache_filter = false,
 };
-bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags)
+int should_failslab(struct kmem_cache *s, gfp_t gfpflags)
 {
 	int flags = 0;
 	/* No fault-injection for bootstrap cache */
 	if (unlikely(s == kmem_cache))
-		return false;
+		return 0;
 	if (gfpflags & __GFP_NOFAIL)
-		return false;
+		return 0;
 	if (failslab.ignore_gfp_reclaim &&
 			(gfpflags & __GFP_DIRECT_RECLAIM))
-		return false;
+		return 0;
 	if (failslab.cache_filter && !(s->flags & SLAB_FAILSLAB))
-		return false;
+		return 0;
 	/*
 	 * In some cases, it expects to specify __GFP_NOWARN
 	if (gfpflags & __GFP_NOWARN)
 		flags |= FAULT_NOWARN;
-	return should_fail_ex(&failslab.attr, s->object_size, flags);
+	return should_fail_ex(&failslab.attr, s->object_size, flags) ? -ENOMEM : 0;
 }
+ALLOW_ERROR_INJECTION(should_failslab, ERRNO);
 static int __init setup_failslab(char *str)
 {

mm/filemap.c+4−4

@@ -177,7 +177,7 @@ static void filemap_unaccount_folio(struct address_space *mapping, · 176 unmodified lines
177	* and we'd rather not leak it: if we're wrong,	177	* and we'd rather not leak it: if we're wrong,
178	* another bad page check should catch it later.	178	* another bad page check should catch it later.
179	*/	179	*/
180	page_mapcount_reset(&folio->page);	180	atomic_set(&folio->_mapcount, -1);
181	folio_ref_sub(folio, mapcount);	181	folio_ref_sub(folio, mapcount);
182	}	182	}
183	}	183	}
@@ -1752,12 +1752,12 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1568 unmodified lines
1752	while (max_scan--) {	1752	while (max_scan--) {
1753	void *entry = xas_next(&xas);	1753	void *entry = xas_next(&xas);
1754	if (!entry \|\| xa_is_value(entry))	1754	if (!entry \|\| xa_is_value(entry))
1755	break;	1755	return xas.xa_index;
1756	if (xas.xa_index == 0)	1756	if (xas.xa_index == 0)
1757	break;	1757	return 0;
1758	}	1758	}
1759		1759
1760	return xas.xa_index;	1760	return index + max_scan;
1761	}	1761	}
1762	EXPORT_SYMBOL(page_cache_next_miss);	1762	EXPORT_SYMBOL(page_cache_next_miss);
1763		1763

mm/folio-compat.c+0−6

 #include <linux/swap.h>
 #include "internal.h"
-struct address_space *page_mapping(struct page *page)
-{
-	return folio_mapping(page_folio(page));
-}
-EXPORT_SYMBOL(page_mapping);
 void unlock_page(struct page *page)
 {
 	return folio_unlock(page_folio(page));

mm/gup.c+271−239

 #include <linux/spinlock.h>
 #include <linux/mm.h>
+#include <linux/memfd.h>
 #include <linux/memremap.h>
 #include <linux/pagemap.h>
 #include <linux/rmap.h>
 #include <linux/hugetlb.h>
 #include <linux/migrate.h>
 #include <linux/mm_inline.h>
+#include <linux/pagevec.h>
 #include <linux/sched/mm.h>
 #include <linux/shmem_fs.h>
 }
 EXPORT_SYMBOL(unpin_user_page);
+/**
+ * unpin_folio() - release a dma-pinned folio
+ * @folio:         pointer to folio to be released
+ *
+ * Folios that were pinned via memfd_pin_folios() or other similar routines
+ * must be released either using unpin_folio() or unpin_folios().
+ */
+void unpin_folio(struct folio *folio)
+{
+	gup_put_folio(folio, 1, FOLL_PIN);
+}
+EXPORT_SYMBOL_GPL(unpin_folio);
 /**
  * folio_add_pin - Try to get an additional pin on a pinned folio
  * @folio: The folio to be pinned
 		 * 1) This code sees the page as already dirty, so it
 		 * skips the call to set_page_dirty(). That could happen
 		 * because clear_page_dirty_for_io() called
-		 * page_mkclean(), followed by set_page_dirty().
+		 * folio_mkclean(), followed by set_page_dirty().
 		 * However, now the page is going to get written back,
 		 * which meets the original intention of setting it
 		 * dirty, so all is well: clear_page_dirty_for_io() goes
 }
 EXPORT_SYMBOL(unpin_user_pages);
+/**
+ * unpin_folios() - release an array of gup-pinned folios.
+ * @folios:  array of folios to be marked dirty and released.
+ * @nfolios: number of folios in the @folios array.
+ *
+ * For each folio in the @folios array, release the folio using gup_put_folio.
+ *
+ * Please see the unpin_folio() documentation for details.
+ */
+void unpin_folios(struct folio **folios, unsigned long nfolios)
+{
+	unsigned long i = 0, j;
+	/*
+	 * If this WARN_ON() fires, then the system *might* be leaking folios
+	 * (by leaving them pinned), but probably not. More likely, gup/pup
+	 * returned a hard -ERRNO error to the caller, who erroneously passed
+	 * it here.
+	 */
+	if (WARN_ON(IS_ERR_VALUE(nfolios)))
+		return;
+	while (i < nfolios) {
+		for (j = i + 1; j < nfolios; j++)
+			if (folios[i] != folios[j])
+				break;
+		if (folios[i])
+			gup_put_folio(folios[i], j - i, FOLL_PIN);
+		i = j;
+	}
+}
+EXPORT_SYMBOL_GPL(unpin_folios);
 /*
  * Set the MMF_HAS_PINNED if not set yet; after set it'll be there for the mm's
  * lifecycle.  Avoid setting the bit unless necessary, or it might cause write
 #ifdef CONFIG_MMU
-#if defined(CONFIG_ARCH_HAS_HUGEPD) || defined(CONFIG_HAVE_GUP_FAST)
+#ifdef CONFIG_HAVE_GUP_FAST
 static int record_subpages(struct page *page, unsigned long sz,
 			   unsigned long addr, unsigned long end,
 			   struct page **pages)
 	return folio;
 }
-#endif	/* CONFIG_ARCH_HAS_HUGEPD || CONFIG_HAVE_GUP_FAST */
+#endif	/* CONFIG_HAVE_GUP_FAST */
-#ifdef CONFIG_ARCH_HAS_HUGEPD
-static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,
-				      unsigned long sz)
-{
-	unsigned long __boundary = (addr + sz) & ~(sz-1);
-	return (__boundary - 1 < end - 1) ? __boundary : end;
-}
-/*
- * Returns 1 if succeeded, 0 if failed, -EMLINK if unshare needed.
- *
- * NOTE: for the same entry, gup-fast and gup-slow can return different
- * results (0 v.s. -EMLINK) depending on whether vma is available.  This is
- * the expected behavior, where we simply want gup-fast to fallback to
- * gup-slow to take the vma reference first.
- */
-static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz,
-		       unsigned long addr, unsigned long end, unsigned int flags,
-		       struct page **pages, int *nr, bool fast)
-{
-	unsigned long pte_end;
-	struct page *page;
-	struct folio *folio;
-	pte_t pte;
-	int refs;
-	pte_end = (addr + sz) & ~(sz-1);
-	if (pte_end < end)
-		end = pte_end;
-	pte = huge_ptep_get(ptep);
-	if (!pte_access_permitted(pte, flags & FOLL_WRITE))
-		return 0;
-	/* hugepages are never "special" */
-	VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
-	page = pte_page(pte);
-	refs = record_subpages(page, sz, addr, end, pages + *nr);
-	if (fast) {
-		folio = try_grab_folio_fast(page, refs, flags);
-		if (!folio)
-			return 0;
-	} else {
-		folio = page_folio(page);
-		if (try_grab_folio(folio, refs, flags))
-			return 0;
-	}
-	if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {
-		gup_put_folio(folio, refs, flags);
-		return 0;
-	}
-	if (!pte_write(pte) && gup_must_unshare(vma, flags, &folio->page)) {
-		gup_put_folio(folio, refs, flags);
-		return -EMLINK;
-	}
-	*nr += refs;
-	folio_set_referenced(folio);
-	return 1;
-}
-/*
- * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file
- * systems on Power, which does not have issue with folio writeback against
- * GUP updates.  When hugepd will be extended to support non-hugetlbfs or
- * even anonymous memory, we need to do extra check as what we do with most
- * of the other folios. See writable_file_mapping_allowed() and
- * gup_fast_folio_allowed() for more information.
- */
-static int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd,
-		      unsigned long addr, unsigned int pdshift,
-		      unsigned long end, unsigned int flags,
-		      struct page **pages, int *nr, bool fast)
-{
-	pte_t *ptep;
-	unsigned long sz = 1UL << hugepd_shift(hugepd);
-	unsigned long next;
-	int ret;
-	ptep = hugepte_offset(hugepd, addr, pdshift);
-	do {
-		next = hugepte_addr_end(addr, end, sz);
-		ret = gup_hugepte(vma, ptep, sz, addr, end, flags, pages, nr,
-				  fast);
-		if (ret != 1)
-			return ret;
-	} while (ptep++, addr = next, addr != end);
-	return 1;
-}
-static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd,
-				  unsigned long addr, unsigned int pdshift,
-				  unsigned int flags,
-				  struct follow_page_context *ctx)
-{
-	struct page *page;
-	struct hstate *h;
-	spinlock_t *ptl;
-	int nr = 0, ret;
-	pte_t *ptep;
-	/* Only hugetlb supports hugepd */
-	if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma)))
-		return ERR_PTR(-EFAULT);
-	h = hstate_vma(vma);
-	ptep = hugepte_offset(hugepd, addr, pdshift);
-	ptl = huge_pte_lock(h, vma->vm_mm, ptep);
-	ret = gup_hugepd(vma, hugepd, addr, pdshift, addr + PAGE_SIZE,
-			 flags, &page, &nr, false);
-	spin_unlock(ptl);
-	if (ret == 1) {
-		/* GUP succeeded */
-		WARN_ON_ONCE(nr != 1);
-		ctx->page_mask = (1U << huge_page_order(h)) - 1;
-		return page;
-	}
-	/* ret can be either 0 (translates to NULL) or negative */
-	return ERR_PTR(ret);
-}
-#else /* CONFIG_ARCH_HAS_HUGEPD */
-static inline int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd,
-			     unsigned long addr, unsigned int pdshift,
-			     unsigned long end, unsigned int flags,
-			     struct page **pages, int *nr, bool fast)
-{
-	return 0;
-}
-static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd,
-				  unsigned long addr, unsigned int pdshift,
-				  unsigned int flags,
-				  struct follow_page_context *ctx)
-{
-	return NULL;
-}
-#endif /* CONFIG_ARCH_HAS_HUGEPD */
 static struct page *no_page_table(struct vm_area_struct *vma,
 				  unsigned int flags, unsigned long address)
 		return false;
 	/* ... and a write-fault isn't required for other reasons. */
-	if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd))
+	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
 	return !userfaultfd_huge_pmd_wp(vma, pmd);
 }
 		return false;
 	/* ... and a write-fault isn't required for other reasons. */
-	if (vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte))
+	if (pte_needs_soft_dirty_wp(vma, pte))
 		return false;
 	return !userfaultfd_pte_wp(vma, pte);
 }
 		return no_page_table(vma, flags, address);
 	if (!pmd_present(pmdval))
 		return no_page_table(vma, flags, address);
-	if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval)))))
-		return follow_hugepd(vma, __hugepd(pmd_val(pmdval)),
-				     address, PMD_SHIFT, flags, ctx);
 	if (pmd_devmap(pmdval)) {
 		ptl = pmd_lock(mm, pmd);
 		page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap);
 	pud = READ_ONCE(*pudp);
 	if (!pud_present(pud))
 		return no_page_table(vma, flags, address);
-	if (unlikely(is_hugepd(__hugepd(pud_val(pud)))))
-		return follow_hugepd(vma, __hugepd(pud_val(pud)),
-				     address, PUD_SHIFT, flags, ctx);
 	if (pud_leaf(pud)) {
 		ptl = pud_lock(mm, pudp);
 		page = follow_huge_pud(vma, address, pudp, flags, ctx);
 	p4d = READ_ONCE(*p4dp);
 	BUILD_BUG_ON(p4d_leaf(p4d));
-	if (unlikely(is_hugepd(__hugepd(p4d_val(p4d)))))
-		return follow_hugepd(vma, __hugepd(p4d_val(p4d)),
-				     address, P4D_SHIFT, flags, ctx);
 	if (!p4d_present(p4d) || p4d_bad(p4d))
 		return no_page_table(vma, flags, address);
 	ctx->page_mask = 0;
 	pgd = pgd_offset(mm, address);
-	if (unlikely(is_hugepd(__hugepd(pgd_val(*pgd)))))
+	if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
-		page = follow_hugepd(vma, __hugepd(pgd_val(*pgd)),
-				     address, PGDIR_SHIFT, flags, ctx);
-	else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
 		page = no_page_table(vma, flags, address);
 	else
 		page = follow_p4d_mask(vma, address, pgd, flags, ctx);
 #ifdef CONFIG_MIGRATION
 /*
- * Returns the number of collected pages. Return value is always >= 0.
+ * Returns the number of collected folios. Return value is always >= 0.
  */
-static unsigned long collect_longterm_unpinnable_pages(
+static unsigned long collect_longterm_unpinnable_folios(
-					struct list_head *movable_page_list,
+					struct list_head *movable_folio_list,
-					unsigned long nr_pages,
+					unsigned long nr_folios,
-					struct page **pages)
+					struct folio **folios)
 {
 	unsigned long i, collected = 0;
 	struct folio *prev_folio = NULL;
 	bool drain_allow = true;
-	for (i = 0; i < nr_pages; i++) {
+	for (i = 0; i < nr_folios; i++) {
-		struct folio *folio = page_folio(pages[i]);
+		struct folio *folio = folios[i];
 		if (folio == prev_folio)
 			continue;
 			continue;
 		if (folio_test_hugetlb(folio)) {
-			isolate_hugetlb(folio, movable_page_list);
+			isolate_hugetlb(folio, movable_folio_list);
 			continue;
 		}
 		if (!folio_isolate_lru(folio))
 			continue;
-		list_add_tail(&folio->lru, movable_page_list);
+		list_add_tail(&folio->lru, movable_folio_list);
 		node_stat_mod_folio(folio,
 				    NR_ISOLATED_ANON + folio_is_file_lru(folio),
 				    folio_nr_pages(folio));
 }
 /*
- * Unpins all pages and migrates device coherent pages and movable_page_list.
+ * Unpins all folios and migrates device coherent folios and movable_folio_list.
- * Returns -EAGAIN if all pages were successfully migrated or -errno for failure
+ * Returns -EAGAIN if all folios were successfully migrated or -errno for
- * (or partial success).
+ * failure (or partial success).
  */
-static int migrate_longterm_unpinnable_pages(
+static int migrate_longterm_unpinnable_folios(
-					struct list_head *movable_page_list,
+					struct list_head *movable_folio_list,
-					unsigned long nr_pages,
+					unsigned long nr_folios,
-					struct page **pages)
+					struct folio **folios)
 {
 	int ret;
 	unsigned long i;
-	for (i = 0; i < nr_pages; i++) {
+	for (i = 0; i < nr_folios; i++) {
-		struct folio *folio = page_folio(pages[i]);
+		struct folio *folio = folios[i];
 		if (folio_is_device_coherent(folio)) {
 			/*
-			 * Migration will fail if the page is pinned, so convert
+			 * Migration will fail if the folio is pinned, so
-			 * the pin on the source page to a normal reference.
+			 * convert the pin on the source folio to a normal
+			 * reference.
 			 */
-			pages[i] = NULL;
+			folios[i] = NULL;
 			folio_get(folio);
 			gup_put_folio(folio, 1, FOLL_PIN);
 		}
 		/*
-		 * We can't migrate pages with unexpected references, so drop
+		 * We can't migrate folios with unexpected references, so drop
 		 * the reference obtained by __get_user_pages_locked().
-		 * Migrating pages have been added to movable_page_list after
+		 * Migrating folios have been added to movable_folio_list after
 		 * calling folio_isolate_lru() which takes a reference so the
-		 * page won't be freed if it's migrating.
+		 * folio won't be freed if it's migrating.
 		 */
-		unpin_user_page(pages[i]);
+		unpin_folio(folios[i]);
-		pages[i] = NULL;
+		folios[i] = NULL;
 	}
-	if (!list_empty(movable_page_list)) {
+	if (!list_empty(movable_folio_list)) {
 		struct migration_target_control mtc = {
 			.nid = NUMA_NO_NODE,
 			.gfp_mask = GFP_USER | __GFP_NOWARN,
 			.reason = MR_LONGTERM_PIN,
 		};
-		if (migrate_pages(movable_page_list, alloc_migration_target,
+		if (migrate_pages(movable_folio_list, alloc_migration_target,
 				  NULL, (unsigned long)&mtc, MIGRATE_SYNC,
 				  MR_LONGTERM_PIN, NULL)) {
 			ret = -ENOMEM;
 		}
 	}
-	putback_movable_pages(movable_page_list);
+	putback_movable_pages(movable_folio_list);
 	return -EAGAIN;
 err:
-	for (i = 0; i < nr_pages; i++)
+	unpin_folios(folios, nr_folios);
-		if (pages[i])
+	putback_movable_pages(movable_folio_list);
-			unpin_user_page(pages[i]);
-	putback_movable_pages(movable_page_list);
 	return ret;
 }
 /*
- * Check whether all pages are *allowed* to be pinned. Rather confusingly, all
+ * Check whether all folios are *allowed* to be pinned indefinitely (longterm).
- * pages in the range are required to be pinned via FOLL_PIN, before calling
+ * Rather confusingly, all folios in the range are required to be pinned via
- * this routine.
+ * FOLL_PIN, before calling this routine.
  *
- * If any pages in the range are not allowed to be pinned, then this routine
+ * If any folios in the range are not allowed to be pinned, then this routine
- * will migrate those pages away, unpin all the pages in the range and return
+ * will migrate those folios away, unpin all the folios in the range and return
  * -EAGAIN. The caller should re-pin the entire range with FOLL_PIN and then
  * call this routine again.
  *
  * If an error other than -EAGAIN occurs, this indicates a migration failure.
  * The caller should give up, and propagate the error back up the call stack.
  *
- * If everything is OK and all pages in the range are allowed to be pinned, then
+ * If everything is OK and all folios in the range are allowed to be pinned,
- * this routine leaves all pages pinned and returns zero for success.
+ * then this routine leaves all folios pinned and returns zero for success.
  */
-static long check_and_migrate_movable_pages(unsigned long nr_pages,
+static long check_and_migrate_movable_folios(unsigned long nr_folios,
-					    struct page **pages)
+					     struct folio **folios)
 {
 	unsigned long collected;
-	LIST_HEAD(movable_page_list);
+	LIST_HEAD(movable_folio_list);
-	collected = collect_longterm_unpinnable_pages(&movable_page_list,
+	collected = collect_longterm_unpinnable_folios(&movable_folio_list,
-						nr_pages, pages);
+						       nr_folios, folios);
 	if (!collected)
 		return 0;
-	return migrate_longterm_unpinnable_pages(&movable_page_list, nr_pages,
+	return migrate_longterm_unpinnable_folios(&movable_folio_list,
-						pages);
+						  nr_folios, folios);
+}
+/*
+ * This routine just converts all the pages in the @pages array to folios and
+ * calls check_and_migrate_movable_folios() to do the heavy lifting.
+ *
+ * Please see the check_and_migrate_movable_folios() documentation for details.
+ */
+static long check_and_migrate_movable_pages(unsigned long nr_pages,
+					    struct page **pages)
+{
+	struct folio **folios;
+	long i, ret;
+	folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL);
+	if (!folios)
+		return -ENOMEM;
+	for (i = 0; i < nr_pages; i++)
+		folios[i] = page_folio(pages[i]);
+	ret = check_and_migrate_movable_folios(nr_pages, folios);
+	kfree(folios);
+	return ret;
 }
 #else
 static long check_and_migrate_movable_pages(unsigned long nr_pages,
 {
 	return 0;
 }
+static long check_and_migrate_movable_folios(unsigned long nr_folios,
+					     struct folio **folios)
+{
+	return 0;
+}
 #endif /* CONFIG_MIGRATION */
 /*
 				pages, nr))
 				return 0;
-		} else if (unlikely(is_hugepd(__hugepd(pmd_val(pmd))))) {
-			/*
-			 * architecture have different format for hugetlbfs
-			 * pmd format and THP pmd format
-			 */
-			if (gup_hugepd(NULL, __hugepd(pmd_val(pmd)), addr,
-				       PMD_SHIFT, next, flags, pages, nr,
-				       true) != 1)
-				return 0;
 		} else if (!gup_fast_pte_range(pmd, pmdp, addr, next, flags,
 					       pages, nr))
 			return 0;
 			if (!gup_fast_pud_leaf(pud, pudp, addr, next, flags,
 					       pages, nr))
 				return 0;
-		} else if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) {
-			if (gup_hugepd(NULL, __hugepd(pud_val(pud)), addr,
-				       PUD_SHIFT, next, flags, pages, nr,
-				       true) != 1)
-				return 0;
 		} else if (!gup_fast_pmd_range(pudp, pud, addr, next, flags,
 					       pages, nr))
 			return 0;
 		if (!p4d_present(p4d))
 			return 0;
 		BUILD_BUG_ON(p4d_leaf(p4d));
-		if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) {
+		if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags,
-			if (gup_hugepd(NULL, __hugepd(p4d_val(p4d)), addr,
+					pages, nr))
-				       P4D_SHIFT, next, flags, pages, nr,
-				       true) != 1)
-				return 0;
-		} else if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags,
-					       pages, nr))
 			return 0;
 	} while (p4dp++, addr = next, addr != end);
 			if (!gup_fast_pgd_leaf(pgd, pgdp, addr, next, flags,
 					       pages, nr))
 				return;
-		} else if (unlikely(is_hugepd(__hugepd(pgd_val(pgd))))) {
-			if (gup_hugepd(NULL, __hugepd(pgd_val(pgd)), addr,
-				       PGDIR_SHIFT, next, flags, pages, nr,
-				       true) != 1)
-				return;
 		} else if (!gup_fast_p4d_range(pgdp, pgd, addr, next, flags,
 					       pages, nr))
 			return;
 				     &locked, gup_flags);
 }
 EXPORT_SYMBOL(pin_user_pages_unlocked);
+/**
+ * memfd_pin_folios() - pin folios associated with a memfd
+ * @memfd:      the memfd whose folios are to be pinned
+ * @start:      the first memfd offset
+ * @end:        the last memfd offset (inclusive)
+ * @folios:     array that receives pointers to the folios pinned
+ * @max_folios: maximum number of entries in @folios
+ * @offset:     the offset into the first folio
+ *
+ * Attempt to pin folios associated with a memfd in the contiguous range
+ * [start, end]. Given that a memfd is either backed by shmem or hugetlb,
+ * the folios can either be found in the page cache or need to be allocated
+ * if necessary. Once the folios are located, they are all pinned via
+ * FOLL_PIN and @offset is populatedwith the offset into the first folio.
+ * And, eventually, these pinned folios must be released either using
+ * unpin_folios() or unpin_folio().
+ *
+ * It must be noted that the folios may be pinned for an indefinite amount
+ * of time. And, in most cases, the duration of time they may stay pinned
+ * would be controlled by the userspace. This behavior is effectively the
+ * same as using FOLL_LONGTERM with other GUP APIs.
+ *
+ * Returns number of folios pinned, which could be less than @max_folios
+ * as it depends on the folio sizes that cover the range [start, end].
+ * If no folios were pinned, it returns -errno.
+ */
+long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end,
+		      struct folio **folios, unsigned int max_folios,
+		      pgoff_t *offset)
+{
+	unsigned int flags, nr_folios, nr_found;
+	unsigned int i, pgshift = PAGE_SHIFT;
+	pgoff_t start_idx, end_idx, next_idx;
+	struct folio *folio = NULL;
+	struct folio_batch fbatch;
+	struct hstate *h;
+	long ret = -EINVAL;
+	if (start < 0 || start > end || !max_folios)
+		return -EINVAL;
+	if (!memfd)
+		return -EINVAL;
+	if (!shmem_file(memfd) && !is_file_hugepages(memfd))
+		return -EINVAL;
+	if (end >= i_size_read(file_inode(memfd)))
+		return -EINVAL;
+	if (is_file_hugepages(memfd)) {
+		h = hstate_file(memfd);
+		pgshift = huge_page_shift(h);
+	}
+	flags = memalloc_pin_save();
+	do {
+		nr_folios = 0;
+		start_idx = start >> pgshift;
+		end_idx = end >> pgshift;
+		if (is_file_hugepages(memfd)) {
+			start_idx <<= huge_page_order(h);
+			end_idx <<= huge_page_order(h);
+		}
+		folio_batch_init(&fbatch);
+		while (start_idx <= end_idx && nr_folios < max_folios) {
+			/*
+			 * In most cases, we should be able to find the folios
+			 * in the page cache. If we cannot find them for some
+			 * reason, we try to allocate them and add them to the
+			 * page cache.
+			 */
+			nr_found = filemap_get_folios_contig(memfd->f_mapping,
+							     &start_idx,
+							     end_idx,
+							     &fbatch);
+			if (folio) {
+				folio_put(folio);
+				folio = NULL;
+			}
+			next_idx = 0;
+			for (i = 0; i < nr_found; i++) {
+				/*
+				 * As there can be multiple entries for a
+				 * given folio in the batch returned by
+				 * filemap_get_folios_contig(), the below
+				 * check is to ensure that we pin and return a
+				 * unique set of folios between start and end.
+				 */
+				if (next_idx &&
+				    next_idx != folio_index(fbatch.folios[i]))
+					continue;
+				folio = page_folio(&fbatch.folios[i]->page);
+				if (try_grab_folio(folio, 1, FOLL_PIN)) {
+					folio_batch_release(&fbatch);
+					ret = -EINVAL;
+					goto err;
+				}
+				if (nr_folios == 0)
+					*offset = offset_in_folio(folio, start);
+				folios[nr_folios] = folio;
+				next_idx = folio_next_index(folio);
+				if (++nr_folios == max_folios)
+					break;
+			}
+			folio = NULL;
+			folio_batch_release(&fbatch);
+			if (!nr_found) {
+				folio = memfd_alloc_folio(memfd, start_idx);
+				if (IS_ERR(folio)) {
+					ret = PTR_ERR(folio);
+					if (ret != -EEXIST)
+						goto err;
+				}
+			}
+		}
+		ret = check_and_migrate_movable_folios(nr_folios, folios);
+	} while (ret == -EAGAIN);
+	memalloc_pin_restore(flags);
+	return ret ? ret : nr_folios;
+err:
+	memalloc_pin_restore(flags);
+	unpin_folios(folios, nr_folios);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(memfd_pin_folios);

mm/highmem.c+16−5

 }
 #endif
-atomic_long_t _totalhigh_pages __read_mostly;
+unsigned long __nr_free_highpages(void)
-EXPORT_SYMBOL(_totalhigh_pages);
-unsigned int __nr_free_highpages(void)
 {
+	unsigned long pages = 0;
 	struct zone *zone;
-	unsigned int pages = 0;
 	for_each_populated_zone(zone) {
 		if (is_highmem(zone))
 	return pages;
 }
+unsigned long __totalhigh_pages(void)
+{
+	unsigned long pages = 0;
+	struct zone *zone;
+	for_each_populated_zone(zone) {
+		if (is_highmem(zone))
+			pages += zone_managed_pages(zone);
+	}
+	return pages;
+}
+EXPORT_SYMBOL(__totalhigh_pages);
 static int pkmap_count[LAST_PKMAP];
 static  __cacheline_aligned_in_smp DEFINE_SPINLOCK(kmap_lock);

mm/hmm.c+1−1

@@ -480,7 +480,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, · 479 unmodified lines
480	pte_t entry;	480	pte_t entry;
481		481
482	ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte);	482	ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte);
483	entry = huge_ptep_get(pte);	483	entry = huge_ptep_get(walk->mm, addr, pte);
484		484
485	i = (start - range->start) >> PAGE_SHIFT;	485	i = (start - range->start) >> PAGE_SHIFT;
486	pfn_req_flags = range->hmm_pfns[i];	486	pfn_req_flags = range->hmm_pfns[i];

mm/huge_memory.c+136−57

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	#include <linux/swapops.h>	20	#include <linux/swapops.h>
21	#include <linux/backing-dev.h>	21	#include <linux/backing-dev.h>
22	#include <linux/dax.h>	22	#include <linux/dax.h>
		23	#include <linux/mm_types.h>
23	#include <linux/khugepaged.h>	24	#include <linux/khugepaged.h>
24	#include <linux/freezer.h>	25	#include <linux/freezer.h>
25	#include <linux/pfn_t.h>	26	#include <linux/pfn_t.h>
@@ -150,10 +151,15 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, · 124 unmodified lines
150	* Must be done before hugepage flags check since shmem has its	151	* Must be done before hugepage flags check since shmem has its
151	* own flags.	152	* own flags.
152	*/	153	*/
153	if (!in_pf && shmem_file(vma->vm_file))	154	if (!in_pf && shmem_file(vma->vm_file)) {
154	return shmem_is_huge(file_inode(vma->vm_file), vma->vm_pgoff,	155	bool global_huge = shmem_is_huge(file_inode(vma->vm_file), vma->vm_pgoff,
155	!enforce_sysfs, vma->vm_mm, vm_flags)	156	!enforce_sysfs, vma->vm_mm, vm_flags);
156	? orders : 0;	157
		158	if (!vma_is_anon_shmem(vma))
		159	return global_huge ? orders : 0;
		160	return shmem_allowable_huge_orders(file_inode(vma->vm_file),
		161	vma, vma->vm_pgoff, global_huge);
		162	}
157		163
158	if (!vma_is_anonymous(vma)) {	164	if (!vma_is_anonymous(vma)) {
159	/*	165	/*
@@ -449,14 +455,6 @@ static void thpsize_release(struct kobject *kobj); · 289 unmodified lines
449	static DEFINE_SPINLOCK(huge_anon_orders_lock);	455	static DEFINE_SPINLOCK(huge_anon_orders_lock);
450	static LIST_HEAD(thpsize_list);	456	static LIST_HEAD(thpsize_list);
451		457
452	struct thpsize {
453	struct kobject kobj;
454	struct list_head node;
455	int order;
456	};
457
458	#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj)
459
460	static ssize_t thpsize_enabled_show(struct kobject *kobj,	458	static ssize_t thpsize_enabled_show(struct kobject *kobj,
461	struct kobj_attribute attr, char buf)	459	struct kobj_attribute attr, char buf)
462	{	460	{
@@ -509,6 +507,13 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj, · 46 unmodified lines
509	} else	507	} else
510	ret = -EINVAL;	508	ret = -EINVAL;
511		509
		510	if (ret > 0) {
		511	int err;
		512
		513	err = start_stop_khugepaged();
		514	if (err)
		515	ret = err;
		516	}
512	return ret;	517	return ret;
513	}	518	}
514		519
@@ -517,6 +522,9 @@ static struct kobj_attribute thpsize_enabled_attr = · 2 unmodified lines
517		522
518	static struct attribute *thpsize_attrs[] = {	523	static struct attribute *thpsize_attrs[] = {
519	&thpsize_enabled_attr.attr,	524	&thpsize_enabled_attr.attr,
		525	#ifdef CONFIG_SHMEM
		526	&thpsize_shmem_enabled_attr.attr,
		527	#endif
520	NULL,	528	NULL,
521	};	529	};
522		530
@@ -560,6 +568,12 @@ DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); · 37 unmodified lines
560	DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);	568	DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
561	DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT);	569	DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT);
562	DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK);	570	DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK);
		571	DEFINE_MTHP_STAT_ATTR(shmem_alloc, MTHP_STAT_SHMEM_ALLOC);
		572	DEFINE_MTHP_STAT_ATTR(shmem_fallback, MTHP_STAT_SHMEM_FALLBACK);
		573	DEFINE_MTHP_STAT_ATTR(shmem_fallback_charge, MTHP_STAT_SHMEM_FALLBACK_CHARGE);
		574	DEFINE_MTHP_STAT_ATTR(split, MTHP_STAT_SPLIT);
		575	DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED);
		576	DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED);
563		577
564	static struct attribute *stats_attrs[] = {	578	static struct attribute *stats_attrs[] = {
565	&anon_fault_alloc_attr.attr,	579	&anon_fault_alloc_attr.attr,
@@ -567,6 +581,12 @@ static struct attribute *stats_attrs[] = { · 1 unmodified line
567	&anon_fault_fallback_charge_attr.attr,	581	&anon_fault_fallback_charge_attr.attr,
568	&swpout_attr.attr,	582	&swpout_attr.attr,
569	&swpout_fallback_attr.attr,	583	&swpout_fallback_attr.attr,
		584	&shmem_alloc_attr.attr,
		585	&shmem_fallback_attr.attr,
		586	&shmem_fallback_charge_attr.attr,
		587	&split_attr.attr,
		588	&split_failed_attr.attr,
		589	&split_deferred_attr.attr,
570	NULL,	590	NULL,
571	};	591	};
572		592
@@ -942,10 +962,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · 369 unmodified lines
942	goto release;	962	goto release;
943	}	963	}
944		964
945	clear_huge_page(page, vmf->address, HPAGE_PMD_NR);	965	folio_zero_user(folio, vmf->address);
946	/*	966	/*
947	* The memory barrier inside __folio_mark_uptodate makes sure that	967	* The memory barrier inside __folio_mark_uptodate makes sure that
948	* clear_huge_page writes become visible before the set_pmd_at()	968	* folio_zero_user writes become visible before the set_pmd_at()
949	* write.	969	* write.
950	*/	970	*/
951	__folio_mark_uptodate(folio);	971	__folio_mark_uptodate(folio);
@@ -972,7 +992,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · 20 unmodified lines
972		992
973	entry = mk_huge_pmd(page, vma->vm_page_prot);	993	entry = mk_huge_pmd(page, vma->vm_page_prot);
974	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);	994	entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
975	folio_add_new_anon_rmap(folio, vma, haddr);	995	folio_add_new_anon_rmap(folio, vma, haddr, RMAP_EXCLUSIVE);
976	folio_add_lru_vma(folio, vma);	996	folio_add_lru_vma(folio, vma);
977	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);	997	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
978	set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);	998	set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
@@ -1624,7 +1644,7 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, · 645 unmodified lines
1624	return false;	1644	return false;
1625		1645
1626	/* Do we need write faults for softdirty tracking? */	1646	/* Do we need write faults for softdirty tracking? */
1627	if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd))	1647	if (pmd_needs_soft_dirty_wp(vma, pmd))
1628	return false;	1648	return false;
1629		1649
1630	/* Do we need write faults for uffd-wp tracking? */	1650	/* Do we need write faults for uffd-wp tracking? */
@@ -1651,7 +1671,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 20 unmodified lines
1651	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;	1671	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
1652	int nid = NUMA_NO_NODE;	1672	int nid = NUMA_NO_NODE;
1653	int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);	1673	int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
1654	bool migrated = false, writable = false;	1674	bool writable = false;
1655	int flags = 0;	1675	int flags = 0;
1656		1676
1657	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);	1677	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -1687,16 +1707,17 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 29 unmodified lines
1687	if (node_is_toptier(nid))	1707	if (node_is_toptier(nid))
1688	last_cpupid = folio_last_cpupid(folio);	1708	last_cpupid = folio_last_cpupid(folio);
1689	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);	1709	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
1690	if (target_nid == NUMA_NO_NODE) {	1710	if (target_nid == NUMA_NO_NODE)
1691	folio_put(folio);	1711	goto out_map;
		1712	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
		1713	flags \|= TNF_MIGRATE_FAIL;
1692	goto out_map;	1714	goto out_map;
1693	}	1715	}
1694		1716	/* The folio is isolated and isolation code holds a folio reference. */
1695	spin_unlock(vmf->ptl);	1717	spin_unlock(vmf->ptl);
1696	writable = false;	1718	writable = false;
1697		1719
1698	migrated = migrate_misplaced_folio(folio, vma, target_nid);	1720	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
1699	if (migrated) {
1700	flags \|= TNF_MIGRATED;	1721	flags \|= TNF_MIGRATED;
1701	nid = target_nid;	1722	nid = target_nid;
1702	} else {	1723	} else {
@@ -2581,6 +2602,27 @@ static void __split_huge_pmd_locked(struct vm_area_struct vma, pmd_t pmd, · 878 unmodified lines
2581	pmd_populate(mm, pmd, pgtable);	2602	pmd_populate(mm, pmd, pgtable);
2582	}	2603	}
2583		2604
		2605	void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
		2606	pmd_t pmd, bool freeze, struct folio folio)
		2607	{
		2608	VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio));
		2609	VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE));
		2610	VM_WARN_ON_ONCE(folio && !folio_test_locked(folio));
		2611	VM_BUG_ON(freeze && !folio);
		2612
		2613	/*
		2614	* When the caller requests to set up a migration entry, we
		2615	* require a folio to check the PMD against. Otherwise, there
		2616	* is a risk of replacing the wrong folio.
		2617	*/
		2618	if (pmd_trans_huge(pmd) \|\| pmd_devmap(pmd) \|\|
		2619	is_pmd_migration_entry(*pmd)) {
		2620	if (folio && folio != pmd_folio(*pmd))
		2621	return;
		2622	__split_huge_pmd_locked(vma, pmd, address, freeze);
		2623	}
		2624	}
		2625
2584	void __split_huge_pmd(struct vm_area_struct vma, pmd_t pmd,	2626	void __split_huge_pmd(struct vm_area_struct vma, pmd_t pmd,
2585	unsigned long address, bool freeze, struct folio *folio)	2627	unsigned long address, bool freeze, struct folio *folio)
2586	{	2628	{
@@ -2592,26 +2634,7 @@ void __split_huge_pmd(struct vm_area_struct vma, pmd_t pmd, · 5 unmodified lines
2592	(address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE);	2634	(address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE);
2593	mmu_notifier_invalidate_range_start(&range);	2635	mmu_notifier_invalidate_range_start(&range);
2594	ptl = pmd_lock(vma->vm_mm, pmd);	2636	ptl = pmd_lock(vma->vm_mm, pmd);
2595		2637	split_huge_pmd_locked(vma, range.start, pmd, freeze, folio);
2596	/*
2597	* If caller asks to setup a migration entry, we need a folio to check
2598	* pmd against. Otherwise we can end up replacing wrong folio.
2599	*/
2600	VM_BUG_ON(freeze && !folio);
2601	VM_WARN_ON_ONCE(folio && !folio_test_locked(folio));
2602
2603	if (pmd_trans_huge(pmd) \|\| pmd_devmap(pmd) \|\|
2604	is_pmd_migration_entry(*pmd)) {
2605	/*
2606	* It's safe to call pmd_page when folio is set because it's
2607	* guaranteed that pmd is present.
2608	*/
2609	if (folio && folio != pmd_folio(*pmd))
2610	goto out;
2611	__split_huge_pmd_locked(vma, pmd, range.start, freeze);
2612	}
2613
2614	out:
2615	spin_unlock(ptl);	2638	spin_unlock(ptl);
2616	mmu_notifier_invalidate_range_end(&range);	2639	mmu_notifier_invalidate_range_end(&range);
2617	}	2640	}
@@ -2685,6 +2708,71 @@ static void unmap_folio(struct folio *folio) · 67 unmodified lines
2685	try_to_unmap_flush();	2708	try_to_unmap_flush();
2686	}	2709	}
2687		2710
		2711	static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma,
		2712	unsigned long addr, pmd_t *pmdp,
		2713	struct folio *folio)
		2714	{
		2715	struct mm_struct *mm = vma->vm_mm;
		2716	int ref_count, map_count;
		2717	pmd_t orig_pmd = *pmdp;
		2718
		2719	if (folio_test_dirty(folio) \|\| pmd_dirty(orig_pmd))
		2720	return false;
		2721
		2722	orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp);
		2723
		2724	/*
		2725	* Syncing against concurrent GUP-fast:
		2726	* - clear PMD; barrier; read refcount
		2727	* - inc refcount; barrier; read PMD
		2728	*/
		2729	smp_mb();
		2730
		2731	ref_count = folio_ref_count(folio);
		2732	map_count = folio_mapcount(folio);
		2733
		2734	/*
		2735	* Order reads for folio refcount and dirty flag
		2736	* (see comments in __remove_mapping()).
		2737	*/
		2738	smp_rmb();
		2739
		2740	/*
		2741	* If the folio or its PMD is redirtied at this point, or if there
		2742	* are unexpected references, we will give up to discard this folio
		2743	* and remap it.
		2744	*
		2745	* The only folio refs must be one from isolation plus the rmap(s).
		2746	*/
		2747	if (folio_test_dirty(folio) \|\| pmd_dirty(orig_pmd) \|\|
		2748	ref_count != map_count + 1) {
		2749	set_pmd_at(mm, addr, pmdp, orig_pmd);
		2750	return false;
		2751	}
		2752
		2753	folio_remove_rmap_pmd(folio, pmd_page(orig_pmd), vma);
		2754	zap_deposited_table(mm, pmdp);
		2755	add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR);
		2756	if (vma->vm_flags & VM_LOCKED)
		2757	mlock_drain_local();
		2758	folio_put(folio);
		2759
		2760	return true;
		2761	}
		2762
		2763	bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
		2764	pmd_t pmdp, struct folio folio)
		2765	{
		2766	VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio);
		2767	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
		2768	VM_WARN_ON_ONCE(!IS_ALIGNED(addr, HPAGE_PMD_SIZE));
		2769
		2770	if (folio_test_anon(folio) && !folio_test_swapbacked(folio))
		2771	return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio);
		2772
		2773	return false;
		2774	}
		2775
2688	static void remap_page(struct folio *folio, unsigned long nr)	2776	static void remap_page(struct folio *folio, unsigned long nr)
2689	{	2777	{
2690	int i = 0;	2778	int i = 0;
@@ -2838,7 +2926,7 @@ static void __split_huge_page(struct page page, struct list_head list, · 147 unmodified lines
2838	split_page_memcg(head, order, new_order);	2926	split_page_memcg(head, order, new_order);
2839		2927
2840	if (folio_test_anon(folio) && folio_test_swapcache(folio)) {	2928	if (folio_test_anon(folio) && folio_test_swapcache(folio)) {
2841	offset = swp_offset(folio->swap);	2929	offset = swap_cache_index(folio->swap);
2842	swap_cache = swap_address_space(folio->swap);	2930	swap_cache = swap_address_space(folio->swap);
2843	xa_lock(&swap_cache->i_pages);	2931	xa_lock(&swap_cache->i_pages);
2844	}	2932	}
@@ -2998,7 +3086,7 @@ int split_huge_page_to_list_to_order(struct page page, struct list_head list, · 153 unmodified lines
2998	XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order);	3086	XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order);
2999	struct anon_vma *anon_vma = NULL;	3087	struct anon_vma *anon_vma = NULL;
3000	struct address_space *mapping = NULL;	3088	struct address_space *mapping = NULL;
3001	bool is_thp = folio_test_pmd_mappable(folio);	3089	int order = folio_order(folio);
3002	int extra_pins, ret;	3090	int extra_pins, ret;
3003	pgoff_t end;	3091	pgoff_t end;
3004	bool is_hzp;	3092	bool is_hzp;
@@ -3183,27 +3271,17 @@ out_unlock: · 178 unmodified lines
3183	i_mmap_unlock_read(mapping);	3271	i_mmap_unlock_read(mapping);
3184	out:	3272	out:
3185	xas_destroy(&xas);	3273	xas_destroy(&xas);
3186	if (is_thp)	3274	if (order == HPAGE_PMD_ORDER)
3187	count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);	3275	count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
		3276	count_mthp_stat(order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
3188	return ret;	3277	return ret;
3189	}	3278	}
3190		3279
3191	void folio_undo_large_rmappable(struct folio *folio)	3280	void __folio_undo_large_rmappable(struct folio *folio)
3192	{	3281	{
3193	struct deferred_split *ds_queue;	3282	struct deferred_split *ds_queue;
3194	unsigned long flags;	3283	unsigned long flags;
3195		3284
3196	if (folio_order(folio) <= 1)
3197	return;
3198
3199	/*
3200	* At this point, there is no one trying to add the folio to
3201	* deferred_list. If folio is not in deferred_list, it's safe
3202	* to check without acquiring the split_queue_lock.
3203	*/
3204	if (data_race(list_empty(&folio->_deferred_list)))
3205	return;
3206
3207	ds_queue = get_deferred_split_queue(folio);	3285	ds_queue = get_deferred_split_queue(folio);
3208	spin_lock_irqsave(&ds_queue->split_queue_lock, flags);	3286	spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
3209	if (!list_empty(&folio->_deferred_list)) {	3287	if (!list_empty(&folio->_deferred_list)) {
@@ -3248,6 +3326,7 @@ void deferred_split_folio(struct folio *folio) · 38 unmodified lines
3248	if (list_empty(&folio->_deferred_list)) {	3326	if (list_empty(&folio->_deferred_list)) {
3249	if (folio_test_pmd_mappable(folio))	3327	if (folio_test_pmd_mappable(folio))
3250	count_vm_event(THP_DEFERRED_SPLIT_PAGE);	3328	count_vm_event(THP_DEFERRED_SPLIT_PAGE);
		3329	count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
3251	list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);	3330	list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
3252	ds_queue->split_queue_len++;	3331	ds_queue->split_queue_len++;
3253	#ifdef CONFIG_MEMCG	3332	#ifdef CONFIG_MEMCG

mm/hugetlb.c+71−63

 	struct zoneref *z;
 	int node = NUMA_NO_NODE;
+	/* 'nid' should not be NUMA_NO_NODE. Try to catch any misuse of it and rectifiy. */
+	if (nid == NUMA_NO_NODE)
+		nid = numa_node_id();
 	zonelist = node_zonelist(nid, gfp_mask);
 retry_cpuset:
  * pages is zero.
  */
 static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
-		gfp_t gfp_mask, int nid, nodemask_t *nmask,
+		gfp_t gfp_mask, int nid, nodemask_t *nmask)
-		nodemask_t *node_alloc_noretry)
 {
 	struct folio *folio;
-	folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask,
+	folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL);
-						node_alloc_noretry);
 	if (!folio)
 		return NULL;
 		goto out_unlock;
 	spin_unlock_irq(&hugetlb_lock);
-	folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL);
+	folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
 	if (!folio)
 		return NULL;
 	if (hstate_is_gigantic(h))
 		return NULL;
-	folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL);
+	folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
 	if (!folio)
 		return NULL;
 	return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask);
 }
+static nodemask_t *policy_mbind_nodemask(gfp_t gfp)
+{
+#ifdef CONFIG_NUMA
+	struct mempolicy *mpol = get_task_policy(current);
+	/*
+	 * Only enforce MPOL_BIND policy which overlaps with cpuset policy
+	 * (from policy_nodemask) specifically for hugetlb case
+	 */
+	if (mpol->mode == MPOL_BIND &&
+		(apply_policy_zone(mpol, gfp_zone(gfp)) &&
+		 cpuset_nodemask_valid_mems_allowed(&mpol->nodes)))
+		return &mpol->nodes;
+#endif
+	return NULL;
+}
 /*
  * Increase the hugetlb pool such that it can accommodate a reservation
  * of size 'delta'.
 	long i;
 	long needed, allocated;
 	bool alloc_ok = true;
+	int node;
+	nodemask_t *mbind_nodemask = policy_mbind_nodemask(htlb_alloc_mask(h));
 	lockdep_assert_held(&hugetlb_lock);
 	needed = (h->resv_huge_pages + delta) - h->free_huge_pages;
 retry:
 	spin_unlock_irq(&hugetlb_lock);
 	for (i = 0; i < needed; i++) {
-		folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h),
+		folio = NULL;
-				NUMA_NO_NODE, NULL);
+		for_each_node_mask(node, cpuset_current_mems_allowed) {
+			if (!mbind_nodemask || node_isset(node, *mbind_nodemask)) {
+				folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h),
+						node, NULL);
+				if (folio)
+					break;
+			}
+		}
 		if (!folio) {
 			alloc_ok = false;
 			break;
 			gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
 			folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid,
-					&node_states[N_MEMORY], NULL);
+					&node_states[N_MEMORY]);
 			if (!folio)
 				break;
 			free_huge_folio(folio); /* free it into the hugepage allocator */
 	BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE);
 	BUG_ON(order < order_base_2(__NR_USED_SUBPAGE));
 	h = &hstates[hugetlb_max_hstate++];
-	mutex_init(&h->resize_lock);
+	__mutex_init(&h->resize_lock, "resize mutex", &h->resize_key);
 	h->order = order;
 	h->mask = ~(huge_page_size(h) - 1);
 	for (i = 0; i < MAX_NUMNODES; ++i)
 }
 __setup("default_hugepagesz=", default_hugepagesz_setup);
-static nodemask_t *policy_mbind_nodemask(gfp_t gfp)
-{
-#ifdef CONFIG_NUMA
-	struct mempolicy *mpol = get_task_policy(current);
-	/*
-	 * Only enforce MPOL_BIND policy which overlaps with cpuset policy
-	 * (from policy_nodemask) specifically for hugetlb case
-	 */
-	if (mpol->mode == MPOL_BIND &&
-		(apply_policy_zone(mpol, gfp_zone(gfp)) &&
-		 cpuset_nodemask_valid_mems_allowed(&mpol->nodes)))
-		return &mpol->nodes;
-#endif
-	return NULL;
-}
 static unsigned int allowed_mems_nr(struct hstate *h)
 {
 	int node;
 }
 #ifdef CONFIG_SYSCTL
-static int proc_hugetlb_doulongvec_minmax(struct ctl_table *table, int write,
+static int proc_hugetlb_doulongvec_minmax(const struct ctl_table *table, int write,
 					  void *buffer, size_t *length,
 					  loff_t *ppos, unsigned long *out)
 {
 }
 static int hugetlb_sysctl_handler_common(bool obey_mempolicy,
-			 struct ctl_table *table, int write,
+			 const struct ctl_table *table, int write,
 			 void *buffer, size_t *length, loff_t *ppos)
 {
 	struct hstate *h = &default_hstate;
 {
 	pte_t entry;
-	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)));
+	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(vma->vm_mm, address, ptep)));
 	if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1))
 		update_mmu_cache(vma, address, ptep);
 }
 		dst_ptl = huge_pte_lock(h, dst, dst_pte);
 		src_ptl = huge_pte_lockptr(h, src, src_pte);
 		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
-		entry = huge_ptep_get(src_pte);
+		entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte);
 again:
 		if (huge_pte_none(entry)) {
 			/*
 				set_huge_pte_at(dst, addr, dst_pte,
 						make_pte_marker(marker), sz);
 		} else {
-			entry = huge_ptep_get(src_pte);
+			entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte);
 			pte_folio = page_folio(pte_page(entry));
 			folio_get(pte_folio);
 					ret = PTR_ERR(new_folio);
 					break;
 				}
-				ret = copy_user_large_folio(new_folio,
+				ret = copy_user_large_folio(new_folio, pte_folio,
-							    pte_folio,
+						ALIGN_DOWN(addr, sz), dst_vma);
-							    addr, dst_vma);
 				folio_put(pte_folio);
 				if (ret) {
 					folio_put(new_folio);
 				dst_ptl = huge_pte_lock(h, dst, dst_pte);
 				src_ptl = huge_pte_lockptr(h, src, src_pte);
 				spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
-				entry = huge_ptep_get(src_pte);
+				entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte);
 				if (!pte_same(src_pte_old, entry)) {
 					restore_reserve_on_error(h, dst_vma, addr,
 								new_folio);
 			new_addr |= last_addr_mask;
 			continue;
 		}
-		if (huge_pte_none(huge_ptep_get(src_pte)))
+		if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte)))
 			continue;
 		if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) {
 			continue;
 		}
-		pte = huge_ptep_get(ptep);
+		pte = huge_ptep_get(mm, address, ptep);
 		if (huge_pte_none(pte)) {
 			spin_unlock(ptl);
 			continue;
 	struct vm_area_struct *vma = vmf->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
-	pte_t pte = huge_ptep_get(vmf->pte);
+	pte_t pte = huge_ptep_get(mm, vmf->address, vmf->pte);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *old_folio;
 	struct folio *new_folio;
 			vmf->pte = hugetlb_walk(vma, vmf->address,
 					huge_page_size(h));
 			if (likely(vmf->pte &&
-				   pte_same(huge_ptep_get(vmf->pte), pte)))
+				   pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte)))
 				goto retry_avoidcopy;
 			/*
 			 * race occurs while re-acquiring page table
 	 */
 	spin_lock(vmf->ptl);
 	vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h));
-	if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) {
+	if (likely(vmf->pte && pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte))) {
 		pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare);
 		/* Break COW or unshare */
  * Recheck pte with pgtable lock.  Returns true if pte didn't change, or
  * false if pte changed or is changing.
  */
-static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm,
+static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, unsigned long addr,
 			       pte_t *ptep, pte_t old_pte)
 {
 	spinlock_t *ptl;
 	bool same;
 	ptl = huge_pte_lock(h, mm, ptep);
-	same = pte_same(huge_ptep_get(ptep), old_pte);
+	same = pte_same(huge_ptep_get(mm, addr, ptep), old_pte);
 	spin_unlock(ptl);
 	return same;
 			 * never happen on the page after UFFDIO_COPY has
 			 * correctly installed the page and returned.
 			 */
-			if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) {
+			if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) {
 				ret = 0;
 				goto out;
 			}
 			 * here.  Before returning error, get ptl and make
 			 * sure there really is no pte entry.
 			 */
-			if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte))
+			if (hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte))
 				ret = vmf_error(PTR_ERR(folio));
 			else
 				ret = 0;
 			goto out;
 		}
-		clear_huge_page(&folio->page, vmf->real_address,
+		folio_zero_user(folio, vmf->real_address);
-				pages_per_huge_page(h));
 		__folio_mark_uptodate(folio);
 		new_folio = true;
 			folio_unlock(folio);
 			folio_put(folio);
 			/* See comment in userfaultfd_missing() block above */
-			if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) {
+			if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) {
 				ret = 0;
 				goto out;
 			}
 	vmf->ptl = huge_pte_lock(h, mm, vmf->pte);
 	ret = 0;
 	/* If pte changed from under us, retry */
-	if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte))
+	if (!pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), vmf->orig_pte))
 		goto backout;
 	if (anon_rmap)
 		return VM_FAULT_OOM;
 	}
-	vmf.orig_pte = huge_ptep_get(vmf.pte);
+	vmf.orig_pte = huge_ptep_get(mm, vmf.address, vmf.pte);
 	if (huge_pte_none_mostly(vmf.orig_pte)) {
 		if (is_pte_marker(vmf.orig_pte)) {
 			pte_marker marker =
 			 * be released there.
 			 */
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
-			migration_entry_wait_huge(vma, vmf.pte);
+			migration_entry_wait_huge(vma, vmf.address, vmf.pte);
 			return 0;
 		} else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte)))
 			ret = VM_FAULT_HWPOISON_LARGE |
 	vmf.ptl = huge_pte_lock(h, mm, vmf.pte);
 	/* Check for a racing update before calling hugetlb_wp() */
-	if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte))))
+	if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(mm, vmf.address, vmf.pte))))
 		goto out_ptl;
 	/* Handle userfault-wp first, before trying to lock more pages */
-	if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) &&
+	if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(mm, vmf.address, vmf.pte)) &&
 	    (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) {
 		if (!userfaultfd_wp_async(vma)) {
 			spin_unlock(vmf.ptl);
 	struct hstate *h = hstate_vma(dst_vma);
 	struct address_space *mapping = dst_vma->vm_file->f_mapping;
 	pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr);
-	unsigned long size;
+	unsigned long size = huge_page_size(h);
 	int vm_shared = dst_vma->vm_flags & VM_SHARED;
 	pte_t _dst_pte;
 	spinlock_t *ptl;
 		ptl = huge_pte_lock(h, dst_mm, dst_pte);
 		/* Don't overwrite any existing PTEs (even markers) */
-		if (!huge_pte_none(huge_ptep_get(dst_pte))) {
+		if (!huge_pte_none(huge_ptep_get(dst_mm, dst_addr, dst_pte))) {
 			spin_unlock(ptl);
 			return -EEXIST;
 		}
 		_dst_pte = make_pte_marker(PTE_MARKER_POISONED);
-		set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte,
+		set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size);
-				huge_page_size(h));
 		/* No need to invalidate - it was non-present before */
 		update_mmu_cache(dst_vma, dst_addr, dst_pte);
 			*foliop = NULL;
 			goto out;
 		}
-		ret = copy_user_large_folio(folio, *foliop, dst_addr, dst_vma);
+		ret = copy_user_large_folio(folio, *foliop,
+					    ALIGN_DOWN(dst_addr, size), dst_vma);
 		folio_put(*foliop);
 		*foliop = NULL;
 		if (ret) {
 	/* Add shared, newly allocated pages to the page cache. */
 	if (vm_shared && !is_continue) {
-		size = i_size_read(mapping->host) >> huge_page_shift(h);
 		ret = -EFAULT;
-		if (idx >= size)
+		if (idx >= (i_size_read(mapping->host) >> huge_page_shift(h)))
 			goto out_release_nounlock;
 		/*
 	 * page backing it, then access the page.
 	 */
 	ret = -EEXIST;
-	if (!huge_pte_none_mostly(huge_ptep_get(dst_pte)))
+	if (!huge_pte_none_mostly(huge_ptep_get(dst_mm, dst_addr, dst_pte)))
 		goto out_release_unlock;
 	if (folio_in_pagecache)
 	if (wp_enabled)
 		_dst_pte = huge_pte_mkuffd_wp(_dst_pte);
-	set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, huge_page_size(h));
+	set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size);
 	hugetlb_count_add(pages_per_huge_page(h), dst_mm);
 			address |= last_addr_mask;
 			continue;
 		}
-		pte = huge_ptep_get(ptep);
+		pte = huge_ptep_get(mm, address, ptep);
 		if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) {
 			/* Nothing to do. */
 		} else if (unlikely(is_hugetlb_entry_migration(pte))) {

mm/hugetlb_cgroup.c+167−138

@@ -27,7 +27,17 @@ · 26 unmodified lines
27	#define MEMFILE_IDX(val) (((val) >> 16) & 0xffff)	27	#define MEMFILE_IDX(val) (((val) >> 16) & 0xffff)
28	#define MEMFILE_ATTR(val) ((val) & 0xffff)	28	#define MEMFILE_ATTR(val) ((val) & 0xffff)
29		29
		30	/* Use t->m[0] to encode the offset */
		31	#define MEMFILE_OFFSET(t, m0) (((offsetof(t, m0) << 16) \| sizeof_field(t, m0)))
		32	#define MEMFILE_OFFSET0(val) (((val) >> 16) & 0xffff)
		33	#define MEMFILE_FIELD_SIZE(val) ((val) & 0xffff)
		34
		35	#define DFL_TMPL_SIZE ARRAY_SIZE(hugetlb_dfl_tmpl)
		36	#define LEGACY_TMPL_SIZE ARRAY_SIZE(hugetlb_legacy_tmpl)
		37
30	static struct hugetlb_cgroup *root_h_cgroup __read_mostly;	38	static struct hugetlb_cgroup *root_h_cgroup __read_mostly;
		39	static struct cftype *dfl_files;
		40	static struct cftype *legacy_files;
31		41
32	static inline struct page_counter *	42	static inline struct page_counter *
33	__hugetlb_cgroup_counter_from_cgroup(struct hugetlb_cgroup *h_cg, int idx,	43	__hugetlb_cgroup_counter_from_cgroup(struct hugetlb_cgroup *h_cg, int idx,
@@ -460,7 +470,7 @@ static int hugetlb_cgroup_read_numa_stat(struct seq_file seq, void dummy) · 426 unmodified lines
460	int nid;	470	int nid;
461	struct cftype *cft = seq_cft(seq);	471	struct cftype *cft = seq_cft(seq);
462	int idx = MEMFILE_IDX(cft->private);	472	int idx = MEMFILE_IDX(cft->private);
463	bool legacy = MEMFILE_ATTR(cft->private);	473	bool legacy = !cgroup_subsys_on_dfl(hugetlb_cgrp_subsys);
464	struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(seq_css(seq));	474	struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(seq_css(seq));
465	struct cgroup_subsys_state *css;	475	struct cgroup_subsys_state *css;
466	unsigned long usage;	476	unsigned long usage;
@@ -702,166 +712,185 @@ static int hugetlb_events_local_show(struct seq_file seq, void v) · 235 unmodified lines
702	return __hugetlb_events_show(seq, true);	712	return __hugetlb_events_show(seq, true);
703	}	713	}
704		714
705	static void __init __hugetlb_cgroup_file_dfl_init(int idx)	715	static struct cftype hugetlb_dfl_tmpl[] = {
		716	{
		717	.name = "max",
		718	.private = RES_LIMIT,
		719	.seq_show = hugetlb_cgroup_read_u64_max,
		720	.write = hugetlb_cgroup_write_dfl,
		721	.flags = CFTYPE_NOT_ON_ROOT,
		722	},
		723	{
		724	.name = "rsvd.max",
		725	.private = RES_RSVD_LIMIT,
		726	.seq_show = hugetlb_cgroup_read_u64_max,
		727	.write = hugetlb_cgroup_write_dfl,
		728	.flags = CFTYPE_NOT_ON_ROOT,
		729	},
		730	{
		731	.name = "current",
		732	.private = RES_USAGE,
		733	.seq_show = hugetlb_cgroup_read_u64_max,
		734	.flags = CFTYPE_NOT_ON_ROOT,
		735	},
		736	{
		737	.name = "rsvd.current",
		738	.private = RES_RSVD_USAGE,
		739	.seq_show = hugetlb_cgroup_read_u64_max,
		740	.flags = CFTYPE_NOT_ON_ROOT,
		741	},
		742	{
		743	.name = "events",
		744	.seq_show = hugetlb_events_show,
		745	.file_offset = MEMFILE_OFFSET(struct hugetlb_cgroup, events_file[0]),
		746	.flags = CFTYPE_NOT_ON_ROOT,
		747	},
		748	{
		749	.name = "events.local",
		750	.seq_show = hugetlb_events_local_show,
		751	.file_offset = MEMFILE_OFFSET(struct hugetlb_cgroup, events_local_file[0]),
		752	.flags = CFTYPE_NOT_ON_ROOT,
		753	},
		754	{
		755	.name = "numa_stat",
		756	.seq_show = hugetlb_cgroup_read_numa_stat,
		757	.flags = CFTYPE_NOT_ON_ROOT,
		758	},
		759	/* don't need terminator here */
		760	};
		761
		762	static struct cftype hugetlb_legacy_tmpl[] = {
		763	{
		764	.name = "limit_in_bytes",
		765	.private = RES_LIMIT,
		766	.read_u64 = hugetlb_cgroup_read_u64,
		767	.write = hugetlb_cgroup_write_legacy,
		768	},
		769	{
		770	.name = "rsvd.limit_in_bytes",
		771	.private = RES_RSVD_LIMIT,
		772	.read_u64 = hugetlb_cgroup_read_u64,
		773	.write = hugetlb_cgroup_write_legacy,
		774	},
		775	{
		776	.name = "usage_in_bytes",
		777	.private = RES_USAGE,
		778	.read_u64 = hugetlb_cgroup_read_u64,
		779	},
		780	{
		781	.name = "rsvd.usage_in_bytes",
		782	.private = RES_RSVD_USAGE,
		783	.read_u64 = hugetlb_cgroup_read_u64,
		784	},
		785	{
		786	.name = "max_usage_in_bytes",
		787	.private = RES_MAX_USAGE,
		788	.write = hugetlb_cgroup_reset,
		789	.read_u64 = hugetlb_cgroup_read_u64,
		790	},
		791	{
		792	.name = "rsvd.max_usage_in_bytes",
		793	.private = RES_RSVD_MAX_USAGE,
		794	.write = hugetlb_cgroup_reset,
		795	.read_u64 = hugetlb_cgroup_read_u64,
		796	},
		797	{
		798	.name = "failcnt",
		799	.private = RES_FAILCNT,
		800	.write = hugetlb_cgroup_reset,
		801	.read_u64 = hugetlb_cgroup_read_u64,
		802	},
		803	{
		804	.name = "rsvd.failcnt",
		805	.private = RES_RSVD_FAILCNT,
		806	.write = hugetlb_cgroup_reset,
		807	.read_u64 = hugetlb_cgroup_read_u64,
		808	},
		809	{
		810	.name = "numa_stat",
		811	.seq_show = hugetlb_cgroup_read_numa_stat,
		812	},
		813	/* don't need terminator here */
		814	};
		815
		816	static void __init
		817	hugetlb_cgroup_cfttypes_init(struct hstate h, struct cftype cft,
		818	struct cftype *tmpl, int tmpl_size)
706	{	819	{
707	char buf[32];	820	char buf[32];
708	struct cftype *cft;	821	int i, idx = hstate_index(h);
709	struct hstate *h = &hstates[idx];
710		822
711	/* format the size */	823	/* format the size */
712	mem_fmt(buf, sizeof(buf), huge_page_size(h));	824	mem_fmt(buf, sizeof(buf), huge_page_size(h));
713		825
714	/* Add the limit file */	826	for (i = 0; i < tmpl_size; cft++, tmpl++, i++) {
715	cft = &h->cgroup_files_dfl[0];	827	cft = tmpl;
716	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.max", buf);	828	/* rebuild the name */
717	cft->private = MEMFILE_PRIVATE(idx, RES_LIMIT);	829	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name);
718	cft->seq_show = hugetlb_cgroup_read_u64_max;	830	/* rebuild the private */
719	cft->write = hugetlb_cgroup_write_dfl;	831	cft->private = MEMFILE_PRIVATE(idx, tmpl->private);
720	cft->flags = CFTYPE_NOT_ON_ROOT;	832	/* rebuild the file_offset */
721		833	if (tmpl->file_offset) {
722	/* Add the reservation limit file */	834	unsigned int offset = tmpl->file_offset;
723	cft = &h->cgroup_files_dfl[1];	835
724	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.max", buf);	836	cft->file_offset = MEMFILE_OFFSET0(offset) +
725	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_LIMIT);	837	MEMFILE_FIELD_SIZE(offset) * idx;
726	cft->seq_show = hugetlb_cgroup_read_u64_max;	838	}
727	cft->write = hugetlb_cgroup_write_dfl;
728	cft->flags = CFTYPE_NOT_ON_ROOT;
729
730	/* Add the current usage file */
731	cft = &h->cgroup_files_dfl[2];
732	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.current", buf);
733	cft->private = MEMFILE_PRIVATE(idx, RES_USAGE);
734	cft->seq_show = hugetlb_cgroup_read_u64_max;
735	cft->flags = CFTYPE_NOT_ON_ROOT;
736
737	/* Add the current reservation usage file */
738	cft = &h->cgroup_files_dfl[3];
739	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.current", buf);
740	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_USAGE);
741	cft->seq_show = hugetlb_cgroup_read_u64_max;
742	cft->flags = CFTYPE_NOT_ON_ROOT;
743
744	/* Add the events file */
745	cft = &h->cgroup_files_dfl[4];
746	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.events", buf);
747	cft->private = MEMFILE_PRIVATE(idx, 0);
748	cft->seq_show = hugetlb_events_show;
749	cft->file_offset = offsetof(struct hugetlb_cgroup, events_file[idx]);
750	cft->flags = CFTYPE_NOT_ON_ROOT;
751
752	/* Add the events.local file */
753	cft = &h->cgroup_files_dfl[5];
754	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.events.local", buf);
755	cft->private = MEMFILE_PRIVATE(idx, 0);
756	cft->seq_show = hugetlb_events_local_show;
757	cft->file_offset = offsetof(struct hugetlb_cgroup,
758	events_local_file[idx]);
759	cft->flags = CFTYPE_NOT_ON_ROOT;
760
761	/* Add the numa stat file */
762	cft = &h->cgroup_files_dfl[6];
763	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.numa_stat", buf);
764	cft->private = MEMFILE_PRIVATE(idx, 0);
765	cft->seq_show = hugetlb_cgroup_read_numa_stat;
766	cft->flags = CFTYPE_NOT_ON_ROOT;
767
768	/* NULL terminate the last cft */
769	cft = &h->cgroup_files_dfl[7];
770	memset(cft, 0, sizeof(*cft));
771		839
772	WARN_ON(cgroup_add_dfl_cftypes(&hugetlb_cgrp_subsys,	840	lockdep_register_key(&cft->lockdep_key);
773	h->cgroup_files_dfl));	841	}
774	}	842	}
775		843
776	static void __init __hugetlb_cgroup_file_legacy_init(int idx)	844	static void __init __hugetlb_cgroup_file_dfl_init(struct hstate *h)
777	{	845	{
778	char buf[32];	846	int idx = hstate_index(h);
779	struct cftype *cft;
780	struct hstate *h = &hstates[idx];
781		847
782	/* format the size */	848	hugetlb_cgroup_cfttypes_init(h, dfl_files + idx * DFL_TMPL_SIZE,
783	mem_fmt(buf, sizeof(buf), huge_page_size(h));	849	hugetlb_dfl_tmpl, DFL_TMPL_SIZE);
		850	}
784		851
785	/* Add the limit file */	852	static void __init __hugetlb_cgroup_file_legacy_init(struct hstate *h)
786	cft = &h->cgroup_files_legacy[0];	853	{
787	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.limit_in_bytes", buf);	854	int idx = hstate_index(h);
788	cft->private = MEMFILE_PRIVATE(idx, RES_LIMIT);
789	cft->read_u64 = hugetlb_cgroup_read_u64;
790	cft->write = hugetlb_cgroup_write_legacy;
791
792	/* Add the reservation limit file */
793	cft = &h->cgroup_files_legacy[1];
794	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.limit_in_bytes", buf);
795	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_LIMIT);
796	cft->read_u64 = hugetlb_cgroup_read_u64;
797	cft->write = hugetlb_cgroup_write_legacy;
798
799	/* Add the usage file */
800	cft = &h->cgroup_files_legacy[2];
801	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.usage_in_bytes", buf);
802	cft->private = MEMFILE_PRIVATE(idx, RES_USAGE);
803	cft->read_u64 = hugetlb_cgroup_read_u64;
804
805	/* Add the reservation usage file */
806	cft = &h->cgroup_files_legacy[3];
807	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.usage_in_bytes", buf);
808	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_USAGE);
809	cft->read_u64 = hugetlb_cgroup_read_u64;
810
811	/* Add the MAX usage file */
812	cft = &h->cgroup_files_legacy[4];
813	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.max_usage_in_bytes", buf);
814	cft->private = MEMFILE_PRIVATE(idx, RES_MAX_USAGE);
815	cft->write = hugetlb_cgroup_reset;
816	cft->read_u64 = hugetlb_cgroup_read_u64;
817
818	/* Add the MAX reservation usage file */
819	cft = &h->cgroup_files_legacy[5];
820	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.max_usage_in_bytes", buf);
821	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_MAX_USAGE);
822	cft->write = hugetlb_cgroup_reset;
823	cft->read_u64 = hugetlb_cgroup_read_u64;
824
825	/* Add the failcntfile */
826	cft = &h->cgroup_files_legacy[6];
827	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.failcnt", buf);
828	cft->private = MEMFILE_PRIVATE(idx, RES_FAILCNT);
829	cft->write = hugetlb_cgroup_reset;
830	cft->read_u64 = hugetlb_cgroup_read_u64;
831
832	/* Add the reservation failcntfile */
833	cft = &h->cgroup_files_legacy[7];
834	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.failcnt", buf);
835	cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_FAILCNT);
836	cft->write = hugetlb_cgroup_reset;
837	cft->read_u64 = hugetlb_cgroup_read_u64;
838
839	/* Add the numa stat file */
840	cft = &h->cgroup_files_legacy[8];
841	snprintf(cft->name, MAX_CFTYPE_NAME, "%s.numa_stat", buf);
842	cft->private = MEMFILE_PRIVATE(idx, 1);
843	cft->seq_show = hugetlb_cgroup_read_numa_stat;
844
845	/* NULL terminate the last cft */
846	cft = &h->cgroup_files_legacy[9];
847	memset(cft, 0, sizeof(*cft));
848		855
849	WARN_ON(cgroup_add_legacy_cftypes(&hugetlb_cgrp_subsys,	856	hugetlb_cgroup_cfttypes_init(h, legacy_files + idx * LEGACY_TMPL_SIZE,
850	h->cgroup_files_legacy));	857	hugetlb_legacy_tmpl, LEGACY_TMPL_SIZE);
		858	}
		859
		860	static void __init __hugetlb_cgroup_file_init(struct hstate *h)
		861	{
		862	__hugetlb_cgroup_file_dfl_init(h);
		863	__hugetlb_cgroup_file_legacy_init(h);
851	}	864	}
852		865
853	static void __init __hugetlb_cgroup_file_init(int idx)	866	static void __init __hugetlb_cgroup_file_pre_init(void)
854	{	867	{
855	__hugetlb_cgroup_file_dfl_init(idx);	868	int cft_count;
856	__hugetlb_cgroup_file_legacy_init(idx);	869
		870	cft_count = hugetlb_max_hstate * DFL_TMPL_SIZE + 1; /* add terminator */
		871	dfl_files = kcalloc(cft_count, sizeof(struct cftype), GFP_KERNEL);
		872	BUG_ON(!dfl_files);
		873	cft_count = hugetlb_max_hstate * LEGACY_TMPL_SIZE + 1; /* add terminator */
		874	legacy_files = kcalloc(cft_count, sizeof(struct cftype), GFP_KERNEL);
		875	BUG_ON(!legacy_files);
		876	}
		877
		878	static void __init __hugetlb_cgroup_file_post_init(void)
		879	{
		880	WARN_ON(cgroup_add_dfl_cftypes(&hugetlb_cgrp_subsys,
		881	dfl_files));
		882	WARN_ON(cgroup_add_legacy_cftypes(&hugetlb_cgrp_subsys,
		883	legacy_files));
857	}	884	}
858		885
859	void __init hugetlb_cgroup_file_init(void)	886	void __init hugetlb_cgroup_file_init(void)
860	{	887	{
861	struct hstate *h;	888	struct hstate *h;
862		889
		890	__hugetlb_cgroup_file_pre_init();
863	for_each_hstate(h)	891	for_each_hstate(h)
864	__hugetlb_cgroup_file_init(hstate_index(h));	892	__hugetlb_cgroup_file_init(h);
		893	__hugetlb_cgroup_file_post_init();
865	}	894	}
866		895
867	/*	896	/*

mm/hugetlb_vmemmap.c+13−4

  */
 static inline void free_vmemmap_page(struct page *page)
 {
-	if (PageReserved(page))
+	if (PageReserved(page)) {
 		free_bootmem_page(page);
-	else
+		mod_node_page_state(page_pgdat(page), NR_MEMMAP_BOOT, -1);
+	} else {
 		__free_page(page);
+		mod_node_page_state(page_pgdat(page), NR_MEMMAP, -1);
+	}
 }
 /* Free a list of the vmemmap pages */
 		copy_page(page_to_virt(walk.reuse_page),
 			  (void *)walk.reuse_addr);
 		list_add(&walk.reuse_page->lru, vmemmap_pages);
+		mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, 1);
 	}
 	/*
 	unsigned long nr_pages = (end - start) >> PAGE_SHIFT;
 	int nid = page_to_nid((struct page *)start);
 	struct page *page, *next;
+	int i;
-	while (nr_pages--) {
+	for (i = 0; i < nr_pages; i++) {
 		page = alloc_pages_node(nid, gfp_mask, 0);
-		if (!page)
+		if (!page) {
+			mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, i);
 			goto out;
+		}
 		list_add(&page->lru, list);
 	}
+	mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, nr_pages);
 	return 0;
 out:
 	list_for_each_entry_safe(page, next, list, lru)

mm/hwpoison-inject.c+1−0

@@ -110,4 +110,5 @@ static int __init pfn_inject_init(void) · 109 unmodified lines
110		110
111	module_init(pfn_inject_init);	111	module_init(pfn_inject_init);
112	module_exit(pfn_inject_exit);	112	module_exit(pfn_inject_exit);
		113	MODULE_DESCRIPTION("HWPoison pages injector");
113	MODULE_LICENSE("GPL");	114	MODULE_LICENSE("GPL");

mm/internal.h+69−6

@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · 210 unmodified lines
211	}	211	}
212		212
213	/**	213	/**
214	* pte_next_swp_offset - Increment the swap entry offset field of a swap pte.	214	* pte_move_swp_offset - Move the swap entry offset field of a swap pte
		215	* forward or backward by delta
215	* @pte: The initial pte state; is_swap_pte(pte) must be true and	216	* @pte: The initial pte state; is_swap_pte(pte) must be true and
216	* non_swap_entry() must be false.	217	* non_swap_entry() must be false.
		218	* @delta: The direction and the offset we are moving; forward if delta
		219	* is positive; backward if delta is negative
217	*	220	*
218	* Increments the swap offset, while maintaining all other fields, including	221	* Moves the swap offset, while maintaining all other fields, including
219	* swap type, and any swp pte bits. The resulting pte is returned.	222	* swap type, and any swp pte bits. The resulting pte is returned.
220	*/	223	*/
221	static inline pte_t pte_next_swp_offset(pte_t pte)	224	static inline pte_t pte_move_swp_offset(pte_t pte, long delta)
222	{	225	{
223	swp_entry_t entry = pte_to_swp_entry(pte);	226	swp_entry_t entry = pte_to_swp_entry(pte);
224	pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry),	227	pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry),
225	(swp_offset(entry) + 1)));	228	(swp_offset(entry) + delta)));
226		229
227	if (pte_swp_soft_dirty(pte))	230	if (pte_swp_soft_dirty(pte))
228	new = pte_swp_mksoft_dirty(new);	231	new = pte_swp_mksoft_dirty(new);
@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) · 5 unmodified lines
234	return new;	237	return new;
235	}	238	}
236		239
		240
		241	/**
		242	* pte_next_swp_offset - Increment the swap entry offset field of a swap pte.
		243	* @pte: The initial pte state; is_swap_pte(pte) must be true and
		244	* non_swap_entry() must be false.
		245	*
		246	* Increments the swap offset, while maintaining all other fields, including
		247	* swap type, and any swp pte bits. The resulting pte is returned.
		248	*/
		249	static inline pte_t pte_next_swp_offset(pte_t pte)
		250	{
		251	return pte_move_swp_offset(pte, 1);
		252	}
		253
237	/**	254	/**
238	* swap_pte_batch - detect a PTE batch for a set of contiguous swap entries	255	* swap_pte_batch - detect a PTE batch for a set of contiguous swap entries
239	* @start_ptep: Page table pointer for the first entry.	256	* @start_ptep: Page table pointer for the first entry.
@@ -587,7 +604,8 @@ extern void __putback_isolated_page(struct page *page, unsigned int order, · 347 unmodified lines
587	int mt);	604	int mt);
588	extern void memblock_free_pages(struct page *page, unsigned long pfn,	605	extern void memblock_free_pages(struct page *page, unsigned long pfn,
589	unsigned int order);	606	unsigned int order);
590	extern void __free_pages_core(struct page *page, unsigned int order);	607	extern void __free_pages_core(struct page *page, unsigned int order,
		608	enum meminit_context context);
591		609
592	/*	610	/*
593	* This will have no effect, other than possibly generating a warning, if the	611	* This will have no effect, other than possibly generating a warning, if the
@@ -604,7 +622,22 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) · 10 unmodified lines
604	#endif	622	#endif
605	}	623	}
606		624
607	void folio_undo_large_rmappable(struct folio *folio);	625	void __folio_undo_large_rmappable(struct folio *folio);
		626	static inline void folio_undo_large_rmappable(struct folio *folio)
		627	{
		628	if (folio_order(folio) <= 1 \|\| !folio_test_large_rmappable(folio))
		629	return;
		630
		631	/*
		632	* At this point, there is no one trying to add the folio to
		633	* deferred_list. If folio is not in deferred_list, it's safe
		634	* to check without acquiring the split_queue_lock.
		635	*/
		636	if (data_race(list_empty(&folio->_deferred_list)))
		637	return;
		638
		639	__folio_undo_large_rmappable(folio);
		640	}
608		641
609	static inline struct folio page_rmappable_folio(struct page page)	642	static inline struct folio page_rmappable_folio(struct page page)
610	{	643	{
@@ -1045,12 +1078,23 @@ extern u64 hwpoison_filter_flags_mask; · 434 unmodified lines
1045	extern u64 hwpoison_filter_flags_value;	1078	extern u64 hwpoison_filter_flags_value;
1046	extern u64 hwpoison_filter_memcg;	1079	extern u64 hwpoison_filter_memcg;
1047	extern u32 hwpoison_filter_enable;	1080	extern u32 hwpoison_filter_enable;
		1081	#define MAGIC_HWPOISON 0x48575053U /* HWPS */
		1082	void SetPageHWPoisonTakenOff(struct page *page);
		1083	void ClearPageHWPoisonTakenOff(struct page *page);
		1084	bool take_page_off_buddy(struct page *page);
		1085	bool put_page_back_buddy(struct page *page);
		1086	struct task_struct task_early_kill(struct task_struct tsk, int force_early);
		1087	void add_to_kill_ksm(struct task_struct tsk, struct page p,
		1088	struct vm_area_struct vma, struct list_head to_kill,
		1089	unsigned long ksm_addr);
		1090	unsigned long page_mapped_in_vma(struct page page, struct vm_area_struct vma);
1048		1091
1049	extern unsigned long __must_check vm_mmap_pgoff(struct file *, unsigned long,	1092	extern unsigned long __must_check vm_mmap_pgoff(struct file *, unsigned long,
1050	unsigned long, unsigned long,	1093	unsigned long, unsigned long,
1051	unsigned long, unsigned long);	1094	unsigned long, unsigned long);
1052		1095
1053	extern void set_pageblock_order(void);	1096	extern void set_pageblock_order(void);
		1097	struct folio alloc_migrate_folio(struct folio src, unsigned long private);
1054	unsigned long reclaim_pages(struct list_head *folio_list);	1098	unsigned long reclaim_pages(struct list_head *folio_list);
1055	unsigned int reclaim_clean_pages_from_list(struct zone *zone,	1099	unsigned int reclaim_clean_pages_from_list(struct zone *zone,
1056	struct list_head *folio_list);	1100	struct list_head *folio_list);
@@ -1316,6 +1360,16 @@ static inline bool vma_soft_dirty_enabled(struct vm_area_struct *vma) · 259 unmodified lines
1316	return !(vma->vm_flags & VM_SOFTDIRTY);	1360	return !(vma->vm_flags & VM_SOFTDIRTY);
1317	}	1361	}
1318		1362
		1363	static inline bool pmd_needs_soft_dirty_wp(struct vm_area_struct *vma, pmd_t pmd)
		1364	{
		1365	return vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd);
		1366	}
		1367
		1368	static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte)
		1369	{
		1370	return vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte);
		1371	}
		1372
1319	static inline void vma_iter_config(struct vma_iterator *vmi,	1373	static inline void vma_iter_config(struct vma_iterator *vmi,
1320	unsigned long index, unsigned long last)	1374	unsigned long index, unsigned long last)
1321	{	1375	{
@@ -1515,4 +1569,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, · 193 unmodified lines
1515	void workingset_update_node(struct xa_node *node);	1569	void workingset_update_node(struct xa_node *node);
1516	extern struct list_lru shadow_nodes;	1570	extern struct list_lru shadow_nodes;
1517		1571
		1572	struct unlink_vma_file_batch {
		1573	int count;
		1574	struct vm_area_struct *vmas[8];
		1575	};
		1576
		1577	void unlink_file_vma_batch_init(struct unlink_vma_file_batch *);
		1578	void unlink_file_vma_batch_add(struct unlink_vma_file_batch , struct vm_area_struct );
		1579	void unlink_file_vma_batch_final(struct unlink_vma_file_batch *);
		1580
1518	#endif /* __MM_INTERNAL_H */	1581	#endif /* __MM_INTERNAL_H */

mm/kfence/core.c+12−5

 	WRITE_ONCE(meta->state, next);
 }
+#ifdef CONFIG_KMSAN
+#define check_canary_attributes noinline __no_kmsan_checks
+#else
+#define check_canary_attributes inline
+#endif
 /* Check canary byte at @addr. */
-static inline bool check_canary_byte(u8 *addr)
+static check_canary_attributes bool check_canary_byte(u8 *addr)
 {
 	struct kfence_metadata *meta;
 	unsigned long flags;
 		*((u64 *)addr) = KFENCE_CANARY_PATTERN_U64;
 }
-static inline void check_canary(const struct kfence_metadata *meta)
+static check_canary_attributes void
+check_canary(const struct kfence_metadata *meta)
 {
 	const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE);
 	unsigned long addr = pageaddr;
 			continue;
 		__folio_set_slab(slab_folio(slab));
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 		slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts |
 				 MEMCG_DATA_OBJEXTS;
 #endif
 		if (!i || (i % 2))
 			continue;
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 		slab->obj_exts = 0;
 #endif
 		__folio_clear_slab(slab_folio(slab));
 {
 	struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
-#ifdef CONFIG_MEMCG_KMEM
+#ifdef CONFIG_MEMCG
 	KFENCE_WARN_ON(meta->obj_exts.objcg);
 #endif
 	/*

mm/kfence/kfence.h+1−1

@@ -97,7 +97,7 @@ struct kfence_metadata { · 96 unmodified lines
97	struct kfence_track free_track;	97	struct kfence_track free_track;
98	/* For updating alloc_covered on frees. */	98	/* For updating alloc_covered on frees. */
99	u32 alloc_stack_hash;	99	u32 alloc_stack_hash;
100	#ifdef CONFIG_MEMCG_KMEM	100	#ifdef CONFIG_MEMCG
101	struct slabobj_ext obj_exts;	101	struct slabobj_ext obj_exts;
102	#endif	102	#endif
103	};	103	};

mm/kfence/kfence_test.c+1−0

@@ -852,3 +852,4 @@ kunit_test_suites(&kfence_test_suite); · 851 unmodified lines
852		852
853	MODULE_LICENSE("GPL v2");	853	MODULE_LICENSE("GPL v2");
854	MODULE_AUTHOR("Alexander Potapenko <glider@google.com>, Marco Elver <elver@google.com>");	854	MODULE_AUTHOR("Alexander Potapenko <glider@google.com>, Marco Elver <elver@google.com>");
		855	MODULE_DESCRIPTION("kfence unit test suite");

mm/khugepaged.c+28−12

 int __init khugepaged_init(void)
 {
-	mm_slot_cache = kmem_cache_create("khugepaged_mm_slot",
+	mm_slot_cache = KMEM_CACHE(khugepaged_mm_slot, 0);
-					  sizeof(struct khugepaged_mm_slot),
-					  __alignof__(struct khugepaged_mm_slot),
-					  0, NULL);
 	if (!mm_slot_cache)
 		return -ENOMEM;
 	       test_bit(MMF_DISABLE_THP, &mm->flags);
 }
+static bool hugepage_pmd_enabled(void)
+{
+	/*
+	 * We cover both the anon and the file-backed case here; file-backed
+	 * hugepages, when configured in, are determined by the global control.
+	 * Anon pmd-sized hugepages are determined by the pmd-size control.
+	 */
+	if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
+	    hugepage_global_enabled())
+		return true;
+	if (test_bit(PMD_ORDER, &huge_anon_orders_always))
+		return true;
+	if (test_bit(PMD_ORDER, &huge_anon_orders_madvise))
+		return true;
+	if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) &&
+	    hugepage_global_enabled())
+		return true;
+	return false;
+}
 void __khugepaged_enter(struct mm_struct *mm)
 {
 	struct khugepaged_mm_slot *mm_slot;
 			  unsigned long vm_flags)
 {
 	if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) &&
-	    hugepage_flags_enabled()) {
+	    hugepage_pmd_enabled()) {
 		if (thp_vma_allowable_order(vma, vm_flags, TVA_ENFORCE_SYSFS,
 					    PMD_ORDER))
 			__khugepaged_enter(vma->vm_mm);
 	spin_lock(pmd_ptl);
 	BUG_ON(!pmd_none(*pmd));
-	folio_add_new_anon_rmap(folio, vma, address);
+	folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
 	folio_add_lru_vma(folio, vma);
 	pgtable_trans_huge_deposit(mm, pmd, pgtable);
 	set_pmd_at(mm, address, pmd, _pmd);
 static int khugepaged_has_work(void)
 {
-	return !list_empty(&khugepaged_scan.mm_head) &&
+	return !list_empty(&khugepaged_scan.mm_head) && hugepage_pmd_enabled();
-		hugepage_flags_enabled();
 }
 static int khugepaged_wait_event(void)
 		return;
 	}
-	if (hugepage_flags_enabled())
+	if (hugepage_pmd_enabled())
 		wait_event_freezable(khugepaged_wait, khugepaged_wait_event());
 }
 	int nr_zones = 0;
 	unsigned long recommended_min;
-	if (!hugepage_flags_enabled()) {
+	if (!hugepage_pmd_enabled()) {
 		calculate_min_free_kbytes();
 		goto update_wmarks;
 	}
 	int err = 0;
 	mutex_lock(&khugepaged_mutex);
-	if (hugepage_flags_enabled()) {
+	if (hugepage_pmd_enabled()) {
 		if (!khugepaged_thread)
 			khugepaged_thread = kthread_run(khugepaged, NULL,
 							"khugepaged");
 void khugepaged_min_free_kbytes_update(void)
 {
 	mutex_lock(&khugepaged_mutex);
-	if (hugepage_flags_enabled() && khugepaged_thread)
+	if (hugepage_pmd_enabled() && khugepaged_thread)
 		set_recommended_min_free_kbytes();
 	mutex_unlock(&khugepaged_mutex);
 }

mm/kmemleak.c+3−3

@@ -657,10 +657,10 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) · 656 unmodified lines
657	/* task information */	657	/* task information */
658	if (in_hardirq()) {	658	if (in_hardirq()) {
659	object->pid = 0;	659	object->pid = 0;
660	strncpy(object->comm, "hardirq", sizeof(object->comm));	660	strscpy(object->comm, "hardirq");
661	} else if (in_serving_softirq()) {	661	} else if (in_serving_softirq()) {
662	object->pid = 0;	662	object->pid = 0;
663	strncpy(object->comm, "softirq", sizeof(object->comm));	663	strscpy(object->comm, "softirq");
664	} else {	664	} else {
665	object->pid = current->pid;	665	object->pid = current->pid;
666	/*	666	/*
@@ -669,7 +669,7 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) · 2 unmodified lines
669	* dependency issues with current->alloc_lock. In the worst	669	* dependency issues with current->alloc_lock. In the worst
670	* case, the command line is not correct.	670	* case, the command line is not correct.
671	*/	671	*/
672	strncpy(object->comm, current->comm, sizeof(object->comm));	672	strscpy(object->comm, current->comm);
673	}	673	}
674		674
675	/* kernel backtrace */	675	/* kernel backtrace */

mm/kmsan/core.c+2−3

 	struct thread_info *info = current_thread_info();
 	__memset(ctx, 0, sizeof(*ctx));
-	ctx->allow_reporting = true;
 	kmsan_internal_unpoison_memory(info, sizeof(*info), false);
 }
 		return NULL;
 }
-void kmsan_internal_check_memory(void *addr, size_t size, const void *user_addr,
+void kmsan_internal_check_memory(void *addr, size_t size,
-				 int reason)
+				 const void __user *user_addr, int reason)
 {
 	depot_stack_handle_t cur_origin = 0, new_origin = 0;
 	unsigned long addr64 = (unsigned long)addr;

mm/kmsan/hooks.c+26−12

 void kmsan_task_exit(struct task_struct *task)
 {
-	struct kmsan_ctx *ctx = &task->kmsan_ctx;
 	if (!kmsan_enabled || kmsan_in_runtime())
 		return;
-	ctx->allow_reporting = false;
+	kmsan_disable_current();
 }
 void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags)
 		return;
 	/* RCU slabs could be legally used after free within the RCU period */
-	if (unlikely(s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)))
+	if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU))
 		return;
 	/*
 	 * If there's a constructor, freed memory must remain in the same state
 		return;
 	ua_flags = user_access_save();
-	if ((u64)to < TASK_SIZE) {
+	if (!IS_ENABLED(CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE) ||
+	    (u64)to < TASK_SIZE) {
 		/* This is a user memory access, check it. */
 		kmsan_internal_check_memory((void *)from, to_copy - left, to,
 					    REASON_COPY_TO_USER);
 	if (is_out)
 		kmsan_internal_check_memory(urb->transfer_buffer,
 					    urb->transfer_buffer_length,
-					    /*user_addr*/ 0, REASON_SUBMIT_URB);
+					    /*user_addr*/ NULL,
+					    REASON_SUBMIT_URB);
 	else
 		kmsan_internal_unpoison_memory(urb->transfer_buffer,
 					       urb->transfer_buffer_length,
 {
 	switch (dir) {
 	case DMA_BIDIRECTIONAL:
-		kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0,
+		kmsan_internal_check_memory((void *)addr, size,
-					    REASON_ANY);
+					    /*user_addr*/ NULL, REASON_ANY);
 		kmsan_internal_unpoison_memory((void *)addr, size,
 					       /*checked*/ false);
 		break;
 	case DMA_TO_DEVICE:
-		kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0,
+		kmsan_internal_check_memory((void *)addr, size,
-					    REASON_ANY);
+					    /*user_addr*/ NULL, REASON_ANY);
 		break;
 	case DMA_FROM_DEVICE:
 		kmsan_internal_unpoison_memory((void *)addr, size,
 {
 	if (!kmsan_enabled)
 		return;
-	return kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0,
+	return kmsan_internal_check_memory((void *)addr, size,
-					   REASON_ANY);
+					   /*user_addr*/ NULL, REASON_ANY);
 }
 EXPORT_SYMBOL(kmsan_check_memory);
+void kmsan_enable_current(void)
+{
+	KMSAN_WARN_ON(current->kmsan_ctx.depth == 0);
+	current->kmsan_ctx.depth--;
+}
+EXPORT_SYMBOL(kmsan_enable_current);
+void kmsan_disable_current(void)
+{
+	current->kmsan_ctx.depth++;
+	KMSAN_WARN_ON(current->kmsan_ctx.depth == 0);
+}
+EXPORT_SYMBOL(kmsan_disable_current);

mm/kmsan/init.c+6−3

 	bool merged = false;
 	KMSAN_WARN_ON(future_index == NUM_FUTURE_RANGES);
-	KMSAN_WARN_ON((nstart >= nend) || !nstart || !nend);
+	KMSAN_WARN_ON((nstart >= nend) ||
+		      /* Virtual address 0 is valid on s390. */
+		      (!IS_ENABLED(CONFIG_S390) && !nstart) ||
+		      !nend);
 	nstart = ALIGN_DOWN(nstart, PAGE_SIZE);
 	nend = ALIGN(nend, PAGE_SIZE);
  */
 void __init kmsan_init_shadow(void)
 {
-	const size_t nd_size = roundup(sizeof(pg_data_t), PAGE_SIZE);
+	const size_t nd_size = sizeof(pg_data_t);
 	phys_addr_t p_start, p_end;
 	u64 loop;
 	int nid;
 		shadow = smallstack_pop(&collect);
 		origin = smallstack_pop(&collect);
 		kmsan_setup_meta(page, shadow, origin, collect.order);
-		__free_pages_core(page, collect.order);
+		__free_pages_core(page, collect.order, MEMINIT_EARLY);
 	}
 }

mm/kmsan/instrumentation.c+8−7

 #include "kmsan.h"
 #include <linux/gfp.h>
+#include <linux/kmsan.h>
 #include <linux/kmsan_string.h>
 #include <linux/mm.h>
 #include <linux/uaccess.h>
 static inline bool is_bad_asm_addr(void *addr, uintptr_t size, bool is_store)
 {
-	if ((u64)addr < TASK_SIZE)
+	if (IS_ENABLED(CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE) &&
+	    (u64)addr < TASK_SIZE)
 		return true;
 	if (!kmsan_get_metadata(addr, KMSAN_META_SHADOW))
 		return true;
 	ua_flags = user_access_save();
 	/*
-	 * Most of the accesses are below 32 bytes. The two exceptions so far
+	 * Most of the accesses are below 32 bytes. The exceptions so far are
-	 * are clwb() (64 bytes) and FPU state (512 bytes).
+	 * clwb() (64 bytes), FPU state (512 bytes) and chsc() (4096 bytes).
-	 * It's unlikely that the assembly will touch more than 512 bytes.
 	 */
-	if (size > 512) {
+	if (size > 4096) {
 		WARN_ONCE(1, "assembly store size too big: %ld\n", size);
 		size = 8;
 	}
 	if (!kmsan_enabled || kmsan_in_runtime())
 		return;
 	kmsan_enter_runtime();
-	kmsan_report(origin, /*address*/ 0, /*size*/ 0,
+	kmsan_report(origin, /*address*/ NULL, /*size*/ 0,
-		     /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ 0,
+		     /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ NULL,
 		     REASON_ANY);
 	kmsan_leave_runtime();
 }

mm/kmsan/kmsan.h+8−31

 #ifndef __MM_KMSAN_KMSAN_H
 #define __MM_KMSAN_KMSAN_H
-#include <asm/pgtable_64_types.h>
 #include <linux/irqflags.h>
+#include <linux/kmsan.h>
+#include <linux/mm.h>
+#include <linux/nmi.h>
+#include <linux/pgtable.h>
+#include <linux/printk.h>
 #include <linux/sched.h>
 #include <linux/stackdepot.h>
 #include <linux/stacktrace.h>
-#include <linux/nmi.h>
-#include <linux/mm.h>
-#include <linux/printk.h>
 #define KMSAN_ALLOCA_MAGIC_ORIGIN 0xabcd0100
 #define KMSAN_CHAIN_MAGIC_ORIGIN 0xabcd0200
 #define KMSAN_META_SHADOW (false)
 #define KMSAN_META_ORIGIN (true)
-extern bool kmsan_enabled;
-extern int panic_on_kmsan;
-/*
- * KMSAN performs a lot of consistency checks that are currently enabled by
- * default. BUG_ON is normally discouraged in the kernel, unless used for
- * debugging, but KMSAN itself is a debugging tool, so it makes little sense to
- * recover if something goes wrong.
- */
-#define KMSAN_WARN_ON(cond)                                           \
-	({                                                            \
-		const bool __cond = WARN_ON(cond);                    \
-		if (unlikely(__cond)) {                               \
-			WRITE_ONCE(kmsan_enabled, false);             \
-			if (panic_on_kmsan) {                         \
-				/* Can't call panic() here because */ \
-				/* of uaccess checks. */              \
-				BUG();                                \
-			}                                             \
-		}                                                     \
-		__cond;                                               \
-	})
 /*
  * A pair of metadata pointers to be returned by the instrumentation functions.
  */
 struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *addr, u64 size,
 						     bool store);
-void *kmsan_get_metadata(void *addr, bool is_origin);
 void __init kmsan_init_alloc_meta_for_range(void *start, void *end);
 enum kmsan_bug_reason {
  * @off_last corresponding to different @origin values.
  */
 void kmsan_report(depot_stack_handle_t origin, void *address, int size,
-		  int off_first, int off_last, const void *user_addr,
+		  int off_first, int off_last, const void __user *user_addr,
 		  enum kmsan_bug_reason reason);
 DECLARE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx);
 void kmsan_internal_task_create(struct task_struct *task);
 bool kmsan_metadata_is_contiguous(void *addr, size_t size);
-void kmsan_internal_check_memory(void *addr, size_t size, const void *user_addr,
+void kmsan_internal_check_memory(void *addr, size_t size,
-				 int reason);
+				 const void __user *user_addr, int reason);
 struct page *kmsan_vmalloc_to_page_or_null(void *vaddr);
 void kmsan_setup_meta(struct page *page, struct page *shadow,

mm/kmsan/kmsan_test.c+32−0

@@ -614,6 +614,32 @@ static void test_stackdepot_roundtrip(struct kunit *test) · 613 unmodified lines
614	KUNIT_EXPECT_TRUE(test, report_matches(&expect));	614	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
615	}	615	}
616		616
		617	/*
		618	* Test case: ensure that kmsan_unpoison_memory() and the instrumentation work
		619	* the same.
		620	*/
		621	static void test_unpoison_memory(struct kunit *test)
		622	{
		623	EXPECTATION_UNINIT_VALUE_FN(expect, "test_unpoison_memory");
		624	volatile char a[4], b[4];
		625
		626	kunit_info(
		627	test,
		628	"unpoisoning via the instrumentation vs. kmsan_unpoison_memory() (2 UMR reports)\n");
		629
		630	/* Initialize a[0] and check a[1]--a[3]. */
		631	a[0] = 0;
		632	kmsan_check_memory((char *)&a[1], 3);
		633	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
		634
		635	report_reset();
		636
		637	/* Initialize b[0] and check b[1]--b[3]. */
		638	kmsan_unpoison_memory((char *)&b[0], 1);
		639	kmsan_check_memory((char *)&b[1], 3);
		640	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
		641	}
		642
617	static struct kunit_case kmsan_test_cases[] = {	643	static struct kunit_case kmsan_test_cases[] = {
618	KUNIT_CASE(test_uninit_kmalloc),	644	KUNIT_CASE(test_uninit_kmalloc),
619	KUNIT_CASE(test_init_kmalloc),	645	KUNIT_CASE(test_init_kmalloc),
@@ -637,6 +663,7 @@ static struct kunit_case kmsan_test_cases[] = { · 17 unmodified lines
637	KUNIT_CASE(test_memset64),	663	KUNIT_CASE(test_memset64),
638	KUNIT_CASE(test_long_origin_chain),	664	KUNIT_CASE(test_long_origin_chain),
639	KUNIT_CASE(test_stackdepot_roundtrip),	665	KUNIT_CASE(test_stackdepot_roundtrip),
		666	KUNIT_CASE(test_unpoison_memory),
640	{},	667	{},
641	};	668	};
642		669
@@ -659,9 +686,13 @@ static void test_exit(struct kunit *test) · 16 unmodified lines
659	{	686	{
660	}	687	}
661		688
		689	static int orig_panic_on_kmsan;
		690
662	static int kmsan_suite_init(struct kunit_suite *suite)	691	static int kmsan_suite_init(struct kunit_suite *suite)
663	{	692	{
664	register_trace_console(probe_console, NULL);	693	register_trace_console(probe_console, NULL);
		694	orig_panic_on_kmsan = panic_on_kmsan;
		695	panic_on_kmsan = 0;
665	return 0;	696	return 0;
666	}	697	}
667		698
@@ -669,6 +700,7 @@ static void kmsan_suite_exit(struct kunit_suite *suite) · 1 unmodified line
669	{	700	{
670	unregister_trace_console(probe_console, NULL);	701	unregister_trace_console(probe_console, NULL);
671	tracepoint_synchronize_unregister();	702	tracepoint_synchronize_unregister();
		703	panic_on_kmsan = orig_panic_on_kmsan;
672	}	704	}
673		705
674	static struct kunit_suite kmsan_test_suite = {	706	static struct kunit_suite kmsan_test_suite = {

mm/kmsan/report.c+6−4

  */
 #include <linux/console.h>
+#include <linux/kmsan.h>
 #include <linux/moduleparam.h>
 #include <linux/stackdepot.h>
 #include <linux/stacktrace.h>
 /* Protected by kmsan_report_lock */
 static char report_local_descr[DESCR_SIZE];
 int panic_on_kmsan __read_mostly;
+EXPORT_SYMBOL_GPL(panic_on_kmsan);
 #ifdef MODULE_PARAM_PREFIX
 #undef MODULE_PARAM_PREFIX
 }
 void kmsan_report(depot_stack_handle_t origin, void *address, int size,
-		  int off_first, int off_last, const void *user_addr,
+		  int off_first, int off_last, const void __user *user_addr,
 		  enum kmsan_bug_reason reason)
 {
 	unsigned long stack_entries[KMSAN_STACK_DEPTH];
 	if (!kmsan_enabled)
 		return;
-	if (!current->kmsan_ctx.allow_reporting)
+	if (current->kmsan_ctx.depth)
 		return;
 	if (!origin)
 		return;
-	current->kmsan_ctx.allow_reporting = false;
+	kmsan_disable_current();
 	ua_flags = user_access_save();
 	raw_spin_lock(&kmsan_report_lock);
 	pr_err("=====================================================\n");
 	if (panic_on_kmsan)
 		panic("kmsan.panic set ...\n");
 	user_access_restore(ua_flags);
-	current->kmsan_ctx.allow_reporting = true;
+	kmsan_enable_current();
 }

mm/kmsan/shadow.c+3−6

  */
 void *kmsan_get_metadata(void *address, bool is_origin)
 {
-	u64 addr = (u64)address, pad, off;
+	u64 addr = (u64)address, off;
 	struct page *page;
 	void *ret;
-	if (is_origin && !IS_ALIGNED(addr, KMSAN_ORIGIN_SIZE)) {
+	if (is_origin)
-		pad = addr % KMSAN_ORIGIN_SIZE;
+		addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE);
-		addr -= pad;
-	}
 	address = (void *)addr;
 	if (kmsan_internal_is_vmalloc_addr(address) ||
 	    kmsan_internal_is_module_addr(address))
 		s_pages[i] = shadow_page_for(pages[i]);
 		o_pages[i] = origin_page_for(pages[i]);
 	}
-	prot = __pgprot(pgprot_val(prot) | _PAGE_NX);
 	prot = PAGE_KERNEL;
 	origin_start = vmalloc_meta((void *)start, KMSAN_META_ORIGIN);

mm/ksm.c+85−176

@@ -488,21 +488,17 @@ static DECLARE_WAIT_QUEUE_HEAD(ksm_iter_wait); · 487 unmodified lines
488	static DEFINE_MUTEX(ksm_thread_mutex);	488	static DEFINE_MUTEX(ksm_thread_mutex);
489	static DEFINE_SPINLOCK(ksm_mmlist_lock);	489	static DEFINE_SPINLOCK(ksm_mmlist_lock);
490		490
491	#define KSM_KMEM_CACHE(__struct, __flags) kmem_cache_create(#__struct,\
492	sizeof(struct __struct), __alignof__(struct __struct),\
493	(__flags), NULL)
494
495	static int __init ksm_slab_init(void)	491	static int __init ksm_slab_init(void)
496	{	492	{
497	rmap_item_cache = KSM_KMEM_CACHE(ksm_rmap_item, 0);	493	rmap_item_cache = KMEM_CACHE(ksm_rmap_item, 0);
498	if (!rmap_item_cache)	494	if (!rmap_item_cache)
499	goto out;	495	goto out;
500		496
501	stable_node_cache = KSM_KMEM_CACHE(ksm_stable_node, 0);	497	stable_node_cache = KMEM_CACHE(ksm_stable_node, 0);
502	if (!stable_node_cache)	498	if (!stable_node_cache)
503	goto out_free1;	499	goto out_free1;
504		500
505	mm_slot_cache = KSM_KMEM_CACHE(ksm_mm_slot, 0);	501	mm_slot_cache = KMEM_CACHE(ksm_mm_slot, 0);
506	if (!mm_slot_cache)	502	if (!mm_slot_cache)
507	goto out_free2;	503	goto out_free2;
508		504
@@ -1531,6 +1527,44 @@ out: · 1022 unmodified lines
1531	return err;	1527	return err;
1532	}	1528	}
1533		1529
		1530	/*
		1531	* This function returns 0 if the pages were merged or if they are
		1532	* no longer merging candidates (e.g., VMA stale), -EFAULT otherwise.
		1533	*/
		1534	static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item,
		1535	struct page *page)
		1536	{
		1537	struct mm_struct *mm = rmap_item->mm;
		1538	int err = -EFAULT;
		1539
		1540	/*
		1541	* Same checksum as an empty page. We attempt to merge it with the
		1542	* appropriate zero page if the user enabled this via sysfs.
		1543	*/
		1544	if (ksm_use_zero_pages && (rmap_item->oldchecksum == zero_checksum)) {
		1545	struct vm_area_struct *vma;
		1546
		1547	mmap_read_lock(mm);
		1548	vma = find_mergeable_vma(mm, rmap_item->address);
		1549	if (vma) {
		1550	err = try_to_merge_one_page(vma, page,
		1551	ZERO_PAGE(rmap_item->address));
		1552	trace_ksm_merge_one_page(
		1553	page_to_pfn(ZERO_PAGE(rmap_item->address)),
		1554	rmap_item, mm, err);
		1555	} else {
		1556	/*
		1557	* If the vma is out of date, we do not need to
		1558	* continue.
		1559	*/
		1560	err = 0;
		1561	}
		1562	mmap_read_unlock(mm);
		1563	}
		1564
		1565	return err;
		1566	}
		1567
1534	/*	1568	/*
1535	* try_to_merge_with_ksm_page - like try_to_merge_two_pages,	1569	* try_to_merge_with_ksm_page - like try_to_merge_two_pages,
1536	* but no new kernel page is allocated: kpage must already be a ksm page.	1570	* but no new kernel page is allocated: kpage must already be a ksm page.
@@ -1625,7 +1659,6 @@ static struct folio stable_node_dup(struct ksm_stable_node *_stable_node_dup, · 88 unmodified lines
1625	struct ksm_stable_node dup, found = NULL, stable_node = _stable_node;	1659	struct ksm_stable_node dup, found = NULL, stable_node = _stable_node;
1626	struct hlist_node *hlist_safe;	1660	struct hlist_node *hlist_safe;
1627	struct folio folio, tree_folio = NULL;	1661	struct folio folio, tree_folio = NULL;
1628	int nr = 0;
1629	int found_rmap_hlist_len;	1662	int found_rmap_hlist_len;
1630		1663
1631	if (!prune_stale_stable_nodes \|\|	1664	if (!prune_stale_stable_nodes \|\|
@@ -1652,33 +1685,26 @@ static struct folio stable_node_dup(struct ksm_stable_node *_stable_node_dup, · 20 unmodified lines
1652	folio = ksm_get_folio(dup, KSM_GET_FOLIO_NOLOCK);	1685	folio = ksm_get_folio(dup, KSM_GET_FOLIO_NOLOCK);
1653	if (!folio)	1686	if (!folio)
1654	continue;	1687	continue;
1655	nr += 1;	1688	/* Pick the best candidate if possible. */
1656	if (is_page_sharing_candidate(dup)) {	1689	if (!found \|\| (is_page_sharing_candidate(dup) &&
1657	if (!found \|\|	1690	(!is_page_sharing_candidate(found) \|\|
1658	dup->rmap_hlist_len > found_rmap_hlist_len) {	1691	dup->rmap_hlist_len > found_rmap_hlist_len))) {
1659	if (found)	1692	if (found)
1660	folio_put(tree_folio);	1693	folio_put(tree_folio);
1661	found = dup;	1694	found = dup;
1662	found_rmap_hlist_len = found->rmap_hlist_len;	1695	found_rmap_hlist_len = found->rmap_hlist_len;
1663	tree_folio = folio;	1696	tree_folio = folio;
1664		1697	/* skip put_page for found candidate */
1665	/* skip put_page for found dup */	1698	if (!prune_stale_stable_nodes &&
1666	if (!prune_stale_stable_nodes)	1699	is_page_sharing_candidate(found))
1667	break;	1700	break;
1668	continue;	1701	continue;
1669	}
1670	}	1702	}
1671	folio_put(folio);	1703	folio_put(folio);
1672	}	1704	}
1673		1705
1674	if (found) {	1706	if (found) {
1675	/*	1707	if (hlist_is_singular_node(&found->hlist_dup, &stable_node->hlist)) {
1676	* nr is counting all dups in the chain only if
1677	* prune_stale_stable_nodes is true, otherwise we may
1678	* break the loop at nr == 1 even if there are
1679	* multiple entries.
1680	*/
1681	if (prune_stale_stable_nodes && nr == 1) {
1682	/*	1708	/*
1683	* If there's not just one entry it would	1709	* If there's not just one entry it would
1684	* corrupt memory, better BUG_ON. In KSM	1710	* corrupt memory, better BUG_ON. In KSM
@@ -1730,25 +1756,15 @@ static struct folio stable_node_dup(struct ksm_stable_node *_stable_node_dup, · 45 unmodified lines
1730	hlist_add_head(&found->hlist_dup,	1756	hlist_add_head(&found->hlist_dup,
1731	&stable_node->hlist);	1757	&stable_node->hlist);
1732	}	1758	}
		1759	} else {
		1760	/* Its hlist must be empty if no one found. */
		1761	free_stable_node_chain(stable_node, root);
1733	}	1762	}
1734		1763
1735	*_stable_node_dup = found;	1764	*_stable_node_dup = found;
1736	return tree_folio;	1765	return tree_folio;
1737	}	1766	}
1738		1767
1739	static struct ksm_stable_node stable_node_dup_any(struct ksm_stable_node stable_node,
1740	struct rb_root *root)
1741	{
1742	if (!is_stable_node_chain(stable_node))
1743	return stable_node;
1744	if (hlist_empty(&stable_node->hlist)) {
1745	free_stable_node_chain(stable_node, root);
1746	return NULL;
1747	}
1748	return hlist_entry(stable_node->hlist.first,
1749	typeof(*stable_node), hlist_dup);
1750	}
1751
1752	/*	1768	/*
1753	* Like for ksm_get_folio, this function can free the *_stable_node and	1769	* Like for ksm_get_folio, this function can free the *_stable_node and
1754	* *_stable_node_dup if the returned tree_page is NULL.	1770	* *_stable_node_dup if the returned tree_page is NULL.
@@ -1769,17 +1785,10 @@ static struct folio __stable_node_chain(struct ksm_stable_node *_stable_node_d · 14 unmodified lines
1769	bool prune_stale_stable_nodes)	1785	bool prune_stale_stable_nodes)
1770	{	1786	{
1771	struct ksm_stable_node stable_node = _stable_node;	1787	struct ksm_stable_node stable_node = _stable_node;
		1788
1772	if (!is_stable_node_chain(stable_node)) {	1789	if (!is_stable_node_chain(stable_node)) {
1773	if (is_page_sharing_candidate(stable_node)) {	1790	*_stable_node_dup = stable_node;
1774	*_stable_node_dup = stable_node;	1791	return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK);
1775	return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK);
1776	}
1777	/*
1778	* _stable_node_dup set to NULL means the stable_node
1779	* reached the ksm_max_page_sharing limit.
1780	*/
1781	*_stable_node_dup = NULL;
1782	return NULL;
1783	}	1792	}
1784	return stable_node_dup(_stable_node_dup, _stable_node, root,	1793	return stable_node_dup(_stable_node_dup, _stable_node, root,
1785	prune_stale_stable_nodes);	1794	prune_stale_stable_nodes);
@@ -1793,16 +1802,10 @@ static __always_inline struct folio chain_prune(struct ksm_stable_node *s_n_d, · 7 unmodified lines
1793	}	1802	}
1794		1803
1795	static __always_inline struct folio chain(struct ksm_stable_node *s_n_d,	1804	static __always_inline struct folio chain(struct ksm_stable_node *s_n_d,
1796	struct ksm_stable_node *s_n,	1805	struct ksm_stable_node **s_n,
1797	struct rb_root *root)	1806	struct rb_root *root)
1798	{	1807	{
1799	struct ksm_stable_node *old_stable_node = s_n;	1808	return __stable_node_chain(s_n_d, s_n, root, false);
1800	struct folio *tree_folio;
1801
1802	tree_folio = __stable_node_chain(s_n_d, &s_n, root, false);
1803	/* not pruning dups so s_n cannot have changed */
1804	VM_BUG_ON(s_n != old_stable_node);
1805	return tree_folio;
1806	}	1809	}
1807		1810
1808	/*	1811	/*
@@ -1820,7 +1823,7 @@ static struct page stable_tree_search(struct page page) · 11 unmodified lines
1820	struct rb_root *root;	1823	struct rb_root *root;
1821	struct rb_node **new;	1824	struct rb_node **new;
1822	struct rb_node *parent;	1825	struct rb_node *parent;
1823	struct ksm_stable_node stable_node, stable_node_dup, *stable_node_any;	1826	struct ksm_stable_node stable_node, stable_node_dup;
1824	struct ksm_stable_node *page_node;	1827	struct ksm_stable_node *page_node;
1825	struct folio *folio;	1828	struct folio *folio;
1826		1829
@@ -1844,45 +1847,7 @@ again: · 17 unmodified lines
1844		1847
1845	cond_resched();	1848	cond_resched();
1846	stable_node = rb_entry(*new, struct ksm_stable_node, node);	1849	stable_node = rb_entry(*new, struct ksm_stable_node, node);
1847	stable_node_any = NULL;
1848	tree_folio = chain_prune(&stable_node_dup, &stable_node, root);	1850	tree_folio = chain_prune(&stable_node_dup, &stable_node, root);
1849	/*
1850	* NOTE: stable_node may have been freed by
1851	* chain_prune() if the returned stable_node_dup is
1852	* not NULL. stable_node_dup may have been inserted in
1853	* the rbtree instead as a regular stable_node (in
1854	* order to collapse the stable_node chain if a single
1855	* stable_node dup was found in it). In such case the
1856	* stable_node is overwritten by the callee to point
1857	* to the stable_node_dup that was collapsed in the
1858	* stable rbtree and stable_node will be equal to
1859	* stable_node_dup like if the chain never existed.
1860	*/
1861	if (!stable_node_dup) {
1862	/*
1863	* Either all stable_node dups were full in
1864	* this stable_node chain, or this chain was
1865	* empty and should be rb_erased.
1866	*/
1867	stable_node_any = stable_node_dup_any(stable_node,
1868	root);
1869	if (!stable_node_any) {
1870	/* rb_erase just run */
1871	goto again;
1872	}
1873	/*
1874	* Take any of the stable_node dups page of
1875	* this stable_node chain to let the tree walk
1876	* continue. All KSM pages belonging to the
1877	* stable_node dups in a stable_node chain
1878	* have the same content and they're
1879	* write protected at all times. Any will work
1880	* fine to continue the walk.
1881	*/
1882	tree_folio = ksm_get_folio(stable_node_any,
1883	KSM_GET_FOLIO_NOLOCK);
1884	}
1885	VM_BUG_ON(!stable_node_dup ^ !!stable_node_any);
1886	if (!tree_folio) {	1851	if (!tree_folio) {
1887	/*	1852	/*
1888	* If we walked over a stale stable_node,	1853	* If we walked over a stale stable_node,
@@ -1920,7 +1885,7 @@ again: · 31 unmodified lines
1920	goto chain_append;	1885	goto chain_append;
1921	}	1886	}
1922		1887
1923	if (!stable_node_dup) {	1888	if (!is_page_sharing_candidate(stable_node_dup)) {
1924	/*	1889	/*
1925	* If the stable_node is a chain and	1890	* If the stable_node is a chain and
1926	* we got a payload match in memcmp	1891	* we got a payload match in memcmp
@@ -2029,9 +1994,6 @@ replace: · 102 unmodified lines
2029	return &folio->page;	1994	return &folio->page;
2030		1995
2031	chain_append:	1996	chain_append:
2032	/* stable_node_dup could be null if it reached the limit */
2033	if (!stable_node_dup)
2034	stable_node_dup = stable_node_any;
2035	/*	1997	/*
2036	* If stable_node was a chain and chain_prune collapsed it,	1998	* If stable_node was a chain and chain_prune collapsed it,
2037	* stable_node has been updated to be the new regular	1999	* stable_node has been updated to be the new regular
@@ -2076,7 +2038,7 @@ static struct ksm_stable_node stable_tree_insert(struct folio kfolio) · 38 unmodified lines
2076	struct rb_root *root;	2038	struct rb_root *root;
2077	struct rb_node **new;	2039	struct rb_node **new;
2078	struct rb_node *parent;	2040	struct rb_node *parent;
2079	struct ksm_stable_node stable_node, stable_node_dup, *stable_node_any;	2041	struct ksm_stable_node stable_node, stable_node_dup;
2080	bool need_chain = false;	2042	bool need_chain = false;
2081		2043
2082	kpfn = folio_pfn(kfolio);	2044	kpfn = folio_pfn(kfolio);
@@ -2092,33 +2054,7 @@ again: · 9 unmodified lines
2092		2054
2093	cond_resched();	2055	cond_resched();
2094	stable_node = rb_entry(*new, struct ksm_stable_node, node);	2056	stable_node = rb_entry(*new, struct ksm_stable_node, node);
2095	stable_node_any = NULL;	2057	tree_folio = chain(&stable_node_dup, &stable_node, root);
2096	tree_folio = chain(&stable_node_dup, stable_node, root);
2097	if (!stable_node_dup) {
2098	/*
2099	* Either all stable_node dups were full in
2100	* this stable_node chain, or this chain was
2101	* empty and should be rb_erased.
2102	*/
2103	stable_node_any = stable_node_dup_any(stable_node,
2104	root);
2105	if (!stable_node_any) {
2106	/* rb_erase just run */
2107	goto again;
2108	}
2109	/*
2110	* Take any of the stable_node dups page of
2111	* this stable_node chain to let the tree walk
2112	* continue. All KSM pages belonging to the
2113	* stable_node dups in a stable_node chain
2114	* have the same content and they're
2115	* write protected at all times. Any will work
2116	* fine to continue the walk.
2117	*/
2118	tree_folio = ksm_get_folio(stable_node_any,
2119	KSM_GET_FOLIO_NOLOCK);
2120	}
2121	VM_BUG_ON(!stable_node_dup ^ !!stable_node_any);
2122	if (!tree_folio) {	2058	if (!tree_folio) {
2123	/*	2059	/*
2124	* If we walked over a stale stable_node,	2060	* If we walked over a stale stable_node,
@@ -2306,7 +2242,6 @@ static void stable_tree_append(struct ksm_rmap_item *rmap_item, · 181 unmodified lines
2306	*/	2242	*/
2307	static void cmp_and_merge_page(struct page page, struct ksm_rmap_item rmap_item)	2243	static void cmp_and_merge_page(struct page page, struct ksm_rmap_item rmap_item)
2308	{	2244	{
2309	struct mm_struct *mm = rmap_item->mm;
2310	struct ksm_rmap_item *tree_rmap_item;	2245	struct ksm_rmap_item *tree_rmap_item;
2311	struct page *tree_page = NULL;	2246	struct page *tree_page = NULL;
2312	struct ksm_stable_node *stable_node;	2247	struct ksm_stable_node *stable_node;
@@ -2333,6 +2268,23 @@ static void cmp_and_merge_page(struct page page, struct ksm_rmap_item rmap_ite · 20 unmodified lines
2333	*/	2268	*/
2334	if (!is_page_sharing_candidate(stable_node))	2269	if (!is_page_sharing_candidate(stable_node))
2335	max_page_sharing_bypass = true;	2270	max_page_sharing_bypass = true;
		2271	} else {
		2272	remove_rmap_item_from_tree(rmap_item);
		2273
		2274	/*
		2275	* If the hash value of the page has changed from the last time
		2276	* we calculated it, this page is changing frequently: therefore we
		2277	* don't want to insert it in the unstable tree, and we don't want
		2278	* to waste our time searching for something identical to it there.
		2279	*/
		2280	checksum = calc_checksum(page);
		2281	if (rmap_item->oldchecksum != checksum) {
		2282	rmap_item->oldchecksum = checksum;
		2283	return;
		2284	}
		2285
		2286	if (!try_to_merge_with_zero_page(rmap_item, page))
		2287	return;
2336	}	2288	}
2337		2289
2338	/* We first start with searching the page inside the stable tree */	2290	/* We first start with searching the page inside the stable tree */
@@ -2363,48 +2315,6 @@ static void cmp_and_merge_page(struct page page, struct ksm_rmap_item rmap_ite · 24 unmodified lines
2363	return;	2315	return;
2364	}	2316	}
2365		2317
2366	/*
2367	* If the hash value of the page has changed from the last time
2368	* we calculated it, this page is changing frequently: therefore we
2369	* don't want to insert it in the unstable tree, and we don't want
2370	* to waste our time searching for something identical to it there.
2371	*/
2372	checksum = calc_checksum(page);
2373	if (rmap_item->oldchecksum != checksum) {
2374	rmap_item->oldchecksum = checksum;
2375	return;
2376	}
2377
2378	/*
2379	* Same checksum as an empty page. We attempt to merge it with the
2380	* appropriate zero page if the user enabled this via sysfs.
2381	*/
2382	if (ksm_use_zero_pages && (checksum == zero_checksum)) {
2383	struct vm_area_struct *vma;
2384
2385	mmap_read_lock(mm);
2386	vma = find_mergeable_vma(mm, rmap_item->address);
2387	if (vma) {
2388	err = try_to_merge_one_page(vma, page,
2389	ZERO_PAGE(rmap_item->address));
2390	trace_ksm_merge_one_page(
2391	page_to_pfn(ZERO_PAGE(rmap_item->address)),
2392	rmap_item, mm, err);
2393	} else {
2394	/*
2395	* If the vma is out of date, we do not need to
2396	* continue.
2397	*/
2398	err = 0;
2399	}
2400	mmap_read_unlock(mm);
2401	/*
2402	* In case of failure, the page was not really empty, so we
2403	* need to continue. Otherwise we're done.
2404	*/
2405	if (!err)
2406	return;
2407	}
2408	tree_rmap_item =	2318	tree_rmap_item =
2409	unstable_tree_search_insert(rmap_item, page, &tree_page);	2319	unstable_tree_search_insert(rmap_item, page, &tree_page);
2410	if (tree_rmap_item) {	2320	if (tree_rmap_item) {
@@ -3088,7 +2998,6 @@ struct folio ksm_might_need_to_copy(struct folio folio, · 677 unmodified lines
3088	if (copy_mc_user_highpage(folio_page(new_folio, 0), page,	2998	if (copy_mc_user_highpage(folio_page(new_folio, 0), page,
3089	addr, vma)) {	2999	addr, vma)) {
3090	folio_put(new_folio);	3000	folio_put(new_folio);
3091	memory_failure_queue(folio_pfn(folio), 0);
3092	return ERR_PTR(-EHWPOISON);	3001	return ERR_PTR(-EHWPOISON);
3093	}	3002	}
3094	folio_set_dirty(new_folio);	3003	folio_set_dirty(new_folio);

mm/list_lru.c+7−7

@@ -15,7 +15,7 @@ · 14 unmodified lines
15	#include "slab.h"	15	#include "slab.h"
16	#include "internal.h"	16	#include "internal.h"
17		17
18	#ifdef CONFIG_MEMCG_KMEM	18	#ifdef CONFIG_MEMCG
19	static LIST_HEAD(memcg_list_lrus);	19	static LIST_HEAD(memcg_list_lrus);
20	static DEFINE_MUTEX(list_lrus_mutex);	20	static DEFINE_MUTEX(list_lrus_mutex);
21		21
@@ -83,7 +83,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 61 unmodified lines
83	{	83	{
84	return &lru->node[nid].lru;	84	return &lru->node[nid].lru;
85	}	85	}
86	#endif /* CONFIG_MEMCG_KMEM */	86	#endif /* CONFIG_MEMCG */
87		87
88	bool list_lru_add(struct list_lru lru, struct list_head item, int nid,	88	bool list_lru_add(struct list_lru lru, struct list_head item, int nid,
89	struct mem_cgroup *memcg)	89	struct mem_cgroup *memcg)
@@ -294,7 +294,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, · 204 unmodified lines
294	isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg,	294	isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg,
295	nr_to_walk);	295	nr_to_walk);
296		296
297	#ifdef CONFIG_MEMCG_KMEM	297	#ifdef CONFIG_MEMCG
298	if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) {	298	if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) {
299	struct list_lru_memcg *mlru;	299	struct list_lru_memcg *mlru;
300	unsigned long index;	300	unsigned long index;
@@ -324,7 +324,7 @@ static void init_one_lru(struct list_lru_one *l) · 23 unmodified lines
324	l->nr_items = 0;	324	l->nr_items = 0;
325	}	325	}
326		326
327	#ifdef CONFIG_MEMCG_KMEM	327	#ifdef CONFIG_MEMCG
328	static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp)	328	static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp)
329	{	329	{
330	int nid;	330	int nid;
@@ -544,14 +544,14 @@ static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) · 213 unmodified lines
544	static void memcg_destroy_list_lru(struct list_lru *lru)	544	static void memcg_destroy_list_lru(struct list_lru *lru)
545	{	545	{
546	}	546	}
547	#endif /* CONFIG_MEMCG_KMEM */	547	#endif /* CONFIG_MEMCG */
548		548
549	int __list_lru_init(struct list_lru *lru, bool memcg_aware,	549	int __list_lru_init(struct list_lru *lru, bool memcg_aware,
550	struct lock_class_key key, struct shrinker shrinker)	550	struct lock_class_key key, struct shrinker shrinker)
551	{	551	{
552	int i;	552	int i;
553		553
554	#ifdef CONFIG_MEMCG_KMEM	554	#ifdef CONFIG_MEMCG
555	if (shrinker)	555	if (shrinker)
556	lru->shrinker_id = shrinker->id;	556	lru->shrinker_id = shrinker->id;
557	else	557	else
@@ -591,7 +591,7 @@ void list_lru_destroy(struct list_lru *lru) · 33 unmodified lines
591	kfree(lru->node);	591	kfree(lru->node);
592	lru->node = NULL;	592	lru->node = NULL;
593		593
594	#ifdef CONFIG_MEMCG_KMEM	594	#ifdef CONFIG_MEMCG
595	lru->shrinker_id = -1;	595	lru->shrinker_id = -1;
596	#endif	596	#endif
597	}	597	}

mm/madvise.c+1−1

@@ -1147,7 +1147,7 @@ static int madvise_inject_error(int behavior, · 1146 unmodified lines
1147	} else {	1147	} else {
1148	pr_info("Injecting memory failure for pfn %#lx at process virtual address %#lx\n",	1148	pr_info("Injecting memory failure for pfn %#lx at process virtual address %#lx\n",
1149	pfn, start);	1149	pfn, start);
1150	ret = memory_failure(pfn, MF_COUNT_INCREASED \| MF_SW_SIMULATED);	1150	ret = memory_failure(pfn, MF_ACTION_REQUIRED \| MF_COUNT_INCREASED \| MF_SW_SIMULATED);
1151	if (ret == -EOPNOTSUPP)	1151	if (ret == -EOPNOTSUPP)
1152	ret = 0;	1152	ret = 0;
1153	}	1153	}

mm/memcontrol-v1.c+2969−0

@@ -0,0 +1,2969 @@
	1	// SPDX-License-Identifier: GPL-2.0-or-later
	2
	3	#include <linux/memcontrol.h>
	4	#include <linux/swap.h>
	5	#include <linux/mm_inline.h>
	6	#include <linux/pagewalk.h>
	7	#include <linux/backing-dev.h>
	8	#include <linux/swap_cgroup.h>
	9	#include <linux/eventfd.h>
	10	#include <linux/poll.h>
	11	#include <linux/sort.h>
	12	#include <linux/file.h>
	13	#include <linux/seq_buf.h>
	14
	15	#include "internal.h"
	16	#include "swap.h"
	17	#include "memcontrol-v1.h"
	18
	19	/*
	20	* Cgroups above their limits are maintained in a RB-Tree, independent of
	21	* their hierarchy representation
	22	*/
	23
	24	struct mem_cgroup_tree_per_node {
	25	struct rb_root rb_root;
	26	struct rb_node *rb_rightmost;
	27	spinlock_t lock;
	28	};
	29
	30	struct mem_cgroup_tree {
	31	struct mem_cgroup_tree_per_node *rb_tree_per_node[MAX_NUMNODES];
	32	};
	33
	34	static struct mem_cgroup_tree soft_limit_tree __read_mostly;
	35
	36	/*
	37	* Maximum loops in mem_cgroup_soft_reclaim(), used for soft
	38	* limit reclaim to prevent infinite loops, if they ever occur.
	39	*/
	40	#define MEM_CGROUP_MAX_RECLAIM_LOOPS 100
	41	#define MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS 2
	42
	43	/* Stuffs for move charges at task migration. */
	44	/*
	45	* Types of charges to be moved.
	46	*/
	47	#define MOVE_ANON 0x1ULL
	48	#define MOVE_FILE 0x2ULL
	49	#define MOVE_MASK (MOVE_ANON \| MOVE_FILE)
	50
	51	/* "mc" and its members are protected by cgroup_mutex */
	52	static struct move_charge_struct {
	53	spinlock_t lock; /* for from, to */
	54	struct mm_struct *mm;
	55	struct mem_cgroup *from;
	56	struct mem_cgroup *to;
	57	unsigned long flags;
	58	unsigned long precharge;
	59	unsigned long moved_charge;
	60	unsigned long moved_swap;
	61	struct task_struct moving_task; / a task moving charges */
	62	wait_queue_head_t waitq; /* a waitq for other context */
	63	} mc = {
	64	.lock = __SPIN_LOCK_UNLOCKED(mc.lock),
	65	.waitq = __WAIT_QUEUE_HEAD_INITIALIZER(mc.waitq),
	66	};
	67
	68	/* for OOM */
	69	struct mem_cgroup_eventfd_list {
	70	struct list_head list;
	71	struct eventfd_ctx *eventfd;
	72	};
	73
	74	/*
	75	* cgroup_event represents events which userspace want to receive.
	76	*/
	77	struct mem_cgroup_event {
	78	/*
	79	* memcg which the event belongs to.
	80	*/
	81	struct mem_cgroup *memcg;
	82	/*
	83	* eventfd to signal userspace about the event.
	84	*/
	85	struct eventfd_ctx *eventfd;
	86	/*
	87	* Each of these stored in a list by the cgroup.
	88	*/
	89	struct list_head list;
	90	/*
	91	* register_event() callback will be used to add new userspace
	92	* waiter for changes related to this event. Use eventfd_signal()
	93	* on eventfd to send notification to userspace.
	94	*/
	95	int (register_event)(struct mem_cgroup memcg,
	96	struct eventfd_ctx eventfd, const char args);
	97	/*
	98	* unregister_event() callback will be called when userspace closes
	99	* the eventfd or on cgroup removing. This callback must be set,
	100	* if you want provide notification functionality.
	101	*/
	102	void (unregister_event)(struct mem_cgroup memcg,
	103	struct eventfd_ctx *eventfd);
	104	/*
	105	* All fields below needed to unregister event when
	106	* userspace closes eventfd.
	107	*/
	108	poll_table pt;
	109	wait_queue_head_t *wqh;
	110	wait_queue_entry_t wait;
	111	struct work_struct remove;
	112	};
	113
	114	#define MEMFILE_PRIVATE(x, val) ((x) << 16 \| (val))
	115	#define MEMFILE_TYPE(val) ((val) >> 16 & 0xffff)
	116	#define MEMFILE_ATTR(val) ((val) & 0xffff)
	117
	118	enum {
	119	RES_USAGE,
	120	RES_LIMIT,
	121	RES_MAX_USAGE,
	122	RES_FAILCNT,
	123	RES_SOFT_LIMIT,
	124	};
	125
	126	#ifdef CONFIG_LOCKDEP
	127	static struct lockdep_map memcg_oom_lock_dep_map = {
	128	.name = "memcg_oom_lock",
	129	};
	130	#endif
	131
	132	DEFINE_SPINLOCK(memcg_oom_lock);
	133
	134	static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz,
	135	struct mem_cgroup_tree_per_node *mctz,
	136	unsigned long new_usage_in_excess)
	137	{
	138	struct rb_node **p = &mctz->rb_root.rb_node;
	139	struct rb_node *parent = NULL;
	140	struct mem_cgroup_per_node *mz_node;
	141	bool rightmost = true;
	142
	143	if (mz->on_tree)
	144	return;
	145
	146	mz->usage_in_excess = new_usage_in_excess;
	147	if (!mz->usage_in_excess)
	148	return;
	149	while (*p) {
	150	parent = *p;
	151	mz_node = rb_entry(parent, struct mem_cgroup_per_node,
	152	tree_node);
	153	if (mz->usage_in_excess < mz_node->usage_in_excess) {
	154	p = &(*p)->rb_left;
	155	rightmost = false;
	156	} else {
	157	p = &(*p)->rb_right;
	158	}
	159	}
	160
	161	if (rightmost)
	162	mctz->rb_rightmost = &mz->tree_node;
	163
	164	rb_link_node(&mz->tree_node, parent, p);
	165	rb_insert_color(&mz->tree_node, &mctz->rb_root);
	166	mz->on_tree = true;
	167	}
	168
	169	static void __mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz,
	170	struct mem_cgroup_tree_per_node *mctz)
	171	{
	172	if (!mz->on_tree)
	173	return;
	174
	175	if (&mz->tree_node == mctz->rb_rightmost)
	176	mctz->rb_rightmost = rb_prev(&mz->tree_node);
	177
	178	rb_erase(&mz->tree_node, &mctz->rb_root);
	179	mz->on_tree = false;
	180	}
	181
	182	static void mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz,
	183	struct mem_cgroup_tree_per_node *mctz)
	184	{
	185	unsigned long flags;
	186
	187	spin_lock_irqsave(&mctz->lock, flags);
	188	__mem_cgroup_remove_exceeded(mz, mctz);
	189	spin_unlock_irqrestore(&mctz->lock, flags);
	190	}
	191
	192	static unsigned long soft_limit_excess(struct mem_cgroup *memcg)
	193	{
	194	unsigned long nr_pages = page_counter_read(&memcg->memory);
	195	unsigned long soft_limit = READ_ONCE(memcg->soft_limit);
	196	unsigned long excess = 0;
	197
	198	if (nr_pages > soft_limit)
	199	excess = nr_pages - soft_limit;
	200
	201	return excess;
	202	}
	203
	204	static void memcg1_update_tree(struct mem_cgroup *memcg, int nid)
	205	{
	206	unsigned long excess;
	207	struct mem_cgroup_per_node *mz;
	208	struct mem_cgroup_tree_per_node *mctz;
	209
	210	if (lru_gen_enabled()) {
	211	if (soft_limit_excess(memcg))
	212	lru_gen_soft_reclaim(memcg, nid);
	213	return;
	214	}
	215
	216	mctz = soft_limit_tree.rb_tree_per_node[nid];
	217	if (!mctz)
	218	return;
	219	/*
	220	* Necessary to update all ancestors when hierarchy is used.
	221	* because their event counter is not touched.
	222	*/
	223	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
	224	mz = memcg->nodeinfo[nid];
	225	excess = soft_limit_excess(memcg);
	226	/*
	227	* We have to update the tree if mz is on RB-tree or
	228	* mem is over its softlimit.
	229	*/
	230	if (excess \|\| mz->on_tree) {
	231	unsigned long flags;
	232
	233	spin_lock_irqsave(&mctz->lock, flags);
	234	/* if on-tree, remove it */
	235	if (mz->on_tree)
	236	__mem_cgroup_remove_exceeded(mz, mctz);
	237	/*
	238	* Insert again. mz->usage_in_excess will be updated.
	239	* If excess is 0, no tree ops.
	240	*/
	241	__mem_cgroup_insert_exceeded(mz, mctz, excess);
	242	spin_unlock_irqrestore(&mctz->lock, flags);
	243	}
	244	}
	245	}
	246
	247	void memcg1_remove_from_trees(struct mem_cgroup *memcg)
	248	{
	249	struct mem_cgroup_tree_per_node *mctz;
	250	struct mem_cgroup_per_node *mz;
	251	int nid;
	252
	253	for_each_node(nid) {
	254	mz = memcg->nodeinfo[nid];
	255	mctz = soft_limit_tree.rb_tree_per_node[nid];
	256	if (mctz)
	257	mem_cgroup_remove_exceeded(mz, mctz);
	258	}
	259	}
	260
	261	static struct mem_cgroup_per_node *
	262	__mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)
	263	{
	264	struct mem_cgroup_per_node *mz;
	265
	266	retry:
	267	mz = NULL;
	268	if (!mctz->rb_rightmost)
	269	goto done; /* Nothing to reclaim from */
	270
	271	mz = rb_entry(mctz->rb_rightmost,
	272	struct mem_cgroup_per_node, tree_node);
	273	/*
	274	* Remove the node now but someone else can add it back,
	275	* we will to add it back at the end of reclaim to its correct
	276	* position in the tree.
	277	*/
	278	__mem_cgroup_remove_exceeded(mz, mctz);
	279	if (!soft_limit_excess(mz->memcg) \|\|
	280	!css_tryget(&mz->memcg->css))
	281	goto retry;
	282	done:
	283	return mz;
	284	}
	285
	286	static struct mem_cgroup_per_node *
	287	mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)
	288	{
	289	struct mem_cgroup_per_node *mz;
	290
	291	spin_lock_irq(&mctz->lock);
	292	mz = __mem_cgroup_largest_soft_limit_node(mctz);
	293	spin_unlock_irq(&mctz->lock);
	294	return mz;
	295	}
	296
	297	static int mem_cgroup_soft_reclaim(struct mem_cgroup *root_memcg,
	298	pg_data_t *pgdat,
	299	gfp_t gfp_mask,
	300	unsigned long *total_scanned)
	301	{
	302	struct mem_cgroup *victim = NULL;
	303	int total = 0;
	304	int loop = 0;
	305	unsigned long excess;
	306	unsigned long nr_scanned;
	307	struct mem_cgroup_reclaim_cookie reclaim = {
	308	.pgdat = pgdat,
	309	};
	310
	311	excess = soft_limit_excess(root_memcg);
	312
	313	while (1) {
	314	victim = mem_cgroup_iter(root_memcg, victim, &reclaim);
	315	if (!victim) {
	316	loop++;
	317	if (loop >= 2) {
	318	/*
	319	* If we have not been able to reclaim
	320	* anything, it might because there are
	321	* no reclaimable pages under this hierarchy
	322	*/
	323	if (!total)
	324	break;
	325	/*
	326	* We want to do more targeted reclaim.
	327	* excess >> 2 is not to excessive so as to
	328	* reclaim too much, nor too less that we keep
	329	* coming back to reclaim from this cgroup
	330	*/
	331	if (total >= (excess >> 2) \|\|
	332	(loop > MEM_CGROUP_MAX_RECLAIM_LOOPS))
	333	break;
	334	}
	335	continue;
	336	}
	337	total += mem_cgroup_shrink_node(victim, gfp_mask, false,
	338	pgdat, &nr_scanned);
	339	*total_scanned += nr_scanned;
	340	if (!soft_limit_excess(root_memcg))
	341	break;
	342	}
	343	mem_cgroup_iter_break(root_memcg, victim);
	344	return total;
	345	}
	346
	347	unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
	348	gfp_t gfp_mask,
	349	unsigned long *total_scanned)
	350	{
	351	unsigned long nr_reclaimed = 0;
	352	struct mem_cgroup_per_node mz, next_mz = NULL;
	353	unsigned long reclaimed;
	354	int loop = 0;
	355	struct mem_cgroup_tree_per_node *mctz;
	356	unsigned long excess;
	357
	358	if (lru_gen_enabled())
	359	return 0;
	360
	361	if (order > 0)
	362	return 0;
	363
	364	mctz = soft_limit_tree.rb_tree_per_node[pgdat->node_id];
	365
	366	/*
	367	* Do not even bother to check the largest node if the root
	368	* is empty. Do it lockless to prevent lock bouncing. Races
	369	* are acceptable as soft limit is best effort anyway.
	370	*/
	371	if (!mctz \|\| RB_EMPTY_ROOT(&mctz->rb_root))
	372	return 0;
	373
	374	/*
	375	* This loop can run a while, specially if mem_cgroup's continuously
	376	* keep exceeding their soft limit and putting the system under
	377	* pressure
	378	*/
	379	do {
	380	if (next_mz)
	381	mz = next_mz;
	382	else
	383	mz = mem_cgroup_largest_soft_limit_node(mctz);
	384	if (!mz)
	385	break;
	386
	387	reclaimed = mem_cgroup_soft_reclaim(mz->memcg, pgdat,
	388	gfp_mask, total_scanned);
	389	nr_reclaimed += reclaimed;
	390	spin_lock_irq(&mctz->lock);
	391
	392	/*
	393	* If we failed to reclaim anything from this memory cgroup
	394	* it is time to move on to the next cgroup
	395	*/
	396	next_mz = NULL;
	397	if (!reclaimed)
	398	next_mz = __mem_cgroup_largest_soft_limit_node(mctz);
	399
	400	excess = soft_limit_excess(mz->memcg);
	401	/*
	402	* One school of thought says that we should not add
	403	* back the node to the tree if reclaim returns 0.
	404	* But our reclaim could return 0, simply because due
	405	* to priority we are exposing a smaller subset of
	406	* memory to reclaim from. Consider this as a longer
	407	* term TODO.
	408	*/
	409	/* If excess == 0, no tree ops */
	410	__mem_cgroup_insert_exceeded(mz, mctz, excess);
	411	spin_unlock_irq(&mctz->lock);
	412	css_put(&mz->memcg->css);
	413	loop++;
	414	/*
	415	* Could not reclaim anything and there are no more
	416	* mem cgroups to try or we seem to be looping without
	417	* reclaiming anything.
	418	*/
	419	if (!nr_reclaimed &&
	420	(next_mz == NULL \|\|
	421	loop > MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS))
	422	break;
	423	} while (!nr_reclaimed);
	424	if (next_mz)
	425	css_put(&next_mz->memcg->css);
	426	return nr_reclaimed;
	427	}
	428
	429	/*
	430	* A routine for checking "mem" is under move_account() or not.
	431	*
	432	* Checking a cgroup is mc.from or mc.to or under hierarchy of
	433	* moving cgroups. This is for waiting at high-memory pressure
	434	* caused by "move".
	435	*/
	436	static bool mem_cgroup_under_move(struct mem_cgroup *memcg)
	437	{
	438	struct mem_cgroup *from;
	439	struct mem_cgroup *to;
	440	bool ret = false;
	441	/*
	442	* Unlike task_move routines, we access mc.to, mc.from not under
	443	* mutual exclusion by cgroup_mutex. Here, we take spinlock instead.
	444	*/
	445	spin_lock(&mc.lock);
	446	from = mc.from;
	447	to = mc.to;
	448	if (!from)
	449	goto unlock;
	450
	451	ret = mem_cgroup_is_descendant(from, memcg) \|\|
	452	mem_cgroup_is_descendant(to, memcg);
	453	unlock:
	454	spin_unlock(&mc.lock);
	455	return ret;
	456	}
	457
	458	bool memcg1_wait_acct_move(struct mem_cgroup *memcg)
	459	{
	460	if (mc.moving_task && current != mc.moving_task) {
	461	if (mem_cgroup_under_move(memcg)) {
	462	DEFINE_WAIT(wait);
	463	prepare_to_wait(&mc.waitq, &wait, TASK_INTERRUPTIBLE);
	464	/* moving charge context might have finished. */
	465	if (mc.moving_task)
	466	schedule();
	467	finish_wait(&mc.waitq, &wait);
	468	return true;
	469	}
	470	}
	471	return false;
	472	}
	473
	474	/**
	475	* folio_memcg_lock - Bind a folio to its memcg.
	476	* @folio: The folio.
	477	*
	478	* This function prevents unlocked LRU folios from being moved to
	479	* another cgroup.
	480	*
	481	* It ensures lifetime of the bound memcg. The caller is responsible
	482	* for the lifetime of the folio.
	483	*/
	484	void folio_memcg_lock(struct folio *folio)
	485	{
	486	struct mem_cgroup *memcg;
	487	unsigned long flags;
	488
	489	/*
	490	* The RCU lock is held throughout the transaction. The fast
	491	* path can get away without acquiring the memcg->move_lock
	492	* because page moving starts with an RCU grace period.
	493	*/
	494	rcu_read_lock();
	495
	496	if (mem_cgroup_disabled())
	497	return;
	498	again:
	499	memcg = folio_memcg(folio);
	500	if (unlikely(!memcg))
	501	return;
	502
	503	#ifdef CONFIG_PROVE_LOCKING
	504	local_irq_save(flags);
	505	might_lock(&memcg->move_lock);
	506	local_irq_restore(flags);
	507	#endif
	508
	509	if (atomic_read(&memcg->moving_account) <= 0)
	510	return;
	511
	512	spin_lock_irqsave(&memcg->move_lock, flags);
	513	if (memcg != folio_memcg(folio)) {
	514	spin_unlock_irqrestore(&memcg->move_lock, flags);
	515	goto again;
	516	}
	517
	518	/*
	519	* When charge migration first begins, we can have multiple
	520	* critical sections holding the fast-path RCU lock and one
	521	* holding the slowpath move_lock. Track the task who has the
	522	* move_lock for folio_memcg_unlock().
	523	*/
	524	memcg->move_lock_task = current;
	525	memcg->move_lock_flags = flags;
	526	}
	527
	528	static void __folio_memcg_unlock(struct mem_cgroup *memcg)
	529	{
	530	if (memcg && memcg->move_lock_task == current) {
	531	unsigned long flags = memcg->move_lock_flags;
	532
	533	memcg->move_lock_task = NULL;
	534	memcg->move_lock_flags = 0;
	535
	536	spin_unlock_irqrestore(&memcg->move_lock, flags);
	537	}
	538
	539	rcu_read_unlock();
	540	}
	541
	542	/**
	543	* folio_memcg_unlock - Release the binding between a folio and its memcg.
	544	* @folio: The folio.
	545	*
	546	* This releases the binding created by folio_memcg_lock(). This does
	547	* not change the accounting of this folio to its memcg, but it does
	548	* permit others to change it.
	549	*/
	550	void folio_memcg_unlock(struct folio *folio)
	551	{
	552	__folio_memcg_unlock(folio_memcg(folio));
	553	}
	554
	555	#ifdef CONFIG_SWAP
	556	/**
	557	* mem_cgroup_move_swap_account - move swap charge and swap_cgroup's record.
	558	* @entry: swap entry to be moved
	559	* @from: mem_cgroup which the entry is moved from
	560	* @to: mem_cgroup which the entry is moved to
	561	*
	562	* It succeeds only when the swap_cgroup's record for this entry is the same
	563	* as the mem_cgroup's id of @from.
	564	*
	565	* Returns 0 on success, -EINVAL on failure.
	566	*
	567	* The caller must have charged to @to, IOW, called page_counter_charge() about
	568	* both res and memsw, and called css_get().
	569	*/
	570	static int mem_cgroup_move_swap_account(swp_entry_t entry,
	571	struct mem_cgroup from, struct mem_cgroup to)
	572	{
	573	unsigned short old_id, new_id;
	574
	575	old_id = mem_cgroup_id(from);
	576	new_id = mem_cgroup_id(to);
	577
	578	if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) {
	579	mod_memcg_state(from, MEMCG_SWAP, -1);
	580	mod_memcg_state(to, MEMCG_SWAP, 1);
	581	return 0;
	582	}
	583	return -EINVAL;
	584	}
	585	#else
	586	static inline int mem_cgroup_move_swap_account(swp_entry_t entry,
	587	struct mem_cgroup from, struct mem_cgroup to)
	588	{
	589	return -EINVAL;
	590	}
	591	#endif
	592
	593	static u64 mem_cgroup_move_charge_read(struct cgroup_subsys_state *css,
	594	struct cftype *cft)
	595	{
	596	return mem_cgroup_from_css(css)->move_charge_at_immigrate;
	597	}
	598
	599	#ifdef CONFIG_MMU
	600	static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css,
	601	struct cftype *cft, u64 val)
	602	{
	603	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	604
	605	pr_warn_once("Cgroup memory moving (move_charge_at_immigrate) is deprecated. "
	606	"Please report your usecase to linux-mm@kvack.org if you "
	607	"depend on this functionality.\n");
	608
	609	if (val & ~MOVE_MASK)
	610	return -EINVAL;
	611
	612	/*
	613	* No kind of locking is needed in here, because ->can_attach() will
	614	* check this value once in the beginning of the process, and then carry
	615	* on with stale data. This means that changes to this value will only
	616	* affect task migrations starting after the change.
	617	*/
	618	memcg->move_charge_at_immigrate = val;
	619	return 0;
	620	}
	621	#else
	622	static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css,
	623	struct cftype *cft, u64 val)
	624	{
	625	return -ENOSYS;
	626	}
	627	#endif
	628
	629	#ifdef CONFIG_MMU
	630	/* Handlers for move charge at task migration. */
	631	static int mem_cgroup_do_precharge(unsigned long count)
	632	{
	633	int ret;
	634
	635	/* Try a single bulk charge without reclaim first, kswapd may wake */
	636	ret = try_charge(mc.to, GFP_KERNEL & ~__GFP_DIRECT_RECLAIM, count);
	637	if (!ret) {
	638	mc.precharge += count;
	639	return ret;
	640	}
	641
	642	/* Try charges one by one with reclaim, but do not retry */
	643	while (count--) {
	644	ret = try_charge(mc.to, GFP_KERNEL \| __GFP_NORETRY, 1);
	645	if (ret)
	646	return ret;
	647	mc.precharge++;
	648	cond_resched();
	649	}
	650	return 0;
	651	}
	652
	653	union mc_target {
	654	struct folio *folio;
	655	swp_entry_t ent;
	656	};
	657
	658	enum mc_target_type {
	659	MC_TARGET_NONE = 0,
	660	MC_TARGET_PAGE,
	661	MC_TARGET_SWAP,
	662	MC_TARGET_DEVICE,
	663	};
	664
	665	static struct page mc_handle_present_pte(struct vm_area_struct vma,
	666	unsigned long addr, pte_t ptent)
	667	{
	668	struct page *page = vm_normal_page(vma, addr, ptent);
	669
	670	if (!page)
	671	return NULL;
	672	if (PageAnon(page)) {
	673	if (!(mc.flags & MOVE_ANON))
	674	return NULL;
	675	} else {
	676	if (!(mc.flags & MOVE_FILE))
	677	return NULL;
	678	}
	679	get_page(page);
	680
	681	return page;
	682	}
	683
	684	#if defined(CONFIG_SWAP) \|\| defined(CONFIG_DEVICE_PRIVATE)
	685	static struct page mc_handle_swap_pte(struct vm_area_struct vma,
	686	pte_t ptent, swp_entry_t *entry)
	687	{
	688	struct page *page = NULL;
	689	swp_entry_t ent = pte_to_swp_entry(ptent);
	690
	691	if (!(mc.flags & MOVE_ANON))
	692	return NULL;
	693
	694	/*
	695	* Handle device private pages that are not accessible by the CPU, but
	696	* stored as special swap entries in the page table.
	697	*/
	698	if (is_device_private_entry(ent)) {
	699	page = pfn_swap_entry_to_page(ent);
	700	if (!get_page_unless_zero(page))
	701	return NULL;
	702	return page;
	703	}
	704
	705	if (non_swap_entry(ent))
	706	return NULL;
	707
	708	/*
	709	* Because swap_cache_get_folio() updates some statistics counter,
	710	* we call find_get_page() with swapper_space directly.
	711	*/
	712	page = find_get_page(swap_address_space(ent), swap_cache_index(ent));
	713	entry->val = ent.val;
	714
	715	return page;
	716	}
	717	#else
	718	static struct page mc_handle_swap_pte(struct vm_area_struct vma,
	719	pte_t ptent, swp_entry_t *entry)
	720	{
	721	return NULL;
	722	}
	723	#endif
	724
	725	static struct page mc_handle_file_pte(struct vm_area_struct vma,
	726	unsigned long addr, pte_t ptent)
	727	{
	728	unsigned long index;
	729	struct folio *folio;
	730
	731	if (!vma->vm_file) /* anonymous vma */
	732	return NULL;
	733	if (!(mc.flags & MOVE_FILE))
	734	return NULL;
	735
	736	/* folio is moved even if it's not RSS of this task(page-faulted). */
	737	/* shmem/tmpfs may report page out on swap: account for that too. */
	738	index = linear_page_index(vma, addr);
	739	folio = filemap_get_incore_folio(vma->vm_file->f_mapping, index);
	740	if (IS_ERR(folio))
	741	return NULL;
	742	return folio_file_page(folio, index);
	743	}
	744
	745	/**
	746	* mem_cgroup_move_account - move account of the folio
	747	* @folio: The folio.
	748	* @compound: charge the page as compound or small page
	749	* @from: mem_cgroup which the folio is moved from.
	750	* @to: mem_cgroup which the folio is moved to. @from != @to.
	751	*
	752	* The folio must be locked and not on the LRU.
	753	*
	754	* This function doesn't do "charge" to new cgroup and doesn't do "uncharge"
	755	* from old cgroup.
	756	*/
	757	static int mem_cgroup_move_account(struct folio *folio,
	758	bool compound,
	759	struct mem_cgroup *from,
	760	struct mem_cgroup *to)
	761	{
	762	struct lruvec from_vec, to_vec;
	763	struct pglist_data *pgdat;
	764	unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1;
	765	int nid, ret;
	766
	767	VM_BUG_ON(from == to);
	768	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
	769	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
	770	VM_BUG_ON(compound && !folio_test_large(folio));
	771
	772	ret = -EINVAL;
	773	if (folio_memcg(folio) != from)
	774	goto out;
	775
	776	pgdat = folio_pgdat(folio);
	777	from_vec = mem_cgroup_lruvec(from, pgdat);
	778	to_vec = mem_cgroup_lruvec(to, pgdat);
	779
	780	folio_memcg_lock(folio);
	781
	782	if (folio_test_anon(folio)) {
	783	if (folio_mapped(folio)) {
	784	__mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages);
	785	__mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages);
	786	if (folio_test_pmd_mappable(folio)) {
	787	__mod_lruvec_state(from_vec, NR_ANON_THPS,
	788	-nr_pages);
	789	__mod_lruvec_state(to_vec, NR_ANON_THPS,
	790	nr_pages);
	791	}
	792	}
	793	} else {
	794	__mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages);
	795	__mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages);
	796
	797	if (folio_test_swapbacked(folio)) {
	798	__mod_lruvec_state(from_vec, NR_SHMEM, -nr_pages);
	799	__mod_lruvec_state(to_vec, NR_SHMEM, nr_pages);
	800	}
	801
	802	if (folio_mapped(folio)) {
	803	__mod_lruvec_state(from_vec, NR_FILE_MAPPED, -nr_pages);
	804	__mod_lruvec_state(to_vec, NR_FILE_MAPPED, nr_pages);
	805	}
	806
	807	if (folio_test_dirty(folio)) {
	808	struct address_space *mapping = folio_mapping(folio);
	809
	810	if (mapping_can_writeback(mapping)) {
	811	__mod_lruvec_state(from_vec, NR_FILE_DIRTY,
	812	-nr_pages);
	813	__mod_lruvec_state(to_vec, NR_FILE_DIRTY,
	814	nr_pages);
	815	}
	816	}
	817	}
	818
	819	#ifdef CONFIG_SWAP
	820	if (folio_test_swapcache(folio)) {
	821	__mod_lruvec_state(from_vec, NR_SWAPCACHE, -nr_pages);
	822	__mod_lruvec_state(to_vec, NR_SWAPCACHE, nr_pages);
	823	}
	824	#endif
	825	if (folio_test_writeback(folio)) {
	826	__mod_lruvec_state(from_vec, NR_WRITEBACK, -nr_pages);
	827	__mod_lruvec_state(to_vec, NR_WRITEBACK, nr_pages);
	828	}
	829
	830	/*
	831	* All state has been migrated, let's switch to the new memcg.
	832	*
	833	* It is safe to change page's memcg here because the page
	834	* is referenced, charged, isolated, and locked: we can't race
	835	* with (un)charging, migration, LRU putback, or anything else
	836	* that would rely on a stable page's memory cgroup.
	837	*
	838	* Note that folio_memcg_lock is a memcg lock, not a page lock,
	839	* to save space. As soon as we switch page's memory cgroup to a
	840	* new memcg that isn't locked, the above state can change
	841	* concurrently again. Make sure we're truly done with it.
	842	*/
	843	smp_mb();
	844
	845	css_get(&to->css);
	846	css_put(&from->css);
	847
	848	folio->memcg_data = (unsigned long)to;
	849
	850	__folio_memcg_unlock(from);
	851
	852	ret = 0;
	853	nid = folio_nid(folio);
	854
	855	local_irq_disable();
	856	mem_cgroup_charge_statistics(to, nr_pages);
	857	memcg1_check_events(to, nid);
	858	mem_cgroup_charge_statistics(from, -nr_pages);
	859	memcg1_check_events(from, nid);
	860	local_irq_enable();
	861	out:
	862	return ret;
	863	}
	864
	865	/**
	866	* get_mctgt_type - get target type of moving charge
	867	* @vma: the vma the pte to be checked belongs
	868	* @addr: the address corresponding to the pte to be checked
	869	* @ptent: the pte to be checked
	870	* @target: the pointer the target page or swap ent will be stored(can be NULL)
	871	*
	872	* Context: Called with pte lock held.
	873	* Return:
	874	* * MC_TARGET_NONE - If the pte is not a target for move charge.
	875	* * MC_TARGET_PAGE - If the page corresponding to this pte is a target for
	876	* move charge. If @target is not NULL, the folio is stored in target->folio
	877	* with extra refcnt taken (Caller should release it).
	878	* * MC_TARGET_SWAP - If the swap entry corresponding to this pte is a
	879	* target for charge migration. If @target is not NULL, the entry is
	880	* stored in target->ent.
	881	* * MC_TARGET_DEVICE - Like MC_TARGET_PAGE but page is device memory and
	882	* thus not on the lru. For now such page is charged like a regular page
	883	* would be as it is just special memory taking the place of a regular page.
	884	* See Documentations/vm/hmm.txt and include/linux/hmm.h
	885	*/
	886	static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma,
	887	unsigned long addr, pte_t ptent, union mc_target *target)
	888	{
	889	struct page *page = NULL;
	890	struct folio *folio;
	891	enum mc_target_type ret = MC_TARGET_NONE;
	892	swp_entry_t ent = { .val = 0 };
	893
	894	if (pte_present(ptent))
	895	page = mc_handle_present_pte(vma, addr, ptent);
	896	else if (pte_none_mostly(ptent))
	897	/*
	898	* PTE markers should be treated as a none pte here, separated
	899	* from other swap handling below.
	900	*/
	901	page = mc_handle_file_pte(vma, addr, ptent);
	902	else if (is_swap_pte(ptent))
	903	page = mc_handle_swap_pte(vma, ptent, &ent);
	904
	905	if (page)
	906	folio = page_folio(page);
	907	if (target && page) {
	908	if (!folio_trylock(folio)) {
	909	folio_put(folio);
	910	return ret;
	911	}
	912	/*
	913	* page_mapped() must be stable during the move. This
	914	* pte is locked, so if it's present, the page cannot
	915	* become unmapped. If it isn't, we have only partial
	916	* control over the mapped state: the page lock will
	917	* prevent new faults against pagecache and swapcache,
	918	* so an unmapped page cannot become mapped. However,
	919	* if the page is already mapped elsewhere, it can
	920	* unmap, and there is nothing we can do about it.
	921	* Alas, skip moving the page in this case.
	922	*/
	923	if (!pte_present(ptent) && page_mapped(page)) {
	924	folio_unlock(folio);
	925	folio_put(folio);
	926	return ret;
	927	}
	928	}
	929
	930	if (!page && !ent.val)
	931	return ret;
	932	if (page) {
	933	/*
	934	* Do only loose check w/o serialization.
	935	* mem_cgroup_move_account() checks the page is valid or
	936	* not under LRU exclusion.
	937	*/
	938	if (folio_memcg(folio) == mc.from) {
	939	ret = MC_TARGET_PAGE;
	940	if (folio_is_device_private(folio) \|\|
	941	folio_is_device_coherent(folio))
	942	ret = MC_TARGET_DEVICE;
	943	if (target)
	944	target->folio = folio;
	945	}
	946	if (!ret \|\| !target) {
	947	if (target)
	948	folio_unlock(folio);
	949	folio_put(folio);
	950	}
	951	}
	952	/*
	953	* There is a swap entry and a page doesn't exist or isn't charged.
	954	* But we cannot move a tail-page in a THP.
	955	*/
	956	if (ent.val && !ret && (!page \|\| !PageTransCompound(page)) &&
	957	mem_cgroup_id(mc.from) == lookup_swap_cgroup_id(ent)) {
	958	ret = MC_TARGET_SWAP;
	959	if (target)
	960	target->ent = ent;
	961	}
	962	return ret;
	963	}
	964
	965	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	966	/*
	967	* We don't consider PMD mapped swapping or file mapped pages because THP does
	968	* not support them for now.
	969	* Caller should make sure that pmd_trans_huge(pmd) is true.
	970	*/
	971	static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
	972	unsigned long addr, pmd_t pmd, union mc_target *target)
	973	{
	974	struct page *page = NULL;
	975	struct folio *folio;
	976	enum mc_target_type ret = MC_TARGET_NONE;
	977
	978	if (unlikely(is_swap_pmd(pmd))) {
	979	VM_BUG_ON(thp_migration_supported() &&
	980	!is_pmd_migration_entry(pmd));
	981	return ret;
	982	}
	983	page = pmd_page(pmd);
	984	VM_BUG_ON_PAGE(!page \|\| !PageHead(page), page);
	985	folio = page_folio(page);
	986	if (!(mc.flags & MOVE_ANON))
	987	return ret;
	988	if (folio_memcg(folio) == mc.from) {
	989	ret = MC_TARGET_PAGE;
	990	if (target) {
	991	folio_get(folio);
	992	if (!folio_trylock(folio)) {
	993	folio_put(folio);
	994	return MC_TARGET_NONE;
	995	}
	996	target->folio = folio;
	997	}
	998	}
	999	return ret;
	1000	}
	1001	#else
	1002	static inline enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
	1003	unsigned long addr, pmd_t pmd, union mc_target *target)
	1004	{
	1005	return MC_TARGET_NONE;
	1006	}
	1007	#endif
	1008
	1009	static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd,
	1010	unsigned long addr, unsigned long end,
	1011	struct mm_walk *walk)
	1012	{
	1013	struct vm_area_struct *vma = walk->vma;
	1014	pte_t *pte;
	1015	spinlock_t *ptl;
	1016
	1017	ptl = pmd_trans_huge_lock(pmd, vma);
	1018	if (ptl) {
	1019	/*
	1020	* Note their can not be MC_TARGET_DEVICE for now as we do not
	1021	* support transparent huge page with MEMORY_DEVICE_PRIVATE but
	1022	* this might change.
	1023	*/
	1024	if (get_mctgt_type_thp(vma, addr, *pmd, NULL) == MC_TARGET_PAGE)
	1025	mc.precharge += HPAGE_PMD_NR;
	1026	spin_unlock(ptl);
	1027	return 0;
	1028	}
	1029
	1030	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
	1031	if (!pte)
	1032	return 0;
	1033	for (; addr != end; pte++, addr += PAGE_SIZE)
	1034	if (get_mctgt_type(vma, addr, ptep_get(pte), NULL))
	1035	mc.precharge++; /* increment precharge temporarily */
	1036	pte_unmap_unlock(pte - 1, ptl);
	1037	cond_resched();
	1038
	1039	return 0;
	1040	}
	1041
	1042	static const struct mm_walk_ops precharge_walk_ops = {
	1043	.pmd_entry = mem_cgroup_count_precharge_pte_range,
	1044	.walk_lock = PGWALK_RDLOCK,
	1045	};
	1046
	1047	static unsigned long mem_cgroup_count_precharge(struct mm_struct *mm)
	1048	{
	1049	unsigned long precharge;
	1050
	1051	mmap_read_lock(mm);
	1052	walk_page_range(mm, 0, ULONG_MAX, &precharge_walk_ops, NULL);
	1053	mmap_read_unlock(mm);
	1054
	1055	precharge = mc.precharge;
	1056	mc.precharge = 0;
	1057
	1058	return precharge;
	1059	}
	1060
	1061	static int mem_cgroup_precharge_mc(struct mm_struct *mm)
	1062	{
	1063	unsigned long precharge = mem_cgroup_count_precharge(mm);
	1064
	1065	VM_BUG_ON(mc.moving_task);
	1066	mc.moving_task = current;
	1067	return mem_cgroup_do_precharge(precharge);
	1068	}
	1069
	1070	/* cancels all extra charges on mc.from and mc.to, and wakes up all waiters. */
	1071	static void __mem_cgroup_clear_mc(void)
	1072	{
	1073	struct mem_cgroup *from = mc.from;
	1074	struct mem_cgroup *to = mc.to;
	1075
	1076	/* we must uncharge all the leftover precharges from mc.to */
	1077	if (mc.precharge) {
	1078	mem_cgroup_cancel_charge(mc.to, mc.precharge);
	1079	mc.precharge = 0;
	1080	}
	1081	/*
	1082	* we didn't uncharge from mc.from at mem_cgroup_move_account(), so
	1083	* we must uncharge here.
	1084	*/
	1085	if (mc.moved_charge) {
	1086	mem_cgroup_cancel_charge(mc.from, mc.moved_charge);
	1087	mc.moved_charge = 0;
	1088	}
	1089	/* we must fixup refcnts and charges */
	1090	if (mc.moved_swap) {
	1091	/* uncharge swap account from the old cgroup */
	1092	if (!mem_cgroup_is_root(mc.from))
	1093	page_counter_uncharge(&mc.from->memsw, mc.moved_swap);
	1094
	1095	mem_cgroup_id_put_many(mc.from, mc.moved_swap);
	1096
	1097	/*
	1098	* we charged both to->memory and to->memsw, so we
	1099	* should uncharge to->memory.
	1100	*/
	1101	if (!mem_cgroup_is_root(mc.to))
	1102	page_counter_uncharge(&mc.to->memory, mc.moved_swap);
	1103
	1104	mc.moved_swap = 0;
	1105	}
	1106	memcg1_oom_recover(from);
	1107	memcg1_oom_recover(to);
	1108	wake_up_all(&mc.waitq);
	1109	}
	1110
	1111	static void mem_cgroup_clear_mc(void)
	1112	{
	1113	struct mm_struct *mm = mc.mm;
	1114
	1115	/*
	1116	* we must clear moving_task before waking up waiters at the end of
	1117	* task migration.
	1118	*/
	1119	mc.moving_task = NULL;
	1120	__mem_cgroup_clear_mc();
	1121	spin_lock(&mc.lock);
	1122	mc.from = NULL;
	1123	mc.to = NULL;
	1124	mc.mm = NULL;
	1125	spin_unlock(&mc.lock);
	1126
	1127	mmput(mm);
	1128	}
	1129
	1130	int memcg1_can_attach(struct cgroup_taskset *tset)
	1131	{
	1132	struct cgroup_subsys_state *css;
	1133	struct mem_cgroup memcg = NULL; / unneeded init to make gcc happy */
	1134	struct mem_cgroup *from;
	1135	struct task_struct leader, p;
	1136	struct mm_struct *mm;
	1137	unsigned long move_flags;
	1138	int ret = 0;
	1139
	1140	/* charge immigration isn't supported on the default hierarchy */
	1141	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
	1142	return 0;
	1143
	1144	/*
	1145	* Multi-process migrations only happen on the default hierarchy
	1146	* where charge immigration is not used. Perform charge
	1147	* immigration if @tset contains a leader and whine if there are
	1148	* multiple.
	1149	*/
	1150	p = NULL;
	1151	cgroup_taskset_for_each_leader(leader, css, tset) {
	1152	WARN_ON_ONCE(p);
	1153	p = leader;
	1154	memcg = mem_cgroup_from_css(css);
	1155	}
	1156	if (!p)
	1157	return 0;
	1158
	1159	/*
	1160	* We are now committed to this value whatever it is. Changes in this
	1161	* tunable will only affect upcoming migrations, not the current one.
	1162	* So we need to save it, and keep it going.
	1163	*/
	1164	move_flags = READ_ONCE(memcg->move_charge_at_immigrate);
	1165	if (!move_flags)
	1166	return 0;
	1167
	1168	from = mem_cgroup_from_task(p);
	1169
	1170	VM_BUG_ON(from == memcg);
	1171
	1172	mm = get_task_mm(p);
	1173	if (!mm)
	1174	return 0;
	1175	/* We move charges only when we move a owner of the mm */
	1176	if (mm->owner == p) {
	1177	VM_BUG_ON(mc.from);
	1178	VM_BUG_ON(mc.to);
	1179	VM_BUG_ON(mc.precharge);
	1180	VM_BUG_ON(mc.moved_charge);
	1181	VM_BUG_ON(mc.moved_swap);
	1182
	1183	spin_lock(&mc.lock);
	1184	mc.mm = mm;
	1185	mc.from = from;
	1186	mc.to = memcg;
	1187	mc.flags = move_flags;
	1188	spin_unlock(&mc.lock);
	1189	/* We set mc.moving_task later */
	1190
	1191	ret = mem_cgroup_precharge_mc(mm);
	1192	if (ret)
	1193	mem_cgroup_clear_mc();
	1194	} else {
	1195	mmput(mm);
	1196	}
	1197	return ret;
	1198	}
	1199
	1200	void memcg1_cancel_attach(struct cgroup_taskset *tset)
	1201	{
	1202	if (mc.to)
	1203	mem_cgroup_clear_mc();
	1204	}
	1205
	1206	static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,
	1207	unsigned long addr, unsigned long end,
	1208	struct mm_walk *walk)
	1209	{
	1210	int ret = 0;
	1211	struct vm_area_struct *vma = walk->vma;
	1212	pte_t *pte;
	1213	spinlock_t *ptl;
	1214	enum mc_target_type target_type;
	1215	union mc_target target;
	1216	struct folio *folio;
	1217
	1218	ptl = pmd_trans_huge_lock(pmd, vma);
	1219	if (ptl) {
	1220	if (mc.precharge < HPAGE_PMD_NR) {
	1221	spin_unlock(ptl);
	1222	return 0;
	1223	}
	1224	target_type = get_mctgt_type_thp(vma, addr, *pmd, &target);
	1225	if (target_type == MC_TARGET_PAGE) {
	1226	folio = target.folio;
	1227	if (folio_isolate_lru(folio)) {
	1228	if (!mem_cgroup_move_account(folio, true,
	1229	mc.from, mc.to)) {
	1230	mc.precharge -= HPAGE_PMD_NR;
	1231	mc.moved_charge += HPAGE_PMD_NR;
	1232	}
	1233	folio_putback_lru(folio);
	1234	}
	1235	folio_unlock(folio);
	1236	folio_put(folio);
	1237	} else if (target_type == MC_TARGET_DEVICE) {
	1238	folio = target.folio;
	1239	if (!mem_cgroup_move_account(folio, true,
	1240	mc.from, mc.to)) {
	1241	mc.precharge -= HPAGE_PMD_NR;
	1242	mc.moved_charge += HPAGE_PMD_NR;
	1243	}
	1244	folio_unlock(folio);
	1245	folio_put(folio);
	1246	}
	1247	spin_unlock(ptl);
	1248	return 0;
	1249	}
	1250
	1251	retry:
	1252	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
	1253	if (!pte)
	1254	return 0;
	1255	for (; addr != end; addr += PAGE_SIZE) {
	1256	pte_t ptent = ptep_get(pte++);
	1257	bool device = false;
	1258	swp_entry_t ent;
	1259
	1260	if (!mc.precharge)
	1261	break;
	1262
	1263	switch (get_mctgt_type(vma, addr, ptent, &target)) {
	1264	case MC_TARGET_DEVICE:
	1265	device = true;
	1266	fallthrough;
	1267	case MC_TARGET_PAGE:
	1268	folio = target.folio;
	1269	/*
	1270	* We can have a part of the split pmd here. Moving it
	1271	* can be done but it would be too convoluted so simply
	1272	* ignore such a partial THP and keep it in original
	1273	* memcg. There should be somebody mapping the head.
	1274	*/
	1275	if (folio_test_large(folio))
	1276	goto put;
	1277	if (!device && !folio_isolate_lru(folio))
	1278	goto put;
	1279	if (!mem_cgroup_move_account(folio, false,
	1280	mc.from, mc.to)) {
	1281	mc.precharge--;
	1282	/* we uncharge from mc.from later. */
	1283	mc.moved_charge++;
	1284	}
	1285	if (!device)
	1286	folio_putback_lru(folio);
	1287	put: /* get_mctgt_type() gets & locks the page */
	1288	folio_unlock(folio);
	1289	folio_put(folio);
	1290	break;
	1291	case MC_TARGET_SWAP:
	1292	ent = target.ent;
	1293	if (!mem_cgroup_move_swap_account(ent, mc.from, mc.to)) {
	1294	mc.precharge--;
	1295	mem_cgroup_id_get_many(mc.to, 1);
	1296	/* we fixup other refcnts and charges later. */
	1297	mc.moved_swap++;
	1298	}
	1299	break;
	1300	default:
	1301	break;
	1302	}
	1303	}
	1304	pte_unmap_unlock(pte - 1, ptl);
	1305	cond_resched();
	1306
	1307	if (addr != end) {
	1308	/*
	1309	* We have consumed all precharges we got in can_attach().
	1310	* We try charge one by one, but don't do any additional
	1311	* charges to mc.to if we have failed in charge once in attach()
	1312	* phase.
	1313	*/
	1314	ret = mem_cgroup_do_precharge(1);
	1315	if (!ret)
	1316	goto retry;
	1317	}
	1318
	1319	return ret;
	1320	}
	1321
	1322	static const struct mm_walk_ops charge_walk_ops = {
	1323	.pmd_entry = mem_cgroup_move_charge_pte_range,
	1324	.walk_lock = PGWALK_RDLOCK,
	1325	};
	1326
	1327	static void mem_cgroup_move_charge(void)
	1328	{
	1329	lru_add_drain_all();
	1330	/*
	1331	* Signal folio_memcg_lock() to take the memcg's move_lock
	1332	* while we're moving its pages to another memcg. Then wait
	1333	* for already started RCU-only updates to finish.
	1334	*/
	1335	atomic_inc(&mc.from->moving_account);
	1336	synchronize_rcu();
	1337	retry:
	1338	if (unlikely(!mmap_read_trylock(mc.mm))) {
	1339	/*
	1340	* Someone who are holding the mmap_lock might be waiting in
	1341	* waitq. So we cancel all extra charges, wake up all waiters,
	1342	* and retry. Because we cancel precharges, we might not be able
	1343	* to move enough charges, but moving charge is a best-effort
	1344	* feature anyway, so it wouldn't be a big problem.
	1345	*/
	1346	__mem_cgroup_clear_mc();
	1347	cond_resched();
	1348	goto retry;
	1349	}
	1350	/*
	1351	* When we have consumed all precharges and failed in doing
	1352	* additional charge, the page walk just aborts.
	1353	*/
	1354	walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, NULL);
	1355	mmap_read_unlock(mc.mm);
	1356	atomic_dec(&mc.from->moving_account);
	1357	}
	1358
	1359	void memcg1_move_task(void)
	1360	{
	1361	if (mc.to) {
	1362	mem_cgroup_move_charge();
	1363	mem_cgroup_clear_mc();
	1364	}
	1365	}
	1366
	1367	#else /* !CONFIG_MMU */
	1368	int memcg1_can_attach(struct cgroup_taskset *tset)
	1369	{
	1370	return 0;
	1371	}
	1372	void memcg1_cancel_attach(struct cgroup_taskset *tset)
	1373	{
	1374	}
	1375	void memcg1_move_task(void)
	1376	{
	1377	}
	1378	#endif
	1379
	1380	static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)
	1381	{
	1382	struct mem_cgroup_threshold_ary *t;
	1383	unsigned long usage;
	1384	int i;
	1385
	1386	rcu_read_lock();
	1387	if (!swap)
	1388	t = rcu_dereference(memcg->thresholds.primary);
	1389	else
	1390	t = rcu_dereference(memcg->memsw_thresholds.primary);
	1391
	1392	if (!t)
	1393	goto unlock;
	1394
	1395	usage = mem_cgroup_usage(memcg, swap);
	1396
	1397	/*
	1398	* current_threshold points to threshold just below or equal to usage.
	1399	* If it's not true, a threshold was crossed after last
	1400	* call of __mem_cgroup_threshold().
	1401	*/
	1402	i = t->current_threshold;
	1403
	1404	/*
	1405	* Iterate backward over array of thresholds starting from
	1406	* current_threshold and check if a threshold is crossed.
	1407	* If none of thresholds below usage is crossed, we read
	1408	* only one element of the array here.
	1409	*/
	1410	for (; i >= 0 && unlikely(t->entries[i].threshold > usage); i--)
	1411	eventfd_signal(t->entries[i].eventfd);
	1412
	1413	/* i = current_threshold + 1 */
	1414	i++;
	1415
	1416	/*
	1417	* Iterate forward over array of thresholds starting from
	1418	* current_threshold+1 and check if a threshold is crossed.
	1419	* If none of thresholds above usage is crossed, we read
	1420	* only one element of the array here.
	1421	*/
	1422	for (; i < t->size && unlikely(t->entries[i].threshold <= usage); i++)
	1423	eventfd_signal(t->entries[i].eventfd);
	1424
	1425	/* Update current_threshold */
	1426	t->current_threshold = i - 1;
	1427	unlock:
	1428	rcu_read_unlock();
	1429	}
	1430
	1431	static void mem_cgroup_threshold(struct mem_cgroup *memcg)
	1432	{
	1433	while (memcg) {
	1434	__mem_cgroup_threshold(memcg, false);
	1435	if (do_memsw_account())
	1436	__mem_cgroup_threshold(memcg, true);
	1437
	1438	memcg = parent_mem_cgroup(memcg);
	1439	}
	1440	}
	1441
	1442	/*
	1443	* Check events in order.
	1444	*
	1445	*/
	1446	void memcg1_check_events(struct mem_cgroup *memcg, int nid)
	1447	{
	1448	if (IS_ENABLED(CONFIG_PREEMPT_RT))
	1449	return;
	1450
	1451	/* threshold event is triggered in finer grain than soft limit */
	1452	if (unlikely(mem_cgroup_event_ratelimit(memcg,
	1453	MEM_CGROUP_TARGET_THRESH))) {
	1454	bool do_softlimit;
	1455
	1456	do_softlimit = mem_cgroup_event_ratelimit(memcg,
	1457	MEM_CGROUP_TARGET_SOFTLIMIT);
	1458	mem_cgroup_threshold(memcg);
	1459	if (unlikely(do_softlimit))
	1460	memcg1_update_tree(memcg, nid);
	1461	}
	1462	}
	1463
	1464	static int compare_thresholds(const void a, const void b)
	1465	{
	1466	const struct mem_cgroup_threshold *_a = a;
	1467	const struct mem_cgroup_threshold *_b = b;
	1468
	1469	if (_a->threshold > _b->threshold)
	1470	return 1;
	1471
	1472	if (_a->threshold < _b->threshold)
	1473	return -1;
	1474
	1475	return 0;
	1476	}
	1477
	1478	static int mem_cgroup_oom_notify_cb(struct mem_cgroup *memcg)
	1479	{
	1480	struct mem_cgroup_eventfd_list *ev;
	1481
	1482	spin_lock(&memcg_oom_lock);
	1483
	1484	list_for_each_entry(ev, &memcg->oom_notify, list)
	1485	eventfd_signal(ev->eventfd);
	1486
	1487	spin_unlock(&memcg_oom_lock);
	1488	return 0;
	1489	}
	1490
	1491	static void mem_cgroup_oom_notify(struct mem_cgroup *memcg)
	1492	{
	1493	struct mem_cgroup *iter;
	1494
	1495	for_each_mem_cgroup_tree(iter, memcg)
	1496	mem_cgroup_oom_notify_cb(iter);
	1497	}
	1498
	1499	static int __mem_cgroup_usage_register_event(struct mem_cgroup *memcg,
	1500	struct eventfd_ctx eventfd, const char args, enum res_type type)
	1501	{
	1502	struct mem_cgroup_thresholds *thresholds;
	1503	struct mem_cgroup_threshold_ary *new;
	1504	unsigned long threshold;
	1505	unsigned long usage;
	1506	int i, size, ret;
	1507
	1508	ret = page_counter_memparse(args, "-1", &threshold);
	1509	if (ret)
	1510	return ret;
	1511
	1512	mutex_lock(&memcg->thresholds_lock);
	1513
	1514	if (type == _MEM) {
	1515	thresholds = &memcg->thresholds;
	1516	usage = mem_cgroup_usage(memcg, false);
	1517	} else if (type == _MEMSWAP) {
	1518	thresholds = &memcg->memsw_thresholds;
	1519	usage = mem_cgroup_usage(memcg, true);
	1520	} else
	1521	BUG();
	1522
	1523	/* Check if a threshold crossed before adding a new one */
	1524	if (thresholds->primary)
	1525	__mem_cgroup_threshold(memcg, type == _MEMSWAP);
	1526
	1527	size = thresholds->primary ? thresholds->primary->size + 1 : 1;
	1528
	1529	/* Allocate memory for new array of thresholds */
	1530	new = kmalloc(struct_size(new, entries, size), GFP_KERNEL);
	1531	if (!new) {
	1532	ret = -ENOMEM;
	1533	goto unlock;
	1534	}
	1535	new->size = size;
	1536
	1537	/* Copy thresholds (if any) to new array */
	1538	if (thresholds->primary)
	1539	memcpy(new->entries, thresholds->primary->entries,
	1540	flex_array_size(new, entries, size - 1));
	1541
	1542	/* Add new threshold */
	1543	new->entries[size - 1].eventfd = eventfd;
	1544	new->entries[size - 1].threshold = threshold;
	1545
	1546	/* Sort thresholds. Registering of new threshold isn't time-critical */
	1547	sort(new->entries, size, sizeof(*new->entries),
	1548	compare_thresholds, NULL);
	1549
	1550	/* Find current threshold */
	1551	new->current_threshold = -1;
	1552	for (i = 0; i < size; i++) {
	1553	if (new->entries[i].threshold <= usage) {
	1554	/*
	1555	* new->current_threshold will not be used until
	1556	* rcu_assign_pointer(), so it's safe to increment
	1557	* it here.
	1558	*/
	1559	++new->current_threshold;
	1560	} else
	1561	break;
	1562	}
	1563
	1564	/* Free old spare buffer and save old primary buffer as spare */
	1565	kfree(thresholds->spare);
	1566	thresholds->spare = thresholds->primary;
	1567
	1568	rcu_assign_pointer(thresholds->primary, new);
	1569
	1570	/* To be sure that nobody uses thresholds */
	1571	synchronize_rcu();
	1572
	1573	unlock:
	1574	mutex_unlock(&memcg->thresholds_lock);
	1575
	1576	return ret;
	1577	}
	1578
	1579	static int mem_cgroup_usage_register_event(struct mem_cgroup *memcg,
	1580	struct eventfd_ctx eventfd, const char args)
	1581	{
	1582	return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEM);
	1583	}
	1584
	1585	static int memsw_cgroup_usage_register_event(struct mem_cgroup *memcg,
	1586	struct eventfd_ctx eventfd, const char args)
	1587	{
	1588	return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEMSWAP);
	1589	}
	1590
	1591	static void __mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
	1592	struct eventfd_ctx *eventfd, enum res_type type)
	1593	{
	1594	struct mem_cgroup_thresholds *thresholds;
	1595	struct mem_cgroup_threshold_ary *new;
	1596	unsigned long usage;
	1597	int i, j, size, entries;
	1598
	1599	mutex_lock(&memcg->thresholds_lock);
	1600
	1601	if (type == _MEM) {
	1602	thresholds = &memcg->thresholds;
	1603	usage = mem_cgroup_usage(memcg, false);
	1604	} else if (type == _MEMSWAP) {
	1605	thresholds = &memcg->memsw_thresholds;
	1606	usage = mem_cgroup_usage(memcg, true);
	1607	} else
	1608	BUG();
	1609
	1610	if (!thresholds->primary)
	1611	goto unlock;
	1612
	1613	/* Check if a threshold crossed before removing */
	1614	__mem_cgroup_threshold(memcg, type == _MEMSWAP);
	1615
	1616	/* Calculate new number of threshold */
	1617	size = entries = 0;
	1618	for (i = 0; i < thresholds->primary->size; i++) {
	1619	if (thresholds->primary->entries[i].eventfd != eventfd)
	1620	size++;
	1621	else
	1622	entries++;
	1623	}
	1624
	1625	new = thresholds->spare;
	1626
	1627	/* If no items related to eventfd have been cleared, nothing to do */
	1628	if (!entries)
	1629	goto unlock;
	1630
	1631	/* Set thresholds array to NULL if we don't have thresholds */
	1632	if (!size) {
	1633	kfree(new);
	1634	new = NULL;
	1635	goto swap_buffers;
	1636	}
	1637
	1638	new->size = size;
	1639
	1640	/* Copy thresholds and find current threshold */
	1641	new->current_threshold = -1;
	1642	for (i = 0, j = 0; i < thresholds->primary->size; i++) {
	1643	if (thresholds->primary->entries[i].eventfd == eventfd)
	1644	continue;
	1645
	1646	new->entries[j] = thresholds->primary->entries[i];
	1647	if (new->entries[j].threshold <= usage) {
	1648	/*
	1649	* new->current_threshold will not be used
	1650	* until rcu_assign_pointer(), so it's safe to increment
	1651	* it here.
	1652	*/
	1653	++new->current_threshold;
	1654	}
	1655	j++;
	1656	}
	1657
	1658	swap_buffers:
	1659	/* Swap primary and spare array */
	1660	thresholds->spare = thresholds->primary;
	1661
	1662	rcu_assign_pointer(thresholds->primary, new);
	1663
	1664	/* To be sure that nobody uses thresholds */
	1665	synchronize_rcu();
	1666
	1667	/* If all events are unregistered, free the spare array */
	1668	if (!new) {
	1669	kfree(thresholds->spare);
	1670	thresholds->spare = NULL;
	1671	}
	1672	unlock:
	1673	mutex_unlock(&memcg->thresholds_lock);
	1674	}
	1675
	1676	static void mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
	1677	struct eventfd_ctx *eventfd)
	1678	{
	1679	return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEM);
	1680	}
	1681
	1682	static void memsw_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
	1683	struct eventfd_ctx *eventfd)
	1684	{
	1685	return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEMSWAP);
	1686	}
	1687
	1688	static int mem_cgroup_oom_register_event(struct mem_cgroup *memcg,
	1689	struct eventfd_ctx eventfd, const char args)
	1690	{
	1691	struct mem_cgroup_eventfd_list *event;
	1692
	1693	event = kmalloc(sizeof(*event), GFP_KERNEL);
	1694	if (!event)
	1695	return -ENOMEM;
	1696
	1697	spin_lock(&memcg_oom_lock);
	1698
	1699	event->eventfd = eventfd;
	1700	list_add(&event->list, &memcg->oom_notify);
	1701
	1702	/* already in OOM ? */
	1703	if (memcg->under_oom)
	1704	eventfd_signal(eventfd);
	1705	spin_unlock(&memcg_oom_lock);
	1706
	1707	return 0;
	1708	}
	1709
	1710	static void mem_cgroup_oom_unregister_event(struct mem_cgroup *memcg,
	1711	struct eventfd_ctx *eventfd)
	1712	{
	1713	struct mem_cgroup_eventfd_list ev, tmp;
	1714
	1715	spin_lock(&memcg_oom_lock);
	1716
	1717	list_for_each_entry_safe(ev, tmp, &memcg->oom_notify, list) {
	1718	if (ev->eventfd == eventfd) {
	1719	list_del(&ev->list);
	1720	kfree(ev);
	1721	}
	1722	}
	1723
	1724	spin_unlock(&memcg_oom_lock);
	1725	}
	1726
	1727	/*
	1728	* DO NOT USE IN NEW FILES.
	1729	*
	1730	* "cgroup.event_control" implementation.
	1731	*
	1732	* This is way over-engineered. It tries to support fully configurable
	1733	* events for each user. Such level of flexibility is completely
	1734	* unnecessary especially in the light of the planned unified hierarchy.
	1735	*
	1736	* Please deprecate this and replace with something simpler if at all
	1737	* possible.
	1738	*/
	1739
	1740	/*
	1741	* Unregister event and free resources.
	1742	*
	1743	* Gets called from workqueue.
	1744	*/
	1745	static void memcg_event_remove(struct work_struct *work)
	1746	{
	1747	struct mem_cgroup_event *event =
	1748	container_of(work, struct mem_cgroup_event, remove);
	1749	struct mem_cgroup *memcg = event->memcg;
	1750
	1751	remove_wait_queue(event->wqh, &event->wait);
	1752
	1753	event->unregister_event(memcg, event->eventfd);
	1754
	1755	/* Notify userspace the event is going away. */
	1756	eventfd_signal(event->eventfd);
	1757
	1758	eventfd_ctx_put(event->eventfd);
	1759	kfree(event);
	1760	css_put(&memcg->css);
	1761	}
	1762
	1763	/*
	1764	* Gets called on EPOLLHUP on eventfd when user closes it.
	1765	*
	1766	* Called with wqh->lock held and interrupts disabled.
	1767	*/
	1768	static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode,
	1769	int sync, void *key)
	1770	{
	1771	struct mem_cgroup_event *event =
	1772	container_of(wait, struct mem_cgroup_event, wait);
	1773	struct mem_cgroup *memcg = event->memcg;
	1774	__poll_t flags = key_to_poll(key);
	1775
	1776	if (flags & EPOLLHUP) {
	1777	/*
	1778	* If the event has been detached at cgroup removal, we
	1779	* can simply return knowing the other side will cleanup
	1780	* for us.
	1781	*
	1782	* We can't race against event freeing since the other
	1783	* side will require wqh->lock via remove_wait_queue(),
	1784	* which we hold.
	1785	*/
	1786	spin_lock(&memcg->event_list_lock);
	1787	if (!list_empty(&event->list)) {
	1788	list_del_init(&event->list);
	1789	/*
	1790	* We are in atomic context, but cgroup_event_remove()
	1791	* may sleep, so we have to call it in workqueue.
	1792	*/
	1793	schedule_work(&event->remove);
	1794	}
	1795	spin_unlock(&memcg->event_list_lock);
	1796	}
	1797
	1798	return 0;
	1799	}
	1800
	1801	static void memcg_event_ptable_queue_proc(struct file *file,
	1802	wait_queue_head_t wqh, poll_table pt)
	1803	{
	1804	struct mem_cgroup_event *event =
	1805	container_of(pt, struct mem_cgroup_event, pt);
	1806
	1807	event->wqh = wqh;
	1808	add_wait_queue(wqh, &event->wait);
	1809	}
	1810
	1811	/*
	1812	* DO NOT USE IN NEW FILES.
	1813	*
	1814	* Parse input and register new cgroup event handler.
	1815	*
	1816	* Input must be in format '<event_fd> <control_fd> <args>'.
	1817	* Interpretation of args is defined by control file implementation.
	1818	*/
	1819	static ssize_t memcg_write_event_control(struct kernfs_open_file *of,
	1820	char *buf, size_t nbytes, loff_t off)
	1821	{
	1822	struct cgroup_subsys_state *css = of_css(of);
	1823	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	1824	struct mem_cgroup_event *event;
	1825	struct cgroup_subsys_state *cfile_css;
	1826	unsigned int efd, cfd;
	1827	struct fd efile;
	1828	struct fd cfile;
	1829	struct dentry *cdentry;
	1830	const char *name;
	1831	char *endp;
	1832	int ret;
	1833
	1834	if (IS_ENABLED(CONFIG_PREEMPT_RT))
	1835	return -EOPNOTSUPP;
	1836
	1837	buf = strstrip(buf);
	1838
	1839	efd = simple_strtoul(buf, &endp, 10);
	1840	if (*endp != ' ')
	1841	return -EINVAL;
	1842	buf = endp + 1;
	1843
	1844	cfd = simple_strtoul(buf, &endp, 10);
	1845	if ((endp != ' ') && (endp != '\0'))
	1846	return -EINVAL;
	1847	buf = endp + 1;
	1848
	1849	event = kzalloc(sizeof(*event), GFP_KERNEL);
	1850	if (!event)
	1851	return -ENOMEM;
	1852
	1853	event->memcg = memcg;
	1854	INIT_LIST_HEAD(&event->list);
	1855	init_poll_funcptr(&event->pt, memcg_event_ptable_queue_proc);
	1856	init_waitqueue_func_entry(&event->wait, memcg_event_wake);
	1857	INIT_WORK(&event->remove, memcg_event_remove);
	1858
	1859	efile = fdget(efd);
	1860	if (!efile.file) {
	1861	ret = -EBADF;
	1862	goto out_kfree;
	1863	}
	1864
	1865	event->eventfd = eventfd_ctx_fileget(efile.file);
	1866	if (IS_ERR(event->eventfd)) {
	1867	ret = PTR_ERR(event->eventfd);
	1868	goto out_put_efile;
	1869	}
	1870
	1871	cfile = fdget(cfd);
	1872	if (!cfile.file) {
	1873	ret = -EBADF;
	1874	goto out_put_eventfd;
	1875	}
	1876
	1877	/* the process need read permission on control file */
	1878	/* AV: shouldn't we check that it's been opened for read instead? */
	1879	ret = file_permission(cfile.file, MAY_READ);
	1880	if (ret < 0)
	1881	goto out_put_cfile;
	1882
	1883	/*
	1884	* The control file must be a regular cgroup1 file. As a regular cgroup
	1885	* file can't be renamed, it's safe to access its name afterwards.
	1886	*/
	1887	cdentry = cfile.file->f_path.dentry;
	1888	if (cdentry->d_sb->s_type != &cgroup_fs_type \|\| !d_is_reg(cdentry)) {
	1889	ret = -EINVAL;
	1890	goto out_put_cfile;
	1891	}
	1892
	1893	/*
	1894	* Determine the event callbacks and set them in @event. This used
	1895	* to be done via struct cftype but cgroup core no longer knows
	1896	* about these events. The following is crude but the whole thing
	1897	* is for compatibility anyway.
	1898	*
	1899	* DO NOT ADD NEW FILES.
	1900	*/
	1901	name = cdentry->d_name.name;
	1902
	1903	if (!strcmp(name, "memory.usage_in_bytes")) {
	1904	event->register_event = mem_cgroup_usage_register_event;
	1905	event->unregister_event = mem_cgroup_usage_unregister_event;
	1906	} else if (!strcmp(name, "memory.oom_control")) {
	1907	event->register_event = mem_cgroup_oom_register_event;
	1908	event->unregister_event = mem_cgroup_oom_unregister_event;
	1909	} else if (!strcmp(name, "memory.pressure_level")) {
	1910	event->register_event = vmpressure_register_event;
	1911	event->unregister_event = vmpressure_unregister_event;
	1912	} else if (!strcmp(name, "memory.memsw.usage_in_bytes")) {
	1913	event->register_event = memsw_cgroup_usage_register_event;
	1914	event->unregister_event = memsw_cgroup_usage_unregister_event;
	1915	} else {
	1916	ret = -EINVAL;
	1917	goto out_put_cfile;
	1918	}
	1919
	1920	/*
	1921	* Verify @cfile should belong to @css. Also, remaining events are
	1922	* automatically removed on cgroup destruction but the removal is
	1923	* asynchronous, so take an extra ref on @css.
	1924	*/
	1925	cfile_css = css_tryget_online_from_dir(cdentry->d_parent,
	1926	&memory_cgrp_subsys);
	1927	ret = -EINVAL;
	1928	if (IS_ERR(cfile_css))
	1929	goto out_put_cfile;
	1930	if (cfile_css != css) {
	1931	css_put(cfile_css);
	1932	goto out_put_cfile;
	1933	}
	1934
	1935	ret = event->register_event(memcg, event->eventfd, buf);
	1936	if (ret)
	1937	goto out_put_css;
	1938
	1939	vfs_poll(efile.file, &event->pt);
	1940
	1941	spin_lock_irq(&memcg->event_list_lock);
	1942	list_add(&event->list, &memcg->event_list);
	1943	spin_unlock_irq(&memcg->event_list_lock);
	1944
	1945	fdput(cfile);
	1946	fdput(efile);
	1947
	1948	return nbytes;
	1949
	1950	out_put_css:
	1951	css_put(css);
	1952	out_put_cfile:
	1953	fdput(cfile);
	1954	out_put_eventfd:
	1955	eventfd_ctx_put(event->eventfd);
	1956	out_put_efile:
	1957	fdput(efile);
	1958	out_kfree:
	1959	kfree(event);
	1960
	1961	return ret;
	1962	}
	1963
	1964	void memcg1_memcg_init(struct mem_cgroup *memcg)
	1965	{
	1966	INIT_LIST_HEAD(&memcg->oom_notify);
	1967	mutex_init(&memcg->thresholds_lock);
	1968	spin_lock_init(&memcg->move_lock);
	1969	INIT_LIST_HEAD(&memcg->event_list);
	1970	spin_lock_init(&memcg->event_list_lock);
	1971	}
	1972
	1973	void memcg1_css_offline(struct mem_cgroup *memcg)
	1974	{
	1975	struct mem_cgroup_event event, tmp;
	1976
	1977	/*
	1978	* Unregister events and notify userspace.
	1979	* Notify userspace about cgroup removing only after rmdir of cgroup
	1980	* directory to avoid race between userspace and kernelspace.
	1981	*/
	1982	spin_lock_irq(&memcg->event_list_lock);
	1983	list_for_each_entry_safe(event, tmp, &memcg->event_list, list) {
	1984	list_del_init(&event->list);
	1985	schedule_work(&event->remove);
	1986	}
	1987	spin_unlock_irq(&memcg->event_list_lock);
	1988	}
	1989
	1990	/*
	1991	* Check OOM-Killer is already running under our hierarchy.
	1992	* If someone is running, return false.
	1993	*/
	1994	static bool mem_cgroup_oom_trylock(struct mem_cgroup *memcg)
	1995	{
	1996	struct mem_cgroup iter, failed = NULL;
	1997
	1998	spin_lock(&memcg_oom_lock);
	1999
	2000	for_each_mem_cgroup_tree(iter, memcg) {
	2001	if (iter->oom_lock) {
	2002	/*
	2003	* this subtree of our hierarchy is already locked
	2004	* so we cannot give a lock.
	2005	*/
	2006	failed = iter;
	2007	mem_cgroup_iter_break(memcg, iter);
	2008	break;
	2009	} else
	2010	iter->oom_lock = true;
	2011	}
	2012
	2013	if (failed) {
	2014	/*
	2015	* OK, we failed to lock the whole subtree so we have
	2016	* to clean up what we set up to the failing subtree
	2017	*/
	2018	for_each_mem_cgroup_tree(iter, memcg) {
	2019	if (iter == failed) {
	2020	mem_cgroup_iter_break(memcg, iter);
	2021	break;
	2022	}
	2023	iter->oom_lock = false;
	2024	}
	2025	} else
	2026	mutex_acquire(&memcg_oom_lock_dep_map, 0, 1, _RET_IP_);
	2027
	2028	spin_unlock(&memcg_oom_lock);
	2029
	2030	return !failed;
	2031	}
	2032
	2033	static void mem_cgroup_oom_unlock(struct mem_cgroup *memcg)
	2034	{
	2035	struct mem_cgroup *iter;
	2036
	2037	spin_lock(&memcg_oom_lock);
	2038	mutex_release(&memcg_oom_lock_dep_map, _RET_IP_);
	2039	for_each_mem_cgroup_tree(iter, memcg)
	2040	iter->oom_lock = false;
	2041	spin_unlock(&memcg_oom_lock);
	2042	}
	2043
	2044	static void mem_cgroup_mark_under_oom(struct mem_cgroup *memcg)
	2045	{
	2046	struct mem_cgroup *iter;
	2047
	2048	spin_lock(&memcg_oom_lock);
	2049	for_each_mem_cgroup_tree(iter, memcg)
	2050	iter->under_oom++;
	2051	spin_unlock(&memcg_oom_lock);
	2052	}
	2053
	2054	static void mem_cgroup_unmark_under_oom(struct mem_cgroup *memcg)
	2055	{
	2056	struct mem_cgroup *iter;
	2057
	2058	/*
	2059	* Be careful about under_oom underflows because a child memcg
	2060	* could have been added after mem_cgroup_mark_under_oom.
	2061	*/
	2062	spin_lock(&memcg_oom_lock);
	2063	for_each_mem_cgroup_tree(iter, memcg)
	2064	if (iter->under_oom > 0)
	2065	iter->under_oom--;
	2066	spin_unlock(&memcg_oom_lock);
	2067	}
	2068
	2069	static DECLARE_WAIT_QUEUE_HEAD(memcg_oom_waitq);
	2070
	2071	struct oom_wait_info {
	2072	struct mem_cgroup *memcg;
	2073	wait_queue_entry_t wait;
	2074	};
	2075
	2076	static int memcg_oom_wake_function(wait_queue_entry_t *wait,
	2077	unsigned mode, int sync, void *arg)
	2078	{
	2079	struct mem_cgroup wake_memcg = (struct mem_cgroup )arg;
	2080	struct mem_cgroup *oom_wait_memcg;
	2081	struct oom_wait_info *oom_wait_info;
	2082
	2083	oom_wait_info = container_of(wait, struct oom_wait_info, wait);
	2084	oom_wait_memcg = oom_wait_info->memcg;
	2085
	2086	if (!mem_cgroup_is_descendant(wake_memcg, oom_wait_memcg) &&
	2087	!mem_cgroup_is_descendant(oom_wait_memcg, wake_memcg))
	2088	return 0;
	2089	return autoremove_wake_function(wait, mode, sync, arg);
	2090	}
	2091
	2092	void memcg1_oom_recover(struct mem_cgroup *memcg)
	2093	{
	2094	/*
	2095	* For the following lockless ->under_oom test, the only required
	2096	* guarantee is that it must see the state asserted by an OOM when
	2097	* this function is called as a result of userland actions
	2098	* triggered by the notification of the OOM. This is trivially
	2099	* achieved by invoking mem_cgroup_mark_under_oom() before
	2100	* triggering notification.
	2101	*/
	2102	if (memcg && memcg->under_oom)
	2103	__wake_up(&memcg_oom_waitq, TASK_NORMAL, 0, memcg);
	2104	}
	2105
	2106	/**
	2107	* mem_cgroup_oom_synchronize - complete memcg OOM handling
	2108	* @handle: actually kill/wait or just clean up the OOM state
	2109	*
	2110	* This has to be called at the end of a page fault if the memcg OOM
	2111	* handler was enabled.
	2112	*
	2113	* Memcg supports userspace OOM handling where failed allocations must
	2114	* sleep on a waitqueue until the userspace task resolves the
	2115	* situation. Sleeping directly in the charge context with all kinds
	2116	* of locks held is not a good idea, instead we remember an OOM state
	2117	* in the task and mem_cgroup_oom_synchronize() has to be called at
	2118	* the end of the page fault to complete the OOM handling.
	2119	*
	2120	* Returns %true if an ongoing memcg OOM situation was detected and
	2121	* completed, %false otherwise.
	2122	*/
	2123	bool mem_cgroup_oom_synchronize(bool handle)
	2124	{
	2125	struct mem_cgroup *memcg = current->memcg_in_oom;
	2126	struct oom_wait_info owait;
	2127	bool locked;
	2128
	2129	/* OOM is global, do not handle */
	2130	if (!memcg)
	2131	return false;
	2132
	2133	if (!handle)
	2134	goto cleanup;
	2135
	2136	owait.memcg = memcg;
	2137	owait.wait.flags = 0;
	2138	owait.wait.func = memcg_oom_wake_function;
	2139	owait.wait.private = current;
	2140	INIT_LIST_HEAD(&owait.wait.entry);
	2141
	2142	prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE);
	2143	mem_cgroup_mark_under_oom(memcg);
	2144
	2145	locked = mem_cgroup_oom_trylock(memcg);
	2146
	2147	if (locked)
	2148	mem_cgroup_oom_notify(memcg);
	2149
	2150	schedule();
	2151	mem_cgroup_unmark_under_oom(memcg);
	2152	finish_wait(&memcg_oom_waitq, &owait.wait);
	2153
	2154	if (locked)
	2155	mem_cgroup_oom_unlock(memcg);
	2156	cleanup:
	2157	current->memcg_in_oom = NULL;
	2158	css_put(&memcg->css);
	2159	return true;
	2160	}
	2161
	2162
	2163	bool memcg1_oom_prepare(struct mem_cgroup memcg, bool locked)
	2164	{
	2165	/*
	2166	* We are in the middle of the charge context here, so we
	2167	* don't want to block when potentially sitting on a callstack
	2168	* that holds all kinds of filesystem and mm locks.
	2169	*
	2170	* cgroup1 allows disabling the OOM killer and waiting for outside
	2171	* handling until the charge can succeed; remember the context and put
	2172	* the task to sleep at the end of the page fault when all locks are
	2173	* released.
	2174	*
	2175	* On the other hand, in-kernel OOM killer allows for an async victim
	2176	* memory reclaim (oom_reaper) and that means that we are not solely
	2177	* relying on the oom victim to make a forward progress and we can
	2178	* invoke the oom killer here.
	2179	*
	2180	* Please note that mem_cgroup_out_of_memory might fail to find a
	2181	* victim and then we have to bail out from the charge path.
	2182	*/
	2183	if (READ_ONCE(memcg->oom_kill_disable)) {
	2184	if (current->in_user_fault) {
	2185	css_get(&memcg->css);
	2186	current->memcg_in_oom = memcg;
	2187	}
	2188	return false;
	2189	}
	2190
	2191	mem_cgroup_mark_under_oom(memcg);
	2192
	2193	*locked = mem_cgroup_oom_trylock(memcg);
	2194
	2195	if (*locked)
	2196	mem_cgroup_oom_notify(memcg);
	2197
	2198	mem_cgroup_unmark_under_oom(memcg);
	2199
	2200	return true;
	2201	}
	2202
	2203	void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked)
	2204	{
	2205	if (locked)
	2206	mem_cgroup_oom_unlock(memcg);
	2207	}
	2208
	2209	static DEFINE_MUTEX(memcg_max_mutex);
	2210
	2211	static int mem_cgroup_resize_max(struct mem_cgroup *memcg,
	2212	unsigned long max, bool memsw)
	2213	{
	2214	bool enlarge = false;
	2215	bool drained = false;
	2216	int ret;
	2217	bool limits_invariant;
	2218	struct page_counter *counter = memsw ? &memcg->memsw : &memcg->memory;
	2219
	2220	do {
	2221	if (signal_pending(current)) {
	2222	ret = -EINTR;
	2223	break;
	2224	}
	2225
	2226	mutex_lock(&memcg_max_mutex);
	2227	/*
	2228	* Make sure that the new limit (memsw or memory limit) doesn't
	2229	* break our basic invariant rule memory.max <= memsw.max.
	2230	*/
	2231	limits_invariant = memsw ? max >= READ_ONCE(memcg->memory.max) :
	2232	max <= memcg->memsw.max;
	2233	if (!limits_invariant) {
	2234	mutex_unlock(&memcg_max_mutex);
	2235	ret = -EINVAL;
	2236	break;
	2237	}
	2238	if (max > counter->max)
	2239	enlarge = true;
	2240	ret = page_counter_set_max(counter, max);
	2241	mutex_unlock(&memcg_max_mutex);
	2242
	2243	if (!ret)
	2244	break;
	2245
	2246	if (!drained) {
	2247	drain_all_stock(memcg);
	2248	drained = true;
	2249	continue;
	2250	}
	2251
	2252	if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
	2253	memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, NULL)) {
	2254	ret = -EBUSY;
	2255	break;
	2256	}
	2257	} while (true);
	2258
	2259	if (!ret && enlarge)
	2260	memcg1_oom_recover(memcg);
	2261
	2262	return ret;
	2263	}
	2264
	2265	/*
	2266	* Reclaims as many pages from the given memcg as possible.
	2267	*
	2268	* Caller is responsible for holding css reference for memcg.
	2269	*/
	2270	static int mem_cgroup_force_empty(struct mem_cgroup *memcg)
	2271	{
	2272	int nr_retries = MAX_RECLAIM_RETRIES;
	2273
	2274	/* we call try-to-free pages for make this cgroup empty */
	2275	lru_add_drain_all();
	2276
	2277	drain_all_stock(memcg);
	2278
	2279	/* try to free all pages in this cgroup */
	2280	while (nr_retries && page_counter_read(&memcg->memory)) {
	2281	if (signal_pending(current))
	2282	return -EINTR;
	2283
	2284	if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
	2285	MEMCG_RECLAIM_MAY_SWAP, NULL))
	2286	nr_retries--;
	2287	}
	2288
	2289	return 0;
	2290	}
	2291
	2292	static ssize_t mem_cgroup_force_empty_write(struct kernfs_open_file *of,
	2293	char *buf, size_t nbytes,
	2294	loff_t off)
	2295	{
	2296	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	2297
	2298	if (mem_cgroup_is_root(memcg))
	2299	return -EINVAL;
	2300	return mem_cgroup_force_empty(memcg) ?: nbytes;
	2301	}
	2302
	2303	static u64 mem_cgroup_hierarchy_read(struct cgroup_subsys_state *css,
	2304	struct cftype *cft)
	2305	{
	2306	return 1;
	2307	}
	2308
	2309	static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css,
	2310	struct cftype *cft, u64 val)
	2311	{
	2312	if (val == 1)
	2313	return 0;
	2314
	2315	pr_warn_once("Non-hierarchical mode is deprecated. "
	2316	"Please report your usecase to linux-mm@kvack.org if you "
	2317	"depend on this functionality.\n");
	2318
	2319	return -EINVAL;
	2320	}
	2321
	2322	static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css,
	2323	struct cftype *cft)
	2324	{
	2325	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	2326	struct page_counter *counter;
	2327
	2328	switch (MEMFILE_TYPE(cft->private)) {
	2329	case _MEM:
	2330	counter = &memcg->memory;
	2331	break;
	2332	case _MEMSWAP:
	2333	counter = &memcg->memsw;
	2334	break;
	2335	case _KMEM:
	2336	counter = &memcg->kmem;
	2337	break;
	2338	case _TCP:
	2339	counter = &memcg->tcpmem;
	2340	break;
	2341	default:
	2342	BUG();
	2343	}
	2344
	2345	switch (MEMFILE_ATTR(cft->private)) {
	2346	case RES_USAGE:
	2347	if (counter == &memcg->memory)
	2348	return (u64)mem_cgroup_usage(memcg, false) * PAGE_SIZE;
	2349	if (counter == &memcg->memsw)
	2350	return (u64)mem_cgroup_usage(memcg, true) * PAGE_SIZE;
	2351	return (u64)page_counter_read(counter) * PAGE_SIZE;
	2352	case RES_LIMIT:
	2353	return (u64)counter->max * PAGE_SIZE;
	2354	case RES_MAX_USAGE:
	2355	return (u64)counter->watermark * PAGE_SIZE;
	2356	case RES_FAILCNT:
	2357	return counter->failcnt;
	2358	case RES_SOFT_LIMIT:
	2359	return (u64)READ_ONCE(memcg->soft_limit) * PAGE_SIZE;
	2360	default:
	2361	BUG();
	2362	}
	2363	}
	2364
	2365	/*
	2366	* This function doesn't do anything useful. Its only job is to provide a read
	2367	* handler for a file so that cgroup_file_mode() will add read permissions.
	2368	*/
	2369	static int mem_cgroup_dummy_seq_show(__always_unused struct seq_file *m,
	2370	__always_unused void *v)
	2371	{
	2372	return -EINVAL;
	2373	}
	2374
	2375	static int memcg_update_tcp_max(struct mem_cgroup *memcg, unsigned long max)
	2376	{
	2377	int ret;
	2378
	2379	mutex_lock(&memcg_max_mutex);
	2380
	2381	ret = page_counter_set_max(&memcg->tcpmem, max);
	2382	if (ret)
	2383	goto out;
	2384
	2385	if (!memcg->tcpmem_active) {
	2386	/*
	2387	* The active flag needs to be written after the static_key
	2388	* update. This is what guarantees that the socket activation
	2389	* function is the last one to run. See mem_cgroup_sk_alloc()
	2390	* for details, and note that we don't mark any socket as
	2391	* belonging to this memcg until that flag is up.
	2392	*
	2393	* We need to do this, because static_keys will span multiple
	2394	* sites, but we can't control their order. If we mark a socket
	2395	* as accounted, but the accounting functions are not patched in
	2396	* yet, we'll lose accounting.
	2397	*
	2398	* We never race with the readers in mem_cgroup_sk_alloc(),
	2399	* because when this value change, the code to process it is not
	2400	* patched in yet.
	2401	*/
	2402	static_branch_inc(&memcg_sockets_enabled_key);
	2403	memcg->tcpmem_active = true;
	2404	}
	2405	out:
	2406	mutex_unlock(&memcg_max_mutex);
	2407	return ret;
	2408	}
	2409
	2410	/*
	2411	* The user of this function is...
	2412	* RES_LIMIT.
	2413	*/
	2414	static ssize_t mem_cgroup_write(struct kernfs_open_file *of,
	2415	char *buf, size_t nbytes, loff_t off)
	2416	{
	2417	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	2418	unsigned long nr_pages;
	2419	int ret;
	2420
	2421	buf = strstrip(buf);
	2422	ret = page_counter_memparse(buf, "-1", &nr_pages);
	2423	if (ret)
	2424	return ret;
	2425
	2426	switch (MEMFILE_ATTR(of_cft(of)->private)) {
	2427	case RES_LIMIT:
	2428	if (mem_cgroup_is_root(memcg)) { /* Can't set limit on root */
	2429	ret = -EINVAL;
	2430	break;
	2431	}
	2432	switch (MEMFILE_TYPE(of_cft(of)->private)) {
	2433	case _MEM:
	2434	ret = mem_cgroup_resize_max(memcg, nr_pages, false);
	2435	break;
	2436	case _MEMSWAP:
	2437	ret = mem_cgroup_resize_max(memcg, nr_pages, true);
	2438	break;
	2439	case _KMEM:
	2440	pr_warn_once("kmem.limit_in_bytes is deprecated and will be removed. "
	2441	"Writing any value to this file has no effect. "
	2442	"Please report your usecase to linux-mm@kvack.org if you "
	2443	"depend on this functionality.\n");
	2444	ret = 0;
	2445	break;
	2446	case _TCP:
	2447	ret = memcg_update_tcp_max(memcg, nr_pages);
	2448	break;
	2449	}
	2450	break;
	2451	case RES_SOFT_LIMIT:
	2452	if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
	2453	ret = -EOPNOTSUPP;
	2454	} else {
	2455	WRITE_ONCE(memcg->soft_limit, nr_pages);
	2456	ret = 0;
	2457	}
	2458	break;
	2459	}
	2460	return ret ?: nbytes;
	2461	}
	2462
	2463	static ssize_t mem_cgroup_reset(struct kernfs_open_file of, char buf,
	2464	size_t nbytes, loff_t off)
	2465	{
	2466	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	2467	struct page_counter *counter;
	2468
	2469	switch (MEMFILE_TYPE(of_cft(of)->private)) {
	2470	case _MEM:
	2471	counter = &memcg->memory;
	2472	break;
	2473	case _MEMSWAP:
	2474	counter = &memcg->memsw;
	2475	break;
	2476	case _KMEM:
	2477	counter = &memcg->kmem;
	2478	break;
	2479	case _TCP:
	2480	counter = &memcg->tcpmem;
	2481	break;
	2482	default:
	2483	BUG();
	2484	}
	2485
	2486	switch (MEMFILE_ATTR(of_cft(of)->private)) {
	2487	case RES_MAX_USAGE:
	2488	page_counter_reset_watermark(counter);
	2489	break;
	2490	case RES_FAILCNT:
	2491	counter->failcnt = 0;
	2492	break;
	2493	default:
	2494	BUG();
	2495	}
	2496
	2497	return nbytes;
	2498	}
	2499
	2500	#ifdef CONFIG_NUMA
	2501
	2502	#define LRU_ALL_FILE (BIT(LRU_INACTIVE_FILE) \| BIT(LRU_ACTIVE_FILE))
	2503	#define LRU_ALL_ANON (BIT(LRU_INACTIVE_ANON) \| BIT(LRU_ACTIVE_ANON))
	2504	#define LRU_ALL ((1 << NR_LRU_LISTS) - 1)
	2505
	2506	static unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg,
	2507	int nid, unsigned int lru_mask, bool tree)
	2508	{
	2509	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
	2510	unsigned long nr = 0;
	2511	enum lru_list lru;
	2512
	2513	VM_BUG_ON((unsigned)nid >= nr_node_ids);
	2514
	2515	for_each_lru(lru) {
	2516	if (!(BIT(lru) & lru_mask))
	2517	continue;
	2518	if (tree)
	2519	nr += lruvec_page_state(lruvec, NR_LRU_BASE + lru);
	2520	else
	2521	nr += lruvec_page_state_local(lruvec, NR_LRU_BASE + lru);
	2522	}
	2523	return nr;
	2524	}
	2525
	2526	static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg,
	2527	unsigned int lru_mask,
	2528	bool tree)
	2529	{
	2530	unsigned long nr = 0;
	2531	enum lru_list lru;
	2532
	2533	for_each_lru(lru) {
	2534	if (!(BIT(lru) & lru_mask))
	2535	continue;
	2536	if (tree)
	2537	nr += memcg_page_state(memcg, NR_LRU_BASE + lru);
	2538	else
	2539	nr += memcg_page_state_local(memcg, NR_LRU_BASE + lru);
	2540	}
	2541	return nr;
	2542	}
	2543
	2544	static int memcg_numa_stat_show(struct seq_file m, void v)
	2545	{
	2546	struct numa_stat {
	2547	const char *name;
	2548	unsigned int lru_mask;
	2549	};
	2550
	2551	static const struct numa_stat stats[] = {
	2552	{ "total", LRU_ALL },
	2553	{ "file", LRU_ALL_FILE },
	2554	{ "anon", LRU_ALL_ANON },
	2555	{ "unevictable", BIT(LRU_UNEVICTABLE) },
	2556	};
	2557	const struct numa_stat *stat;
	2558	int nid;
	2559	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
	2560
	2561	mem_cgroup_flush_stats(memcg);
	2562
	2563	for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) {
	2564	seq_printf(m, "%s=%lu", stat->name,
	2565	mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
	2566	false));
	2567	for_each_node_state(nid, N_MEMORY)
	2568	seq_printf(m, " N%d=%lu", nid,
	2569	mem_cgroup_node_nr_lru_pages(memcg, nid,
	2570	stat->lru_mask, false));
	2571	seq_putc(m, '\n');
	2572	}
	2573
	2574	for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) {
	2575
	2576	seq_printf(m, "hierarchical_%s=%lu", stat->name,
	2577	mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
	2578	true));
	2579	for_each_node_state(nid, N_MEMORY)
	2580	seq_printf(m, " N%d=%lu", nid,
	2581	mem_cgroup_node_nr_lru_pages(memcg, nid,
	2582	stat->lru_mask, true));
	2583	seq_putc(m, '\n');
	2584	}
	2585
	2586	return 0;
	2587	}
	2588	#endif /* CONFIG_NUMA */
	2589
	2590	static const unsigned int memcg1_stats[] = {
	2591	NR_FILE_PAGES,
	2592	NR_ANON_MAPPED,
	2593	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	2594	NR_ANON_THPS,
	2595	#endif
	2596	NR_SHMEM,
	2597	NR_FILE_MAPPED,
	2598	NR_FILE_DIRTY,
	2599	NR_WRITEBACK,
	2600	WORKINGSET_REFAULT_ANON,
	2601	WORKINGSET_REFAULT_FILE,
	2602	#ifdef CONFIG_SWAP
	2603	MEMCG_SWAP,
	2604	NR_SWAPCACHE,
	2605	#endif
	2606	};
	2607
	2608	static const char *const memcg1_stat_names[] = {
	2609	"cache",
	2610	"rss",
	2611	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	2612	"rss_huge",
	2613	#endif
	2614	"shmem",
	2615	"mapped_file",
	2616	"dirty",
	2617	"writeback",
	2618	"workingset_refault_anon",
	2619	"workingset_refault_file",
	2620	#ifdef CONFIG_SWAP
	2621	"swap",
	2622	"swapcached",
	2623	#endif
	2624	};
	2625
	2626	/* Universal VM events cgroup1 shows, original sort order */
	2627	static const unsigned int memcg1_events[] = {
	2628	PGPGIN,
	2629	PGPGOUT,
	2630	PGFAULT,
	2631	PGMAJFAULT,
	2632	};
	2633
	2634	void memcg1_stat_format(struct mem_cgroup memcg, struct seq_buf s)
	2635	{
	2636	unsigned long memory, memsw;
	2637	struct mem_cgroup *mi;
	2638	unsigned int i;
	2639
	2640	BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats));
	2641
	2642	mem_cgroup_flush_stats(memcg);
	2643
	2644	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
	2645	unsigned long nr;
	2646
	2647	nr = memcg_page_state_local_output(memcg, memcg1_stats[i]);
	2648	seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr);
	2649	}
	2650
	2651	for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
	2652	seq_buf_printf(s, "%s %lu\n", vm_event_name(memcg1_events[i]),
	2653	memcg_events_local(memcg, memcg1_events[i]));
	2654
	2655	for (i = 0; i < NR_LRU_LISTS; i++)
	2656	seq_buf_printf(s, "%s %lu\n", lru_list_name(i),
	2657	memcg_page_state_local(memcg, NR_LRU_BASE + i) *
	2658	PAGE_SIZE);
	2659
	2660	/* Hierarchical information */
	2661	memory = memsw = PAGE_COUNTER_MAX;
	2662	for (mi = memcg; mi; mi = parent_mem_cgroup(mi)) {
	2663	memory = min(memory, READ_ONCE(mi->memory.max));
	2664	memsw = min(memsw, READ_ONCE(mi->memsw.max));
	2665	}
	2666	seq_buf_printf(s, "hierarchical_memory_limit %llu\n",
	2667	(u64)memory * PAGE_SIZE);
	2668	seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",
	2669	(u64)memsw * PAGE_SIZE);
	2670
	2671	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
	2672	unsigned long nr;
	2673
	2674	nr = memcg_page_state_output(memcg, memcg1_stats[i]);
	2675	seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i],
	2676	(u64)nr);
	2677	}
	2678
	2679	for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
	2680	seq_buf_printf(s, "total_%s %llu\n",
	2681	vm_event_name(memcg1_events[i]),
	2682	(u64)memcg_events(memcg, memcg1_events[i]));
	2683
	2684	for (i = 0; i < NR_LRU_LISTS; i++)
	2685	seq_buf_printf(s, "total_%s %llu\n", lru_list_name(i),
	2686	(u64)memcg_page_state(memcg, NR_LRU_BASE + i) *
	2687	PAGE_SIZE);
	2688
	2689	#ifdef CONFIG_DEBUG_VM
	2690	{
	2691	pg_data_t *pgdat;
	2692	struct mem_cgroup_per_node *mz;
	2693	unsigned long anon_cost = 0;
	2694	unsigned long file_cost = 0;
	2695
	2696	for_each_online_pgdat(pgdat) {
	2697	mz = memcg->nodeinfo[pgdat->node_id];
	2698
	2699	anon_cost += mz->lruvec.anon_cost;
	2700	file_cost += mz->lruvec.file_cost;
	2701	}
	2702	seq_buf_printf(s, "anon_cost %lu\n", anon_cost);
	2703	seq_buf_printf(s, "file_cost %lu\n", file_cost);
	2704	}
	2705	#endif
	2706	}
	2707
	2708	static u64 mem_cgroup_swappiness_read(struct cgroup_subsys_state *css,
	2709	struct cftype *cft)
	2710	{
	2711	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	2712
	2713	return mem_cgroup_swappiness(memcg);
	2714	}
	2715
	2716	static int mem_cgroup_swappiness_write(struct cgroup_subsys_state *css,
	2717	struct cftype *cft, u64 val)
	2718	{
	2719	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	2720
	2721	if (val > MAX_SWAPPINESS)
	2722	return -EINVAL;
	2723
	2724	if (!mem_cgroup_is_root(memcg))
	2725	WRITE_ONCE(memcg->swappiness, val);
	2726	else
	2727	WRITE_ONCE(vm_swappiness, val);
	2728
	2729	return 0;
	2730	}
	2731
	2732	static int mem_cgroup_oom_control_read(struct seq_file sf, void v)
	2733	{
	2734	struct mem_cgroup *memcg = mem_cgroup_from_seq(sf);
	2735
	2736	seq_printf(sf, "oom_kill_disable %d\n", READ_ONCE(memcg->oom_kill_disable));
	2737	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
	2738	seq_printf(sf, "oom_kill %lu\n",
	2739	atomic_long_read(&memcg->memory_events[MEMCG_OOM_KILL]));
	2740	return 0;
	2741	}
	2742
	2743	static int mem_cgroup_oom_control_write(struct cgroup_subsys_state *css,
	2744	struct cftype *cft, u64 val)
	2745	{
	2746	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	2747
	2748	/* cannot set to root cgroup and only 0 and 1 are allowed */
	2749	if (mem_cgroup_is_root(memcg) \|\| !((val == 0) \|\| (val == 1)))
	2750	return -EINVAL;
	2751
	2752	WRITE_ONCE(memcg->oom_kill_disable, val);
	2753	if (!val)
	2754	memcg1_oom_recover(memcg);
	2755
	2756	return 0;
	2757	}
	2758
	2759	#ifdef CONFIG_SLUB_DEBUG
	2760	static int mem_cgroup_slab_show(struct seq_file m, void p)
	2761	{
	2762	/*
	2763	* Deprecated.
	2764	* Please, take a look at tools/cgroup/memcg_slabinfo.py .
	2765	*/
	2766	return 0;
	2767	}
	2768	#endif
	2769
	2770	struct cftype mem_cgroup_legacy_files[] = {
	2771	{
	2772	.name = "usage_in_bytes",
	2773	.private = MEMFILE_PRIVATE(_MEM, RES_USAGE),
	2774	.read_u64 = mem_cgroup_read_u64,
	2775	},
	2776	{
	2777	.name = "max_usage_in_bytes",
	2778	.private = MEMFILE_PRIVATE(_MEM, RES_MAX_USAGE),
	2779	.write = mem_cgroup_reset,
	2780	.read_u64 = mem_cgroup_read_u64,
	2781	},
	2782	{
	2783	.name = "limit_in_bytes",
	2784	.private = MEMFILE_PRIVATE(_MEM, RES_LIMIT),
	2785	.write = mem_cgroup_write,
	2786	.read_u64 = mem_cgroup_read_u64,
	2787	},
	2788	{
	2789	.name = "soft_limit_in_bytes",
	2790	.private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT),
	2791	.write = mem_cgroup_write,
	2792	.read_u64 = mem_cgroup_read_u64,
	2793	},
	2794	{
	2795	.name = "failcnt",
	2796	.private = MEMFILE_PRIVATE(_MEM, RES_FAILCNT),
	2797	.write = mem_cgroup_reset,
	2798	.read_u64 = mem_cgroup_read_u64,
	2799	},
	2800	{
	2801	.name = "stat",
	2802	.seq_show = memory_stat_show,
	2803	},
	2804	{
	2805	.name = "force_empty",
	2806	.write = mem_cgroup_force_empty_write,
	2807	},
	2808	{
	2809	.name = "use_hierarchy",
	2810	.write_u64 = mem_cgroup_hierarchy_write,
	2811	.read_u64 = mem_cgroup_hierarchy_read,
	2812	},
	2813	{
	2814	.name = "cgroup.event_control", /* XXX: for compat */
	2815	.write = memcg_write_event_control,
	2816	.flags = CFTYPE_NO_PREFIX \| CFTYPE_WORLD_WRITABLE,
	2817	},
	2818	{
	2819	.name = "swappiness",
	2820	.read_u64 = mem_cgroup_swappiness_read,
	2821	.write_u64 = mem_cgroup_swappiness_write,
	2822	},
	2823	{
	2824	.name = "move_charge_at_immigrate",
	2825	.read_u64 = mem_cgroup_move_charge_read,
	2826	.write_u64 = mem_cgroup_move_charge_write,
	2827	},
	2828	{
	2829	.name = "oom_control",
	2830	.seq_show = mem_cgroup_oom_control_read,
	2831	.write_u64 = mem_cgroup_oom_control_write,
	2832	},
	2833	{
	2834	.name = "pressure_level",
	2835	.seq_show = mem_cgroup_dummy_seq_show,
	2836	},
	2837	#ifdef CONFIG_NUMA
	2838	{
	2839	.name = "numa_stat",
	2840	.seq_show = memcg_numa_stat_show,
	2841	},
	2842	#endif
	2843	{
	2844	.name = "kmem.limit_in_bytes",
	2845	.private = MEMFILE_PRIVATE(_KMEM, RES_LIMIT),
	2846	.write = mem_cgroup_write,
	2847	.read_u64 = mem_cgroup_read_u64,
	2848	},
	2849	{
	2850	.name = "kmem.usage_in_bytes",
	2851	.private = MEMFILE_PRIVATE(_KMEM, RES_USAGE),
	2852	.read_u64 = mem_cgroup_read_u64,
	2853	},
	2854	{
	2855	.name = "kmem.failcnt",
	2856	.private = MEMFILE_PRIVATE(_KMEM, RES_FAILCNT),
	2857	.write = mem_cgroup_reset,
	2858	.read_u64 = mem_cgroup_read_u64,
	2859	},
	2860	{
	2861	.name = "kmem.max_usage_in_bytes",
	2862	.private = MEMFILE_PRIVATE(_KMEM, RES_MAX_USAGE),
	2863	.write = mem_cgroup_reset,
	2864	.read_u64 = mem_cgroup_read_u64,
	2865	},
	2866	#ifdef CONFIG_SLUB_DEBUG
	2867	{
	2868	.name = "kmem.slabinfo",
	2869	.seq_show = mem_cgroup_slab_show,
	2870	},
	2871	#endif
	2872	{
	2873	.name = "kmem.tcp.limit_in_bytes",
	2874	.private = MEMFILE_PRIVATE(_TCP, RES_LIMIT),
	2875	.write = mem_cgroup_write,
	2876	.read_u64 = mem_cgroup_read_u64,
	2877	},
	2878	{
	2879	.name = "kmem.tcp.usage_in_bytes",
	2880	.private = MEMFILE_PRIVATE(_TCP, RES_USAGE),
	2881	.read_u64 = mem_cgroup_read_u64,
	2882	},
	2883	{
	2884	.name = "kmem.tcp.failcnt",
	2885	.private = MEMFILE_PRIVATE(_TCP, RES_FAILCNT),
	2886	.write = mem_cgroup_reset,
	2887	.read_u64 = mem_cgroup_read_u64,
	2888	},
	2889	{
	2890	.name = "kmem.tcp.max_usage_in_bytes",
	2891	.private = MEMFILE_PRIVATE(_TCP, RES_MAX_USAGE),
	2892	.write = mem_cgroup_reset,
	2893	.read_u64 = mem_cgroup_read_u64,
	2894	},
	2895	{ }, /* terminate */
	2896	};
	2897
	2898	struct cftype memsw_files[] = {
	2899	{
	2900	.name = "memsw.usage_in_bytes",
	2901	.private = MEMFILE_PRIVATE(_MEMSWAP, RES_USAGE),
	2902	.read_u64 = mem_cgroup_read_u64,
	2903	},
	2904	{
	2905	.name = "memsw.max_usage_in_bytes",
	2906	.private = MEMFILE_PRIVATE(_MEMSWAP, RES_MAX_USAGE),
	2907	.write = mem_cgroup_reset,
	2908	.read_u64 = mem_cgroup_read_u64,
	2909	},
	2910	{
	2911	.name = "memsw.limit_in_bytes",
	2912	.private = MEMFILE_PRIVATE(_MEMSWAP, RES_LIMIT),
	2913	.write = mem_cgroup_write,
	2914	.read_u64 = mem_cgroup_read_u64,
	2915	},
	2916	{
	2917	.name = "memsw.failcnt",
	2918	.private = MEMFILE_PRIVATE(_MEMSWAP, RES_FAILCNT),
	2919	.write = mem_cgroup_reset,
	2920	.read_u64 = mem_cgroup_read_u64,
	2921	},
	2922	{ }, /* terminate */
	2923	};
	2924
	2925	void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages)
	2926	{
	2927	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
	2928	if (nr_pages > 0)
	2929	page_counter_charge(&memcg->kmem, nr_pages);
	2930	else
	2931	page_counter_uncharge(&memcg->kmem, -nr_pages);
	2932	}
	2933	}
	2934
	2935	bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
	2936	gfp_t gfp_mask)
	2937	{
	2938	struct page_counter *fail;
	2939
	2940	if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) {
	2941	memcg->tcpmem_pressure = 0;
	2942	return true;
	2943	}
	2944	memcg->tcpmem_pressure = 1;
	2945	if (gfp_mask & __GFP_NOFAIL) {
	2946	page_counter_charge(&memcg->tcpmem, nr_pages);
	2947	return true;
	2948	}
	2949	return false;
	2950	}
	2951
	2952	static int __init memcg1_init(void)
	2953	{
	2954	int node;
	2955
	2956	for_each_node(node) {
	2957	struct mem_cgroup_tree_per_node *rtpn;
	2958
	2959	rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL, node);
	2960
	2961	rtpn->rb_root = RB_ROOT;
	2962	rtpn->rb_rightmost = NULL;
	2963	spin_lock_init(&rtpn->lock);
	2964	soft_limit_tree.rb_tree_per_node[node] = rtpn;
	2965	}
	2966
	2967	return 0;
	2968	}
	2969	subsys_initcall(memcg1_init);

mm/memcontrol-v1.h+147−0

@@ -0,0 +1,147 @@
	1	/* SPDX-License-Identifier: GPL-2.0-or-later */
	2
	3	#ifndef __MM_MEMCONTROL_V1_H
	4	#define __MM_MEMCONTROL_V1_H
	5
	6	#include <linux/cgroup-defs.h>
	7
	8	/* Cgroup v1 and v2 common declarations */
	9
	10	void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, int nr_pages);
	11	int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
	12	unsigned int nr_pages);
	13
	14	static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
	15	unsigned int nr_pages)
	16	{
	17	if (mem_cgroup_is_root(memcg))
	18	return 0;
	19
	20	return try_charge_memcg(memcg, gfp_mask, nr_pages);
	21	}
	22
	23	void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n);
	24	void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n);
	25
	26	/*
	27	* Iteration constructs for visiting all cgroups (under a tree). If
	28	* loops are exited prematurely (break), mem_cgroup_iter_break() must
	29	* be used for reference counting.
	30	*/
	31	#define for_each_mem_cgroup_tree(iter, root) \
	32	for (iter = mem_cgroup_iter(root, NULL, NULL); \
	33	iter != NULL; \
	34	iter = mem_cgroup_iter(root, iter, NULL))
	35
	36	#define for_each_mem_cgroup(iter) \
	37	for (iter = mem_cgroup_iter(NULL, NULL, NULL); \
	38	iter != NULL; \
	39	iter = mem_cgroup_iter(NULL, iter, NULL))
	40
	41	/* Whether legacy memory+swap accounting is active */
	42	static bool do_memsw_account(void)
	43	{
	44	return !cgroup_subsys_on_dfl(memory_cgrp_subsys);
	45	}
	46
	47	/*
	48	* Per memcg event counter is incremented at every pagein/pageout. With THP,
	49	* it will be incremented by the number of pages. This counter is used
	50	* to trigger some periodic events. This is straightforward and better
	51	* than using jiffies etc. to handle periodic memcg event.
	52	*/
	53	enum mem_cgroup_events_target {
	54	MEM_CGROUP_TARGET_THRESH,
	55	MEM_CGROUP_TARGET_SOFTLIMIT,
	56	MEM_CGROUP_NTARGETS,
	57	};
	58
	59	bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg,
	60	enum mem_cgroup_events_target target);
	61	unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap);
	62
	63	void drain_all_stock(struct mem_cgroup *root_memcg);
	64
	65	unsigned long memcg_events(struct mem_cgroup *memcg, int event);
	66	unsigned long memcg_events_local(struct mem_cgroup *memcg, int event);
	67	unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx);
	68	unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item);
	69	unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item);
	70	int memory_stat_show(struct seq_file m, void v);
	71
	72	/* Cgroup v1-specific declarations */
	73	#ifdef CONFIG_MEMCG_V1
	74	void memcg1_memcg_init(struct mem_cgroup *memcg);
	75	void memcg1_remove_from_trees(struct mem_cgroup *memcg);
	76
	77	static inline void memcg1_soft_limit_reset(struct mem_cgroup *memcg)
	78	{
	79	WRITE_ONCE(memcg->soft_limit, PAGE_COUNTER_MAX);
	80	}
	81
	82	bool memcg1_wait_acct_move(struct mem_cgroup *memcg);
	83
	84	struct cgroup_taskset;
	85	int memcg1_can_attach(struct cgroup_taskset *tset);
	86	void memcg1_cancel_attach(struct cgroup_taskset *tset);
	87	void memcg1_move_task(void);
	88	void memcg1_css_offline(struct mem_cgroup *memcg);
	89
	90	/* for encoding cft->private value on file */
	91	enum res_type {
	92	_MEM,
	93	_MEMSWAP,
	94	_KMEM,
	95	_TCP,
	96	};
	97
	98	bool memcg1_oom_prepare(struct mem_cgroup memcg, bool locked);
	99	void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked);
	100	void memcg1_oom_recover(struct mem_cgroup *memcg);
	101
	102	void memcg1_check_events(struct mem_cgroup *memcg, int nid);
	103
	104	void memcg1_stat_format(struct mem_cgroup memcg, struct seq_buf s);
	105
	106	void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages);
	107	static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg)
	108	{
	109	return memcg->tcpmem_active;
	110	}
	111	bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
	112	gfp_t gfp_mask);
	113	static inline void memcg1_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages)
	114	{
	115	page_counter_uncharge(&memcg->tcpmem, nr_pages);
	116	}
	117
	118	extern struct cftype memsw_files[];
	119	extern struct cftype mem_cgroup_legacy_files[];
	120
	121	#else /* CONFIG_MEMCG_V1 */
	122
	123	static inline void memcg1_memcg_init(struct mem_cgroup *memcg) {}
	124	static inline void memcg1_remove_from_trees(struct mem_cgroup *memcg) {}
	125	static inline void memcg1_soft_limit_reset(struct mem_cgroup *memcg) {}
	126	static inline bool memcg1_wait_acct_move(struct mem_cgroup *memcg) { return false; }
	127	static inline void memcg1_css_offline(struct mem_cgroup *memcg) {}
	128
	129	static inline bool memcg1_oom_prepare(struct mem_cgroup memcg, bool locked) { return true; }
	130	static inline void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked) {}
	131	static inline void memcg1_oom_recover(struct mem_cgroup *memcg) {}
	132
	133	static inline void memcg1_check_events(struct mem_cgroup *memcg, int nid) {}
	134
	135	static inline void memcg1_stat_format(struct mem_cgroup memcg, struct seq_buf s) {}
	136
	137	static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {}
	138	static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; }
	139	static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages,
	140	gfp_t gfp_mask) { return true; }
	141	static inline void memcg1_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) {}
	142
	143	extern struct cftype memsw_files[];
	144	extern struct cftype mem_cgroup_legacy_files[];
	145	#endif /* CONFIG_MEMCG_V1 */
	146
	147	#endif /* __MM_MEMCONTROL_V1_H */

mm/memcontrol.c+649−3768

@@ -28,7 +28,6 @@ · 27 unmodified lines
28	#include <linux/page_counter.h>	28	#include <linux/page_counter.h>
29	#include <linux/memcontrol.h>	29	#include <linux/memcontrol.h>
30	#include <linux/cgroup.h>	30	#include <linux/cgroup.h>
31	#include <linux/pagewalk.h>
32	#include <linux/sched/mm.h>	31	#include <linux/sched/mm.h>
33	#include <linux/shmem_fs.h>	32	#include <linux/shmem_fs.h>
34	#include <linux/hugetlb.h>	33	#include <linux/hugetlb.h>
@@ -45,14 +44,11 @@ · 10 unmodified lines
45	#include <linux/mutex.h>	44	#include <linux/mutex.h>
46	#include <linux/rbtree.h>	45	#include <linux/rbtree.h>
47	#include <linux/slab.h>	46	#include <linux/slab.h>
48	#include <linux/swap.h>
49	#include <linux/swapops.h>	47	#include <linux/swapops.h>
50	#include <linux/spinlock.h>	48	#include <linux/spinlock.h>
51	#include <linux/eventfd.h>
52	#include <linux/poll.h>
53	#include <linux/sort.h>
54	#include <linux/fs.h>	49	#include <linux/fs.h>
55	#include <linux/seq_file.h>	50	#include <linux/seq_file.h>
		51	#include <linux/parser.h>
56	#include <linux/vmpressure.h>	52	#include <linux/vmpressure.h>
57	#include <linux/memremap.h>	53	#include <linux/memremap.h>
58	#include <linux/mm_inline.h>	54	#include <linux/mm_inline.h>
@@ -60,7 +56,6 @@ · 1 unmodified line
60	#include <linux/cpu.h>	56	#include <linux/cpu.h>
61	#include <linux/oom.h>	57	#include <linux/oom.h>
62	#include <linux/lockdep.h>	58	#include <linux/lockdep.h>
63	#include <linux/file.h>
64	#include <linux/resume_user_mode.h>	59	#include <linux/resume_user_mode.h>
65	#include <linux/psi.h>	60	#include <linux/psi.h>
66	#include <linux/seq_buf.h>	61	#include <linux/seq_buf.h>
@@ -70,7 +65,7 @@ · 3 unmodified lines
70	#include <net/sock.h>	65	#include <net/sock.h>
71	#include <net/ip.h>	66	#include <net/ip.h>
72	#include "slab.h"	67	#include "slab.h"
73	#include "swap.h"	68	#include "memcontrol-v1.h"
74		69
75	#include <linux/uaccess.h>	70	#include <linux/uaccess.h>
76		71
@@ -98,140 +93,9 @@ static bool cgroup_memory_nobpf __ro_after_init; · 21 unmodified lines
98	static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq);	93	static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq);
99	#endif	94	#endif
100		95
101	/* Whether legacy memory+swap accounting is active */
102	static bool do_memsw_account(void)
103	{
104	return !cgroup_subsys_on_dfl(memory_cgrp_subsys);
105	}
106
107	#define THRESHOLDS_EVENTS_TARGET 128	96	#define THRESHOLDS_EVENTS_TARGET 128
108	#define SOFTLIMIT_EVENTS_TARGET 1024	97	#define SOFTLIMIT_EVENTS_TARGET 1024
109		98
110	/*
111	* Cgroups above their limits are maintained in a RB-Tree, independent of
112	* their hierarchy representation
113	*/
114
115	struct mem_cgroup_tree_per_node {
116	struct rb_root rb_root;
117	struct rb_node *rb_rightmost;
118	spinlock_t lock;
119	};
120
121	struct mem_cgroup_tree {
122	struct mem_cgroup_tree_per_node *rb_tree_per_node[MAX_NUMNODES];
123	};
124
125	static struct mem_cgroup_tree soft_limit_tree __read_mostly;
126
127	/* for OOM */
128	struct mem_cgroup_eventfd_list {
129	struct list_head list;
130	struct eventfd_ctx *eventfd;
131	};
132
133	/*
134	* cgroup_event represents events which userspace want to receive.
135	*/
136	struct mem_cgroup_event {
137	/*
138	* memcg which the event belongs to.
139	*/
140	struct mem_cgroup *memcg;
141	/*
142	* eventfd to signal userspace about the event.
143	*/
144	struct eventfd_ctx *eventfd;
145	/*
146	* Each of these stored in a list by the cgroup.
147	*/
148	struct list_head list;
149	/*
150	* register_event() callback will be used to add new userspace
151	* waiter for changes related to this event. Use eventfd_signal()
152	* on eventfd to send notification to userspace.
153	*/
154	int (register_event)(struct mem_cgroup memcg,
155	struct eventfd_ctx eventfd, const char args);
156	/*
157	* unregister_event() callback will be called when userspace closes
158	* the eventfd or on cgroup removing. This callback must be set,
159	* if you want provide notification functionality.
160	*/
161	void (unregister_event)(struct mem_cgroup memcg,
162	struct eventfd_ctx *eventfd);
163	/*
164	* All fields below needed to unregister event when
165	* userspace closes eventfd.
166	*/
167	poll_table pt;
168	wait_queue_head_t *wqh;
169	wait_queue_entry_t wait;
170	struct work_struct remove;
171	};
172
173	static void mem_cgroup_threshold(struct mem_cgroup *memcg);
174	static void mem_cgroup_oom_notify(struct mem_cgroup *memcg);
175
176	/* Stuffs for move charges at task migration. */
177	/*
178	* Types of charges to be moved.
179	*/
180	#define MOVE_ANON 0x1U
181	#define MOVE_FILE 0x2U
182	#define MOVE_MASK (MOVE_ANON \| MOVE_FILE)
183
184	/* "mc" and its members are protected by cgroup_mutex */
185	static struct move_charge_struct {
186	spinlock_t lock; /* for from, to */
187	struct mm_struct *mm;
188	struct mem_cgroup *from;
189	struct mem_cgroup *to;
190	unsigned long flags;
191	unsigned long precharge;
192	unsigned long moved_charge;
193	unsigned long moved_swap;
194	struct task_struct moving_task; / a task moving charges */
195	wait_queue_head_t waitq; /* a waitq for other context */
196	} mc = {
197	.lock = __SPIN_LOCK_UNLOCKED(mc.lock),
198	.waitq = __WAIT_QUEUE_HEAD_INITIALIZER(mc.waitq),
199	};
200
201	/*
202	* Maximum loops in mem_cgroup_soft_reclaim(), used for soft
203	* limit reclaim to prevent infinite loops, if they ever occur.
204	*/
205	#define MEM_CGROUP_MAX_RECLAIM_LOOPS 100
206	#define MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS 2
207
208	/* for encoding cft->private value on file */
209	enum res_type {
210	_MEM,
211	_MEMSWAP,
212	_KMEM,
213	_TCP,
214	};
215
216	#define MEMFILE_PRIVATE(x, val) ((x) << 16 \| (val))
217	#define MEMFILE_TYPE(val) ((val) >> 16 & 0xffff)
218	#define MEMFILE_ATTR(val) ((val) & 0xffff)
219
220	/*
221	* Iteration constructs for visiting all cgroups (under a tree). If
222	* loops are exited prematurely (break), mem_cgroup_iter_break() must
223	* be used for reference counting.
224	*/
225	#define for_each_mem_cgroup_tree(iter, root) \
226	for (iter = mem_cgroup_iter(root, NULL, NULL); \
227	iter != NULL; \
228	iter = mem_cgroup_iter(root, iter, NULL))
229
230	#define for_each_mem_cgroup(iter) \
231	for (iter = mem_cgroup_iter(NULL, NULL, NULL); \
232	iter != NULL; \
233	iter = mem_cgroup_iter(NULL, iter, NULL))
234
235	static inline bool task_is_dying(void)	99	static inline bool task_is_dying(void)
236	{	100	{
237	return tsk_is_oom_victim(current) \|\| fatal_signal_pending(current) \|\|	101	return tsk_is_oom_victim(current) \|\| fatal_signal_pending(current) \|\|
@@ -254,7 +118,6 @@ struct mem_cgroup vmpressure_to_memcg(struct vmpressure vmpr) · 16 unmodified lines
254	#define CURRENT_OBJCG_UPDATE_BIT 0	118	#define CURRENT_OBJCG_UPDATE_BIT 0
255	#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT)	119	#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT)
256		120
257	#ifdef CONFIG_MEMCG_KMEM
258	static DEFINE_SPINLOCK(objcg_lock);	121	static DEFINE_SPINLOCK(objcg_lock);
259		122
260	bool mem_cgroup_kmem_disabled(void)	123	bool mem_cgroup_kmem_disabled(void)
@@ -359,7 +222,6 @@ EXPORT_SYMBOL(memcg_kmem_online_key); · 98 unmodified lines
359		222
360	DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);	223	DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);
361	EXPORT_SYMBOL(memcg_bpf_enabled_key);	224	EXPORT_SYMBOL(memcg_bpf_enabled_key);
362	#endif
363		225
364	/**	226	/**
365	* mem_cgroup_css_from_folio - css of the memcg associated with a folio	227	* mem_cgroup_css_from_folio - css of the memcg associated with a folio
@@ -412,169 +274,6 @@ ino_t page_cgroup_ino(struct page *page) · 46 unmodified lines
412	return ino;	274	return ino;
413	}	275	}
414		276
415	static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz,
416	struct mem_cgroup_tree_per_node *mctz,
417	unsigned long new_usage_in_excess)
418	{
419	struct rb_node **p = &mctz->rb_root.rb_node;
420	struct rb_node *parent = NULL;
421	struct mem_cgroup_per_node *mz_node;
422	bool rightmost = true;
423
424	if (mz->on_tree)
425	return;
426
427	mz->usage_in_excess = new_usage_in_excess;
428	if (!mz->usage_in_excess)
429	return;
430	while (*p) {
431	parent = *p;
432	mz_node = rb_entry(parent, struct mem_cgroup_per_node,
433	tree_node);
434	if (mz->usage_in_excess < mz_node->usage_in_excess) {
435	p = &(*p)->rb_left;
436	rightmost = false;
437	} else {
438	p = &(*p)->rb_right;
439	}
440	}
441
442	if (rightmost)
443	mctz->rb_rightmost = &mz->tree_node;
444
445	rb_link_node(&mz->tree_node, parent, p);
446	rb_insert_color(&mz->tree_node, &mctz->rb_root);
447	mz->on_tree = true;
448	}
449
450	static void __mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz,
451	struct mem_cgroup_tree_per_node *mctz)
452	{
453	if (!mz->on_tree)
454	return;
455
456	if (&mz->tree_node == mctz->rb_rightmost)
457	mctz->rb_rightmost = rb_prev(&mz->tree_node);
458
459	rb_erase(&mz->tree_node, &mctz->rb_root);
460	mz->on_tree = false;
461	}
462
463	static void mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz,
464	struct mem_cgroup_tree_per_node *mctz)
465	{
466	unsigned long flags;
467
468	spin_lock_irqsave(&mctz->lock, flags);
469	__mem_cgroup_remove_exceeded(mz, mctz);
470	spin_unlock_irqrestore(&mctz->lock, flags);
471	}
472
473	static unsigned long soft_limit_excess(struct mem_cgroup *memcg)
474	{
475	unsigned long nr_pages = page_counter_read(&memcg->memory);
476	unsigned long soft_limit = READ_ONCE(memcg->soft_limit);
477	unsigned long excess = 0;
478
479	if (nr_pages > soft_limit)
480	excess = nr_pages - soft_limit;
481
482	return excess;
483	}
484
485	static void mem_cgroup_update_tree(struct mem_cgroup *memcg, int nid)
486	{
487	unsigned long excess;
488	struct mem_cgroup_per_node *mz;
489	struct mem_cgroup_tree_per_node *mctz;
490
491	if (lru_gen_enabled()) {
492	if (soft_limit_excess(memcg))
493	lru_gen_soft_reclaim(memcg, nid);
494	return;
495	}
496
497	mctz = soft_limit_tree.rb_tree_per_node[nid];
498	if (!mctz)
499	return;
500	/*
501	* Necessary to update all ancestors when hierarchy is used.
502	* because their event counter is not touched.
503	*/
504	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
505	mz = memcg->nodeinfo[nid];
506	excess = soft_limit_excess(memcg);
507	/*
508	* We have to update the tree if mz is on RB-tree or
509	* mem is over its softlimit.
510	*/
511	if (excess \|\| mz->on_tree) {
512	unsigned long flags;
513
514	spin_lock_irqsave(&mctz->lock, flags);
515	/* if on-tree, remove it */
516	if (mz->on_tree)
517	__mem_cgroup_remove_exceeded(mz, mctz);
518	/*
519	* Insert again. mz->usage_in_excess will be updated.
520	* If excess is 0, no tree ops.
521	*/
522	__mem_cgroup_insert_exceeded(mz, mctz, excess);
523	spin_unlock_irqrestore(&mctz->lock, flags);
524	}
525	}
526	}
527
528	static void mem_cgroup_remove_from_trees(struct mem_cgroup *memcg)
529	{
530	struct mem_cgroup_tree_per_node *mctz;
531	struct mem_cgroup_per_node *mz;
532	int nid;
533
534	for_each_node(nid) {
535	mz = memcg->nodeinfo[nid];
536	mctz = soft_limit_tree.rb_tree_per_node[nid];
537	if (mctz)
538	mem_cgroup_remove_exceeded(mz, mctz);
539	}
540	}
541
542	static struct mem_cgroup_per_node *
543	__mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)
544	{
545	struct mem_cgroup_per_node *mz;
546
547	retry:
548	mz = NULL;
549	if (!mctz->rb_rightmost)
550	goto done; /* Nothing to reclaim from */
551
552	mz = rb_entry(mctz->rb_rightmost,
553	struct mem_cgroup_per_node, tree_node);
554	/*
555	* Remove the node now but someone else can add it back,
556	* we will to add it back at the end of reclaim to its correct
557	* position in the tree.
558	*/
559	__mem_cgroup_remove_exceeded(mz, mctz);
560	if (!soft_limit_excess(mz->memcg) \|\|
561	!css_tryget(&mz->memcg->css))
562	goto retry;
563	done:
564	return mz;
565	}
566
567	static struct mem_cgroup_per_node *
568	mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)
569	{
570	struct mem_cgroup_per_node *mz;
571
572	spin_lock_irq(&mctz->lock);
573	mz = __mem_cgroup_largest_soft_limit_node(mctz);
574	spin_unlock_irq(&mctz->lock);
575	return mz;
576	}
577
578	/* Subset of node_stat_item for memcg stats */	277	/* Subset of node_stat_item for memcg stats */
579	static const unsigned int memcg_node_stat_items[] = {	278	static const unsigned int memcg_node_stat_items[] = {
580	NR_INACTIVE_ANON,	279	NR_INACTIVE_ANON,
@@ -722,7 +421,7 @@ static const unsigned int memcg_vm_event_stat[] = { · 141 unmodified lines
722	PGDEACTIVATE,	421	PGDEACTIVATE,
723	PGLAZYFREE,	422	PGLAZYFREE,
724	PGLAZYFREED,	423	PGLAZYFREED,
725	#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)	424	#ifdef CONFIG_ZSWAP
726	ZSWPIN,	425	ZSWPIN,
727	ZSWPOUT,	426	ZSWPOUT,
728	ZSWPWB,	427	ZSWPWB,
@@ -971,7 +670,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, · 242 unmodified lines
971	}	670	}
972		671
973	/* idx can be of type enum memcg_stat_item or node_stat_item. */	672	/* idx can be of type enum memcg_stat_item or node_stat_item. */
974	static unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)	673	unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
975	{	674	{
976	long x;	675	long x;
977	int i = memcg_stats_index(idx);	676	int i = memcg_stats_index(idx);
@@ -1120,7 +819,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, · 142 unmodified lines
1120	memcg_stats_unlock();	819	memcg_stats_unlock();
1121	}	820	}
1122		821
1123	static unsigned long memcg_events(struct mem_cgroup *memcg, int event)	822	unsigned long memcg_events(struct mem_cgroup *memcg, int event)
1124	{	823	{
1125	int i = memcg_events_index(event);	824	int i = memcg_events_index(event);
1126		825
@@ -1130,7 +829,7 @@ static unsigned long memcg_events(struct mem_cgroup *memcg, int event) · 3 unmodified lines
1130	return READ_ONCE(memcg->vmstats->events[i]);	829	return READ_ONCE(memcg->vmstats->events[i]);
1131	}	830	}
1132		831
1133	static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)	832	unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)
1134	{	833	{
1135	int i = memcg_events_index(event);	834	int i = memcg_events_index(event);
1136		835
@@ -1140,8 +839,7 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) · 3 unmodified lines
1140	return READ_ONCE(memcg->vmstats->events_local[i]);	839	return READ_ONCE(memcg->vmstats->events_local[i]);
1141	}	840	}
1142		841
1143	static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,	842	void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, int nr_pages)
1144	int nr_pages)
1145	{	843	{
1146	/* pagein of a big page is an event. So, ignore page size */	844	/* pagein of a big page is an event. So, ignore page size */
1147	if (nr_pages > 0)	845	if (nr_pages > 0)
@@ -1154,8 +852,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, · 6 unmodified lines
1154	__this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages);	852	__this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages);
1155	}	853	}
1156		854
1157	static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg,	855	bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg,
1158	enum mem_cgroup_events_target target)	856	enum mem_cgroup_events_target target)
1159	{	857	{
1160	unsigned long val, next;	858	unsigned long val, next;
1161		859
@@ -1179,28 +877,6 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, · 17 unmodified lines
1179	return false;	877	return false;
1180	}	878	}
1181		879
1182	/*
1183	* Check events in order.
1184	*
1185	*/
1186	static void memcg_check_events(struct mem_cgroup *memcg, int nid)
1187	{
1188	if (IS_ENABLED(CONFIG_PREEMPT_RT))
1189	return;
1190
1191	/* threshold event is triggered in finer grain than soft limit */
1192	if (unlikely(mem_cgroup_event_ratelimit(memcg,
1193	MEM_CGROUP_TARGET_THRESH))) {
1194	bool do_softlimit;
1195
1196	do_softlimit = mem_cgroup_event_ratelimit(memcg,
1197	MEM_CGROUP_TARGET_SOFTLIMIT);
1198	mem_cgroup_threshold(memcg);
1199	if (unlikely(do_softlimit))
1200	mem_cgroup_update_tree(memcg, nid);
1201	}
1202	}
1203
1204	struct mem_cgroup mem_cgroup_from_task(struct task_struct p)	880	struct mem_cgroup mem_cgroup_from_task(struct task_struct p)
1205	{	881	{
1206	/*	882	/*
@@ -1652,51 +1328,6 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg) · 445 unmodified lines
1652	return margin;	1328	return margin;
1653	}	1329	}
1654		1330
1655	/*
1656	* A routine for checking "mem" is under move_account() or not.
1657	*
1658	* Checking a cgroup is mc.from or mc.to or under hierarchy of
1659	* moving cgroups. This is for waiting at high-memory pressure
1660	* caused by "move".
1661	*/
1662	static bool mem_cgroup_under_move(struct mem_cgroup *memcg)
1663	{
1664	struct mem_cgroup *from;
1665	struct mem_cgroup *to;
1666	bool ret = false;
1667	/*
1668	* Unlike task_move routines, we access mc.to, mc.from not under
1669	* mutual exclusion by cgroup_mutex. Here, we take spinlock instead.
1670	*/
1671	spin_lock(&mc.lock);
1672	from = mc.from;
1673	to = mc.to;
1674	if (!from)
1675	goto unlock;
1676
1677	ret = mem_cgroup_is_descendant(from, memcg) \|\|
1678	mem_cgroup_is_descendant(to, memcg);
1679	unlock:
1680	spin_unlock(&mc.lock);
1681	return ret;
1682	}
1683
1684	static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg)
1685	{
1686	if (mc.moving_task && current != mc.moving_task) {
1687	if (mem_cgroup_under_move(memcg)) {
1688	DEFINE_WAIT(wait);
1689	prepare_to_wait(&mc.waitq, &wait, TASK_INTERRUPTIBLE);
1690	/* moving charge context might have finished. */
1691	if (mc.moving_task)
1692	schedule();
1693	finish_wait(&mc.waitq, &wait);
1694	return true;
1695	}
1696	}
1697	return false;
1698	}
1699
1700	struct memory_stat {	1331	struct memory_stat {
1701	const char *name;	1332	const char *name;
1702	unsigned int idx;	1333	unsigned int idx;
@@ -1713,7 +1344,7 @@ static const struct memory_stat memory_stats[] = { · 10 unmodified lines
1713	{ "sock", MEMCG_SOCK },	1344	{ "sock", MEMCG_SOCK },
1714	{ "vmalloc", MEMCG_VMALLOC },	1345	{ "vmalloc", MEMCG_VMALLOC },
1715	{ "shmem", NR_SHMEM },	1346	{ "shmem", NR_SHMEM },
1716	#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)	1347	#ifdef CONFIG_ZSWAP
1717	{ "zswap", MEMCG_ZSWAP_B },	1348	{ "zswap", MEMCG_ZSWAP_B },
1718	{ "zswapped", MEMCG_ZSWAPPED },	1349	{ "zswapped", MEMCG_ZSWAPPED },
1719	#endif	1350	#endif
@@ -1783,15 +1414,13 @@ static int memcg_page_state_output_unit(int item) · 63 unmodified lines
1783	}	1414	}
1784	}	1415	}
1785		1416
1786	static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg,	1417	unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item)
1787	int item)
1788	{	1418	{
1789	return memcg_page_state(memcg, item) *	1419	return memcg_page_state(memcg, item) *
1790	memcg_page_state_output_unit(item);	1420	memcg_page_state_output_unit(item);
1791	}	1421	}
1792		1422
1793	static inline unsigned long memcg_page_state_local_output(	1423	unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item)
1794	struct mem_cgroup *memcg, int item)
1795	{	1424	{
1796	return memcg_page_state_local(memcg, item) *	1425	return memcg_page_state_local(memcg, item) *
1797	memcg_page_state_output_unit(item);	1426	memcg_page_state_output_unit(item);
@@ -1845,20 +1474,16 @@ static void memcg_stat_format(struct mem_cgroup memcg, struct seq_buf s) · 47 unmodified lines
1845	vm_event_name(memcg_vm_event_stat[i]),	1474	vm_event_name(memcg_vm_event_stat[i]),
1846	memcg_events(memcg, memcg_vm_event_stat[i]));	1475	memcg_events(memcg, memcg_vm_event_stat[i]));
1847	}	1476	}
1848
1849	/* The above should easily fit into one page */
1850	WARN_ON_ONCE(seq_buf_has_overflowed(s));
1851	}	1477	}
1852		1478
1853	static void memcg1_stat_format(struct mem_cgroup memcg, struct seq_buf s);
1854
1855	static void memory_stat_format(struct mem_cgroup memcg, struct seq_buf s)	1479	static void memory_stat_format(struct mem_cgroup memcg, struct seq_buf s)
1856	{	1480	{
1857	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))	1481	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
1858	memcg_stat_format(memcg, s);	1482	memcg_stat_format(memcg, s);
1859	else	1483	else
1860	memcg1_stat_format(memcg, s);	1484	memcg1_stat_format(memcg, s);
1861	WARN_ON_ONCE(seq_buf_has_overflowed(s));	1485	if (seq_buf_has_overflowed(s))
		1486	pr_warn("%s: Warning, stat buffer overflow, please report\n", __func__);
1862	}	1487	}
1863		1488
1864	/**	1489	/**
@@ -1906,6 +1531,7 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) · 41 unmodified lines
1906	pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",	1531	pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
1907	K((u64)page_counter_read(&memcg->swap)),	1532	K((u64)page_counter_read(&memcg->swap)),
1908	K((u64)READ_ONCE(memcg->swap.max)), memcg->swap.failcnt);	1533	K((u64)READ_ONCE(memcg->swap.max)), memcg->swap.failcnt);
		1534	#ifdef CONFIG_MEMCG_V1
1909	else {	1535	else {
1910	pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",	1536	pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",
1911	K((u64)page_counter_read(&memcg->memsw)),	1537	K((u64)page_counter_read(&memcg->memsw)),
@@ -1914,6 +1540,7 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) · 2 unmodified lines
1914	K((u64)page_counter_read(&memcg->kmem)),	1540	K((u64)page_counter_read(&memcg->kmem)),
1915	K((u64)memcg->kmem.max), memcg->kmem.failcnt);	1541	K((u64)memcg->kmem.max), memcg->kmem.failcnt);
1916	}	1542	}
		1543	#endif
1917		1544
1918	pr_info("Memory cgroup stats for ");	1545	pr_info("Memory cgroup stats for ");
1919	pr_cont_cgroup_path(memcg->css.cgroup);	1546	pr_cont_cgroup_path(memcg->css.cgroup);
@@ -1979,180 +1606,6 @@ unlock: · 59 unmodified lines
1979	return ret;	1606	return ret;
1980	}	1607	}
1981		1608
1982	static int mem_cgroup_soft_reclaim(struct mem_cgroup *root_memcg,
1983	pg_data_t *pgdat,
1984	gfp_t gfp_mask,
1985	unsigned long *total_scanned)
1986	{
1987	struct mem_cgroup *victim = NULL;
1988	int total = 0;
1989	int loop = 0;
1990	unsigned long excess;
1991	unsigned long nr_scanned;
1992	struct mem_cgroup_reclaim_cookie reclaim = {
1993	.pgdat = pgdat,
1994	};
1995
1996	excess = soft_limit_excess(root_memcg);
1997
1998	while (1) {
1999	victim = mem_cgroup_iter(root_memcg, victim, &reclaim);
2000	if (!victim) {
2001	loop++;
2002	if (loop >= 2) {
2003	/*
2004	* If we have not been able to reclaim
2005	* anything, it might because there are
2006	* no reclaimable pages under this hierarchy
2007	*/
2008	if (!total)
2009	break;
2010	/*
2011	* We want to do more targeted reclaim.
2012	* excess >> 2 is not to excessive so as to
2013	* reclaim too much, nor too less that we keep
2014	* coming back to reclaim from this cgroup
2015	*/
2016	if (total >= (excess >> 2) \|\|
2017	(loop > MEM_CGROUP_MAX_RECLAIM_LOOPS))
2018	break;
2019	}
2020	continue;
2021	}
2022	total += mem_cgroup_shrink_node(victim, gfp_mask, false,
2023	pgdat, &nr_scanned);
2024	*total_scanned += nr_scanned;
2025	if (!soft_limit_excess(root_memcg))
2026	break;
2027	}
2028	mem_cgroup_iter_break(root_memcg, victim);
2029	return total;
2030	}
2031
2032	#ifdef CONFIG_LOCKDEP
2033	static struct lockdep_map memcg_oom_lock_dep_map = {
2034	.name = "memcg_oom_lock",
2035	};
2036	#endif
2037
2038	static DEFINE_SPINLOCK(memcg_oom_lock);
2039
2040	/*
2041	* Check OOM-Killer is already running under our hierarchy.
2042	* If someone is running, return false.
2043	*/
2044	static bool mem_cgroup_oom_trylock(struct mem_cgroup *memcg)
2045	{
2046	struct mem_cgroup iter, failed = NULL;
2047
2048	spin_lock(&memcg_oom_lock);
2049
2050	for_each_mem_cgroup_tree(iter, memcg) {
2051	if (iter->oom_lock) {
2052	/*
2053	* this subtree of our hierarchy is already locked
2054	* so we cannot give a lock.
2055	*/
2056	failed = iter;
2057	mem_cgroup_iter_break(memcg, iter);
2058	break;
2059	} else
2060	iter->oom_lock = true;
2061	}
2062
2063	if (failed) {
2064	/*
2065	* OK, we failed to lock the whole subtree so we have
2066	* to clean up what we set up to the failing subtree
2067	*/
2068	for_each_mem_cgroup_tree(iter, memcg) {
2069	if (iter == failed) {
2070	mem_cgroup_iter_break(memcg, iter);
2071	break;
2072	}
2073	iter->oom_lock = false;
2074	}
2075	} else
2076	mutex_acquire(&memcg_oom_lock_dep_map, 0, 1, _RET_IP_);
2077
2078	spin_unlock(&memcg_oom_lock);
2079
2080	return !failed;
2081	}
2082
2083	static void mem_cgroup_oom_unlock(struct mem_cgroup *memcg)
2084	{
2085	struct mem_cgroup *iter;
2086
2087	spin_lock(&memcg_oom_lock);
2088	mutex_release(&memcg_oom_lock_dep_map, _RET_IP_);
2089	for_each_mem_cgroup_tree(iter, memcg)
2090	iter->oom_lock = false;
2091	spin_unlock(&memcg_oom_lock);
2092	}
2093
2094	static void mem_cgroup_mark_under_oom(struct mem_cgroup *memcg)
2095	{
2096	struct mem_cgroup *iter;
2097
2098	spin_lock(&memcg_oom_lock);
2099	for_each_mem_cgroup_tree(iter, memcg)
2100	iter->under_oom++;
2101	spin_unlock(&memcg_oom_lock);
2102	}
2103
2104	static void mem_cgroup_unmark_under_oom(struct mem_cgroup *memcg)
2105	{
2106	struct mem_cgroup *iter;
2107
2108	/*
2109	* Be careful about under_oom underflows because a child memcg
2110	* could have been added after mem_cgroup_mark_under_oom.
2111	*/
2112	spin_lock(&memcg_oom_lock);
2113	for_each_mem_cgroup_tree(iter, memcg)
2114	if (iter->under_oom > 0)
2115	iter->under_oom--;
2116	spin_unlock(&memcg_oom_lock);
2117	}
2118
2119	static DECLARE_WAIT_QUEUE_HEAD(memcg_oom_waitq);
2120
2121	struct oom_wait_info {
2122	struct mem_cgroup *memcg;
2123	wait_queue_entry_t wait;
2124	};
2125
2126	static int memcg_oom_wake_function(wait_queue_entry_t *wait,
2127	unsigned mode, int sync, void *arg)
2128	{
2129	struct mem_cgroup wake_memcg = (struct mem_cgroup )arg;
2130	struct mem_cgroup *oom_wait_memcg;
2131	struct oom_wait_info *oom_wait_info;
2132
2133	oom_wait_info = container_of(wait, struct oom_wait_info, wait);
2134	oom_wait_memcg = oom_wait_info->memcg;
2135
2136	if (!mem_cgroup_is_descendant(wake_memcg, oom_wait_memcg) &&
2137	!mem_cgroup_is_descendant(oom_wait_memcg, wake_memcg))
2138	return 0;
2139	return autoremove_wake_function(wait, mode, sync, arg);
2140	}
2141
2142	static void memcg_oom_recover(struct mem_cgroup *memcg)
2143	{
2144	/*
2145	* For the following lockless ->under_oom test, the only required
2146	* guarantee is that it must see the state asserted by an OOM when
2147	* this function is called as a result of userland actions
2148	* triggered by the notification of the OOM. This is trivially
2149	* achieved by invoking mem_cgroup_mark_under_oom() before
2150	* triggering notification.
2151	*/
2152	if (memcg && memcg->under_oom)
2153	__wake_up(&memcg_oom_waitq, TASK_NORMAL, 0, memcg);
2154	}
2155
2156	/*	1609	/*
2157	* Returns true if successfully killed one or more processes. Though in some	1610	* Returns true if successfully killed one or more processes. Though in some
2158	* corner cases it can return true even without killing any process.	1611	* corner cases it can return true even without killing any process.
@@ -2166,104 +1619,16 @@ static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order) · 7 unmodified lines
2166		1619
2167	memcg_memory_event(memcg, MEMCG_OOM);	1620	memcg_memory_event(memcg, MEMCG_OOM);
2168		1621
2169	/*	1622	if (!memcg1_oom_prepare(memcg, &locked))
2170	* We are in the middle of the charge context here, so we
2171	* don't want to block when potentially sitting on a callstack
2172	* that holds all kinds of filesystem and mm locks.
2173	*
2174	* cgroup1 allows disabling the OOM killer and waiting for outside
2175	* handling until the charge can succeed; remember the context and put
2176	* the task to sleep at the end of the page fault when all locks are
2177	* released.
2178	*
2179	* On the other hand, in-kernel OOM killer allows for an async victim
2180	* memory reclaim (oom_reaper) and that means that we are not solely
2181	* relying on the oom victim to make a forward progress and we can
2182	* invoke the oom killer here.
2183	*
2184	* Please note that mem_cgroup_out_of_memory might fail to find a
2185	* victim and then we have to bail out from the charge path.
2186	*/
2187	if (READ_ONCE(memcg->oom_kill_disable)) {
2188	if (current->in_user_fault) {
2189	css_get(&memcg->css);
2190	current->memcg_in_oom = memcg;
2191	}
2192	return false;	1623	return false;
2193	}
2194
2195	mem_cgroup_mark_under_oom(memcg);
2196
2197	locked = mem_cgroup_oom_trylock(memcg);
2198
2199	if (locked)
2200	mem_cgroup_oom_notify(memcg);
2201		1624
2202	mem_cgroup_unmark_under_oom(memcg);
2203	ret = mem_cgroup_out_of_memory(memcg, mask, order);	1625	ret = mem_cgroup_out_of_memory(memcg, mask, order);
2204		1626
2205	if (locked)	1627	memcg1_oom_finish(memcg, locked);
2206	mem_cgroup_oom_unlock(memcg);
2207		1628
2208	return ret;	1629	return ret;
2209	}	1630	}
2210		1631
2211	/**
2212	* mem_cgroup_oom_synchronize - complete memcg OOM handling
2213	* @handle: actually kill/wait or just clean up the OOM state
2214	*
2215	* This has to be called at the end of a page fault if the memcg OOM
2216	* handler was enabled.
2217	*
2218	* Memcg supports userspace OOM handling where failed allocations must
2219	* sleep on a waitqueue until the userspace task resolves the
2220	* situation. Sleeping directly in the charge context with all kinds
2221	* of locks held is not a good idea, instead we remember an OOM state
2222	* in the task and mem_cgroup_oom_synchronize() has to be called at
2223	* the end of the page fault to complete the OOM handling.
2224	*
2225	* Returns %true if an ongoing memcg OOM situation was detected and
2226	* completed, %false otherwise.
2227	*/
2228	bool mem_cgroup_oom_synchronize(bool handle)
2229	{
2230	struct mem_cgroup *memcg = current->memcg_in_oom;
2231	struct oom_wait_info owait;
2232	bool locked;
2233
2234	/* OOM is global, do not handle */
2235	if (!memcg)
2236	return false;
2237
2238	if (!handle)
2239	goto cleanup;
2240
2241	owait.memcg = memcg;
2242	owait.wait.flags = 0;
2243	owait.wait.func = memcg_oom_wake_function;
2244	owait.wait.private = current;
2245	INIT_LIST_HEAD(&owait.wait.entry);
2246
2247	prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE);
2248	mem_cgroup_mark_under_oom(memcg);
2249
2250	locked = mem_cgroup_oom_trylock(memcg);
2251
2252	if (locked)
2253	mem_cgroup_oom_notify(memcg);
2254
2255	schedule();
2256	mem_cgroup_unmark_under_oom(memcg);
2257	finish_wait(&memcg_oom_waitq, &owait.wait);
2258
2259	if (locked)
2260	mem_cgroup_oom_unlock(memcg);
2261	cleanup:
2262	current->memcg_in_oom = NULL;
2263	css_put(&memcg->css);
2264	return true;
2265	}
2266
2267	/**	1632	/**
2268	* mem_cgroup_get_oom_group - get a memory cgroup to clean up after OOM	1633	* mem_cgroup_get_oom_group - get a memory cgroup to clean up after OOM
2269	* @victim: task to be killed by the OOM killer	1634	* @victim: task to be killed by the OOM killer
@@ -2328,99 +1693,16 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) · 58 unmodified lines
2328	pr_cont(" are going to be killed due to memory.oom.group set\n");	1693	pr_cont(" are going to be killed due to memory.oom.group set\n");
2329	}	1694	}
2330		1695
2331	/**
2332	* folio_memcg_lock - Bind a folio to its memcg.
2333	* @folio: The folio.
2334	*
2335	* This function prevents unlocked LRU folios from being moved to
2336	* another cgroup.
2337	*
2338	* It ensures lifetime of the bound memcg. The caller is responsible
2339	* for the lifetime of the folio.
2340	*/
2341	void folio_memcg_lock(struct folio *folio)
2342	{
2343	struct mem_cgroup *memcg;
2344	unsigned long flags;
2345
2346	/*
2347	* The RCU lock is held throughout the transaction. The fast
2348	* path can get away without acquiring the memcg->move_lock
2349	* because page moving starts with an RCU grace period.
2350	*/
2351	rcu_read_lock();
2352
2353	if (mem_cgroup_disabled())
2354	return;
2355	again:
2356	memcg = folio_memcg(folio);
2357	if (unlikely(!memcg))
2358	return;
2359
2360	#ifdef CONFIG_PROVE_LOCKING
2361	local_irq_save(flags);
2362	might_lock(&memcg->move_lock);
2363	local_irq_restore(flags);
2364	#endif
2365
2366	if (atomic_read(&memcg->moving_account) <= 0)
2367	return;
2368
2369	spin_lock_irqsave(&memcg->move_lock, flags);
2370	if (memcg != folio_memcg(folio)) {
2371	spin_unlock_irqrestore(&memcg->move_lock, flags);
2372	goto again;
2373	}
2374
2375	/*
2376	* When charge migration first begins, we can have multiple
2377	* critical sections holding the fast-path RCU lock and one
2378	* holding the slowpath move_lock. Track the task who has the
2379	* move_lock for folio_memcg_unlock().
2380	*/
2381	memcg->move_lock_task = current;
2382	memcg->move_lock_flags = flags;
2383	}
2384
2385	static void __folio_memcg_unlock(struct mem_cgroup *memcg)
2386	{
2387	if (memcg && memcg->move_lock_task == current) {
2388	unsigned long flags = memcg->move_lock_flags;
2389
2390	memcg->move_lock_task = NULL;
2391	memcg->move_lock_flags = 0;
2392
2393	spin_unlock_irqrestore(&memcg->move_lock, flags);
2394	}
2395
2396	rcu_read_unlock();
2397	}
2398
2399	/**
2400	* folio_memcg_unlock - Release the binding between a folio and its memcg.
2401	* @folio: The folio.
2402	*
2403	* This releases the binding created by folio_memcg_lock(). This does
2404	* not change the accounting of this folio to its memcg, but it does
2405	* permit others to change it.
2406	*/
2407	void folio_memcg_unlock(struct folio *folio)
2408	{
2409	__folio_memcg_unlock(folio_memcg(folio));
2410	}
2411
2412	struct memcg_stock_pcp {	1696	struct memcg_stock_pcp {
2413	local_lock_t stock_lock;	1697	local_lock_t stock_lock;
2414	struct mem_cgroup cached; / this never be root cgroup */	1698	struct mem_cgroup cached; / this never be root cgroup */
2415	unsigned int nr_pages;	1699	unsigned int nr_pages;
2416		1700
2417	#ifdef CONFIG_MEMCG_KMEM
2418	struct obj_cgroup *cached_objcg;	1701	struct obj_cgroup *cached_objcg;
2419	struct pglist_data *cached_pgdat;	1702	struct pglist_data *cached_pgdat;
2420	unsigned int nr_bytes;	1703	unsigned int nr_bytes;
2421	int nr_slab_reclaimable_b;	1704	int nr_slab_reclaimable_b;
2422	int nr_slab_unreclaimable_b;	1705	int nr_slab_unreclaimable_b;
2423	#endif
2424		1706
2425	struct work_struct work;	1707	struct work_struct work;
2426	unsigned long flags;	1708	unsigned long flags;
@@ -2431,26 +1713,9 @@ static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { · 4 unmodified lines
2431	};	1713	};
2432	static DEFINE_MUTEX(percpu_charge_mutex);	1714	static DEFINE_MUTEX(percpu_charge_mutex);
2433		1715
2434	#ifdef CONFIG_MEMCG_KMEM
2435	static struct obj_cgroup drain_obj_stock(struct memcg_stock_pcp stock);	1716	static struct obj_cgroup drain_obj_stock(struct memcg_stock_pcp stock);
2436	static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,	1717	static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
2437	struct mem_cgroup *root_memcg);	1718	struct mem_cgroup *root_memcg);
2438	static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages);
2439
2440	#else
2441	static inline struct obj_cgroup drain_obj_stock(struct memcg_stock_pcp stock)
2442	{
2443	return NULL;
2444	}
2445	static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
2446	struct mem_cgroup *root_memcg)
2447	{
2448	return false;
2449	}
2450	static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages)
2451	{
2452	}
2453	#endif
2454		1719
2455	/**	1720	/**
2456	* consume_stock: Try to consume stocked charge on this cpu.	1721	* consume_stock: Try to consume stocked charge on this cpu.
@@ -2567,7 +1832,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) · 110 unmodified lines
2567	* Drains all per-CPU charge caches for given root_memcg resp. subtree	1832	* Drains all per-CPU charge caches for given root_memcg resp. subtree
2568	* of the hierarchy under it.	1833	* of the hierarchy under it.
2569	*/	1834	*/
2570	static void drain_all_stock(struct mem_cgroup *root_memcg)	1835	void drain_all_stock(struct mem_cgroup *root_memcg)
2571	{	1836	{
2572	int cpu, curcpu;	1837	int cpu, curcpu;
2573		1838
@@ -2636,7 +1901,8 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg, · 62 unmodified lines
2636	psi_memstall_enter(&pflags);	1901	psi_memstall_enter(&pflags);
2637	nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages,	1902	nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages,
2638	gfp_mask,	1903	gfp_mask,
2639	MEMCG_RECLAIM_MAY_SWAP);	1904	MEMCG_RECLAIM_MAY_SWAP,
		1905	NULL);
2640	psi_memstall_leave(&pflags);	1906	psi_memstall_leave(&pflags);
2641	} while ((memcg = parent_mem_cgroup(memcg)) &&	1907	} while ((memcg = parent_mem_cgroup(memcg)) &&
2642	!mem_cgroup_is_root(memcg));	1908	!mem_cgroup_is_root(memcg));
@@ -2887,8 +2153,8 @@ out: · 244 unmodified lines
2887	css_put(&memcg->css);	2153	css_put(&memcg->css);
2888	}	2154	}
2889		2155
2890	static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,	2156	int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
2891	unsigned int nr_pages)	2157	unsigned int nr_pages)
2892	{	2158	{
2893	unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);	2159	unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
2894	int nr_retries = MAX_RECLAIM_RETRIES;	2160	int nr_retries = MAX_RECLAIM_RETRIES;
@@ -2942,7 +2208,7 @@ retry: · 47 unmodified lines
2942		2208
2943	psi_memstall_enter(&pflags);	2209	psi_memstall_enter(&pflags);
2944	nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,	2210	nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
2945	gfp_mask, reclaim_options);	2211	gfp_mask, reclaim_options, NULL);
2946	psi_memstall_leave(&pflags);	2212	psi_memstall_leave(&pflags);
2947		2213
2948	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)	2214	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
@@ -2971,7 +2237,7 @@ retry: · 22 unmodified lines
2971	* At task move, charge accounts can be doubly counted. So, it's	2237	* At task move, charge accounts can be doubly counted. So, it's
2972	* better to wait until the end of task_move if something is going on.	2238	* better to wait until the end of task_move if something is going on.
2973	*/	2239	*/
2974	if (mem_cgroup_wait_acct_move(mem_over_limit))	2240	if (memcg1_wait_acct_move(mem_over_limit))
2975	goto retry;	2241	goto retry;
2976		2242
2977	if (nr_retries--)	2243	if (nr_retries--)
@@ -3083,15 +2349,6 @@ done_restock: · 105 unmodified lines
3083	return 0;	2349	return 0;
3084	}	2350	}
3085		2351
3086	static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
3087	unsigned int nr_pages)
3088	{
3089	if (mem_cgroup_is_root(memcg))
3090	return 0;
3091
3092	return try_charge_memcg(memcg, gfp_mask, nr_pages);
3093	}
3094
3095	/**	2352	/**
3096	* mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call.	2353	* mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call.
3097	* @memcg: memcg previously charged.	2354	* @memcg: memcg previously charged.
@@ -3134,12 +2391,10 @@ void mem_cgroup_commit_charge(struct folio folio, struct mem_cgroup memcg) · 36 unmodified lines
3134		2391
3135	local_irq_disable();	2392	local_irq_disable();
3136	mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio));	2393	mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio));
3137	memcg_check_events(memcg, folio_nid(folio));	2394	memcg1_check_events(memcg, folio_nid(folio));
3138	local_irq_enable();	2395	local_irq_enable();
3139	}	2396	}
3140		2397
3141	#ifdef CONFIG_MEMCG_KMEM
3142
3143	static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg,	2398	static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg,
3144	struct pglist_data *pgdat,	2399	struct pglist_data *pgdat,
3145	enum node_stat_item idx, int nr)	2400	enum node_stat_item idx, int nr)
@@ -3367,18 +2622,6 @@ struct obj_cgroup get_obj_cgroup_from_folio(struct folio folio) · 221 unmodified lines
3367	return objcg;	2622	return objcg;
3368	}	2623	}
3369		2624
3370	static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages)
3371	{
3372	mod_memcg_state(memcg, MEMCG_KMEM, nr_pages);
3373	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
3374	if (nr_pages > 0)
3375	page_counter_charge(&memcg->kmem, nr_pages);
3376	else
3377	page_counter_uncharge(&memcg->kmem, -nr_pages);
3378	}
3379	}
3380
3381
3382	/*	2625	/*
3383	* obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg	2626	* obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg
3384	* @objcg: object cgroup to uncharge	2627	* @objcg: object cgroup to uncharge
@@ -3391,7 +2634,8 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, · 6 unmodified lines
3391		2634
3392	memcg = get_mem_cgroup_from_objcg(objcg);	2635	memcg = get_mem_cgroup_from_objcg(objcg);
3393		2636
3394	memcg_account_kmem(memcg, -nr_pages);	2637	mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
		2638	memcg1_account_kmem(memcg, -nr_pages);
3395	refill_stock(memcg, nr_pages);	2639	refill_stock(memcg, nr_pages);
3396		2640
3397	css_put(&memcg->css);	2641	css_put(&memcg->css);
@@ -3417,7 +2661,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · 19 unmodified lines
3417	if (ret)	2661	if (ret)
3418	goto out;	2662	goto out;
3419		2663
3420	memcg_account_kmem(memcg, nr_pages);	2664	mod_memcg_state(memcg, MEMCG_KMEM, nr_pages);
		2665	memcg1_account_kmem(memcg, nr_pages);
3421	out:	2666	out:
3422	css_put(&memcg->css);	2667	css_put(&memcg->css);
3423		2668
@@ -3570,7 +2815,8 @@ static struct obj_cgroup drain_obj_stock(struct memcg_stock_pcp stock) · 146 unmodified lines
3570		2815
3571	memcg = get_mem_cgroup_from_objcg(old);	2816	memcg = get_mem_cgroup_from_objcg(old);
3572		2817
3573	memcg_account_kmem(memcg, -nr_pages);	2818	mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
		2819	memcg1_account_kmem(memcg, -nr_pages);
3574	__refill_stock(memcg, nr_pages);	2820	__refill_stock(memcg, nr_pages);
3575		2821
3576	css_put(&memcg->css);	2822	css_put(&memcg->css);
@@ -3804,10 +3050,9 @@ void __memcg_slab_free_hook(struct kmem_cache s, struct slab slab, · 227 unmodified lines
3804	obj_cgroup_put(objcg);	3050	obj_cgroup_put(objcg);
3805	}	3051	}
3806	}	3052	}
3807	#endif /* CONFIG_MEMCG_KMEM */
3808		3053
3809	/*	3054	/*
3810	* Because page_memcg(head) is not set on tails, set it now.	3055	* Because folio_memcg(head) is not set on tails, set it now.
3811	*/	3056	*/
3812	void split_page_memcg(struct page *head, int old_order, int new_order)	3057	void split_page_memcg(struct page *head, int old_order, int new_order)
3813	{	3058	{
@@ -3829,240 +3074,7 @@ void split_page_memcg(struct page *head, int old_order, int new_order) · 15 unmodified lines
3829	css_get_many(&memcg->css, old_nr / new_nr - 1);	3074	css_get_many(&memcg->css, old_nr / new_nr - 1);
3830	}	3075	}
3831		3076
3832	#ifdef CONFIG_SWAP	3077	unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
3833	/**
3834	* mem_cgroup_move_swap_account - move swap charge and swap_cgroup's record.
3835	* @entry: swap entry to be moved
3836	* @from: mem_cgroup which the entry is moved from
3837	* @to: mem_cgroup which the entry is moved to
3838	*
3839	* It succeeds only when the swap_cgroup's record for this entry is the same
3840	* as the mem_cgroup's id of @from.
3841	*
3842	* Returns 0 on success, -EINVAL on failure.
3843	*
3844	* The caller must have charged to @to, IOW, called page_counter_charge() about
3845	* both res and memsw, and called css_get().
3846	*/
3847	static int mem_cgroup_move_swap_account(swp_entry_t entry,
3848	struct mem_cgroup from, struct mem_cgroup to)
3849	{
3850	unsigned short old_id, new_id;
3851
3852	old_id = mem_cgroup_id(from);
3853	new_id = mem_cgroup_id(to);
3854
3855	if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) {
3856	mod_memcg_state(from, MEMCG_SWAP, -1);
3857	mod_memcg_state(to, MEMCG_SWAP, 1);
3858	return 0;
3859	}
3860	return -EINVAL;
3861	}
3862	#else
3863	static inline int mem_cgroup_move_swap_account(swp_entry_t entry,
3864	struct mem_cgroup from, struct mem_cgroup to)
3865	{
3866	return -EINVAL;
3867	}
3868	#endif
3869
3870	static DEFINE_MUTEX(memcg_max_mutex);
3871
3872	static int mem_cgroup_resize_max(struct mem_cgroup *memcg,
3873	unsigned long max, bool memsw)
3874	{
3875	bool enlarge = false;
3876	bool drained = false;
3877	int ret;
3878	bool limits_invariant;
3879	struct page_counter *counter = memsw ? &memcg->memsw : &memcg->memory;
3880
3881	do {
3882	if (signal_pending(current)) {
3883	ret = -EINTR;
3884	break;
3885	}
3886
3887	mutex_lock(&memcg_max_mutex);
3888	/*
3889	* Make sure that the new limit (memsw or memory limit) doesn't
3890	* break our basic invariant rule memory.max <= memsw.max.
3891	*/
3892	limits_invariant = memsw ? max >= READ_ONCE(memcg->memory.max) :
3893	max <= memcg->memsw.max;
3894	if (!limits_invariant) {
3895	mutex_unlock(&memcg_max_mutex);
3896	ret = -EINVAL;
3897	break;
3898	}
3899	if (max > counter->max)
3900	enlarge = true;
3901	ret = page_counter_set_max(counter, max);
3902	mutex_unlock(&memcg_max_mutex);
3903
3904	if (!ret)
3905	break;
3906
3907	if (!drained) {
3908	drain_all_stock(memcg);
3909	drained = true;
3910	continue;
3911	}
3912
3913	if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
3914	memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP)) {
3915	ret = -EBUSY;
3916	break;
3917	}
3918	} while (true);
3919
3920	if (!ret && enlarge)
3921	memcg_oom_recover(memcg);
3922
3923	return ret;
3924	}
3925
3926	unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,
3927	gfp_t gfp_mask,
3928	unsigned long *total_scanned)
3929	{
3930	unsigned long nr_reclaimed = 0;
3931	struct mem_cgroup_per_node mz, next_mz = NULL;
3932	unsigned long reclaimed;
3933	int loop = 0;
3934	struct mem_cgroup_tree_per_node *mctz;
3935	unsigned long excess;
3936
3937	if (lru_gen_enabled())
3938	return 0;
3939
3940	if (order > 0)
3941	return 0;
3942
3943	mctz = soft_limit_tree.rb_tree_per_node[pgdat->node_id];
3944
3945	/*
3946	* Do not even bother to check the largest node if the root
3947	* is empty. Do it lockless to prevent lock bouncing. Races
3948	* are acceptable as soft limit is best effort anyway.
3949	*/
3950	if (!mctz \|\| RB_EMPTY_ROOT(&mctz->rb_root))
3951	return 0;
3952
3953	/*
3954	* This loop can run a while, specially if mem_cgroup's continuously
3955	* keep exceeding their soft limit and putting the system under
3956	* pressure
3957	*/
3958	do {
3959	if (next_mz)
3960	mz = next_mz;
3961	else
3962	mz = mem_cgroup_largest_soft_limit_node(mctz);
3963	if (!mz)
3964	break;
3965
3966	reclaimed = mem_cgroup_soft_reclaim(mz->memcg, pgdat,
3967	gfp_mask, total_scanned);
3968	nr_reclaimed += reclaimed;
3969	spin_lock_irq(&mctz->lock);
3970
3971	/*
3972	* If we failed to reclaim anything from this memory cgroup
3973	* it is time to move on to the next cgroup
3974	*/
3975	next_mz = NULL;
3976	if (!reclaimed)
3977	next_mz = __mem_cgroup_largest_soft_limit_node(mctz);
3978
3979	excess = soft_limit_excess(mz->memcg);
3980	/*
3981	* One school of thought says that we should not add
3982	* back the node to the tree if reclaim returns 0.
3983	* But our reclaim could return 0, simply because due
3984	* to priority we are exposing a smaller subset of
3985	* memory to reclaim from. Consider this as a longer
3986	* term TODO.
3987	*/
3988	/* If excess == 0, no tree ops */
3989	__mem_cgroup_insert_exceeded(mz, mctz, excess);
3990	spin_unlock_irq(&mctz->lock);
3991	css_put(&mz->memcg->css);
3992	loop++;
3993	/*
3994	* Could not reclaim anything and there are no more
3995	* mem cgroups to try or we seem to be looping without
3996	* reclaiming anything.
3997	*/
3998	if (!nr_reclaimed &&
3999	(next_mz == NULL \|\|
4000	loop > MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS))
4001	break;
4002	} while (!nr_reclaimed);
4003	if (next_mz)
4004	css_put(&next_mz->memcg->css);
4005	return nr_reclaimed;
4006	}
4007
4008	/*
4009	* Reclaims as many pages from the given memcg as possible.
4010	*
4011	* Caller is responsible for holding css reference for memcg.
4012	*/
4013	static int mem_cgroup_force_empty(struct mem_cgroup *memcg)
4014	{
4015	int nr_retries = MAX_RECLAIM_RETRIES;
4016
4017	/* we call try-to-free pages for make this cgroup empty */
4018	lru_add_drain_all();
4019
4020	drain_all_stock(memcg);
4021
4022	/* try to free all pages in this cgroup */
4023	while (nr_retries && page_counter_read(&memcg->memory)) {
4024	if (signal_pending(current))
4025	return -EINTR;
4026
4027	if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL,
4028	MEMCG_RECLAIM_MAY_SWAP))
4029	nr_retries--;
4030	}
4031
4032	return 0;
4033	}
4034
4035	static ssize_t mem_cgroup_force_empty_write(struct kernfs_open_file *of,
4036	char *buf, size_t nbytes,
4037	loff_t off)
4038	{
4039	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
4040
4041	if (mem_cgroup_is_root(memcg))
4042	return -EINVAL;
4043	return mem_cgroup_force_empty(memcg) ?: nbytes;
4044	}
4045
4046	static u64 mem_cgroup_hierarchy_read(struct cgroup_subsys_state *css,
4047	struct cftype *cft)
4048	{
4049	return 1;
4050	}
4051
4052	static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css,
4053	struct cftype *cft, u64 val)
4054	{
4055	if (val == 1)
4056	return 0;
4057
4058	pr_warn_once("Non-hierarchical mode is deprecated. "
4059	"Please report your usecase to linux-mm@kvack.org if you "
4060	"depend on this functionality.\n");
4061
4062	return -EINVAL;
4063	}
4064
4065	static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
4066	{	3078	{
4067	unsigned long val;	3079	unsigned long val;
4068		3080
@@ -4084,68 +3096,6 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) · 15 unmodified lines
4084	return val;	3096	return val;
4085	}	3097	}
4086		3098
4087	enum {
4088	RES_USAGE,
4089	RES_LIMIT,
4090	RES_MAX_USAGE,
4091	RES_FAILCNT,
4092	RES_SOFT_LIMIT,
4093	};
4094
4095	static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css,
4096	struct cftype *cft)
4097	{
4098	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
4099	struct page_counter *counter;
4100
4101	switch (MEMFILE_TYPE(cft->private)) {
4102	case _MEM:
4103	counter = &memcg->memory;
4104	break;
4105	case _MEMSWAP:
4106	counter = &memcg->memsw;
4107	break;
4108	case _KMEM:
4109	counter = &memcg->kmem;
4110	break;
4111	case _TCP:
4112	counter = &memcg->tcpmem;
4113	break;
4114	default:
4115	BUG();
4116	}
4117
4118	switch (MEMFILE_ATTR(cft->private)) {
4119	case RES_USAGE:
4120	if (counter == &memcg->memory)
4121	return (u64)mem_cgroup_usage(memcg, false) * PAGE_SIZE;
4122	if (counter == &memcg->memsw)
4123	return (u64)mem_cgroup_usage(memcg, true) * PAGE_SIZE;
4124	return (u64)page_counter_read(counter) * PAGE_SIZE;
4125	case RES_LIMIT:
4126	return (u64)counter->max * PAGE_SIZE;
4127	case RES_MAX_USAGE:
4128	return (u64)counter->watermark * PAGE_SIZE;
4129	case RES_FAILCNT:
4130	return counter->failcnt;
4131	case RES_SOFT_LIMIT:
4132	return (u64)READ_ONCE(memcg->soft_limit) * PAGE_SIZE;
4133	default:
4134	BUG();
4135	}
4136	}
4137
4138	/*
4139	* This function doesn't do anything useful. Its only job is to provide a read
4140	* handler for a file so that cgroup_file_mode() will add read permissions.
4141	*/
4142	static int mem_cgroup_dummy_seq_show(__always_unused struct seq_file *m,
4143	__always_unused void *v)
4144	{
4145	return -EINVAL;
4146	}
4147
4148	#ifdef CONFIG_MEMCG_KMEM
4149	static int memcg_online_kmem(struct mem_cgroup *memcg)	3099	static int memcg_online_kmem(struct mem_cgroup *memcg)
4150	{	3100	{
4151	struct obj_cgroup *objcg;	3101	struct obj_cgroup *objcg;
@@ -4196,2625 +3146,725 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) · 44 unmodified lines
4196	*/	3146	*/
4197	memcg_reparent_list_lrus(memcg, parent);	3147	memcg_reparent_list_lrus(memcg, parent);
4198	}	3148	}
4199	#else	3149
4200	static int memcg_online_kmem(struct mem_cgroup *memcg)	3150	#ifdef CONFIG_CGROUP_WRITEBACK
		3151
		3152	#include <trace/events/writeback.h>
		3153
		3154	static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
4201	{	3155	{
4202	return 0;	3156	return wb_domain_init(&memcg->cgwb_domain, gfp);
4203	}	3157	}
4204	static void memcg_offline_kmem(struct mem_cgroup *memcg)	3158
		3159	static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
4205	{	3160	{
		3161	wb_domain_exit(&memcg->cgwb_domain);
4206	}	3162	}
4207	#endif /* CONFIG_MEMCG_KMEM */
4208		3163
4209	static int memcg_update_tcp_max(struct mem_cgroup *memcg, unsigned long max)	3164	static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
4210	{	3165	{
4211	int ret;	3166	wb_domain_size_changed(&memcg->cgwb_domain);
4212
4213	mutex_lock(&memcg_max_mutex);
4214
4215	ret = page_counter_set_max(&memcg->tcpmem, max);
4216	if (ret)
4217	goto out;
4218
4219	if (!memcg->tcpmem_active) {
4220	/*
4221	* The active flag needs to be written after the static_key
4222	* update. This is what guarantees that the socket activation
4223	* function is the last one to run. See mem_cgroup_sk_alloc()
4224	* for details, and note that we don't mark any socket as
4225	* belonging to this memcg until that flag is up.
4226	*
4227	* We need to do this, because static_keys will span multiple
4228	* sites, but we can't control their order. If we mark a socket
4229	* as accounted, but the accounting functions are not patched in
4230	* yet, we'll lose accounting.
4231	*
4232	* We never race with the readers in mem_cgroup_sk_alloc(),
4233	* because when this value change, the code to process it is not
4234	* patched in yet.
4235	*/
4236	static_branch_inc(&memcg_sockets_enabled_key);
4237	memcg->tcpmem_active = true;
4238	}
4239	out:
4240	mutex_unlock(&memcg_max_mutex);
4241	return ret;
4242	}	3167	}
4243		3168
4244	/*	3169	struct wb_domain mem_cgroup_wb_domain(struct bdi_writeback wb)
4245	* The user of this function is...
4246	* RES_LIMIT.
4247	*/
4248	static ssize_t mem_cgroup_write(struct kernfs_open_file *of,
4249	char *buf, size_t nbytes, loff_t off)
4250	{	3170	{
4251	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));	3171	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
4252	unsigned long nr_pages;
4253	int ret;
4254		3172
4255	buf = strstrip(buf);	3173	if (!memcg->css.parent)
4256	ret = page_counter_memparse(buf, "-1", &nr_pages);	3174	return NULL;
4257	if (ret)
4258	return ret;
4259		3175
4260	switch (MEMFILE_ATTR(of_cft(of)->private)) {	3176	return &memcg->cgwb_domain;
4261	case RES_LIMIT:
4262	if (mem_cgroup_is_root(memcg)) { /* Can't set limit on root */
4263	ret = -EINVAL;
4264	break;
4265	}
4266	switch (MEMFILE_TYPE(of_cft(of)->private)) {
4267	case _MEM:
4268	ret = mem_cgroup_resize_max(memcg, nr_pages, false);
4269	break;
4270	case _MEMSWAP:
4271	ret = mem_cgroup_resize_max(memcg, nr_pages, true);
4272	break;
4273	case _KMEM:
4274	pr_warn_once("kmem.limit_in_bytes is deprecated and will be removed. "
4275	"Writing any value to this file has no effect. "
4276	"Please report your usecase to linux-mm@kvack.org if you "
4277	"depend on this functionality.\n");
4278	ret = 0;
4279	break;
4280	case _TCP:
4281	ret = memcg_update_tcp_max(memcg, nr_pages);
4282	break;
4283	}
4284	break;
4285	case RES_SOFT_LIMIT:
4286	if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
4287	ret = -EOPNOTSUPP;
4288	} else {
4289	WRITE_ONCE(memcg->soft_limit, nr_pages);
4290	ret = 0;
4291	}
4292	break;
4293	}
4294	return ret ?: nbytes;
4295	}	3177	}
4296		3178
4297	static ssize_t mem_cgroup_reset(struct kernfs_open_file of, char buf,	3179	/**
4298	size_t nbytes, loff_t off)	3180	* mem_cgroup_wb_stats - retrieve writeback related stats from its memcg
		3181	* @wb: bdi_writeback in question
		3182	* @pfilepages: out parameter for number of file pages
		3183	* @pheadroom: out parameter for number of allocatable pages according to memcg
		3184	* @pdirty: out parameter for number of dirty pages
		3185	* @pwriteback: out parameter for number of pages under writeback
		3186	*
		3187	* Determine the numbers of file, headroom, dirty, and writeback pages in
		3188	* @wb's memcg. File, dirty and writeback are self-explanatory. Headroom
		3189	* is a bit more involved.
		3190	*
		3191	* A memcg's headroom is "min(max, high) - used". In the hierarchy, the
		3192	* headroom is calculated as the lowest headroom of itself and the
		3193	* ancestors. Note that this doesn't consider the actual amount of
		3194	* available memory in the system. The caller should further cap
		3195	* *@pheadroom accordingly.
		3196	*/
		3197	void mem_cgroup_wb_stats(struct bdi_writeback wb, unsigned long pfilepages,
		3198	unsigned long pheadroom, unsigned long pdirty,
		3199	unsigned long *pwriteback)
4299	{	3200	{
4300	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));	3201	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
4301	struct page_counter *counter;	3202	struct mem_cgroup *parent;
4302		3203
4303	switch (MEMFILE_TYPE(of_cft(of)->private)) {	3204	mem_cgroup_flush_stats_ratelimited(memcg);
4304	case _MEM:
4305	counter = &memcg->memory;
4306	break;
4307	case _MEMSWAP:
4308	counter = &memcg->memsw;
4309	break;
4310	case _KMEM:
4311	counter = &memcg->kmem;
4312	break;
4313	case _TCP:
4314	counter = &memcg->tcpmem;
4315	break;
4316	default:
4317	BUG();
4318	}
4319		3205
4320	switch (MEMFILE_ATTR(of_cft(of)->private)) {	3206	*pdirty = memcg_page_state(memcg, NR_FILE_DIRTY);
4321	case RES_MAX_USAGE:	3207	*pwriteback = memcg_page_state(memcg, NR_WRITEBACK);
4322	page_counter_reset_watermark(counter);	3208	*pfilepages = memcg_page_state(memcg, NR_INACTIVE_FILE) +
4323	break;	3209	memcg_page_state(memcg, NR_ACTIVE_FILE);
4324	case RES_FAILCNT:
4325	counter->failcnt = 0;
4326	break;
4327	default:
4328	BUG();
4329	}
4330		3210
4331	return nbytes;	3211	*pheadroom = PAGE_COUNTER_MAX;
4332	}	3212	while ((parent = parent_mem_cgroup(memcg))) {
		3213	unsigned long ceiling = min(READ_ONCE(memcg->memory.max),
		3214	READ_ONCE(memcg->memory.high));
		3215	unsigned long used = page_counter_read(&memcg->memory);
4333		3216
4334	static u64 mem_cgroup_move_charge_read(struct cgroup_subsys_state *css,	3217	pheadroom = min(pheadroom, ceiling - min(ceiling, used));
4335	struct cftype *cft)	3218	memcg = parent;
4336	{	3219	}
4337	return mem_cgroup_from_css(css)->move_charge_at_immigrate;
4338	}	3220	}
4339		3221
4340	#ifdef CONFIG_MMU	3222	/*
4341	static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css,	3223	* Foreign dirty flushing
4342	struct cftype *cft, u64 val)	3224	*
		3225	* There's an inherent mismatch between memcg and writeback. The former
		3226	* tracks ownership per-page while the latter per-inode. This was a
		3227	* deliberate design decision because honoring per-page ownership in the
		3228	* writeback path is complicated, may lead to higher CPU and IO overheads
		3229	* and deemed unnecessary given that write-sharing an inode across
		3230	* different cgroups isn't a common use-case.
		3231	*
		3232	* Combined with inode majority-writer ownership switching, this works well
		3233	* enough in most cases but there are some pathological cases. For
		3234	* example, let's say there are two cgroups A and B which keep writing to
		3235	* different but confined parts of the same inode. B owns the inode and
		3236	* A's memory is limited far below B's. A's dirty ratio can rise enough to
		3237	* trigger balance_dirty_pages() sleeps but B's can be low enough to avoid
		3238	* triggering background writeback. A will be slowed down without a way to
		3239	* make writeback of the dirty pages happen.
		3240	*
		3241	* Conditions like the above can lead to a cgroup getting repeatedly and
		3242	* severely throttled after making some progress after each
		3243	* dirty_expire_interval while the underlying IO device is almost
		3244	* completely idle.
		3245	*
		3246	* Solving this problem completely requires matching the ownership tracking
		3247	* granularities between memcg and writeback in either direction. However,
		3248	* the more egregious behaviors can be avoided by simply remembering the
		3249	* most recent foreign dirtying events and initiating remote flushes on
		3250	* them when local writeback isn't enough to keep the memory clean enough.
		3251	*
		3252	* The following two functions implement such mechanism. When a foreign
		3253	* page - a page whose memcg and writeback ownerships don't match - is
		3254	* dirtied, mem_cgroup_track_foreign_dirty() records the inode owning
		3255	* bdi_writeback on the page owning memcg. When balance_dirty_pages()
		3256	* decides that the memcg needs to sleep due to high dirty ratio, it calls
		3257	* mem_cgroup_flush_foreign() which queues writeback on the recorded
		3258	* foreign bdi_writebacks which haven't expired. Both the numbers of
		3259	* recorded bdi_writebacks and concurrent in-flight foreign writebacks are
		3260	* limited to MEMCG_CGWB_FRN_CNT.
		3261	*
		3262	* The mechanism only remembers IDs and doesn't hold any object references.
		3263	* As being wrong occasionally doesn't matter, updates and accesses to the
		3264	* records are lockless and racy.
		3265	*/
		3266	void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
		3267	struct bdi_writeback *wb)
4343	{	3268	{
4344	struct mem_cgroup *memcg = mem_cgroup_from_css(css);	3269	struct mem_cgroup *memcg = folio_memcg(folio);
4345		3270	struct memcg_cgwb_frn *frn;
4346	pr_warn_once("Cgroup memory moving (move_charge_at_immigrate) is deprecated. "	3271	u64 now = get_jiffies_64();
4347	"Please report your usecase to linux-mm@kvack.org if you "	3272	u64 oldest_at = now;
4348	"depend on this functionality.\n");	3273	int oldest = -1;
		3274	int i;
4349		3275
4350	if (val & ~MOVE_MASK)	3276	trace_track_foreign_dirty(folio, wb);
4351	return -EINVAL;
4352		3277
4353	/*	3278	/*
4354	* No kind of locking is needed in here, because ->can_attach() will	3279	* Pick the slot to use. If there is already a slot for @wb, keep
4355	* check this value once in the beginning of the process, and then carry	3280	* using it. If not replace the oldest one which isn't being
4356	* on with stale data. This means that changes to this value will only	3281	* written out.
4357	* affect task migrations starting after the change.
4358	*/	3282	*/
4359	memcg->move_charge_at_immigrate = val;	3283	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
4360	return 0;	3284	frn = &memcg->cgwb_frn[i];
4361	}	3285	if (frn->bdi_id == wb->bdi->id &&
4362	#else	3286	frn->memcg_id == wb->memcg_css->id)
4363	static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css,	3287	break;
4364	struct cftype *cft, u64 val)	3288	if (time_before64(frn->at, oldest_at) &&
4365	{	3289	atomic_read(&frn->done.cnt) == 1) {
4366	return -ENOSYS;	3290	oldest = i;
4367	}	3291	oldest_at = frn->at;
4368	#endif	3292	}
4369
4370	#ifdef CONFIG_NUMA
4371
4372	#define LRU_ALL_FILE (BIT(LRU_INACTIVE_FILE) \| BIT(LRU_ACTIVE_FILE))
4373	#define LRU_ALL_ANON (BIT(LRU_INACTIVE_ANON) \| BIT(LRU_ACTIVE_ANON))
4374	#define LRU_ALL ((1 << NR_LRU_LISTS) - 1)
4375
4376	static unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg,
4377	int nid, unsigned int lru_mask, bool tree)
4378	{
4379	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
4380	unsigned long nr = 0;
4381	enum lru_list lru;
4382
4383	VM_BUG_ON((unsigned)nid >= nr_node_ids);
4384
4385	for_each_lru(lru) {
4386	if (!(BIT(lru) & lru_mask))
4387	continue;
4388	if (tree)
4389	nr += lruvec_page_state(lruvec, NR_LRU_BASE + lru);
4390	else
4391	nr += lruvec_page_state_local(lruvec, NR_LRU_BASE + lru);
4392	}	3293	}
4393	return nr;
4394	}
4395		3294
4396	static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg,	3295	if (i < MEMCG_CGWB_FRN_CNT) {
4397	unsigned int lru_mask,	3296	/*
4398	bool tree)	3297	* Re-using an existing one. Update timestamp lazily to
4399	{	3298	* avoid making the cacheline hot. We want them to be
4400	unsigned long nr = 0;	3299	* reasonably up-to-date and significantly shorter than
4401	enum lru_list lru;	3300	* dirty_expire_interval as that's what expires the record.
		3301	* Use the shorter of 1s and dirty_expire_interval / 8.
		3302	*/
		3303	unsigned long update_intv =
		3304	min_t(unsigned long, HZ,
		3305	msecs_to_jiffies(dirty_expire_interval * 10) / 8);
4402		3306
4403	for_each_lru(lru) {	3307	if (time_before64(frn->at, now - update_intv))
4404	if (!(BIT(lru) & lru_mask))	3308	frn->at = now;
4405	continue;	3309	} else if (oldest >= 0) {
4406	if (tree)	3310	/* replace the oldest free one */
4407	nr += memcg_page_state(memcg, NR_LRU_BASE + lru);	3311	frn = &memcg->cgwb_frn[oldest];
4408	else	3312	frn->bdi_id = wb->bdi->id;
4409	nr += memcg_page_state_local(memcg, NR_LRU_BASE + lru);	3313	frn->memcg_id = wb->memcg_css->id;
		3314	frn->at = now;
4410	}	3315	}
4411	return nr;
4412	}	3316	}
4413		3317
4414	static int memcg_numa_stat_show(struct seq_file m, void v)	3318	/* issue foreign writeback flushes for recorded foreign dirtying events */
		3319	void mem_cgroup_flush_foreign(struct bdi_writeback *wb)
4415	{	3320	{
4416	struct numa_stat {	3321	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
4417	const char *name;	3322	unsigned long intv = msecs_to_jiffies(dirty_expire_interval * 10);
4418	unsigned int lru_mask;	3323	u64 now = jiffies_64;
4419	};	3324	int i;
4420
4421	static const struct numa_stat stats[] = {
4422	{ "total", LRU_ALL },
4423	{ "file", LRU_ALL_FILE },
4424	{ "anon", LRU_ALL_ANON },
4425	{ "unevictable", BIT(LRU_UNEVICTABLE) },
4426	};
4427	const struct numa_stat *stat;
4428	int nid;
4429	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
4430		3325
4431	mem_cgroup_flush_stats(memcg);	3326	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
		3327	struct memcg_cgwb_frn *frn = &memcg->cgwb_frn[i];
4432		3328
4433	for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) {	3329	/*
4434	seq_printf(m, "%s=%lu", stat->name,	3330	* If the record is older than dirty_expire_interval,
4435	mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,	3331	* writeback on it has already started. No need to kick it
4436	false));	3332	* off again. Also, don't start a new one if there's
4437	for_each_node_state(nid, N_MEMORY)	3333	* already one in flight.
4438	seq_printf(m, " N%d=%lu", nid,	3334	*/
4439	mem_cgroup_node_nr_lru_pages(memcg, nid,	3335	if (time_after64(frn->at, now - intv) &&
4440	stat->lru_mask, false));	3336	atomic_read(&frn->done.cnt) == 1) {
4441	seq_putc(m, '\n');	3337	frn->at = 0;
		3338	trace_flush_foreign(wb, frn->bdi_id, frn->memcg_id);
		3339	cgroup_writeback_by_id(frn->bdi_id, frn->memcg_id,
		3340	WB_REASON_FOREIGN_FLUSH,
		3341	&frn->done);
		3342	}
4442	}	3343	}
		3344	}
4443		3345
4444	for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) {	3346	#else /* CONFIG_CGROUP_WRITEBACK */
4445
4446	seq_printf(m, "hierarchical_%s=%lu", stat->name,
4447	mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
4448	true));
4449	for_each_node_state(nid, N_MEMORY)
4450	seq_printf(m, " N%d=%lu", nid,
4451	mem_cgroup_node_nr_lru_pages(memcg, nid,
4452	stat->lru_mask, true));
4453	seq_putc(m, '\n');
4454	}
4455		3347
		3348	static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
		3349	{
4456	return 0;	3350	return 0;
4457	}	3351	}
4458	#endif /* CONFIG_NUMA */
4459
4460	static const unsigned int memcg1_stats[] = {
4461	NR_FILE_PAGES,
4462	NR_ANON_MAPPED,
4463	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
4464	NR_ANON_THPS,
4465	#endif
4466	NR_SHMEM,
4467	NR_FILE_MAPPED,
4468	NR_FILE_DIRTY,
4469	NR_WRITEBACK,
4470	WORKINGSET_REFAULT_ANON,
4471	WORKINGSET_REFAULT_FILE,
4472	#ifdef CONFIG_SWAP
4473	MEMCG_SWAP,
4474	NR_SWAPCACHE,
4475	#endif
4476	};
4477
4478	static const char *const memcg1_stat_names[] = {
4479	"cache",
4480	"rss",
4481	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
4482	"rss_huge",
4483	#endif
4484	"shmem",
4485	"mapped_file",
4486	"dirty",
4487	"writeback",
4488	"workingset_refault_anon",
4489	"workingset_refault_file",
4490	#ifdef CONFIG_SWAP
4491	"swap",
4492	"swapcached",
4493	#endif
4494	};
4495
4496	/* Universal VM events cgroup1 shows, original sort order */
4497	static const unsigned int memcg1_events[] = {
4498	PGPGIN,
4499	PGPGOUT,
4500	PGFAULT,
4501	PGMAJFAULT,
4502	};
4503		3352
4504	static void memcg1_stat_format(struct mem_cgroup memcg, struct seq_buf s)	3353	static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
4505	{	3354	{
4506	unsigned long memory, memsw;
4507	struct mem_cgroup *mi;
4508	unsigned int i;
4509
4510	BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats));
4511
4512	mem_cgroup_flush_stats(memcg);
4513
4514	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
4515	unsigned long nr;
4516
4517	nr = memcg_page_state_local_output(memcg, memcg1_stats[i]);
4518	seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr);
4519	}
4520
4521	for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
4522	seq_buf_printf(s, "%s %lu\n", vm_event_name(memcg1_events[i]),
4523	memcg_events_local(memcg, memcg1_events[i]));
4524
4525	for (i = 0; i < NR_LRU_LISTS; i++)
4526	seq_buf_printf(s, "%s %lu\n", lru_list_name(i),
4527	memcg_page_state_local(memcg, NR_LRU_BASE + i) *
4528	PAGE_SIZE);
4529
4530	/* Hierarchical information */
4531	memory = memsw = PAGE_COUNTER_MAX;
4532	for (mi = memcg; mi; mi = parent_mem_cgroup(mi)) {
4533	memory = min(memory, READ_ONCE(mi->memory.max));
4534	memsw = min(memsw, READ_ONCE(mi->memsw.max));
4535	}
4536	seq_buf_printf(s, "hierarchical_memory_limit %llu\n",
4537	(u64)memory * PAGE_SIZE);
4538	seq_buf_printf(s, "hierarchical_memsw_limit %llu\n",
4539	(u64)memsw * PAGE_SIZE);
4540
4541	for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
4542	unsigned long nr;
4543
4544	nr = memcg_page_state_output(memcg, memcg1_stats[i]);
4545	seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i],
4546	(u64)nr);
4547	}
4548
4549	for (i = 0; i < ARRAY_SIZE(memcg1_events); i++)
4550	seq_buf_printf(s, "total_%s %llu\n",
4551	vm_event_name(memcg1_events[i]),
4552	(u64)memcg_events(memcg, memcg1_events[i]));
4553
4554	for (i = 0; i < NR_LRU_LISTS; i++)
4555	seq_buf_printf(s, "total_%s %llu\n", lru_list_name(i),
4556	(u64)memcg_page_state(memcg, NR_LRU_BASE + i) *
4557	PAGE_SIZE);
4558
4559	#ifdef CONFIG_DEBUG_VM
4560	{
4561	pg_data_t *pgdat;
4562	struct mem_cgroup_per_node *mz;
4563	unsigned long anon_cost = 0;
4564	unsigned long file_cost = 0;
4565
4566	for_each_online_pgdat(pgdat) {
4567	mz = memcg->nodeinfo[pgdat->node_id];
4568
4569	anon_cost += mz->lruvec.anon_cost;
4570	file_cost += mz->lruvec.file_cost;
4571	}
4572	seq_buf_printf(s, "anon_cost %lu\n", anon_cost);
4573	seq_buf_printf(s, "file_cost %lu\n", file_cost);
4574	}
4575	#endif
4576	}	3355	}
4577		3356
4578	static u64 mem_cgroup_swappiness_read(struct cgroup_subsys_state *css,	3357	static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
4579	struct cftype *cft)
4580	{	3358	{
4581	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
4582
4583	return mem_cgroup_swappiness(memcg);
4584	}	3359	}
4585		3360
4586	static int mem_cgroup_swappiness_write(struct cgroup_subsys_state *css,	3361	#endif /* CONFIG_CGROUP_WRITEBACK */
4587	struct cftype *cft, u64 val)
4588	{
4589	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
4590		3362
4591	if (val > 200)	3363	/*
4592	return -EINVAL;	3364	* Private memory cgroup IDR
4593
4594	if (!mem_cgroup_is_root(memcg))
4595	WRITE_ONCE(memcg->swappiness, val);
4596	else
4597	WRITE_ONCE(vm_swappiness, val);
4598
4599	return 0;
4600	}
4601
4602	static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)
4603	{
4604	struct mem_cgroup_threshold_ary *t;
4605	unsigned long usage;
4606	int i;
4607
4608	rcu_read_lock();
4609	if (!swap)
4610	t = rcu_dereference(memcg->thresholds.primary);
4611	else
4612	t = rcu_dereference(memcg->memsw_thresholds.primary);
4613
4614	if (!t)
4615	goto unlock;
4616
4617	usage = mem_cgroup_usage(memcg, swap);
4618
4619	/*
4620	* current_threshold points to threshold just below or equal to usage.
4621	* If it's not true, a threshold was crossed after last
4622	* call of __mem_cgroup_threshold().
4623	*/
4624	i = t->current_threshold;
4625
4626	/*
4627	* Iterate backward over array of thresholds starting from
4628	* current_threshold and check if a threshold is crossed.
4629	* If none of thresholds below usage is crossed, we read
4630	* only one element of the array here.
4631	*/
4632	for (; i >= 0 && unlikely(t->entries[i].threshold > usage); i--)
4633	eventfd_signal(t->entries[i].eventfd);
4634
4635	/* i = current_threshold + 1 */
4636	i++;
4637
4638	/*
4639	* Iterate forward over array of thresholds starting from
4640	* current_threshold+1 and check if a threshold is crossed.
4641	* If none of thresholds above usage is crossed, we read
4642	* only one element of the array here.
4643	*/
4644	for (; i < t->size && unlikely(t->entries[i].threshold <= usage); i++)
4645	eventfd_signal(t->entries[i].eventfd);
4646
4647	/* Update current_threshold */
4648	t->current_threshold = i - 1;
4649	unlock:
4650	rcu_read_unlock();
4651	}
4652
4653	static void mem_cgroup_threshold(struct mem_cgroup *memcg)
4654	{
4655	while (memcg) {
4656	__mem_cgroup_threshold(memcg, false);
4657	if (do_memsw_account())
4658	__mem_cgroup_threshold(memcg, true);
4659
4660	memcg = parent_mem_cgroup(memcg);
4661	}
4662	}
4663
4664	static int compare_thresholds(const void a, const void b)
4665	{
4666	const struct mem_cgroup_threshold *_a = a;
4667	const struct mem_cgroup_threshold *_b = b;
4668
4669	if (_a->threshold > _b->threshold)
4670	return 1;
4671
4672	if (_a->threshold < _b->threshold)
4673	return -1;
4674
4675	return 0;
4676	}
4677
4678	static int mem_cgroup_oom_notify_cb(struct mem_cgroup *memcg)
4679	{
4680	struct mem_cgroup_eventfd_list *ev;
4681
4682	spin_lock(&memcg_oom_lock);
4683
4684	list_for_each_entry(ev, &memcg->oom_notify, list)
4685	eventfd_signal(ev->eventfd);
4686
4687	spin_unlock(&memcg_oom_lock);
4688	return 0;
4689	}
4690
4691	static void mem_cgroup_oom_notify(struct mem_cgroup *memcg)
4692	{
4693	struct mem_cgroup *iter;
4694
4695	for_each_mem_cgroup_tree(iter, memcg)
4696	mem_cgroup_oom_notify_cb(iter);
4697	}
4698
4699	static int __mem_cgroup_usage_register_event(struct mem_cgroup *memcg,
4700	struct eventfd_ctx eventfd, const char args, enum res_type type)
4701	{
4702	struct mem_cgroup_thresholds *thresholds;
4703	struct mem_cgroup_threshold_ary *new;
4704	unsigned long threshold;
4705	unsigned long usage;
4706	int i, size, ret;
4707
4708	ret = page_counter_memparse(args, "-1", &threshold);
4709	if (ret)
4710	return ret;
4711
4712	mutex_lock(&memcg->thresholds_lock);
4713
4714	if (type == _MEM) {
4715	thresholds = &memcg->thresholds;
4716	usage = mem_cgroup_usage(memcg, false);
4717	} else if (type == _MEMSWAP) {
4718	thresholds = &memcg->memsw_thresholds;
4719	usage = mem_cgroup_usage(memcg, true);
4720	} else
4721	BUG();
4722
4723	/* Check if a threshold crossed before adding a new one */
4724	if (thresholds->primary)
4725	__mem_cgroup_threshold(memcg, type == _MEMSWAP);
4726
4727	size = thresholds->primary ? thresholds->primary->size + 1 : 1;
4728
4729	/* Allocate memory for new array of thresholds */
4730	new = kmalloc(struct_size(new, entries, size), GFP_KERNEL);
4731	if (!new) {
4732	ret = -ENOMEM;
4733	goto unlock;
4734	}
4735	new->size = size;
4736
4737	/* Copy thresholds (if any) to new array */
4738	if (thresholds->primary)
4739	memcpy(new->entries, thresholds->primary->entries,
4740	flex_array_size(new, entries, size - 1));
4741
4742	/* Add new threshold */
4743	new->entries[size - 1].eventfd = eventfd;
4744	new->entries[size - 1].threshold = threshold;
4745
4746	/* Sort thresholds. Registering of new threshold isn't time-critical */
4747	sort(new->entries, size, sizeof(*new->entries),
4748	compare_thresholds, NULL);
4749
4750	/* Find current threshold */
4751	new->current_threshold = -1;
4752	for (i = 0; i < size; i++) {
4753	if (new->entries[i].threshold <= usage) {
4754	/*
4755	* new->current_threshold will not be used until
4756	* rcu_assign_pointer(), so it's safe to increment
4757	* it here.
4758	*/
4759	++new->current_threshold;
4760	} else
4761	break;
4762	}
4763
4764	/* Free old spare buffer and save old primary buffer as spare */
4765	kfree(thresholds->spare);
4766	thresholds->spare = thresholds->primary;
4767
4768	rcu_assign_pointer(thresholds->primary, new);
4769
4770	/* To be sure that nobody uses thresholds */
4771	synchronize_rcu();
4772
4773	unlock:
4774	mutex_unlock(&memcg->thresholds_lock);
4775
4776	return ret;
4777	}
4778
4779	static int mem_cgroup_usage_register_event(struct mem_cgroup *memcg,
4780	struct eventfd_ctx eventfd, const char args)
4781	{
4782	return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEM);
4783	}
4784
4785	static int memsw_cgroup_usage_register_event(struct mem_cgroup *memcg,
4786	struct eventfd_ctx eventfd, const char args)
4787	{
4788	return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEMSWAP);
4789	}
4790
4791	static void __mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
4792	struct eventfd_ctx *eventfd, enum res_type type)
4793	{
4794	struct mem_cgroup_thresholds *thresholds;
4795	struct mem_cgroup_threshold_ary *new;
4796	unsigned long usage;
4797	int i, j, size, entries;
4798
4799	mutex_lock(&memcg->thresholds_lock);
4800
4801	if (type == _MEM) {
4802	thresholds = &memcg->thresholds;
4803	usage = mem_cgroup_usage(memcg, false);
4804	} else if (type == _MEMSWAP) {
4805	thresholds = &memcg->memsw_thresholds;
4806	usage = mem_cgroup_usage(memcg, true);
4807	} else
4808	BUG();
4809
4810	if (!thresholds->primary)
4811	goto unlock;
4812
4813	/* Check if a threshold crossed before removing */
4814	__mem_cgroup_threshold(memcg, type == _MEMSWAP);
4815
4816	/* Calculate new number of threshold */
4817	size = entries = 0;
4818	for (i = 0; i < thresholds->primary->size; i++) {
4819	if (thresholds->primary->entries[i].eventfd != eventfd)
4820	size++;
4821	else
4822	entries++;
4823	}
4824
4825	new = thresholds->spare;
4826
4827	/* If no items related to eventfd have been cleared, nothing to do */
4828	if (!entries)
4829	goto unlock;
4830
4831	/* Set thresholds array to NULL if we don't have thresholds */
4832	if (!size) {
4833	kfree(new);
4834	new = NULL;
4835	goto swap_buffers;
4836	}
4837
4838	new->size = size;
4839
4840	/* Copy thresholds and find current threshold */
4841	new->current_threshold = -1;
4842	for (i = 0, j = 0; i < thresholds->primary->size; i++) {
4843	if (thresholds->primary->entries[i].eventfd == eventfd)
4844	continue;
4845
4846	new->entries[j] = thresholds->primary->entries[i];
4847	if (new->entries[j].threshold <= usage) {
4848	/*
4849	* new->current_threshold will not be used
4850	* until rcu_assign_pointer(), so it's safe to increment
4851	* it here.
4852	*/
4853	++new->current_threshold;
4854	}
4855	j++;
4856	}
4857
4858	swap_buffers:
4859	/* Swap primary and spare array */
4860	thresholds->spare = thresholds->primary;
4861
4862	rcu_assign_pointer(thresholds->primary, new);
4863
4864	/* To be sure that nobody uses thresholds */
4865	synchronize_rcu();
4866
4867	/* If all events are unregistered, free the spare array */
4868	if (!new) {
4869	kfree(thresholds->spare);
4870	thresholds->spare = NULL;
4871	}
4872	unlock:
4873	mutex_unlock(&memcg->thresholds_lock);
4874	}
4875
4876	static void mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
4877	struct eventfd_ctx *eventfd)
4878	{
4879	return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEM);
4880	}
4881
4882	static void memsw_cgroup_usage_unregister_event(struct mem_cgroup *memcg,
4883	struct eventfd_ctx *eventfd)
4884	{
4885	return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEMSWAP);
4886	}
4887
4888	static int mem_cgroup_oom_register_event(struct mem_cgroup *memcg,
4889	struct eventfd_ctx eventfd, const char args)
4890	{
4891	struct mem_cgroup_eventfd_list *event;
4892
4893	event = kmalloc(sizeof(*event), GFP_KERNEL);
4894	if (!event)
4895	return -ENOMEM;
4896
4897	spin_lock(&memcg_oom_lock);
4898
4899	event->eventfd = eventfd;
4900	list_add(&event->list, &memcg->oom_notify);
4901
4902	/* already in OOM ? */
4903	if (memcg->under_oom)
4904	eventfd_signal(eventfd);
4905	spin_unlock(&memcg_oom_lock);
4906
4907	return 0;
4908	}
4909
4910	static void mem_cgroup_oom_unregister_event(struct mem_cgroup *memcg,
4911	struct eventfd_ctx *eventfd)
4912	{
4913	struct mem_cgroup_eventfd_list ev, tmp;
4914
4915	spin_lock(&memcg_oom_lock);
4916
4917	list_for_each_entry_safe(ev, tmp, &memcg->oom_notify, list) {
4918	if (ev->eventfd == eventfd) {
4919	list_del(&ev->list);
4920	kfree(ev);
4921	}
4922	}
4923
4924	spin_unlock(&memcg_oom_lock);
4925	}
4926
4927	static int mem_cgroup_oom_control_read(struct seq_file sf, void v)
4928	{
4929	struct mem_cgroup *memcg = mem_cgroup_from_seq(sf);
4930
4931	seq_printf(sf, "oom_kill_disable %d\n", READ_ONCE(memcg->oom_kill_disable));
4932	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
4933	seq_printf(sf, "oom_kill %lu\n",
4934	atomic_long_read(&memcg->memory_events[MEMCG_OOM_KILL]));
4935	return 0;
4936	}
4937
4938	static int mem_cgroup_oom_control_write(struct cgroup_subsys_state *css,
4939	struct cftype *cft, u64 val)
4940	{
4941	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
4942
4943	/* cannot set to root cgroup and only 0 and 1 are allowed */
4944	if (mem_cgroup_is_root(memcg) \|\| !((val == 0) \|\| (val == 1)))
4945	return -EINVAL;
4946
4947	WRITE_ONCE(memcg->oom_kill_disable, val);
4948	if (!val)
4949	memcg_oom_recover(memcg);
4950
4951	return 0;
4952	}
4953
4954	#ifdef CONFIG_CGROUP_WRITEBACK
4955
4956	#include <trace/events/writeback.h>
4957
4958	static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
4959	{
4960	return wb_domain_init(&memcg->cgwb_domain, gfp);
4961	}
4962
4963	static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
4964	{
4965	wb_domain_exit(&memcg->cgwb_domain);
4966	}
4967
4968	static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
4969	{
4970	wb_domain_size_changed(&memcg->cgwb_domain);
4971	}
4972
4973	struct wb_domain mem_cgroup_wb_domain(struct bdi_writeback wb)
4974	{
4975	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
4976
4977	if (!memcg->css.parent)
4978	return NULL;
4979
4980	return &memcg->cgwb_domain;
4981	}
4982
4983	/**
4984	* mem_cgroup_wb_stats - retrieve writeback related stats from its memcg
4985	* @wb: bdi_writeback in question
4986	* @pfilepages: out parameter for number of file pages
4987	* @pheadroom: out parameter for number of allocatable pages according to memcg
4988	* @pdirty: out parameter for number of dirty pages
4989	* @pwriteback: out parameter for number of pages under writeback
4990	*	3365	*
4991	* Determine the numbers of file, headroom, dirty, and writeback pages in	3366	* Swap-out records and page cache shadow entries need to store memcg
4992	* @wb's memcg. File, dirty and writeback are self-explanatory. Headroom	3367	* references in constrained space, so we maintain an ID space that is
4993	* is a bit more involved.	3368	* limited to 16 bit (MEM_CGROUP_ID_MAX), limiting the total number of
		3369	* memory-controlled cgroups to 64k.
4994	*	3370	*
4995	* A memcg's headroom is "min(max, high) - used". In the hierarchy, the	3371	* However, there usually are many references to the offline CSS after
4996	* headroom is calculated as the lowest headroom of itself and the	3372	* the cgroup has been destroyed, such as page cache or reclaimable
4997	* ancestors. Note that this doesn't consider the actual amount of	3373	* slab objects, that don't need to hang on to the ID. We want to keep
4998	* available memory in the system. The caller should further cap	3374	* those dead CSS from occupying IDs, or we might quickly exhaust the
4999	* *@pheadroom accordingly.	3375	* relatively small ID space and prevent the creation of new cgroups
5000	*/	3376	* even when there are much fewer than 64k cgroups - possibly none.
5001	void mem_cgroup_wb_stats(struct bdi_writeback wb, unsigned long pfilepages,
5002	unsigned long pheadroom, unsigned long pdirty,
5003	unsigned long *pwriteback)
5004	{
5005	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
5006	struct mem_cgroup *parent;
5007
5008	mem_cgroup_flush_stats_ratelimited(memcg);
5009
5010	*pdirty = memcg_page_state(memcg, NR_FILE_DIRTY);
5011	*pwriteback = memcg_page_state(memcg, NR_WRITEBACK);
5012	*pfilepages = memcg_page_state(memcg, NR_INACTIVE_FILE) +
5013	memcg_page_state(memcg, NR_ACTIVE_FILE);
5014
5015	*pheadroom = PAGE_COUNTER_MAX;
5016	while ((parent = parent_mem_cgroup(memcg))) {
5017	unsigned long ceiling = min(READ_ONCE(memcg->memory.max),
5018	READ_ONCE(memcg->memory.high));
5019	unsigned long used = page_counter_read(&memcg->memory);
5020
5021	pheadroom = min(pheadroom, ceiling - min(ceiling, used));
5022	memcg = parent;
5023	}
5024	}
5025
5026	/*
5027	* Foreign dirty flushing
5028	*	3377	*
5029	* There's an inherent mismatch between memcg and writeback. The former	3378	* Maintain a private 16-bit ID space for memcg, and allow the ID to
5030	* tracks ownership per-page while the latter per-inode. This was a	3379	* be freed and recycled when it's no longer needed, which is usually
5031	* deliberate design decision because honoring per-page ownership in the	3380	* when the CSS is offlined.
5032	* writeback path is complicated, may lead to higher CPU and IO overheads
5033	* and deemed unnecessary given that write-sharing an inode across
5034	* different cgroups isn't a common use-case.
5035	*	3381	*
5036	* Combined with inode majority-writer ownership switching, this works well
5037	* enough in most cases but there are some pathological cases. For
5038	* example, let's say there are two cgroups A and B which keep writing to
5039	* different but confined parts of the same inode. B owns the inode and
5040	* A's memory is limited far below B's. A's dirty ratio can rise enough to
5041	* trigger balance_dirty_pages() sleeps but B's can be low enough to avoid
5042	* triggering background writeback. A will be slowed down without a way to
5043	* make writeback of the dirty pages happen.
5044	*
5045	* Conditions like the above can lead to a cgroup getting repeatedly and
5046	* severely throttled after making some progress after each
5047	* dirty_expire_interval while the underlying IO device is almost
5048	* completely idle.
5049	*
5050	* Solving this problem completely requires matching the ownership tracking
5051	* granularities between memcg and writeback in either direction. However,
5052	* the more egregious behaviors can be avoided by simply remembering the
5053	* most recent foreign dirtying events and initiating remote flushes on
5054	* them when local writeback isn't enough to keep the memory clean enough.
5055	*
5056	* The following two functions implement such mechanism. When a foreign
5057	* page - a page whose memcg and writeback ownerships don't match - is
5058	* dirtied, mem_cgroup_track_foreign_dirty() records the inode owning
5059	* bdi_writeback on the page owning memcg. When balance_dirty_pages()
5060	* decides that the memcg needs to sleep due to high dirty ratio, it calls
5061	* mem_cgroup_flush_foreign() which queues writeback on the recorded
5062	* foreign bdi_writebacks which haven't expired. Both the numbers of
5063	* recorded bdi_writebacks and concurrent in-flight foreign writebacks are
5064	* limited to MEMCG_CGWB_FRN_CNT.
5065	*
5066	* The mechanism only remembers IDs and doesn't hold any object references.
5067	* As being wrong occasionally doesn't matter, updates and accesses to the
5068	* records are lockless and racy.
5069	*/
5070	void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
5071	struct bdi_writeback *wb)
5072	{
5073	struct mem_cgroup *memcg = folio_memcg(folio);
5074	struct memcg_cgwb_frn *frn;
5075	u64 now = get_jiffies_64();
5076	u64 oldest_at = now;
5077	int oldest = -1;
5078	int i;
5079
5080	trace_track_foreign_dirty(folio, wb);
5081
5082	/*
5083	* Pick the slot to use. If there is already a slot for @wb, keep
5084	* using it. If not replace the oldest one which isn't being
5085	* written out.
5086	*/
5087	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
5088	frn = &memcg->cgwb_frn[i];
5089	if (frn->bdi_id == wb->bdi->id &&
5090	frn->memcg_id == wb->memcg_css->id)
5091	break;
5092	if (time_before64(frn->at, oldest_at) &&
5093	atomic_read(&frn->done.cnt) == 1) {
5094	oldest = i;
5095	oldest_at = frn->at;
5096	}
5097	}
5098
5099	if (i < MEMCG_CGWB_FRN_CNT) {
5100	/*
5101	* Re-using an existing one. Update timestamp lazily to
5102	* avoid making the cacheline hot. We want them to be
5103	* reasonably up-to-date and significantly shorter than
5104	* dirty_expire_interval as that's what expires the record.
5105	* Use the shorter of 1s and dirty_expire_interval / 8.
5106	*/
5107	unsigned long update_intv =
5108	min_t(unsigned long, HZ,
5109	msecs_to_jiffies(dirty_expire_interval * 10) / 8);
5110
5111	if (time_before64(frn->at, now - update_intv))
5112	frn->at = now;
5113	} else if (oldest >= 0) {
5114	/* replace the oldest free one */
5115	frn = &memcg->cgwb_frn[oldest];
5116	frn->bdi_id = wb->bdi->id;
5117	frn->memcg_id = wb->memcg_css->id;
5118	frn->at = now;
5119	}
5120	}
5121
5122	/* issue foreign writeback flushes for recorded foreign dirtying events */
5123	void mem_cgroup_flush_foreign(struct bdi_writeback *wb)
5124	{
5125	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
5126	unsigned long intv = msecs_to_jiffies(dirty_expire_interval * 10);
5127	u64 now = jiffies_64;
5128	int i;
5129
5130	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
5131	struct memcg_cgwb_frn *frn = &memcg->cgwb_frn[i];
5132
5133	/*
5134	* If the record is older than dirty_expire_interval,
5135	* writeback on it has already started. No need to kick it
5136	* off again. Also, don't start a new one if there's
5137	* already one in flight.
5138	*/
5139	if (time_after64(frn->at, now - intv) &&
5140	atomic_read(&frn->done.cnt) == 1) {
5141	frn->at = 0;
5142	trace_flush_foreign(wb, frn->bdi_id, frn->memcg_id);
5143	cgroup_writeback_by_id(frn->bdi_id, frn->memcg_id,
5144	WB_REASON_FOREIGN_FLUSH,
5145	&frn->done);
5146	}
5147	}
5148	}
5149
5150	#else /* CONFIG_CGROUP_WRITEBACK */
5151
5152	static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
5153	{
5154	return 0;
5155	}
5156
5157	static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
5158	{
5159	}
5160
5161	static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
5162	{
5163	}
5164
5165	#endif /* CONFIG_CGROUP_WRITEBACK */
5166
5167	/*
5168	* DO NOT USE IN NEW FILES.
5169	*
5170	* "cgroup.event_control" implementation.
5171	*
5172	* This is way over-engineered. It tries to support fully configurable
5173	* events for each user. Such level of flexibility is completely
5174	* unnecessary especially in the light of the planned unified hierarchy.
5175	*
5176	* Please deprecate this and replace with something simpler if at all
5177	* possible.
5178	*/
5179
5180	/*
5181	* Unregister event and free resources.
5182	*
5183	* Gets called from workqueue.
5184	*/
5185	static void memcg_event_remove(struct work_struct *work)
5186	{
5187	struct mem_cgroup_event *event =
5188	container_of(work, struct mem_cgroup_event, remove);
5189	struct mem_cgroup *memcg = event->memcg;
5190
5191	remove_wait_queue(event->wqh, &event->wait);
5192
5193	event->unregister_event(memcg, event->eventfd);
5194
5195	/* Notify userspace the event is going away. */
5196	eventfd_signal(event->eventfd);
5197
5198	eventfd_ctx_put(event->eventfd);
5199	kfree(event);
5200	css_put(&memcg->css);
5201	}
5202
5203	/*
5204	* Gets called on EPOLLHUP on eventfd when user closes it.
5205	*
5206	* Called with wqh->lock held and interrupts disabled.
5207	*/
5208	static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode,
5209	int sync, void *key)
5210	{
5211	struct mem_cgroup_event *event =
5212	container_of(wait, struct mem_cgroup_event, wait);
5213	struct mem_cgroup *memcg = event->memcg;
5214	__poll_t flags = key_to_poll(key);
5215
5216	if (flags & EPOLLHUP) {
5217	/*
5218	* If the event has been detached at cgroup removal, we
5219	* can simply return knowing the other side will cleanup
5220	* for us.
5221	*
5222	* We can't race against event freeing since the other
5223	* side will require wqh->lock via remove_wait_queue(),
5224	* which we hold.
5225	*/
5226	spin_lock(&memcg->event_list_lock);
5227	if (!list_empty(&event->list)) {
5228	list_del_init(&event->list);
5229	/*
5230	* We are in atomic context, but cgroup_event_remove()
5231	* may sleep, so we have to call it in workqueue.
5232	*/
5233	schedule_work(&event->remove);
5234	}
5235	spin_unlock(&memcg->event_list_lock);
5236	}
5237
5238	return 0;
5239	}
5240
5241	static void memcg_event_ptable_queue_proc(struct file *file,
5242	wait_queue_head_t wqh, poll_table pt)
5243	{
5244	struct mem_cgroup_event *event =
5245	container_of(pt, struct mem_cgroup_event, pt);
5246
5247	event->wqh = wqh;
5248	add_wait_queue(wqh, &event->wait);
5249	}
5250
5251	/*
5252	* DO NOT USE IN NEW FILES.
5253	*
5254	* Parse input and register new cgroup event handler.
5255	*
5256	* Input must be in format '<event_fd> <control_fd> <args>'.
5257	* Interpretation of args is defined by control file implementation.
5258	*/
5259	static ssize_t memcg_write_event_control(struct kernfs_open_file *of,
5260	char *buf, size_t nbytes, loff_t off)
5261	{
5262	struct cgroup_subsys_state *css = of_css(of);
5263	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
5264	struct mem_cgroup_event *event;
5265	struct cgroup_subsys_state *cfile_css;
5266	unsigned int efd, cfd;
5267	struct fd efile;
5268	struct fd cfile;
5269	struct dentry *cdentry;
5270	const char *name;
5271	char *endp;
5272	int ret;
5273
5274	if (IS_ENABLED(CONFIG_PREEMPT_RT))
5275	return -EOPNOTSUPP;
5276
5277	buf = strstrip(buf);
5278
5279	efd = simple_strtoul(buf, &endp, 10);
5280	if (*endp != ' ')
5281	return -EINVAL;
5282	buf = endp + 1;
5283
5284	cfd = simple_strtoul(buf, &endp, 10);
5285	if ((endp != ' ') && (endp != '\0'))
5286	return -EINVAL;
5287	buf = endp + 1;
5288
5289	event = kzalloc(sizeof(*event), GFP_KERNEL);
5290	if (!event)
5291	return -ENOMEM;
5292
5293	event->memcg = memcg;
5294	INIT_LIST_HEAD(&event->list);
5295	init_poll_funcptr(&event->pt, memcg_event_ptable_queue_proc);
5296	init_waitqueue_func_entry(&event->wait, memcg_event_wake);
5297	INIT_WORK(&event->remove, memcg_event_remove);
5298
5299	efile = fdget(efd);
5300	if (!efile.file) {
5301	ret = -EBADF;
5302	goto out_kfree;
5303	}
5304
5305	event->eventfd = eventfd_ctx_fileget(efile.file);
5306	if (IS_ERR(event->eventfd)) {
5307	ret = PTR_ERR(event->eventfd);
5308	goto out_put_efile;
5309	}
5310
5311	cfile = fdget(cfd);
5312	if (!cfile.file) {
5313	ret = -EBADF;
5314	goto out_put_eventfd;
5315	}
5316
5317	/* the process need read permission on control file */
5318	/* AV: shouldn't we check that it's been opened for read instead? */
5319	ret = file_permission(cfile.file, MAY_READ);
5320	if (ret < 0)
5321	goto out_put_cfile;
5322
5323	/*
5324	* The control file must be a regular cgroup1 file. As a regular cgroup
5325	* file can't be renamed, it's safe to access its name afterwards.
5326	*/
5327	cdentry = cfile.file->f_path.dentry;
5328	if (cdentry->d_sb->s_type != &cgroup_fs_type \|\| !d_is_reg(cdentry)) {
5329	ret = -EINVAL;
5330	goto out_put_cfile;
5331	}
5332
5333	/*
5334	* Determine the event callbacks and set them in @event. This used
5335	* to be done via struct cftype but cgroup core no longer knows
5336	* about these events. The following is crude but the whole thing
5337	* is for compatibility anyway.
5338	*
5339	* DO NOT ADD NEW FILES.
5340	*/
5341	name = cdentry->d_name.name;
5342
5343	if (!strcmp(name, "memory.usage_in_bytes")) {
5344	event->register_event = mem_cgroup_usage_register_event;
5345	event->unregister_event = mem_cgroup_usage_unregister_event;
5346	} else if (!strcmp(name, "memory.oom_control")) {
5347	event->register_event = mem_cgroup_oom_register_event;
5348	event->unregister_event = mem_cgroup_oom_unregister_event;
5349	} else if (!strcmp(name, "memory.pressure_level")) {
5350	event->register_event = vmpressure_register_event;
5351	event->unregister_event = vmpressure_unregister_event;
5352	} else if (!strcmp(name, "memory.memsw.usage_in_bytes")) {
5353	event->register_event = memsw_cgroup_usage_register_event;
5354	event->unregister_event = memsw_cgroup_usage_unregister_event;
5355	} else {
5356	ret = -EINVAL;
5357	goto out_put_cfile;
5358	}
5359
5360	/*
5361	* Verify @cfile should belong to @css. Also, remaining events are
5362	* automatically removed on cgroup destruction but the removal is
5363	* asynchronous, so take an extra ref on @css.
5364	*/
5365	cfile_css = css_tryget_online_from_dir(cdentry->d_parent,
5366	&memory_cgrp_subsys);
5367	ret = -EINVAL;
5368	if (IS_ERR(cfile_css))
5369	goto out_put_cfile;
5370	if (cfile_css != css) {
5371	css_put(cfile_css);
5372	goto out_put_cfile;
5373	}
5374
5375	ret = event->register_event(memcg, event->eventfd, buf);
5376	if (ret)
5377	goto out_put_css;
5378
5379	vfs_poll(efile.file, &event->pt);
5380
5381	spin_lock_irq(&memcg->event_list_lock);
5382	list_add(&event->list, &memcg->event_list);
5383	spin_unlock_irq(&memcg->event_list_lock);
5384
5385	fdput(cfile);
5386	fdput(efile);
5387
5388	return nbytes;
5389
5390	out_put_css:
5391	css_put(css);
5392	out_put_cfile:
5393	fdput(cfile);
5394	out_put_eventfd:
5395	eventfd_ctx_put(event->eventfd);
5396	out_put_efile:
5397	fdput(efile);
5398	out_kfree:
5399	kfree(event);
5400
5401	return ret;
5402	}
5403
5404	#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_SLUB_DEBUG)
5405	static int mem_cgroup_slab_show(struct seq_file m, void p)
5406	{
5407	/*

2281 more lines aren’t shown here.

mm/memfd.c+45−0

This diff is large: its 52 lines aren’t shown here.

mm/memory-failure.c+138−121

This diff is large: its 688 lines aren’t shown here.

mm/memory-tiers.c+21−33

This diff is large: its 104 lines aren’t shown here.

mm/memory.c+254−122

This diff is large: its 776 lines aren’t shown here.

mm/memory_hotplug.c+24−28

This diff is large: its 146 lines aren’t shown here.

mm/mempolicy.c+26−12

This diff is large: its 102 lines aren’t shown here.

mm/migrate.c+94−119

This diff is large: its 415 lines aren’t shown here.

mm/migrate_device.c+13−11

This diff is large: its 65 lines aren’t shown here.

mm/mincore.c+2−2

This diff is large: its 18 lines aren’t shown here.

mm/mlock.c+4−15

This diff is large: its 31 lines aren’t shown here.

mm/mm_init.c+29−67

This diff is large: its 220 lines aren’t shown here.

mm/mmap.c+41−0

This diff is large: its 48 lines aren’t shown here.

mm/mmap_lock.c+20−155

This diff is large: its 210 lines aren’t shown here.

mm/mprotect.c+3−1

This diff is large: its 18 lines aren’t shown here.

mm/mremap.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/page-writeback.c+173−146

This diff is large: its 424 lines aren’t shown here.

mm/page_alloc.c+48−30

This diff is large: its 180 lines aren’t shown here.

mm/page_counter.c+173−0

This diff is large: its 177 lines aren’t shown here.

mm/page_ext.c+23−9

This diff is large: its 62 lines aren’t shown here.

mm/page_io.c+12−10

This diff is large: its 77 lines aren’t shown here.

mm/pagewalk.c+4−53

This diff is large: its 92 lines aren’t shown here.

mm/percpu-internal.h+3−3

This diff is large: its 27 lines aren’t shown here.

mm/percpu.c+3−3

This diff is large: its 27 lines aren’t shown here.

mm/readahead.c+98−178

This diff is large: its 372 lines aren’t shown here.

mm/rmap.c+87−82

This diff is large: its 331 lines aren’t shown here.

mm/shmem.c+298−61

This diff is large: its 529 lines aren’t shown here.

mm/slab.h+1−1

This diff is large: its 9 lines aren’t shown here.

mm/slab_common.c+5−5

This diff is large: its 40 lines aren’t shown here.

mm/slub.c+30−21

This diff is large: its 135 lines aren’t shown here.

mm/sparse-vmemmap.c+8−0

This diff is large: its 14 lines aren’t shown here.

mm/sparse.c+15−13

This diff is large: its 77 lines aren’t shown here.

mm/swap.c+33−18

This diff is large: its 120 lines aren’t shown here.

mm/swap.h+26−4

This diff is large: its 68 lines aren’t shown here.

mm/swap_state.c+49−71

This diff is large: its 264 lines aren’t shown here.

mm/swapfile.c+61−14

This diff is large: its 119 lines aren’t shown here.

mm/truncate.c+32−38

This diff is large: its 136 lines aren’t shown here.

mm/userfaultfd.c+4−10

This diff is large: its 35 lines aren’t shown here.

mm/util.c+17−0

This diff is large: its 24 lines aren’t shown here.

mm/vmalloc.c+5−4

This diff is large: its 30 lines aren’t shown here.

mm/vmscan.c+120−68

This diff is large: its 428 lines aren’t shown here.

mm/vmstat.c+25−1

This diff is large: its 38 lines aren’t shown here.

mm/zsmalloc.c+101−74

This diff is large: its 508 lines aren’t shown here.

mm/zswap.c+61−65

This diff is large: its 329 lines aren’t shown here.

samples/kmemleak/kmemleak-test.c+1−0

This diff is large: its 6 lines aren’t shown here.

tools/include/uapi/linux/fs.h+552−0

This diff is large: its 553 lines aren’t shown here.

tools/include/uapi/linux/prctl.h+331−0

This diff is large: its 332 lines aren’t shown here.

tools/mm/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/mm/thp_swap_allocator_test.c+234−0

This diff is large: its 235 lines aren’t shown here.

tools/objtool/check.c+2−0

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/cgroup/config+0−1

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/damon/Makefile+2−1

This diff is large: its 17 lines aren’t shown here.

tools/testing/selftests/damon/_damon_sysfs.py+64−1

This diff is large: its 99 lines aren’t shown here.

tools/testing/selftests/damon/access_memory.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/damon/access_memory_even.c+42−0

This diff is large: its 43 lines aren’t shown here.

tools/testing/selftests/damon/damon_nr_regions.py+145−0

This diff is large: its 146 lines aren’t shown here.

tools/testing/selftests/damon/damos_tried_regions.py+65−0

This diff is large: its 66 lines aren’t shown here.

tools/testing/selftests/drivers/dma-buf/udmabuf.c+183−31

This diff is large: its 281 lines aren’t shown here.

tools/testing/selftests/exec/Makefile+0−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/futex/functional/Makefile+1−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/intel_pstate/Makefile+1−1

This diff is large: its 7 lines aren’t shown here.

tools/testing/selftests/iommu/Makefile+0−2

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/kvm/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/lib.mk+3−0

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/mm/.gitignore+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+3−0

This diff is large: its 24 lines aren’t shown here.

tools/testing/selftests/mm/hugepage-mremap.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb-soft-offline.c+228−0

This diff is large: its 229 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb_dio.c+117−0

This diff is large: its 118 lines aren’t shown here.

tools/testing/selftests/mm/ksm_functional_tests.c+1−7

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/mm/memfd_secret.c+1−13

This diff is large: its 32 lines aren’t shown here.

tools/testing/selftests/mm/mkdirty.c+1−7

This diff is large: its 45 lines aren’t shown here.

tools/testing/selftests/mm/mlock2.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/mseal_helpers.h+41−0

This diff is large: its 42 lines aren’t shown here.

tools/testing/selftests/mm/mseal_test.c+48−95

This diff is large: its 479 lines aren’t shown here.

tools/testing/selftests/mm/pagemap_ioctl.c+4−2

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/mm/protection_keys.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/mm/run_vmtests.sh+7−0

This diff is large: its 21 lines aren’t shown here.

tools/testing/selftests/mm/seal_elf.c+3−34

This diff is large: its 58 lines aren’t shown here.

tools/testing/selftests/mm/split_huge_page_test.c+2−1

This diff is large: its 17 lines aren’t shown here.

tools/testing/selftests/mm/thuge-gen.c+10−5

This diff is large: its 39 lines aren’t shown here.

tools/testing/selftests/mm/uffd-common.c+0−4

This diff is large: its 12 lines aren’t shown here.

tools/testing/selftests/mm/uffd-stress.c+12−19

This diff is large: its 72 lines aren’t shown here.

tools/testing/selftests/mm/uffd-unit-tests.c+1−13

This diff is large: its 29 lines aren’t shown here.

tools/testing/selftests/mm/va_high_addr_switch.c+232−222

This diff is large: its 499 lines aren’t shown here.

tools/testing/selftests/mm/va_high_addr_switch.sh+0−4

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/net/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/net/tcp_ao/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/proc/Makefile+1−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/proc/proc-pid-vm.c+86−0

This diff is large: its 100 lines aren’t shown here.

tools/testing/selftests/resctrl/Makefile+1−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/ring-buffer/Makefile+0−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/riscv/mm/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/sgx/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/tmpfs/Makefile+0−1

This diff is large: its 7 lines aren’t shown here.

Merge tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code

@@ -132,7 +132,7 @@ CASE 1: Direct IO (DIO) · 131 unmodified lines
132	-----------------------	132	-----------------------
133	There are GUP references to pages that are serving	133	There are GUP references to pages that are serving
134	as DIO buffers. These buffers are needed for a relatively short time (so they	134	as DIO buffers. These buffers are needed for a relatively short time (so they
135	are not "long term"). No special synchronization with page_mkclean() or	135	are not "long term"). No special synchronization with folio_mkclean() or
136	munmap() is provided. Therefore, flags to set at the call site are: ::	136	munmap() is provided. Therefore, flags to set at the call site are: ::
137		137
138	FOLL_PIN	138	FOLL_PIN
@@ -144,7 +144,7 @@ CASE 2: RDMA · 5 unmodified lines
144	------------	144	------------
145	There are GUP references to pages that are serving as DMA	145	There are GUP references to pages that are serving as DMA
146	buffers. These buffers are needed for a long time ("long term"). No special	146	buffers. These buffers are needed for a long time ("long term"). No special
147	synchronization with page_mkclean() or munmap() is provided. Therefore, flags	147	synchronization with folio_mkclean() or munmap() is provided. Therefore, flags
148	to set at the call site are: ::	148	to set at the call site are: ::
149		149
150	FOLL_PIN \| FOLL_LONGTERM	150	FOLL_PIN \| FOLL_LONGTERM
@@ -170,7 +170,7 @@ callback, simply remove the range from the device's page tables. · 19 unmodified lines
170		170
171	Either way, as long as the driver unpins the pages upon mmu notifier callback,	171	Either way, as long as the driver unpins the pages upon mmu notifier callback,
172	then there is proper synchronization with both filesystem and mm	172	then there is proper synchronization with both filesystem and mm
173	(page_mkclean(), munmap(), etc). Therefore, neither flag needs to be set.	173	(folio_mkclean(), munmap(), etc). Therefore, neither flag needs to be set.
174		174
175	CASE 4: Pinning for struct page manipulation only	175	CASE 4: Pinning for struct page manipulation only
176	-------------------------------------------------	176	-------------------------------------------------
@@ -196,20 +196,20 @@ INCORRECT (uses FOLL_GET calls): · 19 unmodified lines
196	write to the data within the pages	196	write to the data within the pages
197	put_page()	197	put_page()
198		198
199	page_maybe_dma_pinned(): the whole point of pinning	199	folio_maybe_dma_pinned(): the whole point of pinning
200	===================================================	200	====================================================
201		201
202	The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able	202	The whole point of marking folios as "DMA-pinned" or "gup-pinned" is to be able
203	to query, "is this page DMA-pinned?" That allows code such as page_mkclean()	203	to query, "is this folio DMA-pinned?" That allows code such as folio_mkclean()
204	(and file system writeback code in general) to make informed decisions about	204	(and file system writeback code in general) to make informed decisions about
205	what to do when a page cannot be unmapped due to such pins.	205	what to do when a folio cannot be unmapped due to such pins.
206		206
207	What to do in those cases is the subject of a years-long series of discussions	207	What to do in those cases is the subject of a years-long series of discussions
208	and debates (see the References at the end of this document). It's a TODO item	208	and debates (see the References at the end of this document). It's a TODO item
209	here: fill in the details once that's worked out. Meanwhile, it's safe to say	209	here: fill in the details once that's worked out. Meanwhile, it's safe to say
210	that having this available: ::	210	that having this available: ::
211		211
212	static inline bool page_maybe_dma_pinned(struct page *page)	212	static inline bool folio_maybe_dma_pinned(struct folio *folio)
213		213
214	...is a prerequisite to solving the long-running gup+DMA problem.	214	...is a prerequisite to solving the long-running gup+DMA problem.
215		215

@@ -191,13 +191,13 @@ have become evictable again (via munlock() for example) and have been "rescued" · 190 unmodified lines
191	from the unevictable list. However, there may be situations where we decide,	191	from the unevictable list. However, there may be situations where we decide,
192	for the sake of expediency, to leave an unevictable folio on one of the regular	192	for the sake of expediency, to leave an unevictable folio on one of the regular
193	active/inactive LRU lists for vmscan to deal with. vmscan checks for such	193	active/inactive LRU lists for vmscan to deal with. vmscan checks for such
194	folios in all of the shrink_{active\|inactive\|page}_list() functions and will	194	folios in all of the shrink_{active\|inactive\|folio}_list() functions and will
195	"cull" such folios that it encounters: that is, it diverts those folios to the	195	"cull" such folios that it encounters: that is, it diverts those folios to the
196	unevictable list for the memory cgroup and node being scanned.	196	unevictable list for the memory cgroup and node being scanned.
197		197
198	There may be situations where a folio is mapped into a VM_LOCKED VMA,	198	There may be situations where a folio is mapped into a VM_LOCKED VMA,
199	but the folio does not have the mlocked flag set. Such folios will make	199	but the folio does not have the mlocked flag set. Such folios will make
200	it all the way to shrink_active_list() or shrink_page_list() where they	200	it all the way to shrink_active_list() or shrink_folio_list() where they
201	will be detected when vmscan walks the reverse map in folio_referenced()	201	will be detected when vmscan walks the reverse map in folio_referenced()
202	or try_to_unmap(). The folio is culled to the unevictable list when it	202	or try_to_unmap(). The folio is culled to the unevictable list when it
203	is released by the shrinker.	203	is released by the shrinker.
@@ -269,7 +269,7 @@ the LRU. Such pages can be "noticed" by memory management in several places: · 65 unmodified lines
269		269
270	(4) in the fault path and when a VM_LOCKED stack segment is expanded; or	270	(4) in the fault path and when a VM_LOCKED stack segment is expanded; or
271		271
272	(5) as mentioned above, in vmscan:shrink_page_list() when attempting to	272	(5) as mentioned above, in vmscan:shrink_folio_list() when attempting to
273	reclaim a page in a VM_LOCKED VMA by folio_referenced() or try_to_unmap().	273	reclaim a page in a VM_LOCKED VMA by folio_referenced() or try_to_unmap().
274		274
275	mlocked pages become unlocked and rescued from the unevictable list when:	275	mlocked pages become unlocked and rescued from the unevictable list when:
@@ -548,12 +548,12 @@ Some examples of these unevictable pages on the LRU lists are: · 272 unmodified lines
548	(3) pages still mapped into VM_LOCKED VMAs, which should be marked mlocked,	548	(3) pages still mapped into VM_LOCKED VMAs, which should be marked mlocked,
549	but events left mlock_count too low, so they were munlocked too early.	549	but events left mlock_count too low, so they were munlocked too early.
550		550
551	vmscan's shrink_inactive_list() and shrink_page_list() also divert obviously	551	vmscan's shrink_inactive_list() and shrink_folio_list() also divert obviously
552	unevictable pages found on the inactive lists to the appropriate memory cgroup	552	unevictable pages found on the inactive lists to the appropriate memory cgroup
553	and node unevictable list.	553	and node unevictable list.
554		554
555	rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or	555	rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or
556	shrink_page_list(), and rmap's try_to_unmap_one() called via shrink_page_list(),	556	shrink_folio_list(), and rmap's try_to_unmap_one() called via shrink_folio_list(),
557	check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_folio()	557	check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_folio()
558	to correct them. Such pages are culled to the unevictable list when released	558	to correct them. Such pages are culled to the unevictable list when released
559	by the shrinker.	559	by the shrinker.