lael-labs/linux

Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers+25−0

@@ -0,0 +1,25 @@
	1	What: /sys/devices/virtual/memory_tiering/
	2	Date: August 2022
	3	Contact: Linux memory management mailing list <linux-mm@kvack.org>
	4	Description: A collection of all the memory tiers allocated.
	5
	6	Individual memory tier details are contained in subdirectories
	7	named by the abstract distance of the memory tier.
	8
	9	/sys/devices/virtual/memory_tiering/memory_tierN/
	10
	11
	12	What: /sys/devices/virtual/memory_tiering/memory_tierN/
	13	/sys/devices/virtual/memory_tiering/memory_tierN/nodes
	14	Date: August 2022
	15	Contact: Linux memory management mailing list <linux-mm@kvack.org>
	16	Description: Directory with details of a specific memory tier
	17
	18	This is the directory containing information about a particular
	19	memory tier, memtierN, where N is derived based on abstract distance.
	20
	21	A smaller value of N implies a higher (faster) memory tier in the
	22	hierarchy.
	23
	24	nodes: NUMA nodes that are part of this memory tier.
	25

Documentation/accounting/delay-accounting.rst+1−1

@@ -13,7 +13,7 @@ a) waiting for a CPU (while being runnable) · 12 unmodified lines
13	b) completion of synchronous block I/O initiated by the task	13	b) completion of synchronous block I/O initiated by the task
14	c) swapping in pages	14	c) swapping in pages
15	d) memory reclaim	15	d) memory reclaim
16	e) thrashing page cache	16	e) thrashing
17	f) direct compact	17	f) direct compact
18	g) write-protect copy	18	g) write-protect copy
19		19

Documentation/admin-guide/cgroup-v1/memory.rst+1−3

 lruvec->lru_lock; PG_lru bit of page->flags is cleared before
 isolating a page from its LRU under lruvec->lru_lock.
-2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)
+2.7 Kernel Memory Extension
 -----------------------------------------------
 With the Kernel memory extension, the Memory Controller is able to limit
 a. Enable CONFIG_CGROUPS
 b. Enable CONFIG_MEMCG
-c. Enable CONFIG_MEMCG_SWAP (to use swap extension)
-d. Enable CONFIG_MEMCG_KMEM (to use kmem extension)
 3.1. Prepare the cgroups (see cgroups.txt, Why are cgroups needed?)
 -------------------------------------------------------------------

Documentation/admin-guide/kernel-parameters.txt+8−6

 			Permit 'security.evm' to be updated regardless of
 			current integrity status.
+	early_page_ext [KNL] Enforces page_ext initialization to earlier
+			stages so cover more early boot allocations.
+			Please note that as side effect some optimizations
+			might be disabled to achieve that (e.g. parallelized
+			memory initialization is disabled) so the boot process
+			might take longer, especially on systems with a lot of
+			memory. Available with CONFIG_PAGE_EXTENSION=y.
 	failslab=
 	fail_usercopy=
 	fail_page_alloc=
 			This parameter controls use of the Protected
 			Execution Facility on pSeries.
-	swapaccount=	[KNL]
-			Format: [0|1]
-			Enable accounting of swap in memory resource
-			controller if no parameter or 1 is given or disable
-			it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst)
 	swiotlb=	[ARM,IA-64,PPC,MIPS,X86]
 			Format: { <int> [,<int>] | force | noforce }
 			<int> -- Number of I/O TLB slabs

Documentation/admin-guide/mm/cma_debugfs.rst+5−5

@@ -5,10 +5,10 @@ CMA Debugfs Interface · 4 unmodified lines
5	The CMA debugfs interface is useful to retrieve basic information out of the	5	The CMA debugfs interface is useful to retrieve basic information out of the
6	different CMA areas and to test allocation/release in each of the areas.	6	different CMA areas and to test allocation/release in each of the areas.
7		7
8	Each CMA zone represents a directory under <debugfs>/cma/, indexed by the	8	Each CMA area represents a directory under <debugfs>/cma/, represented by
9	kernel's CMA index. So the first CMA zone would be:	9	its CMA name like below:
10		10
11	<debugfs>/cma/cma-0	11	<debugfs>/cma/<cma_name>
12		12
13	The structure of the files created under that directory is as follows:	13	The structure of the files created under that directory is as follows:
14		14
@@ -18,8 +18,8 @@ The structure of the files created under that directory is as follows: · 3 unmodified lines
18	- [RO] bitmap: The bitmap of page states in the zone.	18	- [RO] bitmap: The bitmap of page states in the zone.
19	- [WO] alloc: Allocate N pages from that CMA area. For example::	19	- [WO] alloc: Allocate N pages from that CMA area. For example::
20		20
21	echo 5 > <debugfs>/cma/cma-2/alloc	21	echo 5 > <debugfs>/cma/<cma_name>/alloc
22		22
23	would try to allocate 5 pages from the cma-2 area.	23	would try to allocate 5 pages from the 'cma_name' area.
24		24
25	- [WO] free: Free N pages from that CMA area, similar to the above.	25	- [WO] free: Free N pages from that CMA area, similar to the above.

Documentation/admin-guide/mm/damon/index.rst+3−3

@@ -1,8 +1,8 @@
1	.. SPDX-License-Identifier: GPL-2.0	1	.. SPDX-License-Identifier: GPL-2.0
2		2
3	========================	3	==========================
4	Monitoring Data Accesses	4	DAMON: Data Access MONitor
5	========================	5	==========================
6		6
7	:doc:`DAMON </mm/damon/index>` allows light-weight data access monitoring.	7	:doc:`DAMON </mm/damon/index>` allows light-weight data access monitoring.
8	Using DAMON, users can analyze the memory access patterns of their systems and	8	Using DAMON, users can analyze the memory access patterns of their systems and

Documentation/admin-guide/mm/damon/start.rst+3−10

 https://github.com/awslabs/damo.  The examples below assume that ``damo`` is on
 your ``$PATH``.  It's not mandatory, though.
-Because DAMO is using the debugfs interface (refer to :doc:`usage` for the
+Because DAMO is using the sysfs interface (refer to :doc:`usage` for the
-detail) of DAMON, you should ensure debugfs is mounted.  Mount it manually as
+detail) of DAMON, you should ensure :doc:`sysfs </filesystems/sysfs>` is
-below::
+mounted.
-    # mount -t debugfs none /sys/kernel/debug/
-or append the following line to your ``/etc/fstab`` file so that your system
-can automatically mount debugfs upon booting::
-    debugfs /sys/kernel/debug debugfs defaults 0 0
 Recording Data Access Patterns

Documentation/admin-guide/mm/damon/usage.rst+5−0

 debugfs Interface
 =================
+.. note::
+  DAMON debugfs interface will be removed after next LTS kernel is released, so
+  users should move to the :ref:`sysfs interface <sysfs_interface>`.
 DAMON exports eight files, ``attrs``, ``target_ids``, ``init_regions``,
 ``schemes``, ``monitor_on``, ``kdamond_pid``, ``mk_contexts`` and
 ``rm_contexts`` under its debugfs directory, ``<debugfs>/damon/``.

Documentation/admin-guide/mm/index.rst+1−0

@@ -32,6 +32,7 @@ the Linux memory management. · 31 unmodified lines
32	idle_page_tracking	32	idle_page_tracking
33	ksm	33	ksm
34	memory-hotplug	34	memory-hotplug
		35	multigen_lru
35	nommu-mmap	36	nommu-mmap
36	numa_memory_policy	37	numa_memory_policy
37	numaperf	38	numaperf

Documentation/admin-guide/mm/ksm.rst+36−0

@@ -184,6 +184,42 @@ The maximum possible ``pages_sharing/pages_shared`` ratio is limited by the · 183 unmodified lines
184	``max_page_sharing`` tunable. To increase the ratio ``max_page_sharing`` must	184	``max_page_sharing`` tunable. To increase the ratio ``max_page_sharing`` must
185	be increased accordingly.	185	be increased accordingly.
186		186
		187	Monitoring KSM profit
		188	=====================
		189
		190	KSM can save memory by merging identical pages, but also can consume
		191	additional memory, because it needs to generate a number of rmap_items to
		192	save each scanned page's brief rmap information. Some of these pages may
		193	be merged, but some may not be abled to be merged after being checked
		194	several times, which are unprofitable memory consumed.
		195
		196	1) How to determine whether KSM save memory or consume memory in system-wide
		197	range? Here is a simple approximate calculation for reference::
		198
		199	general_profit =~ pages_sharing * sizeof(page) - (all_rmap_items) *
		200	sizeof(rmap_item);
		201
		202	where all_rmap_items can be easily obtained by summing ``pages_sharing``,
		203	``pages_shared``, ``pages_unshared`` and ``pages_volatile``.
		204
		205	2) The KSM profit inner a single process can be similarly obtained by the
		206	following approximate calculation::
		207
		208	process_profit =~ ksm_merging_pages * sizeof(page) -
		209	ksm_rmap_items * sizeof(rmap_item).
		210
		211	where ksm_merging_pages is shown under the directory ``/proc/<pid>/``,
		212	and ksm_rmap_items is shown in ``/proc/<pid>/ksm_stat``.
		213
		214	From the perspective of application, a high ratio of ``ksm_rmap_items`` to
		215	``ksm_merging_pages`` means a bad madvise-applied policy, so developers or
		216	administrators have to rethink how to change madvise policy. Giving an example
		217	for reference, a page's size is usually 4K, and the rmap_item's size is
		218	separately 32B on 32-bit CPU architecture and 64B on 64-bit CPU architecture.
		219	so if the ``ksm_rmap_items/ksm_merging_pages`` ratio exceeds 64 on 64-bit CPU
		220	or exceeds 128 on 32-bit CPU, then the app's madvise policy should be dropped,
		221	because the ksm profit is approximately zero or negative.
		222
187	Monitoring KSM events	223	Monitoring KSM events
188	=====================	224	=====================
189		225

Documentation/admin-guide/mm/multigen_lru.rst+162−0

@@ -0,0 +1,162 @@
	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	=============
	4	Multi-Gen LRU
	5	=============
	6	The multi-gen LRU is an alternative LRU implementation that optimizes
	7	page reclaim and improves performance under memory pressure. Page
	8	reclaim decides the kernel's caching policy and ability to overcommit
	9	memory. It directly impacts the kswapd CPU usage and RAM efficiency.
	10
	11	Quick start
	12	===========
	13	Build the kernel with the following configurations.
	14
	15	* ``CONFIG_LRU_GEN=y``
	16	* ``CONFIG_LRU_GEN_ENABLED=y``
	17
	18	All set!
	19
	20	Runtime options
	21	===============
	22	``/sys/kernel/mm/lru_gen/`` contains stable ABIs described in the
	23	following subsections.
	24
	25	Kill switch
	26	-----------
	27	``enabled`` accepts different values to enable or disable the
	28	following components. Its default value depends on
	29	``CONFIG_LRU_GEN_ENABLED``. All the components should be enabled
	30	unless some of them have unforeseen side effects. Writing to
	31	``enabled`` has no effect when a component is not supported by the
	32	hardware, and valid values will be accepted even when the main switch
	33	is off.
	34
	35	====== ===============================================================
	36	Values Components
	37	====== ===============================================================
	38	0x0001 The main switch for the multi-gen LRU.
	39	0x0002 Clearing the accessed bit in leaf page table entries in large
	40	batches, when MMU sets it (e.g., on x86). This behavior can
	41	theoretically worsen lock contention (mmap_lock). If it is
	42	disabled, the multi-gen LRU will suffer a minor performance
	43	degradation for workloads that contiguously map hot pages,
	44	whose accessed bits can be otherwise cleared by fewer larger
	45	batches.
	46	0x0004 Clearing the accessed bit in non-leaf page table entries as
	47	well, when MMU sets it (e.g., on x86). This behavior was not
	48	verified on x86 varieties other than Intel and AMD. If it is
	49	disabled, the multi-gen LRU will suffer a negligible
	50	performance degradation.
	51	[yYnN] Apply to all the components above.
	52	====== ===============================================================
	53
	54	E.g.,
	55	::
	56
	57	echo y >/sys/kernel/mm/lru_gen/enabled
	58	cat /sys/kernel/mm/lru_gen/enabled
	59	0x0007
	60	echo 5 >/sys/kernel/mm/lru_gen/enabled
	61	cat /sys/kernel/mm/lru_gen/enabled
	62	0x0005
	63
	64	Thrashing prevention
	65	--------------------
	66	Personal computers are more sensitive to thrashing because it can
	67	cause janks (lags when rendering UI) and negatively impact user
	68	experience. The multi-gen LRU offers thrashing prevention to the
	69	majority of laptop and desktop users who do not have ``oomd``.
	70
	71	Users can write ``N`` to ``min_ttl_ms`` to prevent the working set of
	72	``N`` milliseconds from getting evicted. The OOM killer is triggered
	73	if this working set cannot be kept in memory. In other words, this
	74	option works as an adjustable pressure relief valve, and when open, it
	75	terminates applications that are hopefully not being used.
	76
	77	Based on the average human detectable lag (~100ms), ``N=1000`` usually
	78	eliminates intolerable janks due to thrashing. Larger values like
	79	``N=3000`` make janks less noticeable at the risk of premature OOM
	80	kills.
	81
	82	The default value ``0`` means disabled.
	83
	84	Experimental features
	85	=====================
	86	``/sys/kernel/debug/lru_gen`` accepts commands described in the
	87	following subsections. Multiple command lines are supported, so does
	88	concatenation with delimiters ``,`` and ``;``.
	89
	90	``/sys/kernel/debug/lru_gen_full`` provides additional stats for
	91	debugging. ``CONFIG_LRU_GEN_STATS=y`` keeps historical stats from
	92	evicted generations in this file.
	93
	94	Working set estimation
	95	----------------------
	96	Working set estimation measures how much memory an application needs
	97	in a given time interval, and it is usually done with little impact on
	98	the performance of the application. E.g., data centers want to
	99	optimize job scheduling (bin packing) to improve memory utilizations.
	100	When a new job comes in, the job scheduler needs to find out whether
	101	each server it manages can allocate a certain amount of memory for
	102	this new job before it can pick a candidate. To do so, the job
	103	scheduler needs to estimate the working sets of the existing jobs.
	104
	105	When it is read, ``lru_gen`` returns a histogram of numbers of pages
	106	accessed over different time intervals for each memcg and node.
	107	``MAX_NR_GENS`` decides the number of bins for each histogram. The
	108	histograms are noncumulative.
	109	::
	110
	111	memcg memcg_id memcg_path
	112	node node_id
	113	min_gen_nr age_in_ms nr_anon_pages nr_file_pages
	114	...
	115	max_gen_nr age_in_ms nr_anon_pages nr_file_pages
	116
	117	Each bin contains an estimated number of pages that have been accessed
	118	within ``age_in_ms``. E.g., ``min_gen_nr`` contains the coldest pages
	119	and ``max_gen_nr`` contains the hottest pages, since ``age_in_ms`` of
	120	the former is the largest and that of the latter is the smallest.
	121
	122	Users can write the following command to ``lru_gen`` to create a new
	123	generation ``max_gen_nr+1``:
	124
	125	``+ memcg_id node_id max_gen_nr [can_swap [force_scan]]``
	126
	127	``can_swap`` defaults to the swap setting and, if it is set to ``1``,
	128	it forces the scan of anon pages when swap is off, and vice versa.
	129	``force_scan`` defaults to ``1`` and, if it is set to ``0``, it
	130	employs heuristics to reduce the overhead, which is likely to reduce
	131	the coverage as well.
	132
	133	A typical use case is that a job scheduler runs this command at a
	134	certain time interval to create new generations, and it ranks the
	135	servers it manages based on the sizes of their cold pages defined by
	136	this time interval.
	137
	138	Proactive reclaim
	139	-----------------
	140	Proactive reclaim induces page reclaim when there is no memory
	141	pressure. It usually targets cold pages only. E.g., when a new job
	142	comes in, the job scheduler wants to proactively reclaim cold pages on
	143	the server it selected, to improve the chance of successfully landing
	144	this new job.
	145
	146	Users can write the following command to ``lru_gen`` to evict
	147	generations less than or equal to ``min_gen_nr``.
	148
	149	``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
	150
	151	``min_gen_nr`` should be less than ``max_gen_nr-1``, since
	152	``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
	153	the active list) and therefore cannot be evicted. ``swappiness``
	154	overrides the default value in ``/proc/sys/vm/swappiness``.
	155	``nr_to_reclaim`` limits the number of pages to evict.
	156
	157	A typical use case is that a job scheduler runs this command before it
	158	tries to land a new job on a server. If it fails to materialize enough
	159	cold pages because of the overestimation, it retries on the next
	160	server according to the ranking result obtained from the working set
	161	estimation step. This less forceful approach limits the impacts on the
	162	existing jobs.

Documentation/admin-guide/mm/transhuge.rst+11−5

 	/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs
-The khugepaged progress can be seen in the number of pages collapsed::
+The khugepaged progress can be seen in the number of pages collapsed (note
+that this counter may not be an exact count of the number of pages
+collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping
+being replaced by a PMD mapping, or (2) All 4K physical pages replaced by
+one 2M hugepage. Each may happen independently, or together, depending on
+the type of memory and the failures that occur. As such, this value should
+be interpreted roughly as a sign of progress, and counters in /proc/vmstat
+consulted for more accurate accounting)::
 	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
 	page table entry.
 thp_zero_page_alloc
-	is incremented every time a huge zero page is
+	is incremented every time a huge zero page used for thp is
-	successfully allocated. It includes allocations which where
+	successfully allocated. Note, it doesn't count every map of
-	dropped due race with other allocation. Note, it doesn't count
+	the huge zero page, only its allocation.
-	every map of the huge zero page, only its allocation.
 thp_zero_page_alloc_failed
 	is incremented if kernel fails to allocate

Documentation/admin-guide/mm/userfaultfd.rst+38−3

@@ -17,7 +17,10 @@ of the ``PROT_NONE+SIGSEGV`` trick. · 16 unmodified lines
17	Design	17	Design
18	======	18	======
19		19
20	Userfaults are delivered and resolved through the ``userfaultfd`` syscall.	20	Userspace creates a new userfaultfd, initializes it, and registers one or more
		21	regions of virtual memory with it. Then, any page faults which occur within the
		22	region(s) result in a message being delivered to the userfaultfd, notifying
		23	userspace of the fault.
21		24
22	The ``userfaultfd`` (aside from registering and unregistering virtual	25	The ``userfaultfd`` (aside from registering and unregistering virtual
23	memory ranges) provides two primary functionalities:	26	memory ranges) provides two primary functionalities:
@@ -34,12 +37,11 @@ The real advantage of userfaults if compared to regular virtual memory · 10 unmodified lines
34	management of mremap/mprotect is that the userfaults in all their	37	management of mremap/mprotect is that the userfaults in all their
35	operations never involve heavyweight structures like vmas (in fact the	38	operations never involve heavyweight structures like vmas (in fact the
36	``userfaultfd`` runtime load never takes the mmap_lock for writing).	39	``userfaultfd`` runtime load never takes the mmap_lock for writing).
37
38	Vmas are not suitable for page- (or hugepage) granular fault tracking	40	Vmas are not suitable for page- (or hugepage) granular fault tracking
39	when dealing with virtual address spaces that could span	41	when dealing with virtual address spaces that could span
40	Terabytes. Too many vmas would be needed for that.	42	Terabytes. Too many vmas would be needed for that.
41		43
42	The ``userfaultfd`` once opened by invoking the syscall, can also be	44	The ``userfaultfd``, once created, can also be
43	passed using unix domain sockets to a manager process, so the same	45	passed using unix domain sockets to a manager process, so the same
44	manager process could handle the userfaults of a multitude of	46	manager process could handle the userfaults of a multitude of
45	different processes without them being aware about what is going on	47	different processes without them being aware about what is going on
@@ -50,6 +52,39 @@ is a corner case that would currently return ``-EBUSY``). · 4 unmodified lines
50	API	52	API
51	===	53	===
52		54
		55	Creating a userfaultfd
		56	----------------------
		57
		58	There are two ways to create a new userfaultfd, each of which provide ways to
		59	restrict access to this functionality (since historically userfaultfds which
		60	handle kernel page faults have been a useful tool for exploiting the kernel).
		61
		62	The first way, supported since userfaultfd was introduced, is the
		63	userfaultfd(2) syscall. Access to this is controlled in several ways:
		64
		65	- Any user can always create a userfaultfd which traps userspace page faults
		66	only. Such a userfaultfd can be created using the userfaultfd(2) syscall
		67	with the flag UFFD_USER_MODE_ONLY.
		68
		69	- In order to also trap kernel page faults for the address space, either the
		70	process needs the CAP_SYS_PTRACE capability, or the system must have
		71	vm.unprivileged_userfaultfd set to 1. By default, vm.unprivileged_userfaultfd
		72	is set to 0.
		73
		74	The second way, added to the kernel more recently, is by opening
		75	/dev/userfaultfd and issuing a USERFAULTFD_IOC_NEW ioctl to it. This method
		76	yields equivalent userfaultfds to the userfaultfd(2) syscall.
		77
		78	Unlike userfaultfd(2), access to /dev/userfaultfd is controlled via normal
		79	filesystem permissions (user/group/mode), which gives fine grained access to
		80	userfaultfd specifically, without also granting other unrelated privileges at
		81	the same time (as e.g. granting CAP_SYS_PTRACE would do). Users who have access
		82	to /dev/userfaultfd can always create userfaultfds that trap kernel page faults;
		83	vm.unprivileged_userfaultfd is not considered.
		84
		85	Initializing a userfaultfd
		86	--------------------------
		87
53	When first opened the ``userfaultfd`` must be enabled invoking the	88	When first opened the ``userfaultfd`` must be enabled invoking the
54	``UFFDIO_API`` ioctl specifying a ``uffdio_api.api`` value set to ``UFFD_API`` (or	89	``UFFDIO_API`` ioctl specifying a ``uffdio_api.api`` value set to ``UFFD_API`` (or
55	a later API version) which will specify the ``read/POLLIN`` protocol	90	a later API version) which will specify the ``read/POLLIN`` protocol

Documentation/admin-guide/sysctl/kernel.rst+11−0

@@ -635,6 +635,17 @@ different types of memory (represented as different NUMA nodes) to · 634 unmodified lines
635	place the hot pages in the fast memory. This is implemented based on	635	place the hot pages in the fast memory. This is implemented based on
636	unmapping and page fault too.	636	unmapping and page fault too.
637		637
		638	numa_balancing_promote_rate_limit_MBps
		639	======================================
		640
		641	Too high promotion/demotion throughput between different memory types
		642	may hurt application latency. This can be used to rate limit the
		643	promotion throughput. The per-node max promotion throughput in MB/s
		644	will be limited to be no more than the set value.
		645
		646	A rule of thumb is to set this to less than 1/10 of the PMEM node
		647	write bandwidth.
		648
638	oops_all_cpu_backtrace	649	oops_all_cpu_backtrace
639	======================	650	======================
640		651

Documentation/admin-guide/sysctl/vm.rst+3−0

@@ -926,6 +926,9 @@ calls without any restrictions. · 925 unmodified lines
926		926
927	The default value is 0.	927	The default value is 0.
928		928
		929	Another way to control permissions for userfaultfd is to use
		930	/dev/userfaultfd instead of userfaultfd(2). See
		931	Documentation/admin-guide/mm/userfaultfd.rst.
929		932
930	user_reserve_kbytes	933	user_reserve_kbytes
931	===================	934	===================

Documentation/core-api/index.rst+1−0

@@ -37,6 +37,7 @@ Library functionality that is used throughout the kernel. · 36 unmodified lines
37	kref	37	kref
38	assoc_array	38	assoc_array
39	xarray	39	xarray
		40	maple_tree
40	idr	41	idr
41	circular-buffers	42	circular-buffers
42	rbtree	43	rbtree

Documentation/core-api/maple_tree.rst+217−0

@@ -0,0 +1,217 @@
	1	.. SPDX-License-Identifier: GPL-2.0+
	2
	3
	4	==========
	5	Maple Tree
	6	==========
	7
	8	:Author: Liam R. Howlett
	9
	10	Overview
	11	========
	12
	13	The Maple Tree is a B-Tree data type which is optimized for storing
	14	non-overlapping ranges, including ranges of size 1. The tree was designed to
	15	be simple to use and does not require a user written search method. It
	16	supports iterating over a range of entries and going to the previous or next
	17	entry in a cache-efficient manner. The tree can also be put into an RCU-safe
	18	mode of operation which allows reading and writing concurrently. Writers must
	19	synchronize on a lock, which can be the default spinlock, or the user can set
	20	the lock to an external lock of a different type.
	21
	22	The Maple Tree maintains a small memory footprint and was designed to use
	23	modern processor cache efficiently. The majority of the users will be able to
	24	use the normal API. An :ref:`maple-tree-advanced-api` exists for more complex
	25	scenarios. The most important usage of the Maple Tree is the tracking of the
	26	virtual memory areas.
	27
	28	The Maple Tree can store values between ``0`` and ``ULONG_MAX``. The Maple
	29	Tree reserves values with the bottom two bits set to '10' which are below 4096
	30	(ie 2, 6, 10 .. 4094) for internal use. If the entries may use reserved
	31	entries then the users can convert the entries using xa_mk_value() and convert
	32	them back by calling xa_to_value(). If the user needs to use a reserved
	33	value, then the user can convert the value when using the
	34	:ref:`maple-tree-advanced-api`, but are blocked by the normal API.
	35
	36	The Maple Tree can also be configured to support searching for a gap of a given
	37	size (or larger).
	38
	39	Pre-allocating of nodes is also supported using the
	40	:ref:`maple-tree-advanced-api`. This is useful for users who must guarantee a
	41	successful store operation within a given
	42	code segment when allocating cannot be done. Allocations of nodes are
	43	relatively small at around 256 bytes.
	44
	45	.. _maple-tree-normal-api:
	46
	47	Normal API
	48	==========
	49
	50	Start by initialising a maple tree, either with DEFINE_MTREE() for statically
	51	allocated maple trees or mt_init() for dynamically allocated ones. A
	52	freshly-initialised maple tree contains a ``NULL`` pointer for the range ``0``
	53	- ``ULONG_MAX``. There are currently two types of maple trees supported: the
	54	allocation tree and the regular tree. The regular tree has a higher branching
	55	factor for internal nodes. The allocation tree has a lower branching factor
	56	but allows the user to search for a gap of a given size or larger from either
	57	``0`` upwards or ``ULONG_MAX`` down. An allocation tree can be used by
	58	passing in the ``MT_FLAGS_ALLOC_RANGE`` flag when initialising the tree.
	59
	60	You can then set entries using mtree_store() or mtree_store_range().
	61	mtree_store() will overwrite any entry with the new entry and return 0 on
	62	success or an error code otherwise. mtree_store_range() works in the same way
	63	but takes a range. mtree_load() is used to retrieve the entry stored at a
	64	given index. You can use mtree_erase() to erase an entire range by only
	65	knowing one value within that range, or mtree_store() call with an entry of
	66	NULL may be used to partially erase a range or many ranges at once.
	67
	68	If you want to only store a new entry to a range (or index) if that range is
	69	currently ``NULL``, you can use mtree_insert_range() or mtree_insert() which
	70	return -EEXIST if the range is not empty.
	71
	72	You can search for an entry from an index upwards by using mt_find().
	73
	74	You can walk each entry within a range by calling mt_for_each(). You must
	75	provide a temporary variable to store a cursor. If you want to walk each
	76	element of the tree then ``0`` and ``ULONG_MAX`` may be used as the range. If
	77	the caller is going to hold the lock for the duration of the walk then it is
	78	worth looking at the mas_for_each() API in the :ref:`maple-tree-advanced-api`
	79	section.
	80
	81	Sometimes it is necessary to ensure the next call to store to a maple tree does
	82	not allocate memory, please see :ref:`maple-tree-advanced-api` for this use case.
	83
	84	Finally, you can remove all entries from a maple tree by calling
	85	mtree_destroy(). If the maple tree entries are pointers, you may wish to free
	86	the entries first.
	87
	88	Allocating Nodes
	89	----------------
	90
	91	The allocations are handled by the internal tree code. See
	92	:ref:`maple-tree-advanced-alloc` for other options.
	93
	94	Locking
	95	-------
	96
	97	You do not have to worry about locking. See :ref:`maple-tree-advanced-locks`
	98	for other options.
	99
	100	The Maple Tree uses RCU and an internal spinlock to synchronise access:
	101
	102	Takes RCU read lock:
	103	* mtree_load()
	104	* mt_find()
	105	* mt_for_each()
	106	* mt_next()
	107	* mt_prev()
	108
	109	Takes ma_lock internally:
	110	* mtree_store()
	111	* mtree_store_range()
	112	* mtree_insert()
	113	* mtree_insert_range()
	114	* mtree_erase()
	115	* mtree_destroy()
	116	* mt_set_in_rcu()
	117	* mt_clear_in_rcu()
	118
	119	If you want to take advantage of the internal lock to protect the data
	120	structures that you are storing in the Maple Tree, you can call mtree_lock()
	121	before calling mtree_load(), then take a reference count on the object you
	122	have found before calling mtree_unlock(). This will prevent stores from
	123	removing the object from the tree between looking up the object and
	124	incrementing the refcount. You can also use RCU to avoid dereferencing
	125	freed memory, but an explanation of that is beyond the scope of this
	126	document.
	127
	128	.. _maple-tree-advanced-api:
	129
	130	Advanced API
	131	============
	132
	133	The advanced API offers more flexibility and better performance at the
	134	cost of an interface which can be harder to use and has fewer safeguards.
	135	You must take care of your own locking while using the advanced API.
	136	You can use the ma_lock, RCU or an external lock for protection.
	137	You can mix advanced and normal operations on the same array, as long
	138	as the locking is compatible. The :ref:`maple-tree-normal-api` is implemented
	139	in terms of the advanced API.
	140
	141	The advanced API is based around the ma_state, this is where the 'mas'
	142	prefix originates. The ma_state struct keeps track of tree operations to make
	143	life easier for both internal and external tree users.
	144
	145	Initialising the maple tree is the same as in the :ref:`maple-tree-normal-api`.
	146	Please see above.
	147
	148	The maple state keeps track of the range start and end in mas->index and
	149	mas->last, respectively.
	150
	151	mas_walk() will walk the tree to the location of mas->index and set the
	152	mas->index and mas->last according to the range for the entry.
	153
	154	You can set entries using mas_store(). mas_store() will overwrite any entry
	155	with the new entry and return the first existing entry that is overwritten.
	156	The range is passed in as members of the maple state: index and last.
	157
	158	You can use mas_erase() to erase an entire range by setting index and
	159	last of the maple state to the desired range to erase. This will erase
	160	the first range that is found in that range, set the maple state index
	161	and last as the range that was erased and return the entry that existed
	162	at that location.
	163
	164	You can walk each entry within a range by using mas_for_each(). If you want
	165	to walk each element of the tree then ``0`` and ``ULONG_MAX`` may be used as
	166	the range. If the lock needs to be periodically dropped, see the locking
	167	section mas_pause().
	168
	169	Using a maple state allows mas_next() and mas_prev() to function as if the
	170	tree was a linked list. With such a high branching factor the amortized
	171	performance penalty is outweighed by cache optimization. mas_next() will
	172	return the next entry which occurs after the entry at index. mas_prev()
	173	will return the previous entry which occurs before the entry at index.
	174
	175	mas_find() will find the first entry which exists at or above index on
	176	the first call, and the next entry from every subsequent calls.
	177
	178	mas_find_rev() will find the fist entry which exists at or below the last on
	179	the first call, and the previous entry from every subsequent calls.
	180
	181	If the user needs to yield the lock during an operation, then the maple state
	182	must be paused using mas_pause().
	183
	184	There are a few extra interfaces provided when using an allocation tree.
	185	If you wish to search for a gap within a range, then mas_empty_area()
	186	or mas_empty_area_rev() can be used. mas_empty_area() searches for a gap
	187	starting at the lowest index given up to the maximum of the range.
	188	mas_empty_area_rev() searches for a gap starting at the highest index given
	189	and continues downward to the lower bound of the range.
	190
	191	.. _maple-tree-advanced-alloc:
	192
	193	Advanced Allocating Nodes
	194	-------------------------
	195
	196	Allocations are usually handled internally to the tree, however if allocations
	197	need to occur before a write occurs then calling mas_expected_entries() will
	198	allocate the worst-case number of needed nodes to insert the provided number of
	199	ranges. This also causes the tree to enter mass insertion mode. Once
	200	insertions are complete calling mas_destroy() on the maple state will free the
	201	unused allocations.
	202
	203	.. _maple-tree-advanced-locks:
	204
	205	Advanced Locking
	206	----------------
	207
	208	The maple tree uses a spinlock by default, but external locks can be used for
	209	tree updates as well. To use an external lock, the tree must be initialized
	210	with the ``MT_FLAGS_LOCK_EXTERN flag``, this is usually done with the
	211	MTREE_INIT_EXT() #define, which takes an external lock as an argument.
	212
	213	Functions and structures
	214	========================
	215
	216	.. kernel-doc:: include/linux/maple_tree.h
	217	.. kernel-doc:: lib/maple_tree.c

Documentation/core-api/mm-api.rst+0−3

@@ -19,9 +19,6 @@ User Space Memory Access · 18 unmodified lines
19	Memory Allocation Controls	19	Memory Allocation Controls
20	==========================	20	==========================
21		21
22	.. kernel-doc:: include/linux/gfp.h
23	:internal:
24
25	.. kernel-doc:: include/linux/gfp_types.h	22	.. kernel-doc:: include/linux/gfp_types.h
26	:doc: Page mobility and placement hints	23	:doc: Page mobility and placement hints
27		24

Documentation/dev-tools/index.rst+1−0

@@ -24,6 +24,7 @@ Documentation/dev-tools/testing-overview.rst · 23 unmodified lines
24	kcov	24	kcov
25	gcov	25	gcov
26	kasan	26	kasan
		27	kmsan
27	ubsan	28	ubsan
28	kmemleak	29	kmemleak
29	kcsan	30	kcsan

Documentation/dev-tools/kasan.rst+11−6

@@ -111,9 +111,17 @@ parameter can be used to control panic and reporting behaviour: · 110 unmodified lines
111	report or also panic the kernel (default: ``report``). The panic happens even	111	report or also panic the kernel (default: ``report``). The panic happens even
112	if ``kasan_multi_shot`` is enabled.	112	if ``kasan_multi_shot`` is enabled.
113		113
114	Hardware Tag-Based KASAN mode (see the section about various modes below) is	114	Software and Hardware Tag-Based KASAN modes (see the section about various
115	intended for use in production as a security mitigation. Therefore, it supports	115	modes below) support altering stack trace collection behavior:
116	additional boot parameters that allow disabling KASAN or controlling features:	116
		117	- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack
		118	traces collection (default: ``on``).
		119	- ``kasan.stack_ring_size=<number of entries>`` specifies the number of entries
		120	in the stack ring (default: ``32768``).
		121
		122	Hardware Tag-Based KASAN mode is intended for use in production as a security
		123	mitigation. Therefore, it supports additional boot parameters that allow
		124	disabling KASAN altogether or controlling its features:
117		125
118	- ``kasan=off`` or ``=on`` controls whether KASAN is enabled (default: ``on``).	126	- ``kasan=off`` or ``=on`` controls whether KASAN is enabled (default: ``on``).
119		127
@@ -132,9 +140,6 @@ additional boot parameters that allow disabling KASAN or controlling features: · 12 unmodified lines
132	- ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc	140	- ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc
133	allocations (default: ``on``).	141	allocations (default: ``on``).
134		142
135	- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack
136	traces collection (default: ``on``).
137
138	Error reports	143	Error reports
139	~~~~~~~~~~~~~	144	~~~~~~~~~~~~~
140		145

Documentation/dev-tools/kmsan.rst+427−0

@@ -0,0 +1,427 @@
	1	.. SPDX-License-Identifier: GPL-2.0
	2	.. Copyright (C) 2022, Google LLC.
	3
	4	===================================
	5	The Kernel Memory Sanitizer (KMSAN)
	6	===================================
	7
	8	KMSAN is a dynamic error detector aimed at finding uses of uninitialized
	9	values. It is based on compiler instrumentation, and is quite similar to the
	10	userspace `MemorySanitizer tool`_.
	11
	12	An important note is that KMSAN is not intended for production use, because it
	13	drastically increases kernel memory footprint and slows the whole system down.
	14
	15	Usage
	16	=====
	17
	18	Building the kernel
	19	-------------------
	20
	21	In order to build a kernel with KMSAN you will need a fresh Clang (14.0.6+).
	22	Please refer to `LLVM documentation`_ for the instructions on how to build Clang.
	23
	24	Now configure and build the kernel with CONFIG_KMSAN enabled.
	25
	26	Example report
	27	--------------
	28
	29	Here is an example of a KMSAN report::
	30
	31	=====================================================
	32	BUG: KMSAN: uninit-value in test_uninit_kmsan_check_memory+0x1be/0x380 [kmsan_test]
	33	test_uninit_kmsan_check_memory+0x1be/0x380 mm/kmsan/kmsan_test.c:273
	34	kunit_run_case_internal lib/kunit/test.c:333
	35	kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374
	36	kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28
	37	kthread+0x721/0x850 kernel/kthread.c:327
	38	ret_from_fork+0x1f/0x30 ??:?
	39
	40	Uninit was stored to memory at:
	41	do_uninit_local_array+0xfa/0x110 mm/kmsan/kmsan_test.c:260
	42	test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271
	43	kunit_run_case_internal lib/kunit/test.c:333
	44	kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374
	45	kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28
	46	kthread+0x721/0x850 kernel/kthread.c:327
	47	ret_from_fork+0x1f/0x30 ??:?
	48
	49	Local variable uninit created at:
	50	do_uninit_local_array+0x4a/0x110 mm/kmsan/kmsan_test.c:256
	51	test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271
	52
	53	Bytes 4-7 of 8 are uninitialized
	54	Memory access of size 8 starts at ffff888083fe3da0
	55
	56	CPU: 0 PID: 6731 Comm: kunit_try_catch Tainted: G B E 5.16.0-rc3+ #104
	57	Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
	58	=====================================================
	59
	60	The report says that the local variable ``uninit`` was created uninitialized in
	61	``do_uninit_local_array()``. The third stack trace corresponds to the place
	62	where this variable was created.
	63
	64	The first stack trace shows where the uninit value was used (in
	65	``test_uninit_kmsan_check_memory()``). The tool shows the bytes which were left
	66	uninitialized in the local variable, as well as the stack where the value was
	67	copied to another memory location before use.
	68
	69	A use of uninitialized value ``v`` is reported by KMSAN in the following cases:
	70	- in a condition, e.g. ``if (v) { ... }``;
	71	- in an indexing or pointer dereferencing, e.g. ``array[v]`` or ``*v``;
	72	- when it is copied to userspace or hardware, e.g. ``copy_to_user(..., &v, ...)``;
	73	- when it is passed as an argument to a function, and
	74	``CONFIG_KMSAN_CHECK_PARAM_RETVAL`` is enabled (see below).
	75
	76	The mentioned cases (apart from copying data to userspace or hardware, which is
	77	a security issue) are considered undefined behavior from the C11 Standard point
	78	of view.
	79
	80	Disabling the instrumentation
	81	-----------------------------
	82
	83	A function can be marked with ``__no_kmsan_checks``. Doing so makes KMSAN
	84	ignore uninitialized values in that function and mark its output as initialized.
	85	As a result, the user will not get KMSAN reports related to that function.
	86
	87	Another function attribute supported by KMSAN is ``__no_sanitize_memory``.
	88	Applying this attribute to a function will result in KMSAN not instrumenting
	89	it, which can be helpful if we do not want the compiler to interfere with some
	90	low-level code (e.g. that marked with ``noinstr`` which implicitly adds
	91	``__no_sanitize_memory``).
	92
	93	This however comes at a cost: stack allocations from such functions will have
	94	incorrect shadow/origin values, likely leading to false positives. Functions
	95	called from non-instrumented code may also receive incorrect metadata for their
	96	parameters.
	97
	98	As a rule of thumb, avoid using ``__no_sanitize_memory`` explicitly.
	99
	100	It is also possible to disable KMSAN for a single file (e.g. main.o)::
	101
	102	KMSAN_SANITIZE_main.o := n
	103
	104	or for the whole directory::
	105
	106	KMSAN_SANITIZE := n
	107
	108	in the Makefile. Think of this as applying ``__no_sanitize_memory`` to every
	109	function in the file or directory. Most users won't need KMSAN_SANITIZE, unless
	110	their code gets broken by KMSAN (e.g. runs at early boot time).
	111
	112	Support
	113	=======
	114
	115	In order for KMSAN to work the kernel must be built with Clang, which so far is
	116	the only compiler that has KMSAN support. The kernel instrumentation pass is
	117	based on the userspace `MemorySanitizer tool`_.
	118
	119	The runtime library only supports x86_64 at the moment.
	120
	121	How KMSAN works
	122	===============
	123
	124	KMSAN shadow memory
	125	-------------------
	126
	127	KMSAN associates a metadata byte (also called shadow byte) with every byte of
	128	kernel memory. A bit in the shadow byte is set iff the corresponding bit of the
	129	kernel memory byte is uninitialized. Marking the memory uninitialized (i.e.
	130	setting its shadow bytes to ``0xff``) is called poisoning, marking it
	131	initialized (setting the shadow bytes to ``0x00``) is called unpoisoning.
	132
	133	When a new variable is allocated on the stack, it is poisoned by default by
	134	instrumentation code inserted by the compiler (unless it is a stack variable
	135	that is immediately initialized). Any new heap allocation done without
	136	``__GFP_ZERO`` is also poisoned.
	137
	138	Compiler instrumentation also tracks the shadow values as they are used along
	139	the code. When needed, instrumentation code invokes the runtime library in
	140	``mm/kmsan/`` to persist shadow values.
	141
	142	The shadow value of a basic or compound type is an array of bytes of the same
	143	length. When a constant value is written into memory, that memory is unpoisoned.
	144	When a value is read from memory, its shadow memory is also obtained and
	145	propagated into all the operations which use that value. For every instruction
	146	that takes one or more values the compiler generates code that calculates the
	147	shadow of the result depending on those values and their shadows.
	148
	149	Example::
	150
	151	int a = 0xff; // i.e. 0x000000ff
	152	int b;
	153	int c = a \| b;
	154
	155	In this case the shadow of ``a`` is ``0``, shadow of ``b`` is ``0xffffffff``,
	156	shadow of ``c`` is ``0xffffff00``. This means that the upper three bytes of
	157	``c`` are uninitialized, while the lower byte is initialized.
	158
	159	Origin tracking
	160	---------------
	161
	162	Every four bytes of kernel memory also have a so-called origin mapped to them.
	163	This origin describes the point in program execution at which the uninitialized
	164	value was created. Every origin is associated with either the full allocation
	165	stack (for heap-allocated memory), or the function containing the uninitialized
	166	variable (for locals).
	167
	168	When an uninitialized variable is allocated on stack or heap, a new origin
	169	value is created, and that variable's origin is filled with that value. When a
	170	value is read from memory, its origin is also read and kept together with the
	171	shadow. For every instruction that takes one or more values, the origin of the
	172	result is one of the origins corresponding to any of the uninitialized inputs.
	173	If a poisoned value is written into memory, its origin is written to the
	174	corresponding storage as well.
	175
	176	Example 1::
	177
	178	int a = 42;
	179	int b;
	180	int c = a + b;
	181
	182	In this case the origin of ``b`` is generated upon function entry, and is
	183	stored to the origin of ``c`` right before the addition result is written into
	184	memory.
	185
	186	Several variables may share the same origin address, if they are stored in the
	187	same four-byte chunk. In this case every write to either variable updates the
	188	origin for all of them. We have to sacrifice precision in this case, because
	189	storing origins for individual bits (and even bytes) would be too costly.
	190
	191	Example 2::
	192
	193	int combine(short a, short b) {
	194	union ret_t {
	195	int i;
	196	short s[2];
	197	} ret;
	198	ret.s[0] = a;
	199	ret.s[1] = b;
	200	return ret.i;
	201	}
	202
	203	If ``a`` is initialized and ``b`` is not, the shadow of the result would be
	204	0xffff0000, and the origin of the result would be the origin of ``b``.
	205	``ret.s[0]`` would have the same origin, but it will never be used, because
	206	that variable is initialized.
	207
	208	If both function arguments are uninitialized, only the origin of the second
	209	argument is preserved.
	210
	211	Origin chaining
	212	~~~~~~~~~~~~~~~
	213
	214	To ease debugging, KMSAN creates a new origin for every store of an
	215	uninitialized value to memory. The new origin references both its creation stack
	216	and the previous origin the value had. This may cause increased memory
	217	consumption, so we limit the length of origin chains in the runtime.
	218
	219	Clang instrumentation API
	220	-------------------------
	221
	222	Clang instrumentation pass inserts calls to functions defined in
	223	``mm/kmsan/nstrumentation.c`` into the kernel code.
	224
	225	Shadow manipulation
	226	~~~~~~~~~~~~~~~~~~~
	227
	228	For every memory access the compiler emits a call to a function that returns a
	229	pair of pointers to the shadow and origin addresses of the given memory::
	230
	231	typedef struct {
	232	void shadow, origin;
	233	} shadow_origin_ptr_t
	234
	235	shadow_origin_ptr_t __msan_metadata_ptr_for_load_{1,2,4,8}(void *addr)
	236	shadow_origin_ptr_t __msan_metadata_ptr_for_store_{1,2,4,8}(void *addr)
	237	shadow_origin_ptr_t __msan_metadata_ptr_for_load_n(void *addr, uintptr_t size)
	238	shadow_origin_ptr_t __msan_metadata_ptr_for_store_n(void *addr, uintptr_t size)
	239
	240	The function name depends on the memory access size.
	241
	242	The compiler makes sure that for every loaded value its shadow and origin
	243	values are read from memory. When a value is stored to memory, its shadow and
	244	origin are also stored using the metadata pointers.
	245
	246	Handling locals
	247	~~~~~~~~~~~~~~~
	248
	249	A special function is used to create a new origin value for a local variable and
	250	set the origin of that variable to that value::
	251
	252	void __msan_poison_alloca(void addr, uintptr_t size, char descr)
	253
	254	Access to per-task data
	255	~~~~~~~~~~~~~~~~~~~~~~~
	256
	257	At the beginning of every instrumented function KMSAN inserts a call to
	258	``__msan_get_context_state()``::
	259
	260	kmsan_context_state *__msan_get_context_state(void)
	261
	262	``kmsan_context_state`` is declared in ``include/linux/kmsan.h``::
	263
	264	struct kmsan_context_state {
	265	char param_tls[KMSAN_PARAM_SIZE];
	266	char retval_tls[KMSAN_RETVAL_SIZE];
	267	char va_arg_tls[KMSAN_PARAM_SIZE];
	268	char va_arg_origin_tls[KMSAN_PARAM_SIZE];
	269	u64 va_arg_overflow_size_tls;
	270	char param_origin_tls[KMSAN_PARAM_SIZE];
	271	depot_stack_handle_t retval_origin_tls;
	272	};
	273
	274	This structure is used by KMSAN to pass parameter shadows and origins between
	275	instrumented functions (unless the parameters are checked immediately by
	276	``CONFIG_KMSAN_CHECK_PARAM_RETVAL``).
	277
	278	Passing uninitialized values to functions
	279	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	280
	281	Clang's MemorySanitizer instrumentation has an option,
	282	``-fsanitize-memory-param-retval``, which makes the compiler check function
	283	parameters passed by value, as well as function return values.
	284
	285	The option is controlled by ``CONFIG_KMSAN_CHECK_PARAM_RETVAL``, which is
	286	enabled by default to let KMSAN report uninitialized values earlier.
	287	Please refer to the `LKML discussion`_ for more details.
	288
	289	Because of the way the checks are implemented in LLVM (they are only applied to
	290	parameters marked as ``noundef``), not all parameters are guaranteed to be
	291	checked, so we cannot give up the metadata storage in ``kmsan_context_state``.
	292
	293	String functions
	294	~~~~~~~~~~~~~~~~
	295
	296	The compiler replaces calls to ``memcpy()``/``memmove()``/``memset()`` with the
	297	following functions. These functions are also called when data structures are
	298	initialized or copied, making sure shadow and origin values are copied alongside
	299	with the data::
	300
	301	void __msan_memcpy(void dst, void *src, uintptr_t n)
	302	void __msan_memmove(void dst, void *src, uintptr_t n)
	303	void __msan_memset(void dst, int c, uintptr_t n)
	304
	305	Error reporting
	306	~~~~~~~~~~~~~~~
	307
	308	For each use of a value the compiler emits a shadow check that calls
	309	``__msan_warning()`` in the case that value is poisoned::
	310
	311	void __msan_warning(u32 origin)
	312
	313	``__msan_warning()`` causes KMSAN runtime to print an error report.
	314
	315	Inline assembly instrumentation
	316	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	317
	318	KMSAN instruments every inline assembly output with a call to::
	319
	320	void __msan_instrument_asm_store(void *addr, uintptr_t size)
	321
	322	, which unpoisons the memory region.
	323
	324	This approach may mask certain errors, but it also helps to avoid a lot of
	325	false positives in bitwise operations, atomics etc.
	326
	327	Sometimes the pointers passed into inline assembly do not point to valid memory.
	328	In such cases they are ignored at runtime.
	329
	330
	331	Runtime library
	332	---------------
	333
	334	The code is located in ``mm/kmsan/``.
	335
	336	Per-task KMSAN state
	337	~~~~~~~~~~~~~~~~~~~~
	338
	339	Every task_struct has an associated KMSAN task state that holds the KMSAN
	340	context (see above) and a per-task flag disallowing KMSAN reports::
	341
	342	struct kmsan_context {
	343	...
	344	bool allow_reporting;
	345	struct kmsan_context_state cstate;
	346	...
	347	}
	348
	349	struct task_struct {
	350	...
	351	struct kmsan_context kmsan;
	352	...
	353	}
	354
	355	KMSAN contexts
	356	~~~~~~~~~~~~~~
	357
	358	When running in a kernel task context, KMSAN uses ``current->kmsan.cstate`` to
	359	hold the metadata for function parameters and return values.
	360
	361	But in the case the kernel is running in the interrupt, softirq or NMI context,
	362	where ``current`` is unavailable, KMSAN switches to per-cpu interrupt state::
	363
	364	DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx);
	365
	366	Metadata allocation
	367	~~~~~~~~~~~~~~~~~~~
	368
	369	There are several places in the kernel for which the metadata is stored.
	370
	371	1. Each ``struct page`` instance contains two pointers to its shadow and
	372	origin pages::
	373
	374	struct page {
	375	...
	376	struct page shadow, origin;
	377	...
	378	};
	379
	380	At boot-time, the kernel allocates shadow and origin pages for every available
	381	kernel page. This is done quite late, when the kernel address space is already
	382	fragmented, so normal data pages may arbitrarily interleave with the metadata
	383	pages.
	384
	385	This means that in general for two contiguous memory pages their shadow/origin
	386	pages may not be contiguous. Consequently, if a memory access crosses the
	387	boundary of a memory block, accesses to shadow/origin memory may potentially
	388	corrupt other pages or read incorrect values from them.
	389
	390	In practice, contiguous memory pages returned by the same ``alloc_pages()``
	391	call will have contiguous metadata, whereas if these pages belong to two
	392	different allocations their metadata pages can be fragmented.
	393
	394	For the kernel data (``.data``, ``.bss`` etc.) and percpu memory regions
	395	there also are no guarantees on metadata contiguity.
	396
	397	In the case ``__msan_metadata_ptr_for_XXX_YYY()`` hits the border between two
	398	pages with non-contiguous metadata, it returns pointers to fake shadow/origin regions::
	399
	400	char dummy_load_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
	401	char dummy_store_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
	402
	403	``dummy_load_page`` is zero-initialized, so reads from it always yield zeroes.
	404	All stores to ``dummy_store_page`` are ignored.
	405
	406	2. For vmalloc memory and modules, there is a direct mapping between the memory
	407	range, its shadow and origin. KMSAN reduces the vmalloc area by 3/4, making only
	408	the first quarter available to ``vmalloc()``. The second quarter of the vmalloc
	409	area contains shadow memory for the first quarter, the third one holds the
	410	origins. A small part of the fourth quarter contains shadow and origins for the
	411	kernel modules. Please refer to ``arch/x86/include/asm/pgtable_64_types.h`` for
	412	more details.
	413
	414	When an array of pages is mapped into a contiguous virtual memory space, their
	415	shadow and origin pages are similarly mapped into contiguous regions.
	416
	417	References
	418	==========
	419
	420	E. Stepanov, K. Serebryany. `MemorySanitizer: fast detector of uninitialized
	421	memory use in C++
	422	<https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43308.pdf>`_.
	423	In Proceedings of CGO 2015.
	424
	425	.. _MemorySanitizer tool: https://clang.llvm.org/docs/MemorySanitizer.html
	426	.. _LLVM documentation: https://llvm.org/docs/GettingStarted.html
	427	.. _LKML discussion: https://lore.kernel.org/all/20220614144853.3693273-1-glider@google.com/

Documentation/mm/index.rst+1−0

@@ -51,6 +51,7 @@ above structured documentation, or deleted if it has served its purpose. · 50 unmodified lines
51	ksm	51	ksm
52	memory-model	52	memory-model
53	mmu_notifier	53	mmu_notifier
		54	multigen_lru
54	numa	55	numa
55	overcommit-accounting	56	overcommit-accounting
56	page_migration	57	page_migration

Documentation/mm/ksm.rst+1−1

@@ -26,7 +26,7 @@ tree. · 25 unmodified lines
26		26
27	If a KSM page is shared between less than ``max_page_sharing`` VMAs,	27	If a KSM page is shared between less than ``max_page_sharing`` VMAs,
28	the node of the stable tree that represents such KSM page points to a	28	the node of the stable tree that represents such KSM page points to a
29	list of struct rmap_item and the ``page->mapping`` of the	29	list of struct ksm_rmap_item and the ``page->mapping`` of the
30	KSM page points to the stable tree node.	30	KSM page points to the stable tree node.
31		31
32	When the sharing passes this threshold, KSM adds a second dimension to	32	When the sharing passes this threshold, KSM adds a second dimension to

Documentation/mm/multigen_lru.rst+159−0

@@ -0,0 +1,159 @@
	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	=============
	4	Multi-Gen LRU
	5	=============
	6	The multi-gen LRU is an alternative LRU implementation that optimizes
	7	page reclaim and improves performance under memory pressure. Page
	8	reclaim decides the kernel's caching policy and ability to overcommit
	9	memory. It directly impacts the kswapd CPU usage and RAM efficiency.
	10
	11	Design overview
	12	===============
	13	Objectives
	14	----------
	15	The design objectives are:
	16
	17	* Good representation of access recency
	18	* Try to profit from spatial locality
	19	* Fast paths to make obvious choices
	20	* Simple self-correcting heuristics
	21
	22	The representation of access recency is at the core of all LRU
	23	implementations. In the multi-gen LRU, each generation represents a
	24	group of pages with similar access recency. Generations establish a
	25	(time-based) common frame of reference and therefore help make better
	26	choices, e.g., between different memcgs on a computer or different
	27	computers in a data center (for job scheduling).
	28
	29	Exploiting spatial locality improves efficiency when gathering the
	30	accessed bit. A rmap walk targets a single page and does not try to
	31	profit from discovering a young PTE. A page table walk can sweep all
	32	the young PTEs in an address space, but the address space can be too
	33	sparse to make a profit. The key is to optimize both methods and use
	34	them in combination.
	35
	36	Fast paths reduce code complexity and runtime overhead. Unmapped pages
	37	do not require TLB flushes; clean pages do not require writeback.
	38	These facts are only helpful when other conditions, e.g., access
	39	recency, are similar. With generations as a common frame of reference,
	40	additional factors stand out. But obvious choices might not be good
	41	choices; thus self-correction is necessary.
	42
	43	The benefits of simple self-correcting heuristics are self-evident.
	44	Again, with generations as a common frame of reference, this becomes
	45	attainable. Specifically, pages in the same generation can be
	46	categorized based on additional factors, and a feedback loop can
	47	statistically compare the refault percentages across those categories
	48	and infer which of them are better choices.
	49
	50	Assumptions
	51	-----------
	52	The protection of hot pages and the selection of cold pages are based
	53	on page access channels and patterns. There are two access channels:
	54
	55	* Accesses through page tables
	56	* Accesses through file descriptors
	57
	58	The protection of the former channel is by design stronger because:
	59
	60	1. The uncertainty in determining the access patterns of the former
	61	channel is higher due to the approximation of the accessed bit.
	62	2. The cost of evicting the former channel is higher due to the TLB
	63	flushes required and the likelihood of encountering the dirty bit.
	64	3. The penalty of underprotecting the former channel is higher because
	65	applications usually do not prepare themselves for major page
	66	faults like they do for blocked I/O. E.g., GUI applications
	67	commonly use dedicated I/O threads to avoid blocking rendering
	68	threads.
	69
	70	There are also two access patterns:
	71
	72	* Accesses exhibiting temporal locality
	73	* Accesses not exhibiting temporal locality
	74
	75	For the reasons listed above, the former channel is assumed to follow
	76	the former pattern unless ``VM_SEQ_READ`` or ``VM_RAND_READ`` is
	77	present, and the latter channel is assumed to follow the latter
	78	pattern unless outlying refaults have been observed.
	79
	80	Workflow overview
	81	=================
	82	Evictable pages are divided into multiple generations for each
	83	``lruvec``. The youngest generation number is stored in
	84	``lrugen->max_seq`` for both anon and file types as they are aged on
	85	an equal footing. The oldest generation numbers are stored in
	86	``lrugen->min_seq[]`` separately for anon and file types as clean file
	87	pages can be evicted regardless of swap constraints. These three
	88	variables are monotonically increasing.
	89
	90	Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)``
	91	bits in order to fit into the gen counter in ``folio->flags``. Each
	92	truncated generation number is an index to ``lrugen->lists[]``. The
	93	sliding window technique is used to track at least ``MIN_NR_GENS`` and
	94	at most ``MAX_NR_GENS`` generations. The gen counter stores a value
	95	within ``[1, MAX_NR_GENS]`` while a page is on one of
	96	``lrugen->lists[]``; otherwise it stores zero.
	97
	98	Each generation is divided into multiple tiers. A page accessed ``N``
	99	times through file descriptors is in tier ``order_base_2(N)``. Unlike
	100	generations, tiers do not have dedicated ``lrugen->lists[]``. In
	101	contrast to moving across generations, which requires the LRU lock,
	102	moving across tiers only involves atomic operations on
	103	``folio->flags`` and therefore has a negligible cost. A feedback loop
	104	modeled after the PID controller monitors refaults over all the tiers
	105	from anon and file types and decides which tiers from which types to
	106	evict or protect.
	107
	108	There are two conceptually independent procedures: the aging and the
	109	eviction. They form a closed-loop system, i.e., the page reclaim.
	110
	111	Aging
	112	-----
	113	The aging produces young generations. Given an ``lruvec``, it
	114	increments ``max_seq`` when ``max_seq-min_seq+1`` approaches
	115	``MIN_NR_GENS``. The aging promotes hot pages to the youngest
	116	generation when it finds them accessed through page tables; the
	117	demotion of cold pages happens consequently when it increments
	118	``max_seq``. The aging uses page table walks and rmap walks to find
	119	young PTEs. For the former, it iterates ``lruvec_memcg()->mm_list``
	120	and calls ``walk_page_range()`` with each ``mm_struct`` on this list
	121	to scan PTEs, and after each iteration, it increments ``max_seq``. For
	122	the latter, when the eviction walks the rmap and finds a young PTE,
	123	the aging scans the adjacent PTEs. For both, on finding a young PTE,
	124	the aging clears the accessed bit and updates the gen counter of the
	125	page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``.
	126
	127	Eviction
	128	--------
	129	The eviction consumes old generations. Given an ``lruvec``, it
	130	increments ``min_seq`` when ``lrugen->lists[]`` indexed by
	131	``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to
	132	evict from, it first compares ``min_seq[]`` to select the older type.
	133	If both types are equally old, it selects the one whose first tier has
	134	a lower refault percentage. The first tier contains single-use
	135	unmapped clean pages, which are the best bet. The eviction sorts a
	136	page according to its gen counter if the aging has found this page
	137	accessed through page tables and updated its gen counter. It also
	138	moves a page to the next generation, i.e., ``min_seq+1``, if this page
	139	was accessed multiple times through file descriptors and the feedback
	140	loop has detected outlying refaults from the tier this page is in. To
	141	this end, the feedback loop uses the first tier as the baseline, for
	142	the reason stated earlier.
	143
	144	Summary
	145	-------
	146	The multi-gen LRU can be disassembled into the following parts:
	147
	148	* Generations
	149	* Rmap walks
	150	* Page table walks
	151	* Bloom filters
	152	* PID controller
	153
	154	The aging and the eviction form a producer-consumer model;
	155	specifically, the latter drives the former by the sliding window over
	156	generations. Within the aging, rmap walks drive page table walks by
	157	inserting hot densely populated page tables to the Bloom filters.
	158	Within the eviction, the PID controller uses refaults as the feedback
	159	to select types to evict and tiers to protect.

Documentation/mm/page_owner.rst+5−0

 	Page allocated via order XXX, ...
 	PFN XXX ...
 	// Detailed stack
+    By default, it will do full pfn dump, to start with a given pfn,
+    page_owner supports fseek.
+    FILE *fp = fopen("/sys/kernel/debug/page_owner", "r");
+    fseek(fp, pfn_start, SEEK_SET);
    The ``page_owner_sort`` tool ignores ``PFN`` rows, puts the remaining rows
    in buf, uses regexp to extract the page order value, counts the times

MAINTAINERS+26−1

@@ -11004,7 +11004,6 @@ F: arch//include/asm/kasan.h · 11003 unmodified lines
11004	F: arch//mm/kasan_init	11004	F: arch//mm/kasan_init
11005	F: include/linux/kasan*.h	11005	F: include/linux/kasan*.h
11006	F: lib/Kconfig.kasan	11006	F: lib/Kconfig.kasan
11007	F: lib/test_kasan*.c
11008	F: mm/kasan/	11007	F: mm/kasan/
11009	F: scripts/Makefile.kasan	11008	F: scripts/Makefile.kasan
11010		11009
@@ -11438,6 +11437,20 @@ F: kernel/kmod.c · 427 unmodified lines
11438	F: lib/test_kmod.c	11437	F: lib/test_kmod.c
11439	F: tools/testing/selftests/kmod/	11438	F: tools/testing/selftests/kmod/
11440		11439
		11440	KMSAN
		11441	M: Alexander Potapenko <glider@google.com>
		11442	R: Marco Elver <elver@google.com>
		11443	R: Dmitry Vyukov <dvyukov@google.com>
		11444	L: kasan-dev@googlegroups.com
		11445	S: Maintained
		11446	F: Documentation/dev-tools/kmsan.rst
		11447	F: arch/*/include/asm/kmsan.h
		11448	F: arch//mm/kmsan_
		11449	F: include/linux/kmsan*.h
		11450	F: lib/Kconfig.kmsan
		11451	F: mm/kmsan/
		11452	F: scripts/Makefile.kmsan
		11453
11441	KPROBES	11454	KPROBES
11442	M: Naveen N. Rao <naveen.n.rao@linux.ibm.com>	11455	M: Naveen N. Rao <naveen.n.rao@linux.ibm.com>
11443	M: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>	11456	M: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
@@ -12168,6 +12181,18 @@ L: linux-man@vger.kernel.org · 724 unmodified lines
12168	S: Maintained	12181	S: Maintained
12169	W: http://www.kernel.org/doc/man-pages	12182	W: http://www.kernel.org/doc/man-pages
12170		12183
		12184	MAPLE TREE
		12185	M: Liam R. Howlett <Liam.Howlett@oracle.com>
		12186	L: linux-mm@kvack.org
		12187	S: Supported
		12188	F: Documentation/core-api/maple_tree.rst
		12189	F: include/linux/maple_tree.h
		12190	F: include/trace/events/maple_tree.h
		12191	F: lib/maple_tree.c
		12192	F: lib/test_maple_tree.c
		12193	F: tools/testing/radix-tree/linux/maple_tree.h
		12194	F: tools/testing/radix-tree/maple.c
		12195
12171	MARDUK (CREATOR CI40) DEVICE TREE SUPPORT	12196	MARDUK (CREATOR CI40) DEVICE TREE SUPPORT
12172	M: Rahul Bedarkar <rahulbedarkar89@gmail.com>	12197	M: Rahul Bedarkar <rahulbedarkar89@gmail.com>
12173	L: linux-mips@vger.kernel.org	12198	L: linux-mips@vger.kernel.org

Makefile+1−0

@@ -1081,6 +1081,7 @@ include-y := scripts/Makefile.extrawarn · 1080 unmodified lines
1081	include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug	1081	include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug
1082	include-$(CONFIG_KASAN) += scripts/Makefile.kasan	1082	include-$(CONFIG_KASAN) += scripts/Makefile.kasan
1083	include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan	1083	include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan
		1084	include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan
1084	include-$(CONFIG_UBSAN) += scripts/Makefile.ubsan	1085	include-$(CONFIG_UBSAN) += scripts/Makefile.ubsan
1085	include-$(CONFIG_KCOV) += scripts/Makefile.kcov	1086	include-$(CONFIG_KCOV) += scripts/Makefile.kcov
1086	include-$(CONFIG_RANDSTRUCT) += scripts/Makefile.randstruct	1087	include-$(CONFIG_RANDSTRUCT) += scripts/Makefile.randstruct

arch/Kconfig+8−0

@@ -1416,6 +1416,14 @@ config DYNAMIC_SIGFRAME · 1415 unmodified lines
1416	config HAVE_ARCH_NODE_DEV_GROUP	1416	config HAVE_ARCH_NODE_DEV_GROUP
1417	bool	1417	bool
1418		1418
		1419	config ARCH_HAS_NONLEAF_PMD_YOUNG
		1420	bool
		1421	help
		1422	Architectures that select this option are capable of setting the
		1423	accessed bit in non-leaf PMD entries when using them as part of linear
		1424	address translations. Page table walkers that clear the accessed bit
		1425	may use this capability to reduce their search space.
		1426
1419	source "kernel/gcov/Kconfig"	1427	source "kernel/gcov/Kconfig"
1420		1428
1421	source "scripts/gcc-plugins/Kconfig"	1429	source "scripts/gcc-plugins/Kconfig"

arch/alpha/include/uapi/asm/mman.h+2−0

@@ -76,6 +76,8 @@ · 75 unmodified lines
76		76
77	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */	77	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
78		78
		79	#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
		80
79	/* compatibility flags */	81	/* compatibility flags */
80	#define MAP_FILE 0	82	#define MAP_FILE 0
81		83

arch/arc/Kconfig+1−1

@@ -554,7 +554,7 @@ config ARC_BUILTIN_DTB_NAME · 553 unmodified lines
554		554
555	endmenu # "ARC Architecture Configuration"	555	endmenu # "ARC Architecture Configuration"
556		556
557	config FORCE_MAX_ZONEORDER	557	config ARCH_FORCE_MAX_ORDER
558	int "Maximum zone order"	558	int "Maximum zone order"
559	default "12" if ARC_HUGEPAGE_16M	559	default "12" if ARC_HUGEPAGE_16M
560	default "11"	560	default "11"

arch/arm/Kconfig+1−1

@@ -1362,7 +1362,7 @@ config ARM_MODULE_PLTS · 1361 unmodified lines
1362	Disabling this is usually safe for small single-platform	1362	Disabling this is usually safe for small single-platform
1363	configurations. If unsure, say y.	1363	configurations. If unsure, say y.
1364		1364
1365	config FORCE_MAX_ZONEORDER	1365	config ARCH_FORCE_MAX_ORDER
1366	int "Maximum zone order"	1366	int "Maximum zone order"
1367	default "12" if SOC_AM33XX	1367	default "12" if SOC_AM33XX
1368	default "9" if SA1111	1368	default "9" if SA1111

arch/arm/configs/imx_v6_v7_defconfig+1−1

@@ -31,7 +31,7 @@ CONFIG_SOC_VF610=y · 30 unmodified lines
31	CONFIG_SMP=y	31	CONFIG_SMP=y
32	CONFIG_ARM_PSCI=y	32	CONFIG_ARM_PSCI=y
33	CONFIG_HIGHMEM=y	33	CONFIG_HIGHMEM=y
34	CONFIG_FORCE_MAX_ZONEORDER=14	34	CONFIG_ARCH_FORCE_MAX_ORDER=14
35	CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"	35	CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
36	CONFIG_KEXEC=y	36	CONFIG_KEXEC=y
37	CONFIG_CPU_FREQ=y	37	CONFIG_CPU_FREQ=y

arch/arm/configs/milbeaut_m10v_defconfig+1−1

@@ -26,7 +26,7 @@ CONFIG_THUMB2_KERNEL=y · 25 unmodified lines
26	# CONFIG_THUMB2_AVOID_R_ARM_THM_JUMP11 is not set	26	# CONFIG_THUMB2_AVOID_R_ARM_THM_JUMP11 is not set
27	# CONFIG_ARM_PATCH_IDIV is not set	27	# CONFIG_ARM_PATCH_IDIV is not set
28	CONFIG_HIGHMEM=y	28	CONFIG_HIGHMEM=y
29	CONFIG_FORCE_MAX_ZONEORDER=12	29	CONFIG_ARCH_FORCE_MAX_ORDER=12
30	CONFIG_SECCOMP=y	30	CONFIG_SECCOMP=y
31	CONFIG_KEXEC=y	31	CONFIG_KEXEC=y
32	CONFIG_EFI=y	32	CONFIG_EFI=y

arch/arm/configs/oxnas_v6_defconfig+1−1

@@ -12,7 +12,7 @@ CONFIG_ARCH_OXNAS=y · 11 unmodified lines
12	CONFIG_MACH_OX820=y	12	CONFIG_MACH_OX820=y
13	CONFIG_SMP=y	13	CONFIG_SMP=y
14	CONFIG_NR_CPUS=16	14	CONFIG_NR_CPUS=16
15	CONFIG_FORCE_MAX_ZONEORDER=12	15	CONFIG_ARCH_FORCE_MAX_ORDER=12
16	CONFIG_SECCOMP=y	16	CONFIG_SECCOMP=y
17	CONFIG_ARM_APPENDED_DTB=y	17	CONFIG_ARM_APPENDED_DTB=y
18	CONFIG_ARM_ATAG_DTB_COMPAT=y	18	CONFIG_ARM_ATAG_DTB_COMPAT=y

arch/arm/configs/pxa_defconfig+1−1

@@ -21,7 +21,7 @@ CONFIG_MACH_AKITA=y · 20 unmodified lines
21	CONFIG_MACH_BORZOI=y	21	CONFIG_MACH_BORZOI=y
22	CONFIG_PXA_SYSTEMS_CPLDS=y	22	CONFIG_PXA_SYSTEMS_CPLDS=y
23	CONFIG_AEABI=y	23	CONFIG_AEABI=y
24	CONFIG_FORCE_MAX_ZONEORDER=9	24	CONFIG_ARCH_FORCE_MAX_ORDER=9
25	CONFIG_CMDLINE="root=/dev/ram0 ro"	25	CONFIG_CMDLINE="root=/dev/ram0 ro"
26	CONFIG_KEXEC=y	26	CONFIG_KEXEC=y
27	CONFIG_CPU_FREQ=y	27	CONFIG_CPU_FREQ=y

arch/arm/configs/sama7_defconfig+1−1

@@ -19,7 +19,7 @@ CONFIG_ATMEL_CLOCKSOURCE_TCB=y · 18 unmodified lines
19	# CONFIG_CACHE_L2X0 is not set	19	# CONFIG_CACHE_L2X0 is not set
20	# CONFIG_ARM_PATCH_IDIV is not set	20	# CONFIG_ARM_PATCH_IDIV is not set
21	# CONFIG_CPU_SW_DOMAIN_PAN is not set	21	# CONFIG_CPU_SW_DOMAIN_PAN is not set
22	CONFIG_FORCE_MAX_ZONEORDER=15	22	CONFIG_ARCH_FORCE_MAX_ORDER=15
23	CONFIG_UACCESS_WITH_MEMCPY=y	23	CONFIG_UACCESS_WITH_MEMCPY=y
24	# CONFIG_ATAGS is not set	24	# CONFIG_ATAGS is not set
25	CONFIG_CMDLINE="console=ttyS0,115200 earlyprintk ignore_loglevel"	25	CONFIG_CMDLINE="console=ttyS0,115200 earlyprintk ignore_loglevel"

arch/arm/configs/sp7021_defconfig+1−1

@@ -17,7 +17,7 @@ CONFIG_ARCH_SUNPLUS=y · 16 unmodified lines
17	# CONFIG_VDSO is not set	17	# CONFIG_VDSO is not set
18	CONFIG_SMP=y	18	CONFIG_SMP=y
19	CONFIG_THUMB2_KERNEL=y	19	CONFIG_THUMB2_KERNEL=y
20	CONFIG_FORCE_MAX_ZONEORDER=12	20	CONFIG_ARCH_FORCE_MAX_ORDER=12
21	CONFIG_VFP=y	21	CONFIG_VFP=y
22	CONFIG_NEON=y	22	CONFIG_NEON=y
23	CONFIG_MODULES=y	23	CONFIG_MODULES=y

arch/arm64/Kconfig+1−1

@@ -1431,7 +1431,7 @@ config XEN · 1430 unmodified lines
1431	help	1431	help
1432	Say Y if you want to run Linux in a Virtual Machine on Xen on ARM64.	1432	Say Y if you want to run Linux in a Virtual Machine on Xen on ARM64.
1433		1433
1434	config FORCE_MAX_ZONEORDER	1434	config ARCH_FORCE_MAX_ORDER
1435	int	1435	int
1436	default "14" if ARM64_64K_PAGES	1436	default "14" if ARM64_64K_PAGES
1437	default "12" if ARM64_16K_PAGES	1437	default "12" if ARM64_16K_PAGES

arch/arm64/include/asm/pgtable.h+2−13

  * page after fork() + CoW for pfn mappings. We don't always have a
  * hardware-managed access flag on arm64.
  */
-static inline bool arch_faults_on_old_pte(void)
+#define arch_has_hw_pte_young		cpu_has_hw_af
-{
-	/* The register read below requires a stable CPU to make any sense */
-	cant_migrate();
-	return !cpu_has_hw_af();
-}
-#define arch_faults_on_old_pte		arch_faults_on_old_pte
 /*
  * Experimentally, it's cheap to set the access flag in hardware and we
  * benefit from prefaulting mappings as 'old' to start with.
  */
-static inline bool arch_wants_old_prefaulted_pte(void)
+#define arch_wants_old_prefaulted_pte	cpu_has_hw_af
-{
-	return !arch_faults_on_old_pte();
-}
-#define arch_wants_old_prefaulted_pte	arch_wants_old_prefaulted_pte
 static inline bool pud_sect_supported(void)
 {

arch/arm64/kernel/elfcore.c+10−6

 #include <asm/cpufeature.h>
 #include <asm/mte.h>
-#define for_each_mte_vma(tsk, vma)					\
+#define for_each_mte_vma(vmi, vma)					\
 	if (system_supports_mte())					\
-		for (vma = tsk->mm->mmap; vma; vma = vma->vm_next)	\
+		for_each_vma(vmi, vma)					\
 			if (vma->vm_flags & VM_MTE)
 static unsigned long mte_vma_tag_dump_size(struct vm_area_struct *vma)
 {
 	struct vm_area_struct *vma;
 	int vma_count = 0;
+	VMA_ITERATOR(vmi, current->mm, 0);
-	for_each_mte_vma(current, vma)
+	for_each_mte_vma(vmi, vma)
 		vma_count++;
 	return vma_count;
 int elf_core_write_extra_phdrs(struct coredump_params *cprm, loff_t offset)
 {
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, current->mm, 0);
-	for_each_mte_vma(current, vma) {
+	for_each_mte_vma(vmi, vma) {
 		struct elf_phdr phdr;
 		phdr.p_type = PT_AARCH64_MEMTAG_MTE;
 {
 	struct vm_area_struct *vma;
 	size_t data_size = 0;
+	VMA_ITERATOR(vmi, current->mm, 0);
-	for_each_mte_vma(current, vma)
+	for_each_mte_vma(vmi, vma)
 		data_size += mte_vma_tag_dump_size(vma);
 	return data_size;
 int elf_core_write_extra_data(struct coredump_params *cprm)
 {
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, current->mm, 0);
-	for_each_mte_vma(current, vma) {
+	for_each_mte_vma(vmi, vma) {
 		if (vma->vm_flags & VM_DONTDUMP)
 			continue;

arch/arm64/kernel/vdso.c+2−1

 {
 	struct mm_struct *mm = task->mm;
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
 	mmap_read_lock(mm);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		unsigned long size = vma->vm_end - vma->vm_start;
 		if (vma_is_special_mapping(vma, vdso_info[VDSO_ABI_AA64].dm))

arch/arm64/mm/hugetlbpage.c+1−1

@@ -245,7 +245,7 @@ static inline struct folio *hugetlb_swap_entry_to_folio(swp_entry_t entry) · 244 unmodified lines
245	{	245	{
246	VM_BUG_ON(!is_migration_entry(entry) && !is_hwpoison_entry(entry));	246	VM_BUG_ON(!is_migration_entry(entry) && !is_hwpoison_entry(entry));
247		247
248	return page_folio(pfn_to_page(swp_offset(entry)));	248	return page_folio(pfn_to_page(swp_offset_pfn(entry)));
249	}	249	}
250		250
251	void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,	251	void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,

arch/csky/Kconfig+1−1

@@ -332,7 +332,7 @@ config HIGHMEM · 331 unmodified lines
332	select KMAP_LOCAL	332	select KMAP_LOCAL
333	default y	333	default y
334		334
335	config FORCE_MAX_ZONEORDER	335	config ARCH_FORCE_MAX_ORDER
336	int "Maximum zone order"	336	int "Maximum zone order"
337	default "11"	337	default "11"
338		338

arch/ia64/Kconfig+1−1

@@ -200,7 +200,7 @@ config IA64_CYCLONE · 199 unmodified lines
200	Say Y here to enable support for IBM EXA Cyclone time source.	200	Say Y here to enable support for IBM EXA Cyclone time source.
201	If you're unsure, answer N.	201	If you're unsure, answer N.
202		202
203	config FORCE_MAX_ZONEORDER	203	config ARCH_FORCE_MAX_ORDER
204	int "MAX_ORDER (11 - 17)" if !HUGETLB_PAGE	204	int "MAX_ORDER (11 - 17)" if !HUGETLB_PAGE
205	range 11 17 if !HUGETLB_PAGE	205	range 11 17 if !HUGETLB_PAGE
206	default "17" if HUGETLB_PAGE	206	default "17" if HUGETLB_PAGE

arch/ia64/include/asm/sparsemem.h+3−3

@@ -11,10 +11,10 @@ · 10 unmodified lines
11		11
12	#define SECTION_SIZE_BITS (30)	12	#define SECTION_SIZE_BITS (30)
13	#define MAX_PHYSMEM_BITS (50)	13	#define MAX_PHYSMEM_BITS (50)
14	#ifdef CONFIG_FORCE_MAX_ZONEORDER	14	#ifdef CONFIG_ARCH_FORCE_MAX_ORDER
15	#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS)	15	#if ((CONFIG_ARCH_FORCE_MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS)
16	#undef SECTION_SIZE_BITS	16	#undef SECTION_SIZE_BITS
17	#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)	17	#define SECTION_SIZE_BITS (CONFIG_ARCH_FORCE_MAX_ORDER - 1 + PAGE_SHIFT)
18	#endif	18	#endif
19	#endif	19	#endif
20		20

arch/loongarch/Kconfig+1−1

@@ -377,7 +377,7 @@ config NODES_SHIFT · 376 unmodified lines
377	default "6"	377	default "6"
378	depends on NUMA	378	depends on NUMA
379		379
380	config FORCE_MAX_ZONEORDER	380	config ARCH_FORCE_MAX_ORDER
381	int "Maximum zone order"	381	int "Maximum zone order"
382	range 14 64 if PAGE_SIZE_64KB	382	range 14 64 if PAGE_SIZE_64KB
383	default "14" if PAGE_SIZE_64KB	383	default "14" if PAGE_SIZE_64KB

arch/m68k/Kconfig.cpu+1−1

@@ -397,7 +397,7 @@ config SINGLE_MEMORY_CHUNK · 396 unmodified lines
397	order" to save memory that could be wasted for unused memory map.	397	order" to save memory that could be wasted for unused memory map.
398	Say N if not sure.	398	Say N if not sure.
399		399
400	config FORCE_MAX_ZONEORDER	400	config ARCH_FORCE_MAX_ORDER
401	int "Maximum zone order" if ADVANCED	401	int "Maximum zone order" if ADVANCED
402	depends on !SINGLE_MEMORY_CHUNK	402	depends on !SINGLE_MEMORY_CHUNK
403	default "11"	403	default "11"

arch/mips/Kconfig+1−1

@@ -2140,7 +2140,7 @@ config PAGE_SIZE_64KB · 2139 unmodified lines
2140		2140
2141	endchoice	2141	endchoice
2142		2142
2143	config FORCE_MAX_ZONEORDER	2143	config ARCH_FORCE_MAX_ORDER
2144	int "Maximum zone order"	2144	int "Maximum zone order"
2145	range 14 64 if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB	2145	range 14 64 if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB
2146	default "14" if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB	2146	default "14" if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB

arch/mips/configs/db1xxx_defconfig+0−1

@@ -9,7 +9,6 @@ CONFIG_HIGH_RES_TIMERS=y · 8 unmodified lines
9	CONFIG_LOG_BUF_SHIFT=16	9	CONFIG_LOG_BUF_SHIFT=16
10	CONFIG_CGROUPS=y	10	CONFIG_CGROUPS=y
11	CONFIG_MEMCG=y	11	CONFIG_MEMCG=y
12	CONFIG_MEMCG_SWAP=y
13	CONFIG_BLK_CGROUP=y	12	CONFIG_BLK_CGROUP=y
14	CONFIG_CGROUP_SCHED=y	13	CONFIG_CGROUP_SCHED=y
15	CONFIG_CFS_BANDWIDTH=y	14	CONFIG_CFS_BANDWIDTH=y

arch/mips/configs/generic_defconfig+0−1

@@ -3,7 +3,6 @@ CONFIG_NO_HZ_IDLE=y · 2 unmodified lines
3	CONFIG_IKCONFIG=y	3	CONFIG_IKCONFIG=y
4	CONFIG_IKCONFIG_PROC=y	4	CONFIG_IKCONFIG_PROC=y
5	CONFIG_MEMCG=y	5	CONFIG_MEMCG=y
6	CONFIG_MEMCG_SWAP=y
7	CONFIG_BLK_CGROUP=y	6	CONFIG_BLK_CGROUP=y
8	CONFIG_CFS_BANDWIDTH=y	7	CONFIG_CFS_BANDWIDTH=y
9	CONFIG_RT_GROUP_SCHED=y	8	CONFIG_RT_GROUP_SCHED=y

arch/mips/include/uapi/asm/mman.h+2−0

@@ -103,6 +103,8 @@ · 102 unmodified lines
103		103
104	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */	104	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
105		105
		106	#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
		107
106	/* compatibility flags */	108	/* compatibility flags */
107	#define MAP_FILE 0	109	#define MAP_FILE 0
108		110

arch/nios2/Kconfig+1−1

@@ -44,7 +44,7 @@ menu "Kernel features" · 43 unmodified lines
44		44
45	source "kernel/Kconfig.hz"	45	source "kernel/Kconfig.hz"
46		46
47	config FORCE_MAX_ZONEORDER	47	config ARCH_FORCE_MAX_ORDER
48	int "Maximum zone order"	48	int "Maximum zone order"
49	range 9 20	49	range 9 20
50	default "11"	50	default "11"

arch/parisc/include/uapi/asm/mman.h+2−0

@@ -70,6 +70,8 @@ · 69 unmodified lines
70	#define MADV_WIPEONFORK 71 /* Zero memory on fork, child only */	70	#define MADV_WIPEONFORK 71 /* Zero memory on fork, child only */
71	#define MADV_KEEPONFORK 72 /* Undo MADV_WIPEONFORK */	71	#define MADV_KEEPONFORK 72 /* Undo MADV_WIPEONFORK */
72		72
		73	#define MADV_COLLAPSE 73 /* Synchronous hugepage collapse */
		74
73	#define MADV_HWPOISON 100 /* poison a page for testing */	75	#define MADV_HWPOISON 100 /* poison a page for testing */
74	#define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */	76	#define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */
75		77

arch/parisc/kernel/cache.c+7−2

 {
 	struct vm_area_struct *vma;
 	unsigned long usize = 0;
+	VMA_ITERATOR(vmi, mm, 0);
-	for (vma = mm->mmap; vma && usize < parisc_cache_flush_threshold; vma = vma->vm_next)
+	for_each_vma(vmi, vma) {
+		if (usize >= parisc_cache_flush_threshold)
+			break;
 		usize += vma->vm_end - vma->vm_start;
+	}
 	return usize;
 }
 void flush_cache_mm(struct mm_struct *mm)
 {
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
 	/*
 	 * Flushing the whole cache on each cpu takes forever on
 	}
 	/* Flush mm */
-	for (vma = mm->mmap; vma; vma = vma->vm_next)
+	for_each_vma(vmi, vma)
 		flush_cache_pages(vma, vma->vm_start, vma->vm_end);
 }

arch/powerpc/Kconfig+1−1

@@ -846,7 +846,7 @@ config DATA_SHIFT · 845 unmodified lines
846	in that case. If PIN_TLB is selected, it must be aligned to 8M as	846	in that case. If PIN_TLB is selected, it must be aligned to 8M as
847	8M pages will be pinned.	847	8M pages will be pinned.
848		848
849	config FORCE_MAX_ZONEORDER	849	config ARCH_FORCE_MAX_ORDER
850	int "Maximum zone order"	850	int "Maximum zone order"
851	range 8 9 if PPC64 && PPC_64K_PAGES	851	range 8 9 if PPC64 && PPC_64K_PAGES
852	default "9" if PPC64 && PPC_64K_PAGES	852	default "9" if PPC64 && PPC_64K_PAGES

arch/powerpc/configs/85xx/ge_imp3a_defconfig+1−1

@@ -30,7 +30,7 @@ CONFIG_PREEMPT=y · 29 unmodified lines
30	# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set	30	# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
31	CONFIG_BINFMT_MISC=m	31	CONFIG_BINFMT_MISC=m
32	CONFIG_MATH_EMULATION=y	32	CONFIG_MATH_EMULATION=y
33	CONFIG_FORCE_MAX_ZONEORDER=17	33	CONFIG_ARCH_FORCE_MAX_ORDER=17
34	CONFIG_PCI=y	34	CONFIG_PCI=y
35	CONFIG_PCIEPORTBUS=y	35	CONFIG_PCIEPORTBUS=y
36	CONFIG_PCI_MSI=y	36	CONFIG_PCI_MSI=y

arch/powerpc/configs/fsl-emb-nonhw.config+1−1

@@ -41,7 +41,7 @@ CONFIG_FIXED_PHY=y · 40 unmodified lines
41	CONFIG_FONT_8x16=y	41	CONFIG_FONT_8x16=y
42	CONFIG_FONT_8x8=y	42	CONFIG_FONT_8x8=y
43	CONFIG_FONTS=y	43	CONFIG_FONTS=y
44	CONFIG_FORCE_MAX_ZONEORDER=13	44	CONFIG_ARCH_FORCE_MAX_ORDER=13
45	CONFIG_FRAMEBUFFER_CONSOLE=y	45	CONFIG_FRAMEBUFFER_CONSOLE=y
46	CONFIG_FRAME_WARN=1024	46	CONFIG_FRAME_WARN=1024
47	CONFIG_FTL=y	47	CONFIG_FTL=y

arch/powerpc/configs/powernv_defconfig+0−1

@@ -17,7 +17,6 @@ CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 · 16 unmodified lines
17	CONFIG_NUMA_BALANCING=y	17	CONFIG_NUMA_BALANCING=y
18	CONFIG_CGROUPS=y	18	CONFIG_CGROUPS=y
19	CONFIG_MEMCG=y	19	CONFIG_MEMCG=y
20	CONFIG_MEMCG_SWAP=y
21	CONFIG_CGROUP_SCHED=y	20	CONFIG_CGROUP_SCHED=y
22	CONFIG_CGROUP_FREEZER=y	21	CONFIG_CGROUP_FREEZER=y
23	CONFIG_CPUSETS=y	22	CONFIG_CPUSETS=y

arch/powerpc/configs/pseries_defconfig+0−1

@@ -18,7 +18,6 @@ CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 · 17 unmodified lines
18	CONFIG_NUMA_BALANCING=y	18	CONFIG_NUMA_BALANCING=y
19	CONFIG_CGROUPS=y	19	CONFIG_CGROUPS=y
20	CONFIG_MEMCG=y	20	CONFIG_MEMCG=y
21	CONFIG_MEMCG_SWAP=y
22	CONFIG_CGROUP_SCHED=y	21	CONFIG_CGROUP_SCHED=y
23	CONFIG_CGROUP_FREEZER=y	22	CONFIG_CGROUP_FREEZER=y
24	CONFIG_CPUSETS=y	23	CONFIG_CPUSETS=y

arch/powerpc/kernel/vdso.c+3−3

@@ -115,18 +115,18 @@ struct vdso_data arch_get_vdso_data(void vvar_page) · 114 unmodified lines
115	int vdso_join_timens(struct task_struct task, struct time_namespace ns)	115	int vdso_join_timens(struct task_struct task, struct time_namespace ns)
116	{	116	{
117	struct mm_struct *mm = task->mm;	117	struct mm_struct *mm = task->mm;
		118	VMA_ITERATOR(vmi, mm, 0);
118	struct vm_area_struct *vma;	119	struct vm_area_struct *vma;
119		120
120	mmap_read_lock(mm);	121	mmap_read_lock(mm);
121		122	for_each_vma(vmi, vma) {
122	for (vma = mm->mmap; vma; vma = vma->vm_next) {
123	unsigned long size = vma->vm_end - vma->vm_start;	123	unsigned long size = vma->vm_end - vma->vm_start;
124		124
125	if (vma_is_special_mapping(vma, &vvar_spec))	125	if (vma_is_special_mapping(vma, &vvar_spec))
126	zap_page_range(vma, vma->vm_start, size);	126	zap_page_range(vma, vma->vm_start, size);
127	}	127	}
128
129	mmap_read_unlock(mm);	128	mmap_read_unlock(mm);
		129
130	return 0;	130	return 0;
131	}	131	}
132		132

arch/powerpc/mm/book3s32/tlb.c+6−5

 void hash__flush_tlb_mm(struct mm_struct *mm)
 {
 	struct vm_area_struct *mp;
+	VMA_ITERATOR(vmi, mm, 0);
 	/*
-	 * It is safe to go down the mm's list of vmas when called
+	 * It is safe to iterate the vmas when called from dup_mmap,
-	 * from dup_mmap, holding mmap_lock.  It would also be safe from
+	 * holding mmap_lock.  It would also be safe from unmap_region
-	 * unmap_region or exit_mmap, but not from vmtruncate on SMP -
+	 * or exit_mmap, but not from vmtruncate on SMP - but it seems
-	 * but it seems dup_mmap is the only SMP case which gets here.
+	 * dup_mmap is the only SMP case which gets here.
 	 */
-	for (mp = mm->mmap; mp != NULL; mp = mp->vm_next)
+	for_each_vma(vmi, mp)
 		hash__flush_range(mp->vm_mm, mp->vm_start, mp->vm_end);
 }
 EXPORT_SYMBOL(hash__flush_tlb_mm);

arch/powerpc/mm/book3s64/subpage_prot.c+2−11

 				    unsigned long len)
 {
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, addr);
 	/*
 	 * We don't try too hard, we just mark all the vma in that range
 	 * VM_NOHUGEPAGE and split them.
 	 */
-	vma = find_vma(mm, addr);
+	for_each_vma_range(vmi, vma, addr + len) {
-	/*
-	 * If the range is in unmapped range, just return
-	 */
-	if (vma && ((addr + len) <= vma->vm_start))
-		return;
-	while (vma) {
-		if (vma->vm_start >= (addr + len))
-			break;
 		vma->vm_flags |= VM_NOHUGEPAGE;
 		walk_page_vma(vma, &subpage_walk_ops, NULL);
-		vma = vma->vm_next;
 	}
 }
 #else

arch/riscv/kernel/vdso.c+2−1

 {
 	struct mm_struct *mm = task->mm;
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
 	struct __vdso_info *vdso_info = mm->context.vdso_info;
 	mmap_read_lock(mm);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		unsigned long size = vma->vm_end - vma->vm_start;
 		if (vma_is_special_mapping(vma, vdso_info->dm))

arch/s390/kernel/vdso.c+2−1

 int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
 {
 	struct mm_struct *mm = task->mm;
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
 	mmap_read_lock(mm);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		unsigned long size = vma->vm_end - vma->vm_start;
 		if (!vma_is_special_mapping(vma, &vvar_mapping))

arch/s390/lib/uaccess.c+2−1

 	might_fault();
 	if (!should_fail_usercopy()) {
-		instrument_copy_from_user(to, from, n);
+		instrument_copy_from_user_before(to, from, n);
 		res = raw_copy_from_user_key(to, from, n, key);
+		instrument_copy_from_user_after(to, from, n, res);
 	}
 	if (unlikely(res))
 		memset(to + (n - res), 0, res);

arch/s390/mm/gmap.c+4−2

 static inline void thp_split_mm(struct mm_struct *mm)
 {
 	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
-	for (vma = mm->mmap; vma != NULL; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		vma->vm_flags &= ~VM_HUGEPAGE;
 		vma->vm_flags |= VM_NOHUGEPAGE;
 		walk_page_vma(vma, &thp_split_walk_ops, NULL);
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;
 	int ret;
+	VMA_ITERATOR(vmi, mm, 0);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		ret = ksm_madvise(vma, vma->vm_start, vma->vm_end,
 				  MADV_UNMERGEABLE, &vma->vm_flags);
 		if (ret)

arch/s390/mm/hugetlbpage.c+0−10

@@ -237,16 +237,6 @@ int pud_huge(pud_t pud) · 236 unmodified lines
237	return pud_large(pud);	237	return pud_large(pud);
238	}	238	}
239		239
240	struct page *
241	follow_huge_pud(struct mm_struct *mm, unsigned long address,
242	pud_t *pud, int flags)
243	{
244	if (flags & FOLL_GET)
245	return NULL;
246
247	return pud_page(*pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT);
248	}
249
250	bool __init arch_hugetlb_valid_size(unsigned long size)	240	bool __init arch_hugetlb_valid_size(unsigned long size)
251	{	241	{
252	if (MACHINE_HAS_EDAT1 && size == PMD_SIZE)	242	if (MACHINE_HAS_EDAT1 && size == PMD_SIZE)

arch/sh/configs/ecovec24_defconfig+1−1

@@ -8,7 +8,7 @@ CONFIG_MODULES=y · 7 unmodified lines
8	CONFIG_MODULE_UNLOAD=y	8	CONFIG_MODULE_UNLOAD=y
9	# CONFIG_BLK_DEV_BSG is not set	9	# CONFIG_BLK_DEV_BSG is not set
10	CONFIG_CPU_SUBTYPE_SH7724=y	10	CONFIG_CPU_SUBTYPE_SH7724=y
11	CONFIG_FORCE_MAX_ZONEORDER=12	11	CONFIG_ARCH_FORCE_MAX_ORDER=12
12	CONFIG_MEMORY_SIZE=0x10000000	12	CONFIG_MEMORY_SIZE=0x10000000
13	CONFIG_FLATMEM_MANUAL=y	13	CONFIG_FLATMEM_MANUAL=y
14	CONFIG_SH_ECOVEC=y	14	CONFIG_SH_ECOVEC=y

arch/sh/configs/sdk7786_defconfig+0−1

@@ -16,7 +16,6 @@ CONFIG_CPUSETS=y · 15 unmodified lines
16	# CONFIG_PROC_PID_CPUSET is not set	16	# CONFIG_PROC_PID_CPUSET is not set
17	CONFIG_CGROUP_CPUACCT=y	17	CONFIG_CGROUP_CPUACCT=y
18	CONFIG_CGROUP_MEMCG=y	18	CONFIG_CGROUP_MEMCG=y
19	CONFIG_CGROUP_MEMCG_SWAP=y
20	CONFIG_CGROUP_SCHED=y	19	CONFIG_CGROUP_SCHED=y
21	CONFIG_RT_GROUP_SCHED=y	20	CONFIG_RT_GROUP_SCHED=y
22	CONFIG_BLK_CGROUP=y	21	CONFIG_BLK_CGROUP=y

arch/sh/configs/urquell_defconfig+0−1

@@ -14,7 +14,6 @@ CONFIG_CPUSETS=y · 13 unmodified lines
14	# CONFIG_PROC_PID_CPUSET is not set	14	# CONFIG_PROC_PID_CPUSET is not set
15	CONFIG_CGROUP_CPUACCT=y	15	CONFIG_CGROUP_CPUACCT=y
16	CONFIG_CGROUP_MEMCG=y	16	CONFIG_CGROUP_MEMCG=y
17	CONFIG_CGROUP_MEMCG_SWAP=y
18	CONFIG_CGROUP_SCHED=y	17	CONFIG_CGROUP_SCHED=y
19	CONFIG_RT_GROUP_SCHED=y	18	CONFIG_RT_GROUP_SCHED=y
20	CONFIG_BLK_DEV_INITRD=y	19	CONFIG_BLK_DEV_INITRD=y

arch/sh/mm/Kconfig+1−1

@@ -18,7 +18,7 @@ config PAGE_OFFSET · 17 unmodified lines
18	default "0x80000000" if MMU	18	default "0x80000000" if MMU
19	default "0x00000000"	19	default "0x00000000"
20		20
21	config FORCE_MAX_ZONEORDER	21	config ARCH_FORCE_MAX_ORDER
22	int "Maximum zone order"	22	int "Maximum zone order"
23	range 9 64 if PAGE_SIZE_16KB	23	range 9 64 if PAGE_SIZE_16KB
24	default "9" if PAGE_SIZE_16KB	24	default "9" if PAGE_SIZE_16KB

arch/sparc/Kconfig+1−1

@@ -269,7 +269,7 @@ config ARCH_SPARSEMEM_ENABLE · 268 unmodified lines
269	config ARCH_SPARSEMEM_DEFAULT	269	config ARCH_SPARSEMEM_DEFAULT
270	def_bool y if SPARC64	270	def_bool y if SPARC64
271		271
272	config FORCE_MAX_ZONEORDER	272	config ARCH_FORCE_MAX_ORDER
273	int "Maximum zone order"	273	int "Maximum zone order"
274	default "13"	274	default "13"
275	help	275	help

arch/um/kernel/tlb.c+6−8

 void flush_tlb_mm(struct mm_struct *mm)
 {
-	struct vm_area_struct *vma = mm->mmap;
+	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
-	while (vma != NULL) {
+	for_each_vma(vmi, vma)
 		fix_range(mm, vma->vm_start, vma->vm_end, 0);
-		vma = vma->vm_next;
-	}
 }
 void force_flush_all(void)
 {
 	struct mm_struct *mm = current->mm;
-	struct vm_area_struct *vma = mm->mmap;
+	struct vm_area_struct *vma;
+	VMA_ITERATOR(vmi, mm, 0);
-	while (vma != NULL) {
+	for_each_vma(vmi, vma)
 		fix_range(mm, vma->vm_start, vma->vm_end, 1);
-		vma = vma->vm_next;
-	}
 }

arch/x86/Kconfig+9−1

 	select ARCH_HAS_PMEM_API		if X86_64
 	select ARCH_HAS_PTE_DEVMAP		if X86_64
 	select ARCH_HAS_PTE_SPECIAL
+	select ARCH_HAS_NONLEAF_PMD_YOUNG	if PGTABLE_LEVELS > 2
 	select ARCH_HAS_UACCESS_FLUSHCACHE	if X86_64
 	select ARCH_HAS_COPY_MC			if X86_64
 	select ARCH_HAS_SET_MEMORY
 	select CLKEVT_I8253
 	select CLOCKSOURCE_VALIDATE_LAST_CYCLE
 	select CLOCKSOURCE_WATCHDOG
-	select DCACHE_WORD_ACCESS
+	# Word-size accesses may read uninitialized data past the trailing \0
+	# in strings and cause false KMSAN reports.
+	select DCACHE_WORD_ACCESS		if !KMSAN
 	select DYNAMIC_SIGFRAME
 	select EDAC_ATOMIC_SCRUB
 	select EDAC_SUPPORT
 	select HAVE_ARCH_KASAN			if X86_64
 	select HAVE_ARCH_KASAN_VMALLOC		if X86_64
 	select HAVE_ARCH_KFENCE
+	select HAVE_ARCH_KMSAN			if X86_64
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_MMAP_RND_BITS		if MMU
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS	if MMU && COMPAT
 	def_bool y
 	depends on ISA_DMA_API
+config GENERIC_CSUM
+	bool
+	default y if KMSAN || KASAN
 config GENERIC_BUG
 	def_bool y
 	depends on BUG

arch/x86/boot/Makefile+1−0

@@ -12,6 +12,7 @@ · 11 unmodified lines
12	# Sanitizer runtimes are unavailable and cannot be linked for early boot code.	12	# Sanitizer runtimes are unavailable and cannot be linked for early boot code.
13	KASAN_SANITIZE := n	13	KASAN_SANITIZE := n
14	KCSAN_SANITIZE := n	14	KCSAN_SANITIZE := n
		15	KMSAN_SANITIZE := n
15	OBJECT_FILES_NON_STANDARD := y	16	OBJECT_FILES_NON_STANDARD := y
16		17
17	# Kernel does not boot with kcov instrumentation here.	18	# Kernel does not boot with kcov instrumentation here.

arch/x86/boot/compressed/Makefile+1−0

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	# Sanitizer runtimes are unavailable and cannot be linked for early boot code.	20	# Sanitizer runtimes are unavailable and cannot be linked for early boot code.
21	KASAN_SANITIZE := n	21	KASAN_SANITIZE := n
22	KCSAN_SANITIZE := n	22	KCSAN_SANITIZE := n
		23	KMSAN_SANITIZE := n
23	OBJECT_FILES_NON_STANDARD := y	24	OBJECT_FILES_NON_STANDARD := y
24		25
25	# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in.	26	# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in.

arch/x86/entry/vdso/Makefile+3−0

@@ -11,6 +11,9 @@ include $(srctree)/lib/vdso/Makefile · 10 unmodified lines
11		11
12	# Sanitizer runtimes are unavailable and cannot be linked here.	12	# Sanitizer runtimes are unavailable and cannot be linked here.
13	KASAN_SANITIZE := n	13	KASAN_SANITIZE := n
		14	KMSAN_SANITIZE_vclock_gettime.o := n
		15	KMSAN_SANITIZE_vgetcpu.o := n
		16
14	UBSAN_SANITIZE := n	17	UBSAN_SANITIZE := n
15	KCSAN_SANITIZE := n	18	KCSAN_SANITIZE := n
16	OBJECT_FILES_NON_STANDARD := y	19	OBJECT_FILES_NON_STANDARD := y

arch/x86/entry/vdso/vma.c+5−4

@@ -127,17 +127,17 @@ int vdso_join_timens(struct task_struct task, struct time_namespace ns) · 126 unmodified lines
127	{	127	{
128	struct mm_struct *mm = task->mm;	128	struct mm_struct *mm = task->mm;
129	struct vm_area_struct *vma;	129	struct vm_area_struct *vma;
		130	VMA_ITERATOR(vmi, mm, 0);
130		131
131	mmap_read_lock(mm);	132	mmap_read_lock(mm);
132		133	for_each_vma(vmi, vma) {
133	for (vma = mm->mmap; vma; vma = vma->vm_next) {
134	unsigned long size = vma->vm_end - vma->vm_start;	134	unsigned long size = vma->vm_end - vma->vm_start;
135		135
136	if (vma_is_special_mapping(vma, &vvar_mapping))	136	if (vma_is_special_mapping(vma, &vvar_mapping))
137	zap_page_range(vma, vma->vm_start, size);	137	zap_page_range(vma, vma->vm_start, size);
138	}	138	}
139
140	mmap_read_unlock(mm);	139	mmap_read_unlock(mm);
		140
141	return 0;	141	return 0;
142	}	142	}
143	#else	143	#else
@@ -354,6 +354,7 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) · 210 unmodified lines
354	{	354	{
355	struct mm_struct *mm = current->mm;	355	struct mm_struct *mm = current->mm;
356	struct vm_area_struct *vma;	356	struct vm_area_struct *vma;
		357	VMA_ITERATOR(vmi, mm, 0);
357		358
358	mmap_write_lock(mm);	359	mmap_write_lock(mm);
359	/*	360	/*
@@ -363,7 +364,7 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) · 3 unmodified lines
363	* We could search vma near context.vdso, but it's a slowpath,	364	* We could search vma near context.vdso, but it's a slowpath,
364	* so let's explicitly check all VMAs to be completely sure.	365	* so let's explicitly check all VMAs to be completely sure.
365	*/	366	*/
366	for (vma = mm->mmap; vma; vma = vma->vm_next) {	367	for_each_vma(vmi, vma) {
367	if (vma_is_special_mapping(vma, &vdso_mapping) \|\|	368	if (vma_is_special_mapping(vma, &vdso_mapping) \|\|
368	vma_is_special_mapping(vma, &vvar_mapping)) {	369	vma_is_special_mapping(vma, &vvar_mapping)) {
369	mmap_write_unlock(mm);	370	mmap_write_unlock(mm);

arch/x86/include/asm/checksum.h+10−6

 /* SPDX-License-Identifier: GPL-2.0 */
-#define  _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1
+#ifdef CONFIG_GENERIC_CSUM
-#define HAVE_CSUM_COPY_USER
+# include <asm-generic/checksum.h>
-#define _HAVE_ARCH_CSUM_AND_COPY
-#ifdef CONFIG_X86_32
-# include <asm/checksum_32.h>
 #else
-# include <asm/checksum_64.h>
+# define  _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1
+# define HAVE_CSUM_COPY_USER
+# define _HAVE_ARCH_CSUM_AND_COPY
+# ifdef CONFIG_X86_32
+#  include <asm/checksum_32.h>
+# else
+#  include <asm/checksum_64.h>
+# endif
 #endif

arch/x86/include/asm/kmsan.h+87−0

@@ -0,0 +1,87 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* x86 KMSAN support.
	4	*
	5	* Copyright (C) 2022, Google LLC
	6	* Author: Alexander Potapenko <glider@google.com>
	7	*/
	8
	9	#ifndef _ASM_X86_KMSAN_H
	10	#define _ASM_X86_KMSAN_H
	11
	12	#ifndef MODULE
	13
	14	#include <asm/cpu_entry_area.h>
	15	#include <asm/processor.h>
	16	#include <linux/mmzone.h>
	17
	18	DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow);
	19	DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin);
	20
	21	/*
	22	* Functions below are declared in the header to make sure they are inlined.
	23	* They all are called from kmsan_get_metadata() for every memory access in
	24	* the kernel, so speed is important here.
	25	*/
	26
	27	/*
	28	* Compute metadata addresses for the CPU entry area on x86.
	29	*/
	30	static inline void arch_kmsan_get_meta_or_null(void addr, bool is_origin)
	31	{
	32	unsigned long addr64 = (unsigned long)addr;
	33	char *metadata_array;
	34	unsigned long off;
	35	int cpu;
	36
	37	if ((addr64 < CPU_ENTRY_AREA_BASE) \|\|
	38	(addr64 >= (CPU_ENTRY_AREA_BASE + CPU_ENTRY_AREA_MAP_SIZE)))
	39	return NULL;
	40	cpu = (addr64 - CPU_ENTRY_AREA_BASE) / CPU_ENTRY_AREA_SIZE;
	41	off = addr64 - (unsigned long)get_cpu_entry_area(cpu);
	42	if ((off < 0) \|\| (off >= CPU_ENTRY_AREA_SIZE))
	43	return NULL;
	44	metadata_array = is_origin ? cpu_entry_area_origin :
	45	cpu_entry_area_shadow;
	46	return &per_cpu(metadata_array[off], cpu);
	47	}
	48
	49	/*
	50	* Taken from arch/x86/mm/physaddr.h to avoid using an instrumented version.
	51	*/
	52	static inline bool kmsan_phys_addr_valid(unsigned long addr)
	53	{
	54	if (IS_ENABLED(CONFIG_PHYS_ADDR_T_64BIT))
	55	return !(addr >> boot_cpu_data.x86_phys_bits);
	56	else
	57	return true;
	58	}
	59
	60	/*
	61	* Taken from arch/x86/mm/physaddr.c to avoid using an instrumented version.
	62	*/
	63	static inline bool kmsan_virt_addr_valid(void *addr)
	64	{
	65	unsigned long x = (unsigned long)addr;
	66	unsigned long y = x - __START_KERNEL_map;
	67
	68	/* use the carry flag to determine if x was < __START_KERNEL_map */
	69	if (unlikely(x > y)) {
	70	x = y + phys_base;
	71
	72	if (y >= KERNEL_IMAGE_SIZE)
	73	return false;
	74	} else {
	75	x = y + (__START_KERNEL_map - PAGE_OFFSET);
	76
	77	/* carry flag will be set if starting x was >= PAGE_OFFSET */
	78	if ((x > y) \|\| !kmsan_phys_addr_valid(x))
	79	return false;
	80	}
	81
	82	return pfn_valid(x >> PAGE_SHIFT);
	83	}
	84
	85	#endif /* !MODULE */
	86
	87	#endif /* _ASM_X86_KMSAN_H */

arch/x86/include/asm/page_64.h+7−0

 #include <asm/cpufeatures.h>
 #include <asm/alternative.h>
+#include <linux/kmsan-checks.h>
 /* duplicated to the one in bootmem.h */
 extern unsigned long max_pfn;
 extern unsigned long phys_base;
 static inline void clear_page(void *page)
 {
+	/*
+	 * Clean up KMSAN metadata for the page being cleared. The assembly call
+	 * below clobbers @page, so we perform unpoisoning before it.
+	 */
+	kmsan_unpoison_memory(page, PAGE_SIZE);
 	alternative_call_2(clear_page_orig,
 			   clear_page_rep, X86_FEATURE_REP_GOOD,
 			   clear_page_erms, X86_FEATURE_ERMS,

arch/x86/include/asm/pgtable-3level.h+4−4

@@ -256,10 +256,10 @@ static inline pud_t native_pudp_get_and_clear(pud_t *pudp) · 255 unmodified lines
256	/* We always extract/encode the offset by shifting it all the way up, and then down again */	256	/* We always extract/encode the offset by shifting it all the way up, and then down again */
257	#define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS)	257	#define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS)
258		258
259	#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > 5)	259	#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS)
260	#define __swp_type(x) (((x).val) & 0x1f)	260	#define __swp_type(x) (((x).val) & ((1UL << SWP_TYPE_BITS) - 1))
261	#define __swp_offset(x) ((x).val >> 5)	261	#define __swp_offset(x) ((x).val >> SWP_TYPE_BITS)
262	#define __swp_entry(type, offset) ((swp_entry_t){(type) \| (offset) << 5})	262	#define __swp_entry(type, offset) ((swp_entry_t){(type) \| (offset) << SWP_TYPE_BITS})
263		263
264	/*	264	/*
265	* Normally, __swp_entry() converts from arch-independent swp_entry_t to	265	* Normally, __swp_entry() converts from arch-independent swp_entry_t to

arch/x86/include/asm/pgtable.h+5−4

 static inline int pmd_bad(pmd_t pmd)
 {
-	return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE;
+	return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) !=
+	       (_KERNPG_TABLE & ~_PAGE_ACCESSED);
 }
 static inline unsigned long pages_to_mb(unsigned long npg)
 	return boot_cpu_has_bug(X86_BUG_L1TF);
 }
-#define arch_faults_on_old_pte arch_faults_on_old_pte
+#define arch_has_hw_pte_young arch_has_hw_pte_young
-static inline bool arch_faults_on_old_pte(void)
+static inline bool arch_has_hw_pte_young(void)
 {
-	return false;
+	return true;
 }
 #ifdef CONFIG_PAGE_TABLE_CHECK

arch/x86/include/asm/pgtable_64_types.h+46−1

@@ -139,7 +139,52 @@ extern unsigned int ptrs_per_p4d; · 138 unmodified lines
139	# define VMEMMAP_START __VMEMMAP_BASE_L4	139	# define VMEMMAP_START __VMEMMAP_BASE_L4
140	#endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */	140	#endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */
141		141
142	#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1)	142	/*
		143	* End of the region for which vmalloc page tables are pre-allocated.
		144	* For non-KMSAN builds, this is the same as VMALLOC_END.
		145	* For KMSAN builds, VMALLOC_START..VMEMORY_END is 4 times bigger than
		146	* VMALLOC_START..VMALLOC_END (see below).
		147	*/
		148	#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1)
		149
		150	#ifndef CONFIG_KMSAN
		151	#define VMALLOC_END VMEMORY_END
		152	#else
		153	/*
		154	* In KMSAN builds vmalloc area is four times smaller, and the remaining 3/4
		155	* are used to keep the metadata for virtual pages. The memory formerly
		156	* belonging to vmalloc area is now laid out as follows:
		157	*
		158	* 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area
		159	* 2nd quarter: KMSAN_VMALLOC_SHADOW_START to
		160	* VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area shadow
		161	* 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to
		162	* VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area origins
		163	* 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START
		164	* - shadow for modules,
		165	* KMSAN_MODULES_ORIGIN_START to
		166	* KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for modules.
		167	*/
		168	#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2)
		169	#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1)
		170
		171	/*
		172	* vmalloc metadata addresses are calculated by adding shadow/origin offsets
		173	* to vmalloc address.
		174	*/
		175	#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE
		176	#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1)
		177
		178	#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_OFFSET)
		179	#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_OFFSET)
		180
		181	/*
		182	* The shadow/origin for modules are placed one by one in the last 1/4 of
		183	* vmalloc space.
		184	*/
		185	#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFFSET + 1)
		186	#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN)
		187	#endif /* CONFIG_KMSAN */
143		188
144	#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)	189	#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)
145	/* The module sections ends with the start of the fixmap */	190	/* The module sections ends with the start of the fixmap */

arch/x86/include/asm/sparsemem.h+2−0

@@ -2,6 +2,8 @@ · 1 unmodified line
2	#ifndef _ASM_X86_SPARSEMEM_H	2	#ifndef _ASM_X86_SPARSEMEM_H
3	#define _ASM_X86_SPARSEMEM_H	3	#define _ASM_X86_SPARSEMEM_H
4		4
		5	#include <linux/types.h>
		6
5	#ifdef CONFIG_SPARSEMEM	7	#ifdef CONFIG_SPARSEMEM
6	/*	8	/*
7	* generic non-linear memory support:	9	* generic non-linear memory support:

arch/x86/include/asm/string_64.h+21−2

    function. */
 #define __HAVE_ARCH_MEMCPY 1
+#if defined(__SANITIZE_MEMORY__)
+#undef memcpy
+void *__msan_memcpy(void *dst, const void *src, size_t size);
+#define memcpy __msan_memcpy
+#else
 extern void *memcpy(void *to, const void *from, size_t len);
+#endif
 extern void *__memcpy(void *to, const void *from, size_t len);
 #define __HAVE_ARCH_MEMSET
+#if defined(__SANITIZE_MEMORY__)
+extern void *__msan_memset(void *s, int c, size_t n);
+#undef memset
+#define memset __msan_memset
+#else
 void *memset(void *s, int c, size_t n);
+#endif
 void *__memset(void *s, int c, size_t n);
 #define __HAVE_ARCH_MEMSET16
 }
 #define __HAVE_ARCH_MEMMOVE
+#if defined(__SANITIZE_MEMORY__)
+#undef memmove
+void *__msan_memmove(void *dest, const void *src, size_t len);
+#define memmove __msan_memmove
+#else
 void *memmove(void *dest, const void *src, size_t count);
+#endif
 void *__memmove(void *dest, const void *src, size_t count);
 int memcmp(const void *cs, const void *ct, size_t count);
 char *strcat(char *dest, const char *src);
 int strcmp(const char *cs, const char *ct);
-#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)
+#if (defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__))
 /*
  * For files that not instrumented (e.g. mm/slub.c) we
  * should use not instrumented version of mem* functions.
 #undef memcpy
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
+#undef memmove
 #define memmove(dst, src, len) __memmove(dst, src, len)
+#undef memset
 #define memset(s, c, n) __memset(s, c, n)
 #ifndef __NO_FORTIFY

arch/x86/include/asm/uaccess.h+15−7

  * User space memory access functions
  */
 #include <linux/compiler.h>
+#include <linux/instrumented.h>
 #include <linux/kasan-checks.h>
 #include <linux/string.h>
 #include <asm/asm.h>
 		     : "=a" (__ret_gu), "=r" (__val_gu),		\
 			ASM_CALL_CONSTRAINT				\
 		     : "0" (ptr), "i" (sizeof(*(ptr))));		\
+	instrument_get_user(__val_gu);					\
 	(x) = (__force __typeof__(*(ptr))) __val_gu;			\
 	__builtin_expect(__ret_gu, 0);					\
 })
 	int __ret_pu;							\
 	void __user *__ptr_pu;						\
 	register __typeof__(*(ptr)) __val_pu asm("%"_ASM_AX);		\
-	__chk_user_ptr(ptr);						\
+	__typeof__(*(ptr)) __x = (x); /* eval x once */			\
-	__ptr_pu = (ptr);						\
+	__typeof__(ptr) __ptr = (ptr); /* eval ptr once */		\
-	__val_pu = (x);							\
+	__chk_user_ptr(__ptr);						\
+	__ptr_pu = __ptr;						\
+	__val_pu = __x;							\
 	asm volatile("call __" #fn "_%P[size]"				\
 		     : "=c" (__ret_pu),					\
 			ASM_CALL_CONSTRAINT				\
 		       "r" (__val_pu),					\
 		       [size] "i" (sizeof(*(ptr)))			\
 		     :"ebx");						\
+	instrument_put_user(__x, __ptr, sizeof(*(ptr)));		\
 	__builtin_expect(__ret_pu, 0);					\
 })
 #define __put_user_size(x, ptr, size, label)				\
 do {									\
+	__typeof__(*(ptr)) __x = (x); /* eval x once */			\
 	__chk_user_ptr(ptr);						\
 	switch (size) {							\
 	case 1:								\
-		__put_user_goto(x, ptr, "b", "iq", label);		\
+		__put_user_goto(__x, ptr, "b", "iq", label);		\
 		break;							\
 	case 2:								\
-		__put_user_goto(x, ptr, "w", "ir", label);		\
+		__put_user_goto(__x, ptr, "w", "ir", label);		\
 		break;							\
 	case 4:								\
-		__put_user_goto(x, ptr, "l", "ir", label);		\
+		__put_user_goto(__x, ptr, "l", "ir", label);		\
 		break;							\
 	case 8:								\
-		__put_user_goto_u64(x, ptr, label);			\
+		__put_user_goto_u64(__x, ptr, label);			\
 		break;							\
 	default:							\
 		__put_user_bad();					\
 	}								\
+	instrument_put_user(__x, ptr, size);				\
 } while (0)
 #ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT
 	default:							\
 		(x) = __get_user_bad();					\
 	}								\
+	instrument_get_user(x);						\
 } while (0)
 #define __get_user_asm(x, addr, itype, ltype, label)			\

arch/x86/kernel/Makefile+2−0

@@ -29,6 +29,8 @@ KASAN_SANITIZE_sev.o := n · 28 unmodified lines
29	# With some compiler versions the generated code results in boot hangs, caused	29	# With some compiler versions the generated code results in boot hangs, caused
30	# by several compilation units. To be safe, disable all instrumentation.	30	# by several compilation units. To be safe, disable all instrumentation.
31	KCSAN_SANITIZE := n	31	KCSAN_SANITIZE := n
		32	KMSAN_SANITIZE_head$(BITS).o := n
		33	KMSAN_SANITIZE_nmi.o := n
32		34
33	# If instrumentation of this dir is enabled, boot hangs during first second.	35	# If instrumentation of this dir is enabled, boot hangs during first second.
34	# Probably could be more selective here, but note that files related to irqs,	36	# Probably could be more selective here, but note that files related to irqs,

arch/x86/kernel/cpu/Makefile+1−0

@@ -12,6 +12,7 @@ endif · 11 unmodified lines
12	# If these files are instrumented, boot hangs during the first second.	12	# If these files are instrumented, boot hangs during the first second.
13	KCOV_INSTRUMENT_common.o := n	13	KCOV_INSTRUMENT_common.o := n
14	KCOV_INSTRUMENT_perf_event.o := n	14	KCOV_INSTRUMENT_perf_event.o := n
		15	KMSAN_SANITIZE_common.o := n
15		16
16	# As above, instrumenting secondary CPU boot code causes boot hangs.	17	# As above, instrumenting secondary CPU boot code causes boot hangs.
17	KCSAN_SANITIZE_common.o := n	18	KCSAN_SANITIZE_common.o := n

arch/x86/kernel/dumpstack.c+6−0

 	}
 }
+/*
+ * This function reads pointers from the stack and dereferences them. The
+ * pointers may not have their KMSAN shadow set up properly, which may result
+ * in false positive reports. Disable instrumentation to avoid those.
+ */
+__no_kmsan_checks
 static void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
 			unsigned long *stack, const char *log_lvl)
 {

arch/x86/kernel/process_64.c+1−0

@@ -553,6 +553,7 @@ void compat_start_thread(struct pt_regs *regs, u32 new_ip, u32 new_sp, bool x32) · 552 unmodified lines
553	* Kprobes not supported here. Set the probe on schedule instead.	553	* Kprobes not supported here. Set the probe on schedule instead.
554	* Function graph tracer not supported too.	554	* Function graph tracer not supported too.
555	*/	555	*/
		556	__no_kmsan_checks
556	__visible __notrace_funcgraph struct task_struct *	557	__visible __notrace_funcgraph struct task_struct *
557	__switch_to(struct task_struct prev_p, struct task_struct next_p)	558	__switch_to(struct task_struct prev_p, struct task_struct next_p)
558	{	559	{

arch/x86/kernel/tboot.c+1−1

@@ -95,7 +95,7 @@ void __init tboot_probe(void) · 94 unmodified lines
95		95
96	static pgd_t *tboot_pg_dir;	96	static pgd_t *tboot_pg_dir;
97	static struct mm_struct tboot_mm = {	97	static struct mm_struct tboot_mm = {
98	.mm_rb = RB_ROOT,	98	.mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, tboot_mm.mmap_lock),
99	.pgd = swapper_pg_dir,	99	.pgd = swapper_pg_dir,
100	.mm_users = ATOMIC_INIT(2),	100	.mm_users = ATOMIC_INIT(2),
101	.mm_count = ATOMIC_INIT(1),	101	.mm_count = ATOMIC_INIT(1),

arch/x86/kernel/unwind_frame.c+11−0

 }
 #endif
+/*
+ * While walking the stack, KMSAN may stomp on stale locals from other
+ * functions that were marked as uninitialized upon function exit, and
+ * now hold the call frame information for the current function (e.g. the frame
+ * pointer). Because KMSAN does not specifically mark call frames as
+ * initialized, false positive reports are possible. To prevent such reports,
+ * we mark the functions scanning the stack (here and below) with
+ * __no_kmsan_checks.
+ */
+__no_kmsan_checks
 static bool update_stack_state(struct unwind_state *state,
 			       unsigned long *next_bp)
 {
 	return true;
 }
+__no_kmsan_checks
 bool unwind_next_frame(struct unwind_state *state)
 {
 	struct pt_regs *regs;

arch/x86/lib/Makefile+2−0

 endif
 else
         obj-y += iomap_copy_64.o
+ifneq ($(CONFIG_GENERIC_CSUM),y)
         lib-y += csum-partial_64.o csum-copy_64.o csum-wrappers_64.o
+endif
         lib-y += clear_page_64.o copy_page_64.o
         lib-y += memmove_64.o memset_64.o
         lib-y += copy_user_64.o

arch/x86/lib/iomem.c+5−0

 #include <linux/string.h>
 #include <linux/module.h>
 #include <linux/io.h>
+#include <linux/kmsan-checks.h>
 #define movs(type,to,from) \
 	asm volatile("movs" type:"=&D" (to), "=&S" (from):"0" (to), "1" (from):"memory")
 		n-=2;
 	}
 	rep_movs(to, (const void *)from, n);
+	/* KMSAN must treat values read from devices as initialized. */
+	kmsan_unpoison_memory(to, n);
 }
 static void string_memcpy_toio(volatile void __iomem *to, const void *from, size_t n)
 	if (unlikely(!n))
 		return;
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(from, n);
 	/* Align any unaligned destination IO */
 	if (unlikely(1 & (unsigned long)to)) {
 		movs("b", to, from);

arch/x86/mm/Makefile+5−0

 # Disable KCSAN entirely, because otherwise we get warnings that some functions
 # reference __initdata sections.
 KCSAN_SANITIZE := n
+# Avoid recursion by not calling KMSAN hooks for CEA code.
+KMSAN_SANITIZE_cpu_entry_area.o := n
 ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_mem_encrypt.o		= -pg
 KASAN_SANITIZE_kasan_init_$(BITS).o := n
 obj-$(CONFIG_KASAN)		+= kasan_init_$(BITS).o
+KMSAN_SANITIZE_kmsan_shadow.o	:= n
+obj-$(CONFIG_KMSAN)		+= kmsan_shadow.o
 obj-$(CONFIG_MMIOTRACE)		+= mmiotrace.o
 mmiotrace-y			:= kmmio.o pf_in.o mmio-mod.o
 obj-$(CONFIG_MMIOTRACE_TEST)	+= testmmiotrace.o

arch/x86/mm/fault.c+22−1

@@ -260,7 +260,7 @@ static noinline int vmalloc_fault(unsigned long address) · 259 unmodified lines
260	}	260	}
261	NOKPROBE_SYMBOL(vmalloc_fault);	261	NOKPROBE_SYMBOL(vmalloc_fault);
262		262
263	void arch_sync_kernel_mappings(unsigned long start, unsigned long end)	263	static void __arch_sync_kernel_mappings(unsigned long start, unsigned long end)
264	{	264	{
265	unsigned long addr;	265	unsigned long addr;
266		266
@@ -284,6 +284,27 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end) · 17 unmodified lines
284	}	284	}
285	}	285	}
286		286
		287	void arch_sync_kernel_mappings(unsigned long start, unsigned long end)
		288	{
		289	__arch_sync_kernel_mappings(start, end);
		290	#ifdef CONFIG_KMSAN
		291	/*
		292	* KMSAN maintains two additional metadata page mappings for the
		293	* [VMALLOC_START, VMALLOC_END) range. These mappings start at
		294	* KMSAN_VMALLOC_SHADOW_START and KMSAN_VMALLOC_ORIGIN_START and
		295	* have to be synced together with the vmalloc memory mapping.
		296	*/
		297	if (start >= VMALLOC_START && end < VMALLOC_END) {
		298	__arch_sync_kernel_mappings(
		299	start - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START,
		300	end - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START);
		301	__arch_sync_kernel_mappings(
		302	start - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START,
		303	end - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START);
		304	}
		305	#endif
		306	}
		307
287	static bool low_pfn(unsigned long pfn)	308	static bool low_pfn(unsigned long pfn)
288	{	309	{
289	return pfn < max_low_pfn;	310	return pfn < max_low_pfn;

arch/x86/mm/init.c+1−1

@@ -1054,7 +1054,7 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) · 1053 unmodified lines
1054	}	1054	}
1055		1055
1056	#ifdef CONFIG_SWAP	1056	#ifdef CONFIG_SWAP
1057	unsigned long max_swapfile_size(void)	1057	unsigned long arch_max_swapfile_size(void)
1058	{	1058	{
1059	unsigned long pages;	1059	unsigned long pages;
1060		1060

arch/x86/mm/init_64.c+1−1

@@ -1288,7 +1288,7 @@ static void __init preallocate_vmalloc_pages(void) · 1287 unmodified lines
1288	unsigned long addr;	1288	unsigned long addr;
1289	const char *lvl;	1289	const char *lvl;
1290		1290
1291	for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) {	1291	for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
1292	pgd_t *pgd = pgd_offset_k(addr);	1292	pgd_t *pgd = pgd_offset_k(addr);
1293	p4d_t *p4d;	1293	p4d_t *p4d;
1294	pud_t *pud;	1294	pud_t *pud;

arch/x86/mm/ioremap.c+3−0

 #include <linux/cc_platform.h>
 #include <linux/efi.h>
 #include <linux/pgtable.h>
+#include <linux/kmsan.h>
 #include <asm/set_memory.h>
 #include <asm/e820/api.h>
 		return;
 	}
+	kmsan_iounmap_page_range((unsigned long)addr,
+		(unsigned long)addr + get_vm_area_size(p));
 	memtype_free(p->phys_addr, p->phys_addr + get_vm_area_size(p));
 	/* Finally remove it */

arch/x86/mm/kmsan_shadow.c+20−0

@@ -0,0 +1,20 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/*
	3	* x86-specific bits of KMSAN shadow implementation.
	4	*
	5	* Copyright (C) 2022 Google LLC
	6	* Author: Alexander Potapenko <glider@google.com>
	7	*/
	8
	9	#include <asm/cpu_entry_area.h>
	10	#include <linux/percpu-defs.h>
	11
	12	/*
	13	* Addresses within the CPU entry area (including e.g. exception stacks) do not
	14	* have struct page entries corresponding to them, so they need separate
	15	* handling.
	16	* arch_kmsan_get_meta_or_null() (declared in the header) maps the addresses in
	17	* CPU entry area to addresses in cpu_entry_area_shadow/cpu_entry_area_origin.
	18	*/
	19	DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow);
	20	DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin);

arch/x86/mm/pgtable.c+4−1

 	return ret;
 }
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
 int pmdp_test_and_clear_young(struct vm_area_struct *vma,
 			      unsigned long addr, pmd_t *pmdp)
 {
 	return ret;
 }
+#endif
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
 int pudp_test_and_clear_young(struct vm_area_struct *vma,
 			      unsigned long addr, pud_t *pudp)
 {

arch/x86/realmode/rm/Makefile+1−0

@@ -10,6 +10,7 @@ · 9 unmodified lines
10	# Sanitizer runtimes are unavailable and cannot be linked here.	10	# Sanitizer runtimes are unavailable and cannot be linked here.
11	KASAN_SANITIZE := n	11	KASAN_SANITIZE := n
12	KCSAN_SANITIZE := n	12	KCSAN_SANITIZE := n
		13	KMSAN_SANITIZE := n
13	OBJECT_FILES_NON_STANDARD := y	14	OBJECT_FILES_NON_STANDARD := y
14		15
15	# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in.	16	# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in.

arch/xtensa/Kconfig+1−1

@@ -771,7 +771,7 @@ config HIGHMEM · 770 unmodified lines
771		771
772	If unsure, say Y.	772	If unsure, say Y.
773		773
774	config FORCE_MAX_ZONEORDER	774	config ARCH_FORCE_MAX_ORDER
775	int "Maximum zone order"	775	int "Maximum zone order"
776	default "11"	776	default "11"
777	help	777	help

arch/xtensa/include/uapi/asm/mman.h+2−0

@@ -111,6 +111,8 @@ · 110 unmodified lines
111		111
112	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */	112	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
113		113
		114	#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
		115
114	/* compatibility flags */	116	/* compatibility flags */
115	#define MAP_FILE 0	117	#define MAP_FILE 0
116		118

arch/xtensa/kernel/syscall.c+12−6

@@ -58,6 +58,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, · 57 unmodified lines
58	unsigned long len, unsigned long pgoff, unsigned long flags)	58	unsigned long len, unsigned long pgoff, unsigned long flags)
59	{	59	{
60	struct vm_area_struct *vmm;	60	struct vm_area_struct *vmm;
		61	struct vma_iterator vmi;
61		62
62	if (flags & MAP_FIXED) {	63	if (flags & MAP_FIXED) {
63	/* We do not accept a shared mapping if it would violate	64	/* We do not accept a shared mapping if it would violate
@@ -79,15 +80,20 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, · 15 unmodified lines
79	else	80	else
80	addr = PAGE_ALIGN(addr);	81	addr = PAGE_ALIGN(addr);
81		82
82	for (vmm = find_vma(current->mm, addr); ; vmm = vmm->vm_next) {	83	vma_iter_init(&vmi, current->mm, addr);
83	/* At this point: (!vmm \|\| addr < vmm->vm_end). */	84	for_each_vma(vmi, vmm) {
84	if (TASK_SIZE - len < addr)	85	/* At this point: (addr < vmm->vm_end). */
85	return -ENOMEM;	86	if (addr + len <= vm_start_gap(vmm))
86	if (!vmm \|\| addr + len <= vm_start_gap(vmm))	87	break;
87	return addr;	88
88	addr = vmm->vm_end;	89	addr = vmm->vm_end;
89	if (flags & MAP_SHARED)	90	if (flags & MAP_SHARED)
90	addr = COLOUR_ALIGN(addr, pgoff);	91	addr = COLOUR_ALIGN(addr, pgoff);
91	}	92	}
		93
		94	if (TASK_SIZE - len < addr)
		95	return -ENOMEM;
		96
		97	return addr;
92	}	98	}
93	#endif	99	#endif

block/bio.c+2−0

@@ -867,6 +867,8 @@ static inline bool page_is_mergeable(const struct bio_vec *bv, · 866 unmodified lines
867	*same_page = ((vec_end_addr & PAGE_MASK) == page_addr);	867	*same_page = ((vec_end_addr & PAGE_MASK) == page_addr);
868	if (*same_page)	868	if (*same_page)
869	return true;	869	return true;
		870	else if (IS_ENABLED(CONFIG_KMSAN))
		871	return false;
870	return (bv->bv_page + bv_end / PAGE_SIZE) == (page + off / PAGE_SIZE);	872	return (bv->bv_page + bv_end / PAGE_SIZE) == (page + off / PAGE_SIZE);
871	}	873	}
872		874

block/blk.h+7−0

@@ -88,6 +88,13 @@ static inline bool biovec_phys_mergeable(struct request_queue *q, · 87 unmodified lines
88	phys_addr_t addr1 = page_to_phys(vec1->bv_page) + vec1->bv_offset;	88	phys_addr_t addr1 = page_to_phys(vec1->bv_page) + vec1->bv_offset;
89	phys_addr_t addr2 = page_to_phys(vec2->bv_page) + vec2->bv_offset;	89	phys_addr_t addr2 = page_to_phys(vec2->bv_page) + vec2->bv_offset;
90		90
		91	/*
		92	* Merging adjacent physical pages may not work correctly under KMSAN
		93	* if their metadata pages aren't adjacent. Just disable merging.
		94	*/
		95	if (IS_ENABLED(CONFIG_KMSAN))
		96	return false;
		97
91	if (addr1 + vec1->bv_len != addr2)	98	if (addr1 + vec1->bv_len != addr2)
92	return false;	99	return false;
93	if (xen_domain() && !xen_biovec_phys_mergeable(vec1, vec2->bv_page))	100	if (xen_domain() && !xen_biovec_phys_mergeable(vec1, vec2->bv_page))

crypto/Kconfig+2−0

 config CRYPTO_HASH_INFO
 	bool
+if !KMSAN # avoid false positives from assembly
 if ARM
 source "arch/arm/crypto/Kconfig"
 endif
 if X86
 source "arch/x86/crypto/Kconfig"
 endif
+endif
 source "drivers/crypto/Kconfig"
 source "crypto/asymmetric_keys/Kconfig"

drivers/base/memory.c+0−6

 	}
 }
-/* return true if the memory block is offlined, otherwise, return false */
-bool is_memblock_offlined(struct memory_block *mem)
-{
-	return mem->state == MEM_OFFLINE;
-}
 static struct attribute *memory_root_attrs[] = {
 #ifdef CONFIG_ARCH_MEMORY_PROBE
 	&dev_attr_probe.attr,

drivers/base/node.c+6−135

@@ -20,6 +20,7 @@ · 19 unmodified lines
20	#include <linux/pm_runtime.h>	20	#include <linux/pm_runtime.h>
21	#include <linux/swap.h>	21	#include <linux/swap.h>
22	#include <linux/slab.h>	22	#include <linux/slab.h>
		23	#include <linux/hugetlb.h>
23		24
24	static struct bus_type node_subsys = {	25	static struct bus_type node_subsys = {
25	.name = "node",	26	.name = "node",
@@ -589,64 +590,9 @@ static const struct attribute_group *node_dev_groups[] = { · 563 unmodified lines
589	NULL	590	NULL
590	};	591	};
591		592
592	#ifdef CONFIG_HUGETLBFS
593	/*
594	* hugetlbfs per node attributes registration interface:
595	* When/if hugetlb[fs] subsystem initializes [sometime after this module],
596	* it will register its per node attributes for all online nodes with
597	* memory. It will also call register_hugetlbfs_with_node(), below, to
598	* register its attribute registration functions with this node driver.
599	* Once these hooks have been initialized, the node driver will call into
600	* the hugetlb module to [un]register attributes for hot-plugged nodes.
601	*/
602	static node_registration_func_t __hugetlb_register_node;
603	static node_registration_func_t __hugetlb_unregister_node;
604
605	static inline bool hugetlb_register_node(struct node *node)
606	{
607	if (__hugetlb_register_node &&
608	node_state(node->dev.id, N_MEMORY)) {
609	__hugetlb_register_node(node);
610	return true;
611	}
612	return false;
613	}
614
615	static inline void hugetlb_unregister_node(struct node *node)
616	{
617	if (__hugetlb_unregister_node)
618	__hugetlb_unregister_node(node);
619	}
620
621	void register_hugetlbfs_with_node(node_registration_func_t doregister,
622	node_registration_func_t unregister)
623	{
624	__hugetlb_register_node = doregister;
625	__hugetlb_unregister_node = unregister;
626	}
627	#else
628	static inline void hugetlb_register_node(struct node *node) {}
629
630	static inline void hugetlb_unregister_node(struct node *node) {}
631	#endif
632
633	static void node_device_release(struct device *dev)	593	static void node_device_release(struct device *dev)
634	{	594	{
635	struct node *node = to_node(dev);	595	kfree(to_node(dev));
636
637	#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS)
638	/*
639	* We schedule the work only when a memory section is
640	* onlined/offlined on this node. When we come here,
641	* all the memory on this node has been offlined,
642	* so we won't enqueue new work to this work.
643	*
644	* The work is using node->node_work, so we should
645	* flush work before freeing the memory.
646	*/
647	flush_work(&node->node_work);
648	#endif
649	kfree(node);
650	}	596	}
651		597
652	/*	598	/*
@@ -665,13 +611,13 @@ static int register_node(struct node *node, int num) · 12 unmodified lines
665	node->dev.groups = node_dev_groups;	611	node->dev.groups = node_dev_groups;
666	error = device_register(&node->dev);	612	error = device_register(&node->dev);
667		613
668	if (error)	614	if (error) {
669	put_device(&node->dev);	615	put_device(&node->dev);
670	else {	616	} else {
671	hugetlb_register_node(node);	617	hugetlb_register_node(node);
672
673	compaction_register_node(node);	618	compaction_register_node(node);
674	}	619	}
		620
675	return error;	621	return error;
676	}	622	}
677		623
@@ -684,8 +630,8 @@ static int register_node(struct node *node, int num) · 6 unmodified lines
684	*/	630	*/
685	void unregister_node(struct node *node)	631	void unregister_node(struct node *node)
686	{	632	{
		633	hugetlb_unregister_node(node);
687	compaction_unregister_node(node);	634	compaction_unregister_node(node);
688	hugetlb_unregister_node(node); /* no-op, if memoryless node */
689	node_remove_accesses(node);	635	node_remove_accesses(node);
690	node_remove_caches(node);	636	node_remove_caches(node);
691	device_unregister(&node->dev);	637	device_unregister(&node->dev);
@@ -907,74 +853,8 @@ void register_memory_blocks_under_node(int nid, unsigned long start_pfn, · 215 unmodified lines
907	(void *)&nid, func);	853	(void *)&nid, func);
908	return;	854	return;
909	}	855	}
910
911	#ifdef CONFIG_HUGETLBFS
912	/*
913	* Handle per node hstate attribute [un]registration on transistions
914	* to/from memoryless state.
915	*/
916	static void node_hugetlb_work(struct work_struct *work)
917	{
918	struct node *node = container_of(work, struct node, node_work);
919
920	/*
921	* We only get here when a node transitions to/from memoryless state.
922	* We can detect which transition occurred by examining whether the
923	* node has memory now. hugetlb_register_node() already check this
924	* so we try to register the attributes. If that fails, then the
925	* node has transitioned to memoryless, try to unregister the
926	* attributes.
927	*/
928	if (!hugetlb_register_node(node))
929	hugetlb_unregister_node(node);
930	}
931
932	static void init_node_hugetlb_work(int nid)
933	{
934	INIT_WORK(&node_devices[nid]->node_work, node_hugetlb_work);
935	}
936
937	static int node_memory_callback(struct notifier_block *self,
938	unsigned long action, void *arg)
939	{
940	struct memory_notify *mnb = arg;
941	int nid = mnb->status_change_nid;
942
943	switch (action) {
944	case MEM_ONLINE:
945	case MEM_OFFLINE:
946	/*
947	* offload per node hstate [un]registration to a work thread
948	* when transitioning to/from memoryless state.
949	*/
950	if (nid != NUMA_NO_NODE)
951	schedule_work(&node_devices[nid]->node_work);
952	break;
953
954	case MEM_GOING_ONLINE:
955	case MEM_GOING_OFFLINE:
956	case MEM_CANCEL_ONLINE:
957	case MEM_CANCEL_OFFLINE:
958	default:
959	break;
960	}
961
962	return NOTIFY_OK;
963	}
964	#endif /* CONFIG_HUGETLBFS */
965	#endif /* CONFIG_MEMORY_HOTPLUG */	856	#endif /* CONFIG_MEMORY_HOTPLUG */
966		857
967	#if !defined(CONFIG_MEMORY_HOTPLUG) \|\| !defined(CONFIG_HUGETLBFS)
968	static inline int node_memory_callback(struct notifier_block *self,
969	unsigned long action, void *arg)
970	{
971	return NOTIFY_OK;
972	}
973
974	static void init_node_hugetlb_work(int nid) { }
975
976	#endif
977
978	int __register_one_node(int nid)	858	int __register_one_node(int nid)
979	{	859	{
980	int error;	860	int error;
@@ -993,8 +873,6 @@ int __register_one_node(int nid) · 12 unmodified lines
993	}	873	}
994		874
995	INIT_LIST_HEAD(&node_devices[nid]->access_list);	875	INIT_LIST_HEAD(&node_devices[nid]->access_list);
996	/* initialize work queue for memory hot plug */
997	init_node_hugetlb_work(nid);
998	node_init_caches(nid);	876	node_init_caches(nid);
999		877
1000	return error;	878	return error;
@@ -1065,13 +943,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = { · 64 unmodified lines
1065	NULL,	943	NULL,
1066	};	944	};
1067		945
1068	#define NODE_CALLBACK_PRI 2 /* lower than SLAB */
1069	void __init node_dev_init(void)	946	void __init node_dev_init(void)
1070	{	947	{
1071	static struct notifier_block node_memory_callback_nb = {
1072	.notifier_call = node_memory_callback,
1073	.priority = NODE_CALLBACK_PRI,
1074	};
1075	int ret, i;	948	int ret, i;
1076		949
1077	BUILD_BUG_ON(ARRAY_SIZE(node_state_attr) != NR_NODE_STATES);	950	BUILD_BUG_ON(ARRAY_SIZE(node_state_attr) != NR_NODE_STATES);
@@ -1081,8 +954,6 @@ void __init node_dev_init(void) · 3 unmodified lines
1081	if (ret)	954	if (ret)
1082	panic("%s() failed to register subsystem: %d\n", __func__, ret);	955	panic("%s() failed to register subsystem: %d\n", __func__, ret);
1083		956
1084	register_hotmemory_notifier(&node_memory_callback_nb);
1085
1086	/*	957	/*
1087	* Create all node devices, which will properly link the node	958	* Create all node devices, which will properly link the node
1088	* to applicable memory block devices and already created cpu devices.	959	* to applicable memory block devices and already created cpu devices.

drivers/block/zram/zram_drv.c+24−14

 	if (!sysfs_streq(buf, "all")) {
 		/*
-		 * If it did not parse as 'all' try to treat it as an integer when
+		 * If it did not parse as 'all' try to treat it as an integer
-		 * we have memory tracking enabled.
+		 * when we have memory tracking enabled.
 		 */
 		u64 age_sec;
 	if (!init_done(zram))
 		goto out_unlock;
-	/* A cutoff_time of 0 marks everything as idle, this is the "all" behavior */
+	/*
+	 * A cutoff_time of 0 marks everything as idle, this is the
+	 * "all" behavior.
+	 */
 	mark_idle(zram, cutoff_time);
 	rv = len;
 		handle = zs_malloc(zram->mem_pool, comp_len,
 				GFP_NOIO | __GFP_HIGHMEM |
 				__GFP_MOVABLE);
-		if (!IS_ERR((void *)handle))
+		if (IS_ERR((void *)handle))
+			return PTR_ERR((void *)handle);
+		if (comp_len != PAGE_SIZE)
 			goto compress_again;
-		return PTR_ERR((void *)handle);
+		/*
+		 * If the page is not compressible, you need to acquire the
+		 * lock and execute the code below. The zcomp_stream_get()
+		 * call is needed to disable the cpu hotplug and grab the
+		 * zstrm buffer back. It is necessary that the dereferencing
+		 * of the zstrm variable below occurs correctly.
+		 */
+		zstrm = zcomp_stream_get(zram->comp);
 	}
 	alloced_pages = zs_get_total_pages(zram->mem_pool);
 static void zram_reset_device(struct zram *zram)
 {
-	struct zcomp *comp;
-	u64 disksize;
 	down_write(&zram->init_lock);
 	zram->limit_pages = 0;
 		return;
 	}
-	comp = zram->comp;
-	disksize = zram->disksize;
-	zram->disksize = 0;
 	set_capacity_and_notify(zram->disk, 0);
 	part_stat_set_all(zram->disk->part0, 0);
 	/* I/O operation under all of CPU are done so let's free */
-	zram_meta_free(zram, disksize);
+	zram_meta_free(zram, zram->disksize);
+	zram->disksize = 0;
 	memset(&zram->stats, 0, sizeof(zram->stats));
-	zcomp_destroy(comp);
+	zcomp_destroy(zram->comp);
+	zram->comp = NULL;
 	reset_bdev(zram);
 	up_write(&zram->init_lock);
 {
 	int ret;
+	BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > BITS_PER_LONG);
 	ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare",
 				      zcomp_cpu_up_prepare, zcomp_cpu_dead);
 	if (ret < 0)

drivers/block/zram/zram_drv.h+7−8

 /*
- * The lower ZRAM_FLAG_SHIFT bits of table.flags is for
+ * ZRAM is mainly used for memory efficiency so we want to keep memory
- * object size (excluding header), the higher bits is for
+ * footprint small and thus squeeze size and zram pageflags into a flags
- * zram_pageflags.
+ * member. The lower ZRAM_FLAG_SHIFT bits is for object size (excluding
+ * header), which cannot be larger than PAGE_SIZE (requiring PAGE_SHIFT
+ * bits), the higher bits are for zram_pageflags.
  *
- * zram is mainly used for memory efficiency so we want to keep memory
+ * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow.
- * footprint small so we can squeeze size and flags into a field.
- * The lower ZRAM_FLAG_SHIFT bits is for object size (excluding header),
- * the higher bits is for zram_pageflags.
  */
-#define ZRAM_FLAG_SHIFT 24
+#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1)
 /* Flags for zram pages (table[page_no].flags) */
 enum zram_pageflags {

drivers/dax/kmem.c+39−3

 #include <linux/fs.h>
 #include <linux/mm.h>
 #include <linux/mman.h>
+#include <linux/memory-tiers.h>
 #include "dax-private.h"
 #include "bus.h"
+/*
+ * Default abstract distance assigned to the NUMA node onlined
+ * by DAX/kmem if the low level platform driver didn't initialize
+ * one for this NUMA node.
+ */
+#define MEMTIER_DEFAULT_DAX_ADISTANCE	(MEMTIER_ADISTANCE_DRAM * 5)
 /* Memory resource name used for add_memory_driver_managed(). */
 static const char *kmem_name;
 /* Set if any memory will remain added when the driver will be unloaded. */
 	struct resource *res[];
 };
+static struct memory_dev_type *dax_slowmem_type;
 static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 {
 	struct device *dev = &dev_dax->dev;
 		return -EINVAL;
 	}
+	init_node_memory_type(numa_node, dax_slowmem_type);
+	rc = -ENOMEM;
 	data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL);
 	if (!data)
-		return -ENOMEM;
+		goto err_dax_kmem_data;
-	rc = -ENOMEM;
 	data->res_name = kstrdup(dev_name(dev), GFP_KERNEL);
 	if (!data->res_name)
 		goto err_res_name;
 	kfree(data->res_name);
 err_res_name:
 	kfree(data);
+err_dax_kmem_data:
+	clear_node_memory_type(numa_node, dax_slowmem_type);
 	return rc;
 }
 static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
 {
 	int i, success = 0;
+	int node = dev_dax->target_node;
 	struct device *dev = &dev_dax->dev;
 	struct dax_kmem_data *data = dev_get_drvdata(dev);
 		kfree(data->res_name);
 		kfree(data);
 		dev_set_drvdata(dev, NULL);
+		/*
+		 * Clear the memtype association on successful unplug.
+		 * If not, we have memory blocks left which can be
+		 * offlined/onlined later. We need to keep memory_dev_type
+		 * for that. This implies this reference will be around
+		 * till next reboot.
+		 */
+		clear_node_memory_type(node, dax_slowmem_type);
 	}
 }
 #else
 	if (!kmem_name)
 		return -ENOMEM;
+	dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE);
+	if (IS_ERR(dax_slowmem_type)) {
+		rc = PTR_ERR(dax_slowmem_type);
+		goto err_dax_slowmem_type;
+	}
 	rc = dax_driver_register(&device_dax_kmem_driver);
 	if (rc)
-		kfree_const(kmem_name);
+		goto error_dax_driver;
+	return rc;
+error_dax_driver:
+	destroy_memory_type(dax_slowmem_type);
+err_dax_slowmem_type:
+	kfree_const(kmem_name);
 	return rc;
 }
 	dax_driver_unregister(&device_dax_kmem_driver);
 	if (!any_hotremove_failed)
 		kfree_const(kmem_name);
+	destroy_memory_type(dax_slowmem_type);
 }
 MODULE_AUTHOR("Intel Corporation");

drivers/firmware/efi/efi.c+1−1

@@ -59,7 +59,7 @@ static unsigned long __initdata rt_prop = EFI_INVALID_TABLE_ADDR; · 58 unmodified lines
59	static unsigned long __initdata initrd = EFI_INVALID_TABLE_ADDR;	59	static unsigned long __initdata initrd = EFI_INVALID_TABLE_ADDR;
60		60
61	struct mm_struct efi_mm = {	61	struct mm_struct efi_mm = {
62	.mm_rb = RB_ROOT,	62	.mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, efi_mm.mmap_lock),
63	.mm_users = ATOMIC_INIT(2),	63	.mm_users = ATOMIC_INIT(2),
64	.mm_count = ATOMIC_INIT(1),	64	.mm_count = ATOMIC_INIT(1),
65	.write_protect_seq = SEQCNT_ZERO(efi_mm.write_protect_seq),	65	.write_protect_seq = SEQCNT_ZERO(efi_mm.write_protect_seq),

drivers/firmware/efi/libstub/Makefile+1−0

@@ -57,6 +57,7 @@ GCOV_PROFILE := n · 56 unmodified lines
57	# Sanitizer runtimes are unavailable and cannot be linked here.	57	# Sanitizer runtimes are unavailable and cannot be linked here.
58	KASAN_SANITIZE := n	58	KASAN_SANITIZE := n
59	KCSAN_SANITIZE := n	59	KCSAN_SANITIZE := n
		60	KMSAN_SANITIZE := n
60	UBSAN_SANITIZE := n	61	UBSAN_SANITIZE := n
61	OBJECT_FILES_NON_STANDARD := y	62	OBJECT_FILES_NON_STANDARD := y
62		63

drivers/gpu/drm/i915/gem/i915_gem_userptr.c+5−9

 static int
 probe_range(struct mm_struct *mm, unsigned long addr, unsigned long len)
 {
-	const unsigned long end = addr + len;
+	VMA_ITERATOR(vmi, mm, addr);
 	struct vm_area_struct *vma;
-	int ret = -EFAULT;
 	mmap_read_lock(mm);
-	for (vma = find_vma(mm, addr); vma; vma = vma->vm_next) {
+	for_each_vma_range(vmi, vma, addr + len) {
 		/* Check for holes, note that we also update the addr below */
 		if (vma->vm_start > addr)
 			break;
 		if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
 			break;
-		if (vma->vm_end >= end) {
-			ret = 0;
-			break;
-		}
 		addr = vma->vm_end;
 	}
 	mmap_read_unlock(mm);
-	return ret;
+	if (vma)
+		return -EFAULT;
+	return 0;
 }
 /*

drivers/input/serio/libps2.c+4−1

 #include <linux/sched.h>
 #include <linux/interrupt.h>
 #include <linux/input.h>
+#include <linux/kmsan-checks.h>
 #include <linux/serio.h>
 #include <linux/i8042.h>
 #include <linux/libps2.h>
 	serio_pause_rx(ps2dev->serio);
-	if (param)
+	if (param) {
 		for (i = 0; i < receive; i++)
 			param[i] = ps2dev->cmdbuf[(receive - 1) - i];
+		kmsan_unpoison_memory(param, receive);
+	}
 	if (ps2dev->cmdcnt &&
 	    (command != PS2_CMD_RESET_BAT || ps2dev->cmdcnt != 1)) {

drivers/misc/cxl/fault.c+15−30

 		mmput(mm);
 }
-static void cxl_prefault_one(struct cxl_context *ctx, u64 ea)
-{
-	struct mm_struct *mm;
-	mm = get_mem_context(ctx);
-	if (mm == NULL) {
-		pr_devel("cxl_prefault_one unable to get mm %i\n",
-			 pid_nr(ctx->pid));
-		return;
-	}
-	cxl_fault_segment(ctx, mm, ea);
-	mmput(mm);
-}
 static u64 next_segment(u64 ea, u64 vsid)
 {
 	if (vsid & SLB_VSID_B_1T)
 	return ea + 1;
 }
-static void cxl_prefault_vma(struct cxl_context *ctx)
+static void cxl_prefault_vma(struct cxl_context *ctx, struct mm_struct *mm)
 {
 	u64 ea, last_esid = 0;
 	struct copro_slb slb;
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
 	int rc;
-	struct mm_struct *mm;
-	mm = get_mem_context(ctx);
-	if (mm == NULL) {
-		pr_devel("cxl_prefault_vm unable to get mm %i\n",
-			 pid_nr(ctx->pid));
-		return;
-	}
 	mmap_read_lock(mm);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		for (ea = vma->vm_start; ea < vma->vm_end;
 				ea = next_segment(ea, slb.vsid)) {
 			rc = copro_calculate_slb(mm, ea, &slb);
 		}
 	}
 	mmap_read_unlock(mm);
-	mmput(mm);
 }
 void cxl_prefault(struct cxl_context *ctx, u64 wed)
 {
+	struct mm_struct *mm = get_mem_context(ctx);
+	if (mm == NULL) {
+		pr_devel("cxl_prefault unable to get mm %i\n",
+			 pid_nr(ctx->pid));
+		return;
+	}
 	switch (ctx->afu->prefault_mode) {
 	case CXL_PREFAULT_WED:
-		cxl_prefault_one(ctx, wed);
+		cxl_fault_segment(ctx, mm, wed);
 		break;
 	case CXL_PREFAULT_ALL:
-		cxl_prefault_vma(ctx);
+		cxl_prefault_vma(ctx, mm);
 		break;
 	default:
 		break;
 	}
+	mmput(mm);
 }

drivers/net/Kconfig+1−0

@@ -76,6 +76,7 @@ config WIREGUARD · 75 unmodified lines
76	tristate "WireGuard secure network tunnel"	76	tristate "WireGuard secure network tunnel"
77	depends on NET && INET	77	depends on NET && INET
78	depends on IPV6 \|\| !IPV6	78	depends on IPV6 \|\| !IPV6
		79	depends on !KMSAN # KMSAN doesn't support the crypto configs below
79	select NET_UDP_TUNNEL	80	select NET_UDP_TUNNEL
80	select DST_CACHE	81	select DST_CACHE
81	select CRYPTO	82	select CRYPTO

drivers/nvdimm/nd.h+1−1

@@ -652,7 +652,7 @@ void devm_namespace_disable(struct device *dev, · 651 unmodified lines
652	struct nd_namespace_common *ndns);	652	struct nd_namespace_common *ndns);
653	#if IS_ENABLED(CONFIG_ND_CLAIM)	653	#if IS_ENABLED(CONFIG_ND_CLAIM)
654	/* max struct page size independent of kernel config */	654	/* max struct page size independent of kernel config */
655	#define MAX_STRUCT_PAGE_SIZE 64	655	#define MAX_STRUCT_PAGE_SIZE 128
656	int nvdimm_setup_pfn(struct nd_pfn nd_pfn, struct dev_pagemap pgmap);	656	int nvdimm_setup_pfn(struct nd_pfn nd_pfn, struct dev_pagemap pgmap);
657	#else	657	#else
658	static inline int nvdimm_setup_pfn(struct nd_pfn *nd_pfn,	658	static inline int nvdimm_setup_pfn(struct nd_pfn *nd_pfn,

drivers/nvdimm/pfn_devs.c+1−1

@@ -787,7 +787,7 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn) · 786 unmodified lines
787	* when populating the vmemmap. This should be equal to	787	* when populating the vmemmap. This should be equal to
788	* PMD_SIZE for most architectures.	788	* PMD_SIZE for most architectures.
789	*	789	*
790	* Also make sure size of struct page is less than 64. We	790	* Also make sure size of struct page is less than 128. We
791	* want to make sure we use large enough size here so that	791	* want to make sure we use large enough size here so that
792	* we don't have a dynamic reserve space depending on	792	* we don't have a dynamic reserve space depending on
793	* struct page size. But we also want to make sure we notice	793	* struct page size. But we also want to make sure we notice

drivers/tee/optee/call.c+10−8

@@ -492,15 +492,18 @@ static bool is_normal_memory(pgprot_t p) · 491 unmodified lines
492	#endif	492	#endif
493	}	493	}
494		494
495	static int __check_mem_type(struct vm_area_struct *vma, unsigned long end)	495	static int __check_mem_type(struct mm_struct *mm, unsigned long start,
		496	unsigned long end)
496	{	497	{
497	while (vma && is_normal_memory(vma->vm_page_prot)) {	498	struct vm_area_struct *vma;
498	if (vma->vm_end >= end)	499	VMA_ITERATOR(vmi, mm, start);
499	return 0;	500
500	vma = vma->vm_next;	501	for_each_vma_range(vmi, vma, end) {
		502	if (!is_normal_memory(vma->vm_page_prot))
		503	return -EINVAL;
501	}	504	}
502		505
503	return -EINVAL;	506	return 0;
504	}	507	}
505		508
506	int optee_check_mem_type(unsigned long start, size_t num_pages)	509	int optee_check_mem_type(unsigned long start, size_t num_pages)
@@ -516,8 +519,7 @@ int optee_check_mem_type(unsigned long start, size_t num_pages) · 9 unmodified lines
516	return 0;	519	return 0;
517		520
518	mmap_read_lock(mm);	521	mmap_read_lock(mm);
519	rc = __check_mem_type(find_vma(mm, start),	522	rc = __check_mem_type(mm, start, start + num_pages * PAGE_SIZE);
520	start + num_pages * PAGE_SIZE);
521	mmap_read_unlock(mm);	523	mmap_read_unlock(mm);
522		524
523	return rc;	525	return rc;

drivers/usb/core/urb.c+2−0

 #include <linux/bitops.h>
 #include <linux/slab.h>
 #include <linux/log2.h>
+#include <linux/kmsan.h>
 #include <linux/usb.h>
 #include <linux/wait.h>
 #include <linux/usb/hcd.h>
 			URB_SETUP_MAP_SINGLE | URB_SETUP_MAP_LOCAL |
 			URB_DMA_SG_COMBINED);
 	urb->transfer_flags |= (is_out ? URB_DIR_OUT : URB_DIR_IN);
+	kmsan_handle_urb(urb, is_out);
 	if (xfertype != USB_ENDPOINT_XFER_CONTROL &&
 			dev->state < USB_STATE_CONFIGURED)

drivers/virtio/virtio_ring.c+9−1

 #include <linux/module.h>
 #include <linux/hrtimer.h>
 #include <linux/dma-mapping.h>
+#include <linux/kmsan.h>
 #include <linux/spinlock.h>
 #include <xen/xen.h>
 				   struct scatterlist *sg,
 				   enum dma_data_direction direction)
 {
-	if (!vq->use_dma_api)
+	if (!vq->use_dma_api) {
+		/*
+		 * If DMA is not used, KMSAN doesn't know that the scatterlist
+		 * is initialized by the hardware. Explicitly check/unpoison it
+		 * depending on the direction.
+		 */
+		kmsan_handle_dma(sg_page(sg), sg->offset, sg->length, direction);
 		return (dma_addr_t)sg_phys(sg);
+	}
 	/*
 	 * We can't use dma_map_sg, because we don't use scatterlists in

drivers/xen/privcmd.c+1−1

@@ -282,7 +282,7 @@ static long privcmd_ioctl_mmap(struct file file, void __user udata) · 281 unmodified lines
282	struct page, lru);	282	struct page, lru);
283	struct privcmd_mmap_entry *msg = page_address(page);	283	struct privcmd_mmap_entry *msg = page_address(page);
284		284
285	vma = find_vma(mm, msg->va);	285	vma = vma_lookup(mm, msg->va);
286	rc = -EINVAL;	286	rc = -EINVAL;
287		287
288	if (!vma \|\| (msg->va != vma->vm_start) \|\| vma->vm_private_data)	288	if (!vma \|\| (msg->va != vma->vm_start) \|\| vma->vm_private_data)

fs/Kconfig+1−0

@@ -235,6 +235,7 @@ config ARCH_SUPPORTS_HUGETLBFS · 234 unmodified lines
235	config HUGETLBFS	235	config HUGETLBFS
236	bool "HugeTLB file system support"	236	bool "HugeTLB file system support"
237	depends on X86 \|\| IA64 \|\| SPARC64 \|\| ARCH_SUPPORTS_HUGETLBFS \|\| BROKEN	237	depends on X86 \|\| IA64 \|\| SPARC64 \|\| ARCH_SUPPORTS_HUGETLBFS \|\| BROKEN
		238	depends on (SYSFS \|\| SYSCTL)
238	help	239	help
239	hugetlbfs is a filesystem backing for HugeTLB pages, based on	240	hugetlbfs is a filesystem backing for HugeTLB pages, based on
240	ramfs. For architectures that support it, say Y here and read	241	ramfs. For architectures that support it, say Y here and read

fs/btrfs/compression.c+15−16

@@ -8,6 +8,7 @@ · 7 unmodified lines
8	#include <linux/file.h>	8	#include <linux/file.h>
9	#include <linux/fs.h>	9	#include <linux/fs.h>
10	#include <linux/pagemap.h>	10	#include <linux/pagemap.h>
		11	#include <linux/pagevec.h>
11	#include <linux/highmem.h>	12	#include <linux/highmem.h>
12	#include <linux/kthread.h>	13	#include <linux/kthread.h>
13	#include <linux/time.h>	14	#include <linux/time.h>
@@ -219,8 +220,7 @@ static noinline void end_compressed_writeback(struct inode *inode, · 205 unmodified lines
219	struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);	220	struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
220	unsigned long index = cb->start >> PAGE_SHIFT;	221	unsigned long index = cb->start >> PAGE_SHIFT;
221	unsigned long end_index = (cb->start + cb->len - 1) >> PAGE_SHIFT;	222	unsigned long end_index = (cb->start + cb->len - 1) >> PAGE_SHIFT;
222	struct page *pages[16];	223	struct folio_batch fbatch;
223	unsigned long nr_pages = end_index - index + 1;
224	const int errno = blk_status_to_errno(cb->status);	224	const int errno = blk_status_to_errno(cb->status);
225	int i;	225	int i;
226	int ret;	226	int ret;
@@ -228,24 +228,23 @@ static noinline void end_compressed_writeback(struct inode *inode, · 1 unmodified line
228	if (errno)	228	if (errno)
229	mapping_set_error(inode->i_mapping, errno);	229	mapping_set_error(inode->i_mapping, errno);
230		230
231	while (nr_pages > 0) {	231	folio_batch_init(&fbatch);
232	ret = find_get_pages_contig(inode->i_mapping, index,	232	while (index <= end_index) {
233	min_t(unsigned long,	233	ret = filemap_get_folios(inode->i_mapping, &index, end_index,
234	nr_pages, ARRAY_SIZE(pages)), pages);	234	&fbatch);
235	if (ret == 0) {	235
236	nr_pages -= 1;	236	if (ret == 0)
237	index += 1;	237	return;
238	continue;	238
239	}
240	for (i = 0; i < ret; i++) {	239	for (i = 0; i < ret; i++) {
		240	struct folio *folio = fbatch.folios[i];
		241
241	if (errno)	242	if (errno)
242	SetPageError(pages[i]);	243	folio_set_error(folio);
243	btrfs_page_clamp_clear_writeback(fs_info, pages[i],	244	btrfs_page_clamp_clear_writeback(fs_info, &folio->page,
244	cb->start, cb->len);	245	cb->start, cb->len);
245	put_page(pages[i]);
246	}	246	}
247	nr_pages -= ret;	247	folio_batch_release(&fbatch);
248	index += ret;
249	}	248	}
250	/* the inode may be gone now */	249	/* the inode may be gone now */
251	}	250	}

fs/btrfs/extent_io.c+15−18

@@ -270,9 +270,8 @@ static int __process_pages_contig(struct address_space *mapping, · 269 unmodified lines
270	pgoff_t start_index = start >> PAGE_SHIFT;	270	pgoff_t start_index = start >> PAGE_SHIFT;
271	pgoff_t end_index = end >> PAGE_SHIFT;	271	pgoff_t end_index = end >> PAGE_SHIFT;
272	pgoff_t index = start_index;	272	pgoff_t index = start_index;
273	unsigned long nr_pages = end_index - start_index + 1;
274	unsigned long pages_processed = 0;	273	unsigned long pages_processed = 0;
275	struct page *pages[16];	274	struct folio_batch fbatch;
276	int err = 0;	275	int err = 0;
277	int i;	276	int i;
278		277
@@ -281,16 +280,17 @@ static int __process_pages_contig(struct address_space *mapping, · 2 unmodified lines
281	ASSERT(processed_end && *processed_end == start);	280	ASSERT(processed_end && *processed_end == start);
282	}	281	}
283		282
284	if ((page_ops & PAGE_SET_ERROR) && nr_pages > 0)	283	if ((page_ops & PAGE_SET_ERROR) && start_index <= end_index)
285	mapping_set_error(mapping, -EIO);	284	mapping_set_error(mapping, -EIO);
286		285
287	while (nr_pages > 0) {	286	folio_batch_init(&fbatch);
288	int found_pages;	287	while (index <= end_index) {
		288	int found_folios;
		289
		290	found_folios = filemap_get_folios_contig(mapping, &index,
		291	end_index, &fbatch);
289		292
290	found_pages = find_get_pages_contig(mapping, index,	293	if (found_folios == 0) {
291	min_t(unsigned long,
292	nr_pages, ARRAY_SIZE(pages)), pages);
293	if (found_pages == 0) {
294	/*	294	/*
295	* Only if we're going to lock these pages, we can find	295	* Only if we're going to lock these pages, we can find
296	* nothing at @index.	296	* nothing at @index.
@@ -300,23 +300,20 @@ static int __process_pages_contig(struct address_space *mapping, · 3 unmodified lines
300	goto out;	300	goto out;
301	}	301	}
302		302
303	for (i = 0; i < found_pages; i++) {	303	for (i = 0; i < found_folios; i++) {
304	int process_ret;	304	int process_ret;
305		305	struct folio *folio = fbatch.folios[i];
306	process_ret = process_one_page(fs_info, mapping,	306	process_ret = process_one_page(fs_info, mapping,
307	pages[i], locked_page, page_ops,	307	&folio->page, locked_page, page_ops,
308	start, end);	308	start, end);
309	if (process_ret < 0) {	309	if (process_ret < 0) {
310	for (; i < found_pages; i++)
311	put_page(pages[i]);
312	err = -EAGAIN;	310	err = -EAGAIN;
		311	folio_batch_release(&fbatch);
313	goto out;	312	goto out;
314	}	313	}
315	put_page(pages[i]);	314	pages_processed += folio_nr_pages(folio);
316	pages_processed++;
317	}	315	}
318	nr_pages -= found_pages;	316	folio_batch_release(&fbatch);
319	index += found_pages;
320	cond_resched();	317	cond_resched();
321	}	318	}
322	out:	319	out:

fs/btrfs/subpage.c+1−1

@@ -337,7 +337,7 @@ bool btrfs_subpage_end_and_test_writer(const struct btrfs_fs_info *fs_info, · 336 unmodified lines
337	*	337	*
338	* Even with 0 returned, the page still need extra check to make sure	338	* Even with 0 returned, the page still need extra check to make sure
339	* it's really the correct page, as the caller is using	339	* it's really the correct page, as the caller is using
340	* find_get_pages_contig(), which can race with page invalidating.	340	* filemap_get_folios_contig(), which can race with page invalidating.
341	*/	341	*/
342	int btrfs_page_start_writer_lock(const struct btrfs_fs_info *fs_info,	342	int btrfs_page_start_writer_lock(const struct btrfs_fs_info *fs_info,
343	struct page *page, u64 start, u32 len)	343	struct page *page, u64 start, u32 len)

fs/btrfs/tests/extent-io-tests.c+17−15

@@ -4,6 +4,7 @@ · 3 unmodified lines
4	*/	4	*/
5		5
6	#include <linux/pagemap.h>	6	#include <linux/pagemap.h>
		7	#include <linux/pagevec.h>
7	#include <linux/sched.h>	8	#include <linux/sched.h>
8	#include <linux/slab.h>	9	#include <linux/slab.h>
9	#include <linux/sizes.h>	10	#include <linux/sizes.h>
@@ -20,39 +21,40 @@ static noinline int process_page_range(struct inode *inode, u64 start, u64 end, · 10 unmodified lines
20	unsigned long flags)	21	unsigned long flags)
21	{	22	{
22	int ret;	23	int ret;
23	struct page *pages[16];	24	struct folio_batch fbatch;
24	unsigned long index = start >> PAGE_SHIFT;	25	unsigned long index = start >> PAGE_SHIFT;
25	unsigned long end_index = end >> PAGE_SHIFT;	26	unsigned long end_index = end >> PAGE_SHIFT;
26	unsigned long nr_pages = end_index - index + 1;
27	int i;	27	int i;
28	int count = 0;	28	int count = 0;
29	int loops = 0;	29	int loops = 0;
30		30
31	while (nr_pages > 0) {	31	folio_batch_init(&fbatch);
32	ret = find_get_pages_contig(inode->i_mapping, index,	32
33	min_t(unsigned long, nr_pages,	33	while (index <= end_index) {
34	ARRAY_SIZE(pages)), pages);	34	ret = filemap_get_folios_contig(inode->i_mapping, &index,
		35	end_index, &fbatch);
35	for (i = 0; i < ret; i++) {	36	for (i = 0; i < ret; i++) {
		37	struct folio *folio = fbatch.folios[i];
		38
36	if (flags & PROCESS_TEST_LOCKED &&	39	if (flags & PROCESS_TEST_LOCKED &&
37	!PageLocked(pages[i]))	40	!folio_test_locked(folio))
38	count++;	41	count++;
39	if (flags & PROCESS_UNLOCK && PageLocked(pages[i]))	42	if (flags & PROCESS_UNLOCK && folio_test_locked(folio))
40	unlock_page(pages[i]);	43	folio_unlock(folio);
41	put_page(pages[i]);
42	if (flags & PROCESS_RELEASE)	44	if (flags & PROCESS_RELEASE)
43	put_page(pages[i]);	45	folio_put(folio);
44	}	46	}
45	nr_pages -= ret;	47	folio_batch_release(&fbatch);
46	index += ret;
47	cond_resched();	48	cond_resched();
48	loops++;	49	loops++;
49	if (loops > 100000) {	50	if (loops > 100000) {
50	printk(KERN_ERR	51	printk(KERN_ERR
51	"stuck in a loop, start %llu, end %llu, nr_pages %lu, ret %d\n",	52	"stuck in a loop, start %llu, end %llu, ret %d\n",
52	start, end, nr_pages, ret);	53	start, end, ret);
53	break;	54	break;
54	}	55	}
55	}	56	}
		57
56	return count;	58	return count;
57	}	59	}
58		60

fs/buffer.c+65−93

 /*
  * Default synchronous end-of-IO handler..  Just mark it up-to-date and
- * unlock the buffer. This is what ll_rw_block uses too.
+ * unlock the buffer.
  */
 void end_buffer_read_sync(struct buffer_head *bh, int uptodate)
 {
  * all already-submitted IO to complete, but does not queue any new
  * writes to the disk.
  *
- * To do O_SYNC writes, just queue the buffer writes with ll_rw_block as
+ * To do O_SYNC writes, just queue the buffer writes with write_dirty_buffer
- * you dirty the buffers, and then use osync_inode_buffers to wait for
+ * as you dirty the buffers, and then use osync_inode_buffers to wait for
  * completion.  Any other dirty buffers which are not yet queued for
  * write will not be flushed to disk by the osync.
  */
 	struct buffer_head *bh = __find_get_block(bdev, bblock + 1, blocksize);
 	if (bh) {
 		if (buffer_dirty(bh))
-			ll_rw_block(REQ_OP_WRITE, 1, &bh);
+			write_dirty_buffer(bh, 0);
 		put_bh(bh);
 	}
 }
 {
 	struct buffer_head *bh = __getblk(bdev, block, size);
 	if (likely(bh)) {
-		ll_rw_block(REQ_OP_READ | REQ_RAHEAD, 1, &bh);
+		bh_readahead(bh, REQ_RAHEAD);
 		brelse(bh);
 	}
 }
 EXPORT_SYMBOL(__breadahead);
-void __breadahead_gfp(struct block_device *bdev, sector_t block, unsigned size,
-		      gfp_t gfp)
-{
-	struct buffer_head *bh = __getblk_gfp(bdev, block, size, gfp);
-	if (likely(bh)) {
-		ll_rw_block(REQ_OP_READ | REQ_RAHEAD, 1, &bh);
-		brelse(bh);
-	}
-}
-EXPORT_SYMBOL(__breadahead_gfp);
 /**
  *  __bread_gfp() - reads a specified block and returns the bh
  *  @bdev: the block_device to read from
 		/*
 		 * The page was marked dirty, but the buffers were
 		 * clean.  Someone wrote them back by hand with
-		 * ll_rw_block/submit_bh.  A rare case.
+		 * write_dirty_buffer/submit_bh.  A rare case.
 		 */
 		end_page_writeback(page);
 		if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
 		    !buffer_unwritten(bh) &&
 		     (block_start < from || block_end > to)) {
-			ll_rw_block(REQ_OP_READ, 1, &bh);
+			bh_read_nowait(bh, 0);
 			*wait_bh++=bh;
 		}
 	}
 	struct address_space *mapping = inode->i_mapping;
 	const struct address_space_operations *aops = mapping->a_ops;
 	struct page *page;
-	void *fsdata;
+	void *fsdata = NULL;
 	int err;
 	err = inode_newsize_ok(inode, size);
 	const struct address_space_operations *aops = mapping->a_ops;
 	unsigned int blocksize = i_blocksize(inode);
 	struct page *page;
-	void *fsdata;
+	void *fsdata = NULL;
 	pgoff_t index, curidx;
 	loff_t curpos;
 	unsigned zerofrom, offset, len;
 		set_buffer_uptodate(bh);
 	if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh)) {
-		err = -EIO;
+		err = bh_read(bh, 0);
-		ll_rw_block(REQ_OP_READ, 1, &bh);
-		wait_on_buffer(bh);
 		/* Uhhuh. Read error. Complain and punt. */
-		if (!buffer_uptodate(bh))
+		if (err < 0)
 			goto unlock;
 	}
 }
 EXPORT_SYMBOL(submit_bh);
-/**
- * ll_rw_block: low-level access to block devices (DEPRECATED)
- * @opf: block layer request operation and flags.
- * @nr: number of &struct buffer_heads in the array
- * @bhs: array of pointers to &struct buffer_head
- *
- * ll_rw_block() takes an array of pointers to &struct buffer_heads, and
- * requests an I/O operation on them, either a %REQ_OP_READ or a %REQ_OP_WRITE.
- * @opf contains flags modifying the detailed I/O behavior, most notably
- * %REQ_RAHEAD.
- *
- * This function drops any buffer that it cannot get a lock on (with the
- * BH_Lock state bit), any buffer that appears to be clean when doing a write
- * request, and any buffer that appears to be up-to-date when doing read
- * request.  Further it marks as clean buffers that are processed for
- * writing (the buffer cache won't assume that they are actually clean
- * until the buffer gets unlocked).
- *
- * ll_rw_block sets b_end_io to simple completion handler that marks
- * the buffer up-to-date (if appropriate), unlocks the buffer and wakes
- * any waiters.
- *
- * All of the buffers must be for the same device, and must also be a
- * multiple of the current approved size for the device.
- */
-void ll_rw_block(const blk_opf_t opf, int nr, struct buffer_head *bhs[])
-{
-	const enum req_op op = opf & REQ_OP_MASK;
-	int i;
-	for (i = 0; i < nr; i++) {
-		struct buffer_head *bh = bhs[i];
-		if (!trylock_buffer(bh))
-			continue;
-		if (op == REQ_OP_WRITE) {
-			if (test_clear_buffer_dirty(bh)) {
-				bh->b_end_io = end_buffer_write_sync;
-				get_bh(bh);
-				submit_bh(opf, bh);
-				continue;
-			}
-		} else {
-			if (!buffer_uptodate(bh)) {
-				bh->b_end_io = end_buffer_read_sync;
-				get_bh(bh);
-				submit_bh(opf, bh);
-				continue;
-			}
-		}
-		unlock_buffer(bh);
-	}
-}
-EXPORT_SYMBOL(ll_rw_block);
 void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
 {
 	lock_buffer(bh);
 EXPORT_SYMBOL(bh_uptodate_or_lock);
 /**
- * bh_submit_read - Submit a locked buffer for reading
+ * __bh_read - Submit read for a locked buffer
  * @bh: struct buffer_head
+ * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ
+ * @wait: wait until reading finish
  *
- * Returns zero on success and -EIO on error.
+ * Returns zero on success or don't wait, and -EIO on error.
  */
-int bh_submit_read(struct buffer_head *bh)
+int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait)
 {
-	BUG_ON(!buffer_locked(bh));
+	int ret = 0;
-	if (buffer_uptodate(bh)) {
+	BUG_ON(!buffer_locked(bh));
-		unlock_buffer(bh);
-		return 0;
-	}
 	get_bh(bh);
 	bh->b_end_io = end_buffer_read_sync;
-	submit_bh(REQ_OP_READ, bh);
+	submit_bh(REQ_OP_READ | op_flags, bh);
-	wait_on_buffer(bh);
+	if (wait) {
-	if (buffer_uptodate(bh))
+		wait_on_buffer(bh);
-		return 0;
+		if (!buffer_uptodate(bh))
-	return -EIO;
+			ret = -EIO;
+	}
+	return ret;
+}
+EXPORT_SYMBOL(__bh_read);
+/**
+ * __bh_read_batch - Submit read for a batch of unlocked buffers
+ * @nr: entry number of the buffer batch
+ * @bhs: a batch of struct buffer_head
+ * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ
+ * @force_lock: force to get a lock on the buffer if set, otherwise drops any
+ *              buffer that cannot lock.
+ *
+ * Returns zero on success or don't wait, and -EIO on error.
+ */
+void __bh_read_batch(int nr, struct buffer_head *bhs[],
+		     blk_opf_t op_flags, bool force_lock)
+{
+	int i;
+	for (i = 0; i < nr; i++) {
+		struct buffer_head *bh = bhs[i];
+		if (buffer_uptodate(bh))
+			continue;
+		if (force_lock)
+			lock_buffer(bh);
+		else
+			if (!trylock_buffer(bh))
+				continue;
+		if (buffer_uptodate(bh)) {
+			unlock_buffer(bh);
+			continue;
+		}
+		bh->b_end_io = end_buffer_read_sync;
+		get_bh(bh);
+		submit_bh(REQ_OP_READ | op_flags, bh);
+	}
 }
-EXPORT_SYMBOL(bh_submit_read);
+EXPORT_SYMBOL(__bh_read_batch);
 void __init buffer_init(void)
 {

fs/coredump.c+12−22

 	return vma->vm_end - vma->vm_start;
 }
-static struct vm_area_struct *first_vma(struct task_struct *tsk,
-					struct vm_area_struct *gate_vma)
-{
-	struct vm_area_struct *ret = tsk->mm->mmap;
-	if (ret)
-		return ret;
-	return gate_vma;
-}
 /*
  * Helper function for iterating across a vma list.  It ensures that the caller
  * will visit `gate_vma' prior to terminating the search.
  */
-static struct vm_area_struct *next_vma(struct vm_area_struct *this_vma,
+static struct vm_area_struct *coredump_next_vma(struct ma_state *mas,
+				       struct vm_area_struct *vma,
 				       struct vm_area_struct *gate_vma)
 {
-	struct vm_area_struct *ret;
+	if (gate_vma && (vma == gate_vma))
-	ret = this_vma->vm_next;
-	if (ret)
-		return ret;
-	if (this_vma == gate_vma)
 		return NULL;
+	vma = mas_next(mas, ULONG_MAX);
+	if (vma)
+		return vma;
 	return gate_vma;
 }
  */
 static bool dump_vma_snapshot(struct coredump_params *cprm)
 {
-	struct vm_area_struct *vma, *gate_vma;
+	struct vm_area_struct *gate_vma, *vma = NULL;
 	struct mm_struct *mm = current->mm;
-	int i;
+	MA_STATE(mas, &mm->mm_mt, 0, 0);
+	int i = 0;
 	/*
 	 * Once the stack expansion code is fixed to not change VMA bounds
 		return false;
 	}
-	for (i = 0, vma = first_vma(current, gate_vma); vma != NULL;
+	while ((vma = coredump_next_vma(&mas, vma, gate_vma)) != NULL) {
-			vma = next_vma(vma, gate_vma), i++) {
 		struct core_vma_metadata *m = cprm->vma_meta + i;
 		m->start = vma->vm_start;
 		m->flags = vma->vm_flags;
 		m->dump_size = vma_dump_size(vma, cprm->mm_flags);
 		m->pgoff = vma->vm_pgoff;
 		m->file = vma->vm_file;
 		if (m->file)
 			get_file(m->file);
+		i++;
 	}
 	mmap_write_unlock(mm);

fs/exec.c+8−6

 #include <linux/file.h>
 #include <linux/fdtable.h>
 #include <linux/mm.h>
-#include <linux/vmacache.h>
 #include <linux/stat.h>
 #include <linux/fcntl.h>
 #include <linux/swap.h>
 	unsigned long length = old_end - old_start;
 	unsigned long new_start = old_start - shift;
 	unsigned long new_end = old_end - shift;
+	VMA_ITERATOR(vmi, mm, new_start);
+	struct vm_area_struct *next;
 	struct mmu_gather tlb;
 	BUG_ON(new_start > new_end);
 	 * ensure there are no vmas between where we want to go
 	 * and where we are
 	 */
-	if (vma != find_vma(mm, new_start))
+	if (vma != vma_next(&vmi))
 		return -EFAULT;
 	/*
 	lru_add_drain();
 	tlb_gather_mmu(&tlb, mm);
+	next = vma_next(&vmi);
 	if (new_end > old_start) {
 		/*
 		 * when the old and new regions overlap clear from new_end.
 		 */
 		free_pgd_range(&tlb, new_end, old_end, new_end,
-			vma->vm_next ? vma->vm_next->vm_start : USER_PGTABLES_CEILING);
+			next ? next->vm_start : USER_PGTABLES_CEILING);
 	} else {
 		/*
 		 * otherwise, clean from old_start; this is done to not touch
 		 * for the others its just a little faster.
 		 */
 		free_pgd_range(&tlb, old_start, old_end, new_end,
-			vma->vm_next ? vma->vm_next->vm_start : USER_PGTABLES_CEILING);
+			next ? next->vm_start : USER_PGTABLES_CEILING);
 	}
 	tlb_finish_mmu(&tlb);
 	active_mm = tsk->active_mm;
 	tsk->active_mm = mm;
 	tsk->mm = mm;
+	lru_gen_add_mm(mm);
 	/*
 	 * This prevents preemption while active_mm is being loaded and
 	 * it and mm are being updated, which could cause problems for
 	activate_mm(active_mm, mm);
 	if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM))
 		local_irq_enable();
-	tsk->mm->vmacache_seqnum = 0;
-	vmacache_flush(tsk);
 	task_unlock(tsk);
+	lru_gen_use_mm(mm);
 	if (old_mm) {
 		mmap_read_unlock(old_mm);
 		BUG_ON(active_mm != old_mm);

fs/ext2/balloc.c+4−3

@@ -126,6 +126,7 @@ read_block_bitmap(struct super_block *sb, unsigned int block_group) · 125 unmodified lines
126	struct ext2_group_desc * desc;	126	struct ext2_group_desc * desc;
127	struct buffer_head * bh = NULL;	127	struct buffer_head * bh = NULL;
128	ext2_fsblk_t bitmap_blk;	128	ext2_fsblk_t bitmap_blk;
		129	int ret;
129		130
130	desc = ext2_get_group_desc(sb, block_group, NULL);	131	desc = ext2_get_group_desc(sb, block_group, NULL);
131	if (!desc)	132	if (!desc)
@@ -139,10 +140,10 @@ read_block_bitmap(struct super_block *sb, unsigned int block_group) · 7 unmodified lines
139	block_group, le32_to_cpu(desc->bg_block_bitmap));	140	block_group, le32_to_cpu(desc->bg_block_bitmap));
140	return NULL;	141	return NULL;
141	}	142	}
142	if (likely(bh_uptodate_or_lock(bh)))	143	ret = bh_read(bh, 0);
		144	if (ret > 0)
143	return bh;	145	return bh;
144		146	if (ret < 0) {
145	if (bh_submit_read(bh) < 0) {
146	brelse(bh);	147	brelse(bh);
147	ext2_error(sb, __func__,	148	ext2_error(sb, __func__,
148	"Cannot read block bitmap - "	149	"Cannot read block bitmap - "

fs/fuse/dev.c+2−1

 	       1 << PG_active |
 	       1 << PG_workingset |
 	       1 << PG_reclaim |
-	       1 << PG_waiters))) {
+	       1 << PG_waiters |
+	       LRU_GEN_MASK | LRU_REFS_MASK))) {
 		dump_page(page, "fuse: trying to steal weird page");
 		return 1;
 	}

fs/gfs2/meta_io.c+2−5

 	if (buffer_uptodate(first_bh))
 		goto out;
-	if (!buffer_locked(first_bh))
+	bh_read_nowait(first_bh, REQ_META | REQ_PRIO);
-		ll_rw_block(REQ_OP_READ | REQ_META | REQ_PRIO, 1, &first_bh);
 	dblock++;
 	extlen--;
 	while (extlen) {
 		bh = gfs2_getbuf(gl, dblock, CREATE);
-		if (!buffer_uptodate(bh) && !buffer_locked(bh))
+		bh_readahead(bh, REQ_RAHEAD | REQ_META | REQ_PRIO);
-			ll_rw_block(REQ_OP_READ | REQ_RAHEAD | REQ_META |
-				    REQ_PRIO, 1, &bh);
 		brelse(bh);
 		dblock++;
 		extlen--;

fs/gfs2/quota.c+2−6

 		}
 		if (PageUptodate(page))
 			set_buffer_uptodate(bh);
-		if (!buffer_uptodate(bh)) {
+		if (bh_read(bh, REQ_META | REQ_PRIO) < 0)
-			ll_rw_block(REQ_OP_READ | REQ_META | REQ_PRIO, 1, &bh);
+			goto unlock_out;
-			wait_on_buffer(bh);
-			if (!buffer_uptodate(bh))
-				goto unlock_out;
-		}
 		if (gfs2_is_jdata(ip))
 			gfs2_trans_add_data(ip->i_gl, bh);
 		else

fs/hugetlbfs/inode.c+214−86

@@ -364,13 +364,155 @@ static int hugetlbfs_write_end(struct file file, struct address_space mapping, · 363 unmodified lines
364	return -EINVAL;	364	return -EINVAL;
365	}	365	}
366		366
367	static void remove_huge_page(struct page *page)	367	static void hugetlb_delete_from_page_cache(struct page *page)
368	{	368	{
369	ClearPageDirty(page);	369	ClearPageDirty(page);
370	ClearPageUptodate(page);	370	ClearPageUptodate(page);
371	delete_from_page_cache(page);	371	delete_from_page_cache(page);
372	}	372	}
373		373
		374	/*
		375	* Called with i_mmap_rwsem held for inode based vma maps. This makes
		376	* sure vma (and vm_mm) will not go away. We also hold the hugetlb fault
		377	* mutex for the page in the mapping. So, we can not race with page being
		378	* faulted into the vma.
		379	*/
		380	static bool hugetlb_vma_maps_page(struct vm_area_struct *vma,
		381	unsigned long addr, struct page *page)
		382	{
		383	pte_t *ptep, pte;
		384
		385	ptep = huge_pte_offset(vma->vm_mm, addr,
		386	huge_page_size(hstate_vma(vma)));
		387
		388	if (!ptep)
		389	return false;
		390
		391	pte = huge_ptep_get(ptep);
		392	if (huge_pte_none(pte) \|\| !pte_present(pte))
		393	return false;
		394
		395	if (pte_page(pte) == page)
		396	return true;
		397
		398	return false;
		399	}
		400
		401	/*
		402	* Can vma_offset_start/vma_offset_end overflow on 32-bit arches?
		403	* No, because the interval tree returns us only those vmas
		404	* which overlap the truncated area starting at pgoff,
		405	* and no vma on a 32-bit arch can span beyond the 4GB.
		406	*/
		407	static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start)
		408	{
		409	if (vma->vm_pgoff < start)
		410	return (start - vma->vm_pgoff) << PAGE_SHIFT;
		411	else
		412	return 0;
		413	}
		414
		415	static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end)
		416	{
		417	unsigned long t_end;
		418
		419	if (!end)
		420	return vma->vm_end;
		421
		422	t_end = ((end - vma->vm_pgoff) << PAGE_SHIFT) + vma->vm_start;
		423	if (t_end > vma->vm_end)
		424	t_end = vma->vm_end;
		425	return t_end;
		426	}
		427
		428	/*
		429	* Called with hugetlb fault mutex held. Therefore, no more mappings to
		430	* this folio can be created while executing the routine.
		431	*/
		432	static void hugetlb_unmap_file_folio(struct hstate *h,
		433	struct address_space *mapping,
		434	struct folio *folio, pgoff_t index)
		435	{
		436	struct rb_root_cached *root = &mapping->i_mmap;
		437	struct hugetlb_vma_lock *vma_lock;
		438	struct page *page = &folio->page;
		439	struct vm_area_struct *vma;
		440	unsigned long v_start;
		441	unsigned long v_end;
		442	pgoff_t start, end;
		443
		444	start = index * pages_per_huge_page(h);
		445	end = (index + 1) * pages_per_huge_page(h);
		446
		447	i_mmap_lock_write(mapping);
		448	retry:
		449	vma_lock = NULL;
		450	vma_interval_tree_foreach(vma, root, start, end - 1) {
		451	v_start = vma_offset_start(vma, start);
		452	v_end = vma_offset_end(vma, end);
		453
		454	if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page))
		455	continue;
		456
		457	if (!hugetlb_vma_trylock_write(vma)) {
		458	vma_lock = vma->vm_private_data;
		459	/*
		460	* If we can not get vma lock, we need to drop
		461	* immap_sema and take locks in order. First,
		462	* take a ref on the vma_lock structure so that
		463	* we can be guaranteed it will not go away when
		464	* dropping immap_sema.
		465	*/
		466	kref_get(&vma_lock->refs);
		467	break;
		468	}
		469
		470	unmap_hugepage_range(vma, vma->vm_start + v_start, v_end,
		471	NULL, ZAP_FLAG_DROP_MARKER);
		472	hugetlb_vma_unlock_write(vma);
		473	}
		474
		475	i_mmap_unlock_write(mapping);
		476
		477	if (vma_lock) {
		478	/*
		479	* Wait on vma_lock. We know it is still valid as we have
		480	* a reference. We must 'open code' vma locking as we do
		481	* not know if vma_lock is still attached to vma.
		482	*/
		483	down_write(&vma_lock->rw_sema);
		484	i_mmap_lock_write(mapping);
		485
		486	vma = vma_lock->vma;
		487	if (!vma) {
		488	/*
		489	* If lock is no longer attached to vma, then just
		490	* unlock, drop our reference and retry looking for
		491	* other vmas.
		492	*/
		493	up_write(&vma_lock->rw_sema);
		494	kref_put(&vma_lock->refs, hugetlb_vma_lock_release);
		495	goto retry;
		496	}
		497
		498	/*
		499	* vma_lock is still attached to vma. Check to see if vma
		500	* still maps page and if so, unmap.
		501	*/
		502	v_start = vma_offset_start(vma, start);
		503	v_end = vma_offset_end(vma, end);
		504	if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page))
		505	unmap_hugepage_range(vma, vma->vm_start + v_start,
		506	v_end, NULL,
		507	ZAP_FLAG_DROP_MARKER);
		508
		509	kref_put(&vma_lock->refs, hugetlb_vma_lock_release);
		510	hugetlb_vma_unlock_write(vma);
		511
		512	goto retry;
		513	}
		514	}
		515
374	static void	516	static void
375	hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,	517	hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,
376	zap_flags_t zap_flags)	518	zap_flags_t zap_flags)
@@ -383,32 +525,66 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · 6 unmodified lines
383	* an inclusive "last".	525	* an inclusive "last".
384	*/	526	*/
385	vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {	527	vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {
386	unsigned long v_offset;	528	unsigned long v_start;
387	unsigned long v_end;	529	unsigned long v_end;
388		530
		531	if (!hugetlb_vma_trylock_write(vma))
		532	continue;
		533
		534	v_start = vma_offset_start(vma, start);
		535	v_end = vma_offset_end(vma, end);
		536
		537	unmap_hugepage_range(vma, vma->vm_start + v_start, v_end,
		538	NULL, zap_flags);
		539
389	/*	540	/*
390	* Can the expression below overflow on 32-bit arches?	541	* Note that vma lock only exists for shared/non-private
391	* No, because the interval tree returns us only those vmas	542	* vmas. Therefore, lock is not held when calling
392	* which overlap the truncated area starting at pgoff,	543	* unmap_hugepage_range for private vmas.
393	* and no vma on a 32-bit arch can span beyond the 4GB.
394	*/	544	*/
395	if (vma->vm_pgoff < start)	545	hugetlb_vma_unlock_write(vma);
396	v_offset = (start - vma->vm_pgoff) << PAGE_SHIFT;	546	}
397	else	547	}
398	v_offset = 0;
399
400	if (!end)
401	v_end = vma->vm_end;
402	else {
403	v_end = ((end - vma->vm_pgoff) << PAGE_SHIFT)
404	+ vma->vm_start;
405	if (v_end > vma->vm_end)
406	v_end = vma->vm_end;
407	}
408		548
409	unmap_hugepage_range(vma, vma->vm_start + v_offset, v_end,	549	/*
410	NULL, zap_flags);	550	* Called with hugetlb fault mutex held.
		551	* Returns true if page was actually removed, false otherwise.
		552	*/
		553	static bool remove_inode_single_folio(struct hstate h, struct inode inode,
		554	struct address_space *mapping,
		555	struct folio *folio, pgoff_t index,
		556	bool truncate_op)
		557	{
		558	bool ret = false;
		559
		560	/*
		561	* If folio is mapped, it was faulted in after being
		562	* unmapped in caller. Unmap (again) while holding
		563	* the fault mutex. The mutex will prevent faults
		564	* until we finish removing the folio.
		565	*/
		566	if (unlikely(folio_mapped(folio)))
		567	hugetlb_unmap_file_folio(h, mapping, folio, index);
		568
		569	folio_lock(folio);
		570	/*
		571	* We must remove the folio from page cache before removing
		572	* the region/ reserve map (hugetlb_unreserve_pages). In
		573	* rare out of memory conditions, removal of the region/reserve
		574	* map could fail. Correspondingly, the subpool and global
		575	* reserve usage count can need to be adjusted.
		576	*/
		577	VM_BUG_ON(HPageRestoreReserve(&folio->page));
		578	hugetlb_delete_from_page_cache(&folio->page);
		579	ret = true;
		580	if (!truncate_op) {
		581	if (unlikely(hugetlb_unreserve_pages(inode, index,
		582	index + 1, 1)))
		583	hugetlb_fix_reserve_counts(inode);
411	}	584	}
		585
		586	folio_unlock(folio);
		587	return ret;
412	}	588	}
413		589
414	/*	590	/*
@@ -418,10 +594,10 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · 3 unmodified lines
418	* truncation is indicated by end of range being LLONG_MAX	594	* truncation is indicated by end of range being LLONG_MAX
419	* In this case, we first scan the range and release found pages.	595	* In this case, we first scan the range and release found pages.
420	* After releasing pages, hugetlb_unreserve_pages cleans up region/reserve	596	* After releasing pages, hugetlb_unreserve_pages cleans up region/reserve
421	* maps and global counts. Page faults can not race with truncation	597	* maps and global counts. Page faults can race with truncation.
422	* in this routine. hugetlb_no_page() holds i_mmap_rwsem and prevents	598	* During faults, hugetlb_no_page() checks i_size before page allocation,
423	* page faults in the truncated range by checking i_size. i_size is	599	* and again after obtaining page table lock. It will 'back out'
424	* modified while holding i_mmap_rwsem.	600	* allocations in the truncated range.
425	* hole punch is indicated if end is not LLONG_MAX	601	* hole punch is indicated if end is not LLONG_MAX
426	* In the hole punch case we scan the range and release found pages.	602	* In the hole punch case we scan the range and release found pages.
427	* Only when releasing a page is the associated region/reserve map	603	* Only when releasing a page is the associated region/reserve map
@@ -451,61 +627,17 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 23 unmodified lines
451	u32 hash = 0;	627	u32 hash = 0;
452		628
453	index = folio->index;	629	index = folio->index;
454	if (!truncate_op) {	630	hash = hugetlb_fault_mutex_hash(mapping, index);
455	/*	631	mutex_lock(&hugetlb_fault_mutex_table[hash]);
456	* Only need to hold the fault mutex in the
457	* hole punch case. This prevents races with
458	* page faults. Races are not possible in the
459	* case of truncation.
460	*/
461	hash = hugetlb_fault_mutex_hash(mapping, index);
462	mutex_lock(&hugetlb_fault_mutex_table[hash]);
463	}
464		632
465	/*	633	/*
466	* If folio is mapped, it was faulted in after being	634	* Remove folio that was part of folio_batch.
467	* unmapped in caller. Unmap (again) now after taking
468	* the fault mutex. The mutex will prevent faults
469	* until we finish removing the folio.
470	*
471	* This race can only happen in the hole punch case.
472	* Getting here in a truncate operation is a bug.
473	*/	635	*/
474	if (unlikely(folio_mapped(folio))) {	636	if (remove_inode_single_folio(h, inode, mapping, folio,
475	BUG_ON(truncate_op);	637	index, truncate_op))
476		638	freed++;
477	mutex_unlock(&hugetlb_fault_mutex_table[hash]);	639
478	i_mmap_lock_write(mapping);	640	mutex_unlock(&hugetlb_fault_mutex_table[hash]);
479	mutex_lock(&hugetlb_fault_mutex_table[hash]);
480	hugetlb_vmdelete_list(&mapping->i_mmap,
481	index * pages_per_huge_page(h),
482	(index + 1) * pages_per_huge_page(h),
483	ZAP_FLAG_DROP_MARKER);
484	i_mmap_unlock_write(mapping);
485	}
486
487	folio_lock(folio);
488	/*
489	* We must free the huge page and remove from page
490	* cache (remove_huge_page) BEFORE removing the
491	* region/reserve map (hugetlb_unreserve_pages). In
492	* rare out of memory conditions, removal of the
493	* region/reserve map could fail. Correspondingly,
494	* the subpool and global reserve usage count can need
495	* to be adjusted.
496	*/
497	VM_BUG_ON(HPageRestoreReserve(&folio->page));
498	remove_huge_page(&folio->page);
499	freed++;
500	if (!truncate_op) {
501	if (unlikely(hugetlb_unreserve_pages(inode,
502	index, index + 1, 1)))
503	hugetlb_fix_reserve_counts(inode);
504	}
505
506	folio_unlock(folio);
507	if (!truncate_op)
508	mutex_unlock(&hugetlb_fault_mutex_table[hash]);
509	}	641	}
510	folio_batch_release(&fbatch);	642	folio_batch_release(&fbatch);
511	cond_resched();	643	cond_resched();
@@ -543,8 +675,8 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset) · 31 unmodified lines
543	BUG_ON(offset & ~huge_page_mask(h));	675	BUG_ON(offset & ~huge_page_mask(h));
544	pgoff = offset >> PAGE_SHIFT;	676	pgoff = offset >> PAGE_SHIFT;
545		677
546	i_mmap_lock_write(mapping);
547	i_size_write(inode, offset);	678	i_size_write(inode, offset);
		679	i_mmap_lock_write(mapping);
548	if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))	680	if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))
549	hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0,	681	hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0,
550	ZAP_FLAG_DROP_MARKER);	682	ZAP_FLAG_DROP_MARKER);
@@ -703,11 +835,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 152 unmodified lines
703	/* addr is the offset within the file (zero based) */	835	/* addr is the offset within the file (zero based) */
704	addr = index * hpage_size;	836	addr = index * hpage_size;
705		837
706	/*	838	/* mutex taken here, fault path and hole punch */
707	* fault mutex taken here, protects against fault path
708	* and hole punch. inode_lock previously taken protects
709	* against truncation.
710	*/
711	hash = hugetlb_fault_mutex_hash(mapping, index);	839	hash = hugetlb_fault_mutex_hash(mapping, index);
712	mutex_lock(&hugetlb_fault_mutex_table[hash]);	840	mutex_lock(&hugetlb_fault_mutex_table[hash]);
713		841
@@ -737,7 +865,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 23 unmodified lines
737	}	865	}
738	clear_huge_page(page, addr, pages_per_huge_page(h));	866	clear_huge_page(page, addr, pages_per_huge_page(h));
739	__SetPageUptodate(page);	867	__SetPageUptodate(page);
740	error = huge_add_to_page_cache(page, mapping, index);	868	error = hugetlb_add_to_page_cache(page, mapping, index);
741	if (unlikely(error)) {	869	if (unlikely(error)) {
742	restore_reserve_on_error(h, &pseudo_vma, addr, page);	870	restore_reserve_on_error(h, &pseudo_vma, addr, page);
743	put_page(page);	871	put_page(page);
@@ -749,7 +877,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 5 unmodified lines
749		877
750	SetHPageMigratable(page);	878	SetHPageMigratable(page);
751	/*	879	/*
752	* unlock_page because locked by huge_add_to_page_cache()	880	* unlock_page because locked by hugetlb_add_to_page_cache()
753	* put_page() due to reference from alloc_huge_page()	881	* put_page() due to reference from alloc_huge_page()
754	*/	882	*/
755	unlock_page(page);	883	unlock_page(page);
@@ -994,7 +1122,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping, · 238 unmodified lines
994	struct inode *inode = mapping->host;	1122	struct inode *inode = mapping->host;
995	pgoff_t index = page->index;	1123	pgoff_t index = page->index;
996		1124
997	remove_huge_page(page);	1125	hugetlb_delete_from_page_cache(page);
998	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))	1126	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))
999	hugetlb_fix_reserve_counts(inode);	1127	hugetlb_fix_reserve_counts(inode);
1000		1128

fs/isofs/compress.c+1−1

@@ -82,7 +82,7 @@ static loff_t zisofs_uncompress_block(struct inode *inode, loff_t block_start, · 81 unmodified lines
82	return 0;	82	return 0;
83	}	83	}
84	haveblocks = isofs_get_blocks(inode, blocknum, bhs, needblocks);	84	haveblocks = isofs_get_blocks(inode, blocknum, bhs, needblocks);
85	ll_rw_block(REQ_OP_READ, haveblocks, bhs);	85	bh_read_batch(haveblocks, bhs);
86		86
87	curbh = 0;	87	curbh = 0;
88	curpage = 0;	88	curpage = 0;

fs/jbd2/journal.c+6−9

 {
 	struct buffer_head *bh;
 	journal_superblock_t *sb;
-	int err = -EIO;
+	int err;
 	bh = journal->j_sb_buffer;
 	J_ASSERT(bh != NULL);
-	if (!buffer_uptodate(bh)) {
+	err = bh_read(bh, 0);
-		ll_rw_block(REQ_OP_READ, 1, &bh);
+	if (err < 0) {
-		wait_on_buffer(bh);
+		printk(KERN_ERR
-		if (!buffer_uptodate(bh)) {
+			"JBD2: IO error reading journal superblock\n");
-			printk(KERN_ERR
+		goto out;
-				"JBD2: IO error reading journal superblock\n");
-			goto out;
-		}
 	}
 	if (buffer_verified(bh))

fs/jbd2/recovery.c+10−6

 		if (!buffer_uptodate(bh) && !buffer_locked(bh)) {
 			bufs[nbufs++] = bh;
 			if (nbufs == MAXBUF) {
-				ll_rw_block(REQ_OP_READ, nbufs, bufs);
+				bh_readahead_batch(nbufs, bufs, 0);
 				journal_brelse_array(bufs, nbufs);
 				nbufs = 0;
 			}
 	}
 	if (nbufs)
-		ll_rw_block(REQ_OP_READ, nbufs, bufs);
+		bh_readahead_batch(nbufs, bufs, 0);
 	err = 0;
 failed:
 		return -ENOMEM;
 	if (!buffer_uptodate(bh)) {
-		/* If this is a brand new buffer, start readahead.
+		/*
-                   Otherwise, we assume we are already reading it.  */
+		 * If this is a brand new buffer, start readahead.
-		if (!buffer_req(bh))
+		 * Otherwise, we assume we are already reading it.
+		 */
+		bool need_readahead = !buffer_req(bh);
+		bh_read_nowait(bh, 0);
+		if (need_readahead)
 			do_readahead(journal, offset);
 		wait_on_buffer(bh);
 	}
 					mark_buffer_dirty(nbh);
 					BUFFER_TRACE(nbh, "marking uptodate");
 					++info->nr_replays;
-					/* ll_rw_block(WRITE, 1, &nbh); */
 					unlock_buffer(nbh);
 					brelse(obh);
 					brelse(nbh);

fs/namei.c+1−1

@@ -5088,7 +5088,7 @@ int page_symlink(struct inode inode, const char symname, int len) · 5087 unmodified lines
5088	const struct address_space_operations *aops = mapping->a_ops;	5088	const struct address_space_operations *aops = mapping->a_ops;
5089	bool nofs = !mapping_gfp_constraint(mapping, __GFP_FS);	5089	bool nofs = !mapping_gfp_constraint(mapping, __GFP_FS);
5090	struct page *page;	5090	struct page *page;
5091	void *fsdata;	5091	void *fsdata = NULL;
5092	int err;	5092	int err;
5093	unsigned int flags;	5093	unsigned int flags;
5094		5094

fs/nilfs2/page.c+18−27

 					    sector_t start_blk,
 					    sector_t *blkoff)
 {
-	unsigned int i;
+	unsigned int i, nr_folios;
 	pgoff_t index;
-	unsigned int nblocks_in_page;
 	unsigned long length = 0;
-	sector_t b;
+	struct folio_batch fbatch;
-	struct pagevec pvec;
+	struct folio *folio;
-	struct page *page;
 	if (inode->i_mapping->nrpages == 0)
 		return 0;
 	index = start_blk >> (PAGE_SHIFT - inode->i_blkbits);
-	nblocks_in_page = 1U << (PAGE_SHIFT - inode->i_blkbits);
-	pagevec_init(&pvec);
+	folio_batch_init(&fbatch);
 repeat:
-	pvec.nr = find_get_pages_contig(inode->i_mapping, index, PAGEVEC_SIZE,
+	nr_folios = filemap_get_folios_contig(inode->i_mapping, &index, ULONG_MAX,
-					pvec.pages);
+			&fbatch);
-	if (pvec.nr == 0)
+	if (nr_folios == 0)
 		return length;
-	if (length > 0 && pvec.pages[0]->index > index)
-		goto out;
-	b = pvec.pages[0]->index << (PAGE_SHIFT - inode->i_blkbits);
 	i = 0;
 	do {
-		page = pvec.pages[i];
+		folio = fbatch.folios[i];
-		lock_page(page);
+		folio_lock(folio);
-		if (page_has_buffers(page)) {
+		if (folio_buffers(folio)) {
 			struct buffer_head *bh, *head;
+			sector_t b;
-			bh = head = page_buffers(page);
+			b = folio->index << (PAGE_SHIFT - inode->i_blkbits);
+			bh = head = folio_buffers(folio);
 			do {
 				if (b < start_blk)
 					continue;
 		} else {
 			if (length > 0)
 				goto out_locked;
-			b += nblocks_in_page;
 		}
-		unlock_page(page);
+		folio_unlock(folio);
-	} while (++i < pagevec_count(&pvec));
+	} while (++i < nr_folios);
-	index = page->index + 1;
+	folio_batch_release(&fbatch);
-	pagevec_release(&pvec);
 	cond_resched();
 	goto repeat;
 out_locked:
-	unlock_page(page);
+	folio_unlock(folio);
-out:
+	folio_batch_release(&fbatch);
-	pagevec_release(&pvec);
 	return length;
 }

fs/ntfs3/inode.c+2−5

 			bh->b_size = block_size;
 			off = vbo & (PAGE_SIZE - 1);
 			set_bh_page(bh, page, off);
-			ll_rw_block(REQ_OP_READ, 1, &bh);
+			err = bh_read(bh, 0);
-			wait_on_buffer(bh);
+			if (err < 0)
-			if (!buffer_uptodate(bh)) {
-				err = -EIO;
 				goto out;
-			}
 			zero_user_segment(page, off + voff, off + block_size);
 		}
 	}

fs/ocfs2/aops.c+1−1

@@ -636,7 +636,7 @@ int ocfs2_map_page_blocks(struct page page, u64 p_blkno, · 635 unmodified lines
636	!buffer_new(bh) &&	636	!buffer_new(bh) &&
637	ocfs2_should_read_blk(inode, page, block_start) &&	637	ocfs2_should_read_blk(inode, page, block_start) &&
638	(block_start < from \|\| block_end > to)) {	638	(block_start < from \|\| block_end > to)) {
639	ll_rw_block(REQ_OP_READ, 1, &bh);	639	bh_read_nowait(bh, 0);
640	*wait_bh++=bh;	640	*wait_bh++=bh;
641	}	641	}
642		642

fs/ocfs2/super.c+1−3

 	if (!buffer_dirty(*bh))
 		clear_buffer_uptodate(*bh);
 	unlock_buffer(*bh);
-	ll_rw_block(REQ_OP_READ, 1, bh);
+	if (bh_read(*bh, 0) < 0) {
-	wait_on_buffer(*bh);
-	if (!buffer_uptodate(*bh)) {
 		mlog_errno(-EIO);
 		brelse(*bh);
 		*bh = NULL;

fs/posix_acl.c+3−0

 #include <linux/user_namespace.h>
 #include <linux/namei.h>
 #include <linux/mnt_idmapping.h>
+#include <linux/iversion.h>
 static struct posix_acl **acl_by_type(struct inode *inode, int type)
 {
 	}
 	inode->i_ctime = current_time(inode);
+	if (IS_I_VERSION(inode))
+		inode_inc_iversion(inode);
 	set_cached_acl(inode, type, acl);
 	return 0;
 }

fs/proc/base.c+19−1

 	GENRADIX(struct map_files_info) fa;
 	struct map_files_info *p;
 	int ret;
+	struct vma_iterator vmi;
 	genradix_init(&fa);
 	 * routine might require mmap_lock taken in might_fault().
 	 */
-	for (vma = mm->mmap, pos = 2; vma; vma = vma->vm_next) {
+	pos = 2;
+	vma_iter_init(&vmi, mm, 0);
+	for_each_vma(vmi, vma) {
 		if (!vma->vm_file)
 			continue;
 		if (++pos <= ctx->pos)
 	return 0;
 }
+static int proc_pid_ksm_stat(struct seq_file *m, struct pid_namespace *ns,
+				struct pid *pid, struct task_struct *task)
+{
+	struct mm_struct *mm;
+	mm = get_task_mm(task);
+	if (mm) {
+		seq_printf(m, "ksm_rmap_items %lu\n", mm->ksm_rmap_items);
+		mmput(mm);
+	}
+	return 0;
+}
 #endif /* CONFIG_KSM */
 #ifdef CONFIG_STACKLEAK_METRICS
 #endif
 #ifdef CONFIG_KSM
 	ONE("ksm_merging_pages",  S_IRUSR, proc_pid_ksm_merging_pages),
+	ONE("ksm_stat",  S_IRUSR, proc_pid_ksm_stat),
 #endif
 };
 #endif
 #ifdef CONFIG_KSM
 	ONE("ksm_merging_pages",  S_IRUSR, proc_pid_ksm_merging_pages),
+	ONE("ksm_stat",  S_IRUSR, proc_pid_ksm_stat),
 #endif
 };

fs/proc/internal.h+1−1

@@ -285,7 +285,7 @@ struct proc_maps_private { · 284 unmodified lines
285	struct task_struct *task;	285	struct task_struct *task;
286	struct mm_struct *mm;	286	struct mm_struct *mm;
287	#ifdef CONFIG_MMU	287	#ifdef CONFIG_MMU
288	struct vm_area_struct *tail_vma;	288	struct vma_iterator iter;
289	#endif	289	#endif
290	#ifdef CONFIG_NUMA	290	#ifdef CONFIG_NUMA
291	struct mempolicy *task_mempolicy;	291	struct mempolicy *task_mempolicy;

fs/proc/task_mmu.c+59−37

@@ -1,6 +1,5 @@
1	// SPDX-License-Identifier: GPL-2.0	1	// SPDX-License-Identifier: GPL-2.0
2	#include <linux/pagewalk.h>	2	#include <linux/pagewalk.h>
3	#include <linux/vmacache.h>
4	#include <linux/mm_inline.h>	3	#include <linux/mm_inline.h>
5	#include <linux/hugetlb.h>	4	#include <linux/hugetlb.h>
6	#include <linux/huge_mm.h>	5	#include <linux/huge_mm.h>
@@ -124,12 +123,26 @@ static void release_task_mempolicy(struct proc_maps_private *priv) · 117 unmodified lines
124	}	123	}
125	#endif	124	#endif
126		125
		126	static struct vm_area_struct proc_get_vma(struct proc_maps_private priv,
		127	loff_t *ppos)
		128	{
		129	struct vm_area_struct *vma = vma_next(&priv->iter);
		130
		131	if (vma) {
		132	*ppos = vma->vm_start;
		133	} else {
		134	*ppos = -2UL;
		135	vma = get_gate_vma(priv->mm);
		136	}
		137
		138	return vma;
		139	}
		140
127	static void m_start(struct seq_file m, loff_t *ppos)	141	static void m_start(struct seq_file m, loff_t *ppos)
128	{	142	{
129	struct proc_maps_private *priv = m->private;	143	struct proc_maps_private *priv = m->private;
130	unsigned long last_addr = *ppos;	144	unsigned long last_addr = *ppos;
131	struct mm_struct *mm;	145	struct mm_struct *mm;
132	struct vm_area_struct *vma;
133		146
134	/* See m_next(). Zero at the start or after lseek. */	147	/* See m_next(). Zero at the start or after lseek. */
135	if (last_addr == -1UL)	148	if (last_addr == -1UL)
@@ -153,31 +166,21 @@ static void m_start(struct seq_file m, loff_t *ppos) · 17 unmodified lines
153	return ERR_PTR(-EINTR);	166	return ERR_PTR(-EINTR);
154	}	167	}
155		168
		169	vma_iter_init(&priv->iter, mm, last_addr);
156	hold_task_mempolicy(priv);	170	hold_task_mempolicy(priv);
157	priv->tail_vma = get_gate_vma(mm);	171	if (last_addr == -2UL)
158		172	return get_gate_vma(mm);
159	vma = find_vma(mm, last_addr);
160	if (vma)
161	return vma;
162		173
163	return priv->tail_vma;	174	return proc_get_vma(priv, ppos);
164	}	175	}
165		176
166	static void m_next(struct seq_file m, void v, loff_t ppos)	177	static void m_next(struct seq_file m, void v, loff_t ppos)
167	{	178	{
168	struct proc_maps_private *priv = m->private;	179	if (*ppos == -2UL) {
169	struct vm_area_struct next, vma = v;	180	*ppos = -1UL;
170		181	return NULL;
171	if (vma == priv->tail_vma)	182	}
172	next = NULL;	183	return proc_get_vma(m->private, ppos);
173	else if (vma->vm_next)
174	next = vma->vm_next;
175	else
176	next = priv->tail_vma;
177
178	*ppos = next ? next->vm_start : -1UL;
179
180	return next;
181	}	184	}
182		185
183	static void m_stop(struct seq_file m, void v)	186	static void m_stop(struct seq_file m, void v)
@@ -864,7 +867,7 @@ static int show_smap(struct seq_file m, void v) · 680 unmodified lines
864	__show_smap(m, &mss, false);	867	__show_smap(m, &mss, false);
865		868
866	seq_printf(m, "THPeligible: %d\n",	869	seq_printf(m, "THPeligible: %d\n",
867	hugepage_vma_check(vma, vma->vm_flags, true, false));	870	hugepage_vma_check(vma, vma->vm_flags, true, false, true));
868		871
869	if (arch_pkeys_enabled())	872	if (arch_pkeys_enabled())
870	seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma));	873	seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma));
@@ -877,16 +880,16 @@ static int show_smaps_rollup(struct seq_file m, void v) · 6 unmodified lines
877	{	880	{
878	struct proc_maps_private *priv = m->private;	881	struct proc_maps_private *priv = m->private;
879	struct mem_size_stats mss;	882	struct mem_size_stats mss;
880	struct mm_struct *mm;	883	struct mm_struct *mm = priv->mm;
881	struct vm_area_struct *vma;	884	struct vm_area_struct *vma;
882	unsigned long last_vma_end = 0;	885	unsigned long vma_start = 0, last_vma_end = 0;
883	int ret = 0;	886	int ret = 0;
		887	MA_STATE(mas, &mm->mm_mt, 0, 0);
884		888
885	priv->task = get_proc_task(priv->inode);	889	priv->task = get_proc_task(priv->inode);
886	if (!priv->task)	890	if (!priv->task)
887	return -ESRCH;	891	return -ESRCH;
888		892
889	mm = priv->mm;
890	if (!mm \|\| !mmget_not_zero(mm)) {	893	if (!mm \|\| !mmget_not_zero(mm)) {
891	ret = -ESRCH;	894	ret = -ESRCH;
892	goto out_put_task;	895	goto out_put_task;
@@ -899,8 +902,13 @@ static int show_smaps_rollup(struct seq_file m, void v) · 6 unmodified lines
899	goto out_put_mm;	902	goto out_put_mm;
900		903
901	hold_task_mempolicy(priv);	904	hold_task_mempolicy(priv);
		905	vma = mas_find(&mas, 0);
		906
		907	if (unlikely(!vma))
		908	goto empty_set;
902		909
903	for (vma = priv->mm->mmap; vma;) {	910	vma_start = vma->vm_start;
		911	do {
904	smap_gather_stats(vma, &mss, 0);	912	smap_gather_stats(vma, &mss, 0);
905	last_vma_end = vma->vm_end;	913	last_vma_end = vma->vm_end;
906		914
@@ -909,6 +917,7 @@ static int show_smaps_rollup(struct seq_file m, void v) · 2 unmodified lines
909	* access it for write request.	917	* access it for write request.
910	*/	918	*/
911	if (mmap_lock_is_contended(mm)) {	919	if (mmap_lock_is_contended(mm)) {
		920	mas_pause(&mas);
912	mmap_read_unlock(mm);	921	mmap_read_unlock(mm);
913	ret = mmap_read_lock_killable(mm);	922	ret = mmap_read_lock_killable(mm);
914	if (ret) {	923	if (ret) {
@@ -952,7 +961,7 @@ static int show_smaps_rollup(struct seq_file m, void v) · 37 unmodified lines
952	* contains last_vma_end.	961	* contains last_vma_end.
953	* Iterate VMA' from last_vma_end.	962	* Iterate VMA' from last_vma_end.
954	*/	963	*/
955	vma = find_vma(mm, last_vma_end - 1);	964	vma = mas_find(&mas, ULONG_MAX);
956	/* Case 3 above */	965	/* Case 3 above */
957	if (!vma)	966	if (!vma)
958	break;	967	break;
@@ -966,11 +975,10 @@ static int show_smaps_rollup(struct seq_file m, void v) · 7 unmodified lines
966	smap_gather_stats(vma, &mss, last_vma_end);	975	smap_gather_stats(vma, &mss, last_vma_end);
967	}	976	}
968	/* Case 2 above */	977	/* Case 2 above */
969	vma = vma->vm_next;	978	} while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);
970	}
971		979
972	show_vma_header_prefix(m, priv->mm->mmap->vm_start,	980	empty_set:
973	last_vma_end, 0, 0, 0, 0);	981	show_vma_header_prefix(m, vma_start, last_vma_end, 0, 0, 0, 0);
974	seq_pad(m, ' ');	982	seq_pad(m, ' ');
975	seq_puts(m, "[rollup]\n");	983	seq_puts(m, "[rollup]\n");
976		984
@@ -1263,6 +1271,7 @@ static ssize_t clear_refs_write(struct file file, const char __user buf, · 286 unmodified lines
1263	return -ESRCH;	1271	return -ESRCH;
1264	mm = get_task_mm(task);	1272	mm = get_task_mm(task);
1265	if (mm) {	1273	if (mm) {
		1274	MA_STATE(mas, &mm->mm_mt, 0, 0);
1266	struct mmu_notifier_range range;	1275	struct mmu_notifier_range range;
1267	struct clear_refs_private cp = {	1276	struct clear_refs_private cp = {
1268	.type = type,	1277	.type = type,
@@ -1282,7 +1291,7 @@ static ssize_t clear_refs_write(struct file file, const char __user buf, · 13 unmodified lines
1282	}	1291	}
1283		1292
1284	if (type == CLEAR_REFS_SOFT_DIRTY) {	1293	if (type == CLEAR_REFS_SOFT_DIRTY) {
1285	for (vma = mm->mmap; vma; vma = vma->vm_next) {	1294	mas_for_each(&mas, vma, ULONG_MAX) {
1286	if (!(vma->vm_flags & VM_SOFTDIRTY))	1295	if (!(vma->vm_flags & VM_SOFTDIRTY))
1287	continue;	1296	continue;
1288	vma->vm_flags &= ~VM_SOFTDIRTY;	1297	vma->vm_flags &= ~VM_SOFTDIRTY;
@@ -1294,8 +1303,7 @@ static ssize_t clear_refs_write(struct file file, const char __user buf, · 5 unmodified lines
1294	0, NULL, mm, 0, -1UL);	1303	0, NULL, mm, 0, -1UL);
1295	mmu_notifier_invalidate_range_start(&range);	1304	mmu_notifier_invalidate_range_start(&range);
1296	}	1305	}
1297	walk_page_range(mm, 0, mm->highest_vm_end, &clear_refs_walk_ops,	1306	walk_page_range(mm, 0, -1, &clear_refs_walk_ops, &cp);
1298	&cp);
1299	if (type == CLEAR_REFS_SOFT_DIRTY) {	1307	if (type == CLEAR_REFS_SOFT_DIRTY) {
1300	mmu_notifier_invalidate_range_end(&range);	1308	mmu_notifier_invalidate_range_end(&range);
1301	flush_tlb_mm(mm);	1309	flush_tlb_mm(mm);
@@ -1418,9 +1426,19 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 116 unmodified lines
1418	if (pte_swp_uffd_wp(pte))	1426	if (pte_swp_uffd_wp(pte))
1419	flags \|= PM_UFFD_WP;	1427	flags \|= PM_UFFD_WP;
1420	entry = pte_to_swp_entry(pte);	1428	entry = pte_to_swp_entry(pte);
1421	if (pm->show_pfn)	1429	if (pm->show_pfn) {
		1430	pgoff_t offset;
		1431	/*
		1432	* For PFN swap offsets, keeping the offset field
		1433	* to be PFN only to be compatible with old smaps.
		1434	*/
		1435	if (is_pfn_swap_entry(entry))
		1436	offset = swp_offset_pfn(entry);
		1437	else
		1438	offset = swp_offset(entry);
1422	frame = swp_type(entry) \|	1439	frame = swp_type(entry) \|
1423	(swp_offset(entry) << MAX_SWAPFILES_SHIFT);	1440	(offset << MAX_SWAPFILES_SHIFT);
		1441	}
1424	flags \|= PM_SWAP;	1442	flags \|= PM_SWAP;
1425	migration = is_migration_entry(entry);	1443	migration = is_migration_entry(entry);
1426	if (is_pfn_swap_entry(entry))	1444	if (is_pfn_swap_entry(entry))
@@ -1477,7 +1495,11 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 50 unmodified lines
1477	unsigned long offset;	1495	unsigned long offset;
1478		1496
1479	if (pm->show_pfn) {	1497	if (pm->show_pfn) {
1480	offset = swp_offset(entry) +	1498	if (is_pfn_swap_entry(entry))
		1499	offset = swp_offset_pfn(entry);
		1500	else
		1501	offset = swp_offset(entry);
		1502	offset = offset +
1481	((addr & ~PMD_MASK) >> PAGE_SHIFT);	1503	((addr & ~PMD_MASK) >> PAGE_SHIFT);
1482	frame = swp_type(entry) \|	1504	frame = swp_type(entry) \|
1483	(offset << MAX_SWAPFILES_SHIFT);	1505	(offset << MAX_SWAPFILES_SHIFT);

fs/proc/task_nommu.c+21−24

  */
 void task_mem(struct seq_file *m, struct mm_struct *mm)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
 	struct vm_region *region;
-	struct rb_node *p;
 	unsigned long bytes = 0, sbytes = 0, slack = 0, size;
-	mmap_read_lock(mm);
-	for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
-		vma = rb_entry(p, struct vm_area_struct, vm_rb);
+	mmap_read_lock(mm);
+	for_each_vma(vmi, vma) {
 		bytes += kobjsize(vma);
 		region = vma->vm_region;
 unsigned long task_vsize(struct mm_struct *mm)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
-	struct rb_node *p;
 	unsigned long vsize = 0;
 	mmap_read_lock(mm);
-	for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
+	for_each_vma(vmi, vma)
-		vma = rb_entry(p, struct vm_area_struct, vm_rb);
 		vsize += vma->vm_end - vma->vm_start;
-	}
 	mmap_read_unlock(mm);
 	return vsize;
 }
 			 unsigned long *shared, unsigned long *text,
 			 unsigned long *data, unsigned long *resident)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
 	struct vm_region *region;
-	struct rb_node *p;
 	unsigned long size = kobjsize(mm);
 	mmap_read_lock(mm);
-	for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
+	for_each_vma(vmi, vma) {
-		vma = rb_entry(p, struct vm_area_struct, vm_rb);
 		size += kobjsize(vma);
 		region = vma->vm_region;
 		if (region) {
  */
 static int show_map(struct seq_file *m, void *_p)
 {
-	struct rb_node *p = _p;
+	return nommu_vma_show(m, _p);
-	return nommu_vma_show(m, rb_entry(p, struct vm_area_struct, vm_rb));
 }
 static void *m_start(struct seq_file *m, loff_t *pos)
 {
 	struct proc_maps_private *priv = m->private;
 	struct mm_struct *mm;
-	struct rb_node *p;
+	struct vm_area_struct *vma;
-	loff_t n = *pos;
+	unsigned long addr = *pos;
+	/* See m_next(). Zero at the start or after lseek. */
+	if (addr == -1UL)
+		return NULL;
 	/* pin the task and mm whilst we play with them */
 	priv->task = get_proc_task(priv->inode);
 		return ERR_PTR(-EINTR);
 	}
-	/* start from the Nth VMA */
+	/* start the next element from addr */
-	for (p = rb_first(&mm->mm_rb); p; p = rb_next(p))
+	vma = find_vma(mm, addr);
-		if (n-- == 0)
+	if (vma)
-			return p;
+		return vma;
 	mmap_read_unlock(mm);
 	mmput(mm);
 static void *m_next(struct seq_file *m, void *_p, loff_t *pos)
 {
-	struct rb_node *p = _p;
+	struct vm_area_struct *vma = _p;
-	(*pos)++;
+	*pos = vma->vm_end;
-	return p ? rb_next(p) : NULL;
+	return find_vma(vma->vm_mm, vma->vm_end);
 }
 static const struct seq_operations proc_pid_maps_ops = {

fs/ramfs/file-nommu.c+29−21

@@ -203,9 +203,9 @@ static unsigned long ramfs_nommu_get_unmapped_area(struct file *file, · 202 unmodified lines
203	unsigned long addr, unsigned long len,	203	unsigned long addr, unsigned long len,
204	unsigned long pgoff, unsigned long flags)	204	unsigned long pgoff, unsigned long flags)
205	{	205	{
206	unsigned long maxpages, lpages, nr, loop, ret;	206	unsigned long maxpages, lpages, nr_folios, loop, ret, nr_pages, pfn;
207	struct inode *inode = file_inode(file);	207	struct inode *inode = file_inode(file);
208	struct page pages = NULL, ptr, *page;	208	struct folio_batch fbatch;
209	loff_t isize;	209	loff_t isize;
210		210
211	/* the mapping mustn't extend beyond the EOF */	211	/* the mapping mustn't extend beyond the EOF */
@@ -221,31 +221,39 @@ static unsigned long ramfs_nommu_get_unmapped_area(struct file *file, · 9 unmodified lines
221	goto out;	221	goto out;
222		222
223	/* gang-find the pages */	223	/* gang-find the pages */
224	pages = kcalloc(lpages, sizeof(struct page *), GFP_KERNEL);	224	folio_batch_init(&fbatch);
225	if (!pages)	225	nr_pages = 0;
226	goto out_free;	226	repeat:
227		227	nr_folios = filemap_get_folios_contig(inode->i_mapping, &pgoff,
228	nr = find_get_pages_contig(inode->i_mapping, pgoff, lpages, pages);	228	ULONG_MAX, &fbatch);
229	if (nr != lpages)	229	if (!nr_folios) {
230	goto out_free_pages; /* leave if some pages were missing */	230	ret = -ENOSYS;
		231	return ret;
		232	}
231		233
		234	if (ret == -ENOSYS) {
		235	ret = (unsigned long) folio_address(fbatch.folios[0]);
		236	pfn = folio_pfn(fbatch.folios[0]);
		237	}
232	/* check the pages for physical adjacency */	238	/* check the pages for physical adjacency */
233	ptr = pages;	239	for (loop = 0; loop < nr_folios; loop++) {
234	page = *ptr++;	240	if (pfn + nr_pages != folio_pfn(fbatch.folios[loop])) {
235	page++;	241	ret = -ENOSYS;
236	for (loop = lpages; loop > 1; loop--)	242	goto out_free; /* leave if not physical adjacent */
237	if (*ptr++ != page++)	243	}
238	goto out_free_pages;	244	nr_pages += folio_nr_pages(fbatch.folios[loop]);
		245	if (nr_pages >= lpages)
		246	goto out_free; /* successfully found desired pages*/
		247	}
239		248
		249	if (nr_pages < lpages) {
		250	folio_batch_release(&fbatch);
		251	goto repeat; /* loop if pages are missing */
		252	}
240	/* okay - all conditions fulfilled */	253	/* okay - all conditions fulfilled */
241	ret = (unsigned long) page_address(pages[0]);
242		254
243	out_free_pages:
244	ptr = pages;
245	for (loop = nr; loop > 0; loop--)
246	put_page(*ptr++);
247	out_free:	255	out_free:
248	kfree(pages);	256	folio_batch_release(&fbatch);
249	out:	257	out:
250	return ret;	258	return ret;
251	}	259	}

fs/reiserfs/journal.c+6−5

 		 */
 		if (buffer_dirty(bh) && unlikely(bh->b_page->mapping == NULL)) {
 			spin_unlock(lock);
-			ll_rw_block(REQ_OP_WRITE, 1, &bh);
+			write_dirty_buffer(bh, 0);
 			spin_lock(lock);
 		}
 		put_bh(bh);
 		if (tbh) {
 			if (buffer_dirty(tbh)) {
 		            depth = reiserfs_write_unlock_nested(s);
-			    ll_rw_block(REQ_OP_WRITE, 1, &tbh);
+			    write_dirty_buffer(tbh, 0);
 			    reiserfs_write_lock_nested(s, depth);
 			}
 			put_bh(tbh) ;
 		}
 	}
 	/* read in the log blocks, memcpy to the corresponding real block */
-	ll_rw_block(REQ_OP_READ, get_desc_trans_len(desc), log_blocks);
+	bh_read_batch(get_desc_trans_len(desc), log_blocks);
 	for (i = 0; i < get_desc_trans_len(desc); i++) {
 		wait_on_buffer(log_blocks[i]);
 		} else
 			bhlist[j++] = bh;
 	}
-	ll_rw_block(REQ_OP_READ, j, bhlist);
+	bh = bhlist[0];
+	bh_read_nowait(bh, 0);
+	bh_readahead_batch(j - 1, &bhlist[1], 0);
 	for (i = 1; i < j; i++)
 		brelse(bhlist[i]);
-	bh = bhlist[0];
 	wait_on_buffer(bh);
 	if (buffer_uptodate(bh))
 		return bh;

fs/reiserfs/stree.c+2−2

@@ -579,7 +579,7 @@ static int search_by_key_reada(struct super_block *s, · 578 unmodified lines
579	if (!buffer_uptodate(bh[j])) {	579	if (!buffer_uptodate(bh[j])) {
580	if (depth == -1)	580	if (depth == -1)
581	depth = reiserfs_write_unlock_nested(s);	581	depth = reiserfs_write_unlock_nested(s);
582	ll_rw_block(REQ_OP_READ \| REQ_RAHEAD, 1, bh + j);	582	bh_readahead(bh[j], REQ_RAHEAD);
583	}	583	}
584	brelse(bh[j]);	584	brelse(bh[j]);
585	}	585	}
@@ -685,7 +685,7 @@ int search_by_key(struct super_block sb, const struct cpu_key key, · 99 unmodified lines
685	if (!buffer_uptodate(bh) && depth == -1)	685	if (!buffer_uptodate(bh) && depth == -1)
686	depth = reiserfs_write_unlock_nested(sb);	686	depth = reiserfs_write_unlock_nested(sb);
687		687
688	ll_rw_block(REQ_OP_READ, 1, &bh);	688	bh_read_nowait(bh, 0);
689	wait_on_buffer(bh);	689	wait_on_buffer(bh);
690		690
691	if (depth != -1)	691	if (depth != -1)

fs/reiserfs/super.c+1−3

 /* after journal replay, reread all bitmap and super blocks */
 static int reread_meta_blocks(struct super_block *s)
 {
-	ll_rw_block(REQ_OP_READ, 1, &SB_BUFFER_WITH_SB(s));
+	if (bh_read(SB_BUFFER_WITH_SB(s), 0) < 0) {
-	wait_on_buffer(SB_BUFFER_WITH_SB(s));
-	if (!buffer_uptodate(SB_BUFFER_WITH_SB(s))) {
 		reiserfs_warning(s, "reiserfs-2504", "error reading the super");
 		return 1;
 	}

fs/udf/dir.c+1−1

@@ -130,7 +130,7 @@ static int udf_readdir(struct file file, struct dir_context ctx) · 129 unmodified lines
130	brelse(tmp);	130	brelse(tmp);
131	}	131	}
132	if (num) {	132	if (num) {
133	ll_rw_block(REQ_OP_READ \| REQ_RAHEAD, num, bha);	133	bh_readahead_batch(num, bha, REQ_RAHEAD);
134	for (i = 0; i < num; i++)	134	for (i = 0; i < num; i++)
135	brelse(bha[i]);	135	brelse(bha[i]);
136	}	136	}

fs/udf/directory.c+1−1

@@ -89,7 +89,7 @@ struct fileIdentDesc udf_fileident_read(struct inode dir, loff_t *nf_pos, · 88 unmodified lines
89	brelse(tmp);	89	brelse(tmp);
90	}	90	}
91	if (num) {	91	if (num) {
92	ll_rw_block(REQ_OP_READ \| REQ_RAHEAD, num, bha);	92	bh_readahead_batch(num, bha, REQ_RAHEAD);
93	for (i = 0; i < num; i++)	93	for (i = 0; i < num; i++)
94	brelse(bha[i]);	94	brelse(bha[i]);
95	}	95	}

fs/udf/inode.c+1−7

 	if (!bh)
 		return NULL;
-	if (buffer_uptodate(bh))
+	if (bh_read(bh, 0) >= 0)
-		return bh;
-	ll_rw_block(REQ_OP_READ, 1, &bh);
-	wait_on_buffer(bh);
-	if (buffer_uptodate(bh))
 		return bh;
 	brelse(bh);

fs/ufs/balloc.c+4−8

 			if (!buffer_mapped(bh))
 					map_bh(bh, inode->i_sb, oldb + pos);
-			if (!buffer_uptodate(bh)) {
+			if (bh_read(bh, 0) < 0) {
-				ll_rw_block(REQ_OP_READ, 1, &bh);
+				ufs_error(inode->i_sb, __func__,
-				wait_on_buffer(bh);
+					  "read of block failed\n");
-				if (!buffer_uptodate(bh)) {
+				break;
-					ufs_error(inode->i_sb, __func__,
-						  "read of block failed\n");
-					break;
-				}
 			}
 			UFSD(" change from %llu to %llu, pos %u\n",

fs/userfaultfd.c+97−36

 #include <linux/security.h>
 #include <linux/hugetlb.h>
 #include <linux/swapops.h>
+#include <linux/miscdevice.h>
 int sysctl_unprivileged_userfaultfd __read_mostly;
 	if (ctx->features & UFFD_FEATURE_SIGBUS)
 		goto out;
-	if ((vmf->flags & FAULT_FLAG_USER) == 0 &&
+	if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
-	    ctx->flags & UFFD_USER_MODE_ONLY) {
-		printk_once(KERN_WARNING "uffd: Set unprivileged_userfaultfd "
-			"sysctl knob to 1 if kernel faults must be handled "
-			"without obtaining CAP_SYS_PTRACE capability\n");
 		goto out;
-	}
 	/*
 	 * If it's already released don't get it. This avoids to loop
 	if (release_new_ctx) {
 		struct vm_area_struct *vma;
 		struct mm_struct *mm = release_new_ctx->mm;
+		VMA_ITERATOR(vmi, mm, 0);
 		/* the various vma->vm_userfaultfd_ctx still points to it */
 		mmap_write_lock(mm);
-		for (vma = mm->mmap; vma; vma = vma->vm_next)
+		for_each_vma(vmi, vma) {
 			if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) {
 				vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX;
 				vma->vm_flags &= ~__VM_UFFD_FLAGS;
 			}
+		}
 		mmap_write_unlock(mm);
 		userfaultfd_ctx_put(release_new_ctx);
 	return false;
 }
-int userfaultfd_unmap_prep(struct vm_area_struct *vma,
+int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
-			   unsigned long start, unsigned long end,
+			   unsigned long end, struct list_head *unmaps)
-			   struct list_head *unmaps)
 {
-	for ( ; vma && vma->vm_start < end; vma = vma->vm_next) {
+	VMA_ITERATOR(vmi, mm, start);
+	struct vm_area_struct *vma;
+	for_each_vma_range(vmi, vma, end) {
 		struct userfaultfd_unmap_ctx *unmap_ctx;
 		struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
 	/* len == 0 means wake all */
 	struct userfaultfd_wake_range range = { .len = 0, };
 	unsigned long new_flags;
+	MA_STATE(mas, &mm->mm_mt, 0, 0);
 	WRITE_ONCE(ctx->released, true);
 	 */
 	mmap_write_lock(mm);
 	prev = NULL;
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	mas_for_each(&mas, vma, ULONG_MAX) {
 		cond_resched();
 		BUG_ON(!!vma->vm_userfaultfd_ctx.ctx ^
 		       !!(vma->vm_flags & __VM_UFFD_FLAGS));
 				 vma->vm_file, vma->vm_pgoff,
 				 vma_policy(vma),
 				 NULL_VM_UFFD_CTX, anon_vma_name(vma));
-		if (prev)
+		if (prev) {
+			mas_pause(&mas);
 			vma = prev;
-		else
+		} else {
 			prev = vma;
+		}
 		vma->vm_flags = new_flags;
 		vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX;
 	}
 	bool found;
 	bool basic_ioctls;
 	unsigned long start, end, vma_end;
+	MA_STATE(mas, &mm->mm_mt, 0, 0);
 	user_uffdio_register = (struct uffdio_register __user *) arg;
 		goto out;
 	mmap_write_lock(mm);
-	vma = find_vma_prev(mm, start, &prev);
+	mas_set(&mas, start);
+	vma = mas_find(&mas, ULONG_MAX);
 	if (!vma)
 		goto out_unlock;
 	 */
 	found = false;
 	basic_ioctls = false;
-	for (cur = vma; cur && cur->vm_start < end; cur = cur->vm_next) {
+	for (cur = vma; cur; cur = mas_next(&mas, end - 1)) {
 		cond_resched();
 		BUG_ON(!!cur->vm_userfaultfd_ctx.ctx ^
 	}
 	BUG_ON(!found);
-	if (vma->vm_start < start)
+	mas_set(&mas, start);
-		prev = vma;
+	prev = mas_prev(&mas, 0);
+	if (prev != vma)
+		mas_next(&mas, ULONG_MAX);
 	ret = 0;
 	do {
 				 ((struct vm_userfaultfd_ctx){ ctx }),
 				 anon_vma_name(vma));
 		if (prev) {
+			/* vma_merge() invalidated the mas */
+			mas_pause(&mas);
 			vma = prev;
 			goto next;
 		}
 			ret = split_vma(mm, vma, start, 1);
 			if (ret)
 				break;
+			/* split_vma() invalidated the mas */
+			mas_pause(&mas);
 		}
 		if (vma->vm_end > end) {
 			ret = split_vma(mm, vma, end, 0);
 			if (ret)
 				break;
+			/* split_vma() invalidated the mas */
+			mas_pause(&mas);
 		}
 	next:
 		/*
 	skip:
 		prev = vma;
 		start = vma->vm_end;
-		vma = vma->vm_next;
+		vma = mas_next(&mas, end - 1);
-	} while (vma && vma->vm_start < end);
+	} while (vma);
 out_unlock:
 	mmap_write_unlock(mm);
 	mmput(mm);
 	bool found;
 	unsigned long start, end, vma_end;
 	const void __user *buf = (void __user *)arg;
+	MA_STATE(mas, &mm->mm_mt, 0, 0);
 	ret = -EFAULT;
 	if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
 		goto out;
 	mmap_write_lock(mm);
-	vma = find_vma_prev(mm, start, &prev);
+	mas_set(&mas, start);
+	vma = mas_find(&mas, ULONG_MAX);
 	if (!vma)
 		goto out_unlock;
 	 */
 	found = false;
 	ret = -EINVAL;
-	for (cur = vma; cur && cur->vm_start < end; cur = cur->vm_next) {
+	for (cur = vma; cur; cur = mas_next(&mas, end - 1)) {
 		cond_resched();
 		BUG_ON(!!cur->vm_userfaultfd_ctx.ctx ^
 	}
 	BUG_ON(!found);
-	if (vma->vm_start < start)
+	mas_set(&mas, start);
-		prev = vma;
+	prev = mas_prev(&mas, 0);
+	if (prev != vma)
+		mas_next(&mas, ULONG_MAX);
 	ret = 0;
 	do {
 	skip:
 		prev = vma;
 		start = vma->vm_end;
-		vma = vma->vm_next;
+		vma = mas_next(&mas, end - 1);
-	} while (vma && vma->vm_start < end);
+	} while (vma);
 out_unlock:
 	mmap_write_unlock(mm);
 	mmput(mm);
 	seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock);
 }
-SYSCALL_DEFINE1(userfaultfd, int, flags)
+static int new_userfaultfd(int flags)
 {
 	struct userfaultfd_ctx *ctx;
 	int fd;
-	if (!sysctl_unprivileged_userfaultfd &&
-	    (flags & UFFD_USER_MODE_ONLY) == 0 &&
-	    !capable(CAP_SYS_PTRACE)) {
-		printk_once(KERN_WARNING "uffd: Set unprivileged_userfaultfd "
-			"sysctl knob to 1 if kernel faults must be handled "
-			"without obtaining CAP_SYS_PTRACE capability\n");
-		return -EPERM;
-	}
 	BUG_ON(!current->mm);
 	/* Check the UFFD_* constants for consistency.  */
 	return fd;
 }
+static inline bool userfaultfd_syscall_allowed(int flags)
+{
+	/* Userspace-only page faults are always allowed */
+	if (flags & UFFD_USER_MODE_ONLY)
+		return true;
+	/*
+	 * The user is requesting a userfaultfd which can handle kernel faults.
+	 * Privileged users are always allowed to do this.
+	 */
+	if (capable(CAP_SYS_PTRACE))
+		return true;
+	/* Otherwise, access to kernel fault handling is sysctl controlled. */
+	return sysctl_unprivileged_userfaultfd;
+}
+SYSCALL_DEFINE1(userfaultfd, int, flags)
+{
+	if (!userfaultfd_syscall_allowed(flags))
+		return -EPERM;
+	return new_userfaultfd(flags);
+}
+static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags)
+{
+	if (cmd != USERFAULTFD_IOC_NEW)
+		return -EINVAL;
+	return new_userfaultfd(flags);
+}
+static const struct file_operations userfaultfd_dev_fops = {
+	.unlocked_ioctl = userfaultfd_dev_ioctl,
+	.compat_ioctl = userfaultfd_dev_ioctl,
+	.owner = THIS_MODULE,
+	.llseek = noop_llseek,
+};
+static struct miscdevice userfaultfd_misc = {
+	.minor = MISC_DYNAMIC_MINOR,
+	.name = "userfaultfd",
+	.fops = &userfaultfd_dev_fops
+};
 static int __init userfaultfd_init(void)
 {
+	int ret;
+	ret = misc_register(&userfaultfd_misc);
+	if (ret)
+		return ret;
 	userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache",
 						sizeof(struct userfaultfd_ctx),
 						0,

include/asm-generic/cacheflush.h+12−2

 #ifndef _ASM_GENERIC_CACHEFLUSH_H
 #define _ASM_GENERIC_CACHEFLUSH_H
+#include <linux/instrumented.h>
 struct mm_struct;
 struct vm_area_struct;
 struct page;
 #ifndef copy_to_user_page
 #define copy_to_user_page(vma, page, vaddr, dst, src, len)	\
 	do { \
+		instrument_copy_to_user((void __user *)dst, src, len); \
 		memcpy(dst, src, len); \
 		flush_icache_user_page(vma, page, vaddr, len); \
 	} while (0)
 #endif
 #ifndef copy_from_user_page
-#define copy_from_user_page(vma, page, vaddr, dst, src, len) \
+#define copy_from_user_page(vma, page, vaddr, dst, src, len)		  \
-	memcpy(dst, src, len)
+	do {								  \
+		instrument_copy_from_user_before(dst, (void __user *)src, \
+						 len);			  \
+		memcpy(dst, src, len);					  \
+		instrument_copy_from_user_after(dst, (void __user *)src, len, \
+						0);			  \
+	} while (0)
 #endif
 #endif /* _ASM_GENERIC_CACHEFLUSH_H */

include/linux/buffer_head.h+38−10

 void __brelse(struct buffer_head *);
 void __bforget(struct buffer_head *);
 void __breadahead(struct block_device *, sector_t block, unsigned int size);
-void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size,
-		  gfp_t gfp);
 struct buffer_head *__bread_gfp(struct block_device *,
 				sector_t block, unsigned size, gfp_t gfp);
 void invalidate_bh_lrus(void);
 void free_buffer_head(struct buffer_head * bh);
 void unlock_buffer(struct buffer_head *bh);
 void __lock_buffer(struct buffer_head *bh);
-void ll_rw_block(blk_opf_t, int, struct buffer_head * bh[]);
 int sync_dirty_buffer(struct buffer_head *bh);
 int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags);
 void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags);
 void write_boundary_block(struct block_device *bdev,
 			sector_t bblock, unsigned blocksize);
 int bh_uptodate_or_lock(struct buffer_head *bh);
-int bh_submit_read(struct buffer_head *bh);
+int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait);
+void __bh_read_batch(int nr, struct buffer_head *bhs[],
+		     blk_opf_t op_flags, bool force_lock);
 extern int buffer_heads_over_limit;
 	__breadahead(sb->s_bdev, block, sb->s_blocksize);
 }
-static inline void
-sb_breadahead_unmovable(struct super_block *sb, sector_t block)
-{
-	__breadahead_gfp(sb->s_bdev, block, sb->s_blocksize, 0);
-}
 static inline struct buffer_head *
 sb_getblk(struct super_block *sb, sector_t block)
 {
 	return __getblk_gfp(bdev, block, size, __GFP_MOVABLE);
 }
+static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags)
+{
+	if (!buffer_uptodate(bh) && trylock_buffer(bh)) {
+		if (!buffer_uptodate(bh))
+			__bh_read(bh, op_flags, false);
+		else
+			unlock_buffer(bh);
+	}
+}
+static inline void bh_read_nowait(struct buffer_head *bh, blk_opf_t op_flags)
+{
+	if (!bh_uptodate_or_lock(bh))
+		__bh_read(bh, op_flags, false);
+}
+/* Returns 1 if buffer uptodated, 0 on success, and -EIO on error. */
+static inline int bh_read(struct buffer_head *bh, blk_opf_t op_flags)
+{
+	if (bh_uptodate_or_lock(bh))
+		return 1;
+	return __bh_read(bh, op_flags, true);
+}
+static inline void bh_read_batch(int nr, struct buffer_head *bhs[])
+{
+	__bh_read_batch(nr, bhs, 0, true);
+}
+static inline void bh_readahead_batch(int nr, struct buffer_head *bhs[],
+				      blk_opf_t op_flags)
+{
+	__bh_read_batch(nr, bhs, op_flags, false);
+}
 /**
  *  __bread() - reads a specified block and returns the bh
  *  @bdev: the block_device to read from

include/linux/cache.h+13−0

@@ -85,4 +85,17 @@ · 84 unmodified lines
85	#define cache_line_size() L1_CACHE_BYTES	85	#define cache_line_size() L1_CACHE_BYTES
86	#endif	86	#endif
87		87
		88	/*
		89	* Helper to add padding within a struct to ensure data fall into separate
		90	* cachelines.
		91	*/
		92	#if defined(CONFIG_SMP)
		93	struct cacheline_padding {
		94	char x[0];
		95	} ____cacheline_internodealigned_in_smp;
		96	#define CACHELINE_PADDING(name) struct cacheline_padding name
		97	#else
		98	#define CACHELINE_PADDING(name)
		99	#endif
		100
88	#endif /* __LINUX_CACHE_H */	101	#endif /* __LINUX_CACHE_H */

include/linux/cgroup.h+14−1

 	css_put(&cgrp->self);
 }
+extern struct mutex cgroup_mutex;
+static inline void cgroup_lock(void)
+{
+	mutex_lock(&cgroup_mutex);
+}
+static inline void cgroup_unlock(void)
+{
+	mutex_unlock(&cgroup_mutex);
+}
 /**
  * task_css_set_check - obtain a task's css_set with extra access conditions
  * @task: the task to obtain css_set for
  * as locks used during the cgroup_subsys::attach() methods.
  */
 #ifdef CONFIG_PROVE_RCU
-extern struct mutex cgroup_mutex;
 extern spinlock_t css_set_lock;
 #define task_css_set_check(task, __c)					\
 	rcu_dereference_check((task)->cgroups,				\
 static inline u64 cgroup_id(const struct cgroup *cgrp) { return 1; }
 static inline void css_get(struct cgroup_subsys_state *css) {}
 static inline void css_put(struct cgroup_subsys_state *css) {}
+static inline void cgroup_lock(void) {}
+static inline void cgroup_unlock(void) {}
 static inline int cgroup_attach_task_all(struct task_struct *from,
 					 struct task_struct *t) { return 0; }
 static inline int cgroupstats_build(struct cgroupstats *stats,

include/linux/compiler-clang.h+23−0

@@ -51,6 +51,29 @@ · 50 unmodified lines
51	#define __no_sanitize_undefined	51	#define __no_sanitize_undefined
52	#endif	52	#endif
53		53
		54	#if __has_feature(memory_sanitizer)
		55	#define __SANITIZE_MEMORY__
		56	/*
		57	* Unlike other sanitizers, KMSAN still inserts code into functions marked with
		58	* no_sanitize("kernel-memory"). Using disable_sanitizer_instrumentation
		59	* provides the behavior consistent with other __no_sanitize_ attributes,
		60	* guaranteeing that __no_sanitize_memory functions remain uninstrumented.
		61	*/
		62	#define __no_sanitize_memory __disable_sanitizer_instrumentation
		63
		64	/*
		65	* The __no_kmsan_checks attribute ensures that a function does not produce
		66	* false positive reports by:
		67	* - initializing all local variables and memory stores in this function;
		68	* - skipping all shadow checks;
		69	* - passing initialized arguments to this function's callees.
		70	*/
		71	#define __no_kmsan_checks __attribute__((no_sanitize("kernel-memory")))
		72	#else
		73	#define __no_sanitize_memory
		74	#define __no_kmsan_checks
		75	#endif
		76
54	/*	77	/*
55	* Support for __has_feature(coverage_sanitizer) was added in Clang 13 together	78	* Support for __has_feature(coverage_sanitizer) was added in Clang 13 together
56	* with no_sanitize("coverage"). Prior versions of Clang support coverage	79	* with no_sanitize("coverage"). Prior versions of Clang support coverage

include/linux/compiler-gcc.h+6−0

 #define __SANITIZE_ADDRESS__
 #endif
+/*
+ * GCC does not support KMSAN.
+ */
+#define __no_sanitize_memory
+#define __no_kmsan_checks
 /*
  * Turn individual warnings and errors on and off locally, depending
  * on version.

include/linux/compiler_types.h+2−1

@@ -233,7 +233,8 @@ struct ftrace_likely_data { · 232 unmodified lines
233	/* Section for code which can't be instrumented at all */	233	/* Section for code which can't be instrumented at all */
234	#define noinstr \	234	#define noinstr \
235	noinline notrace __attribute((__section__(".noinstr.text"))) \	235	noinline notrace __attribute((__section__(".noinstr.text"))) \
236	__no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage	236	__no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \
		237	__no_sanitize_memory
237		238
238	#endif /* __KERNEL__ */	239	#endif /* __KERNEL__ */
239		240

include/linux/damon.h+54−32

 };
 /**
- * struct damos - Represents a Data Access Monitoring-based Operation Scheme.
+ * struct damos_access_pattern - Target access pattern of the given scheme.
  * @min_sz_region:	Minimum size of target regions.
  * @max_sz_region:	Maximum size of target regions.
  * @min_nr_accesses:	Minimum ``->nr_accesses`` of target regions.
  * @max_nr_accesses:	Maximum ``->nr_accesses`` of target regions.
  * @min_age_region:	Minimum age of target regions.
  * @max_age_region:	Maximum age of target regions.
+ */
+struct damos_access_pattern {
+	unsigned long min_sz_region;
+	unsigned long max_sz_region;
+	unsigned int min_nr_accesses;
+	unsigned int max_nr_accesses;
+	unsigned int min_age_region;
+	unsigned int max_age_region;
+};
+/**
+ * struct damos - Represents a Data Access Monitoring-based Operation Scheme.
+ * @pattern:		Access pattern of target regions.
  * @action:		&damo_action to be applied to the target regions.
  * @quota:		Control the aggressiveness of this scheme.
  * @wmarks:		Watermarks for automated (in)activation of this scheme.
  * @list:		List head for siblings.
  *
  * For each aggregation interval, DAMON finds regions which fit in the
- * condition (&min_sz_region, &max_sz_region, &min_nr_accesses,
+ * &pattern and applies &action to those. To avoid consuming too much
- * &max_nr_accesses, &min_age_region, &max_age_region) and applies &action to
+ * CPU time or IO resources for the &action, &quota is used.
- * those.  To avoid consuming too much CPU time or IO resources for the
- * &action, &quota is used.
  *
  * To do the work only when needed, schemes can be activated for specific
  * system situations using &wmarks.  If all schemes that registered to the
  * &action is applied.
  */
 struct damos {
-	unsigned long min_sz_region;
+	struct damos_access_pattern pattern;
-	unsigned long max_sz_region;
-	unsigned int min_nr_accesses;
-	unsigned int max_nr_accesses;
-	unsigned int min_age_region;
-	unsigned int max_age_region;
 	enum damos_action action;
 	struct damos_quota quota;
 	struct damos_watermarks wmarks;
 	unsigned long (*apply_scheme)(struct damon_ctx *context,
 			struct damon_target *t, struct damon_region *r,
 			struct damos *scheme);
-	bool (*target_valid)(void *target);
+	bool (*target_valid)(struct damon_target *t);
 	void (*cleanup)(struct damon_ctx *context);
 };
 };
 /**
- * struct damon_ctx - Represents a context for each monitoring.  This is the
+ * struct damon_attrs - Monitoring attributes for accuracy/overhead control.
- * main interface that allows users to set the attributes and get the results
- * of the monitoring.
  *
  * @sample_interval:		The time between access samplings.
  * @aggr_interval:		The time between monitor results aggregations.
  * @ops_update_interval:	The time between monitoring operations updates.
+ * @min_nr_regions:		The minimum number of adaptive monitoring
+ *				regions.
+ * @max_nr_regions:		The maximum number of adaptive monitoring
+ *				regions.
  *
  * For each @sample_interval, DAMON checks whether each region is accessed or
  * not.  It aggregates and keeps the access information (number of accesses to
  * @ops_update_interval.  All time intervals are in micro-seconds.
  * Please refer to &struct damon_operations and &struct damon_callback for more
  * detail.
+ */
+struct damon_attrs {
+	unsigned long sample_interval;
+	unsigned long aggr_interval;
+	unsigned long ops_update_interval;
+	unsigned long min_nr_regions;
+	unsigned long max_nr_regions;
+};
+/**
+ * struct damon_ctx - Represents a context for each monitoring.  This is the
+ * main interface that allows users to set the attributes and get the results
+ * of the monitoring.
  *
+ * @attrs:		Monitoring attributes for accuracy/overhead control.
  * @kdamond:		Kernel thread who does the monitoring.
  * @kdamond_lock:	Mutex for the synchronizations with @kdamond.
  *
  * @ops:	Set of monitoring operations for given use cases.
  * @callback:	Set of callbacks for monitoring events notifications.
  *
- * @min_nr_regions:	The minimum number of adaptive monitoring regions.
- * @max_nr_regions:	The maximum number of adaptive monitoring regions.
  * @adaptive_targets:	Head of monitoring targets (&damon_target) list.
  * @schemes:		Head of schemes (&damos) list.
  */
 struct damon_ctx {
-	unsigned long sample_interval;
+	struct damon_attrs attrs;
-	unsigned long aggr_interval;
-	unsigned long ops_update_interval;
 /* private: internal use only */
 	struct timespec64 last_aggregation;
 	struct damon_operations ops;
 	struct damon_callback callback;
-	unsigned long min_nr_regions;
-	unsigned long max_nr_regions;
 	struct list_head adaptive_targets;
 	struct list_head schemes;
 };
 	return list_last_entry(&t->regions_list, struct damon_region, list);
 }
+static inline struct damon_region *damon_first_region(struct damon_target *t)
+{
+	return list_first_entry(&t->regions_list, struct damon_region, list);
+}
 #define damon_for_each_region(r, t) \
 	list_for_each_entry(r, &t->regions_list, list)
+#define damon_for_each_region_from(r, t) \
+	list_for_each_entry_from(r, &t->regions_list, list)
 #define damon_for_each_region_safe(r, next, t) \
 	list_for_each_entry_safe(r, next, &t->regions_list, list)
 int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
 		unsigned int nr_ranges);
-struct damos *damon_new_scheme(
+struct damos *damon_new_scheme(struct damos_access_pattern *pattern,
-		unsigned long min_sz_region, unsigned long max_sz_region,
+			enum damos_action action, struct damos_quota *quota,
-		unsigned int min_nr_accesses, unsigned int max_nr_accesses,
+			struct damos_watermarks *wmarks);
-		unsigned int min_age_region, unsigned int max_age_region,
-		enum damos_action action, struct damos_quota *quota,
-		struct damos_watermarks *wmarks);
 void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);
 void damon_destroy_scheme(struct damos *s);
 struct damon_ctx *damon_new_ctx(void);
 void damon_destroy_ctx(struct damon_ctx *ctx);
-int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
+int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs);
-		unsigned long aggr_int, unsigned long ops_upd_int,
+void damon_set_schemes(struct damon_ctx *ctx,
-		unsigned long min_nr_reg, unsigned long max_nr_reg);
-int damon_set_schemes(struct damon_ctx *ctx,
 			struct damos **schemes, ssize_t nr_schemes);
 int damon_nr_running_ctxs(void);
 bool damon_is_registered_ops(enum damon_ops_id id);
 int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive);
 int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
+int damon_set_region_biggest_system_ram_default(struct damon_target *t,
+				unsigned long *start, unsigned long *end);
 #endif	/* CONFIG_DAMON */
 #endif	/* _DAMON_H */

include/linux/delayacct.h+8−8

@@ -73,8 +73,8 @@ extern int delayacct_add_tsk(struct taskstats , struct task_struct ); · 72 unmodified lines
73	extern __u64 __delayacct_blkio_ticks(struct task_struct *);	73	extern __u64 __delayacct_blkio_ticks(struct task_struct *);
74	extern void __delayacct_freepages_start(void);	74	extern void __delayacct_freepages_start(void);
75	extern void __delayacct_freepages_end(void);	75	extern void __delayacct_freepages_end(void);
76	extern void __delayacct_thrashing_start(void);	76	extern void __delayacct_thrashing_start(bool *in_thrashing);
77	extern void __delayacct_thrashing_end(void);	77	extern void __delayacct_thrashing_end(bool *in_thrashing);
78	extern void __delayacct_swapin_start(void);	78	extern void __delayacct_swapin_start(void);
79	extern void __delayacct_swapin_end(void);	79	extern void __delayacct_swapin_end(void);
80	extern void __delayacct_compact_start(void);	80	extern void __delayacct_compact_start(void);
@@ -143,22 +143,22 @@ static inline void delayacct_freepages_end(void) · 62 unmodified lines
143	__delayacct_freepages_end();	143	__delayacct_freepages_end();
144	}	144	}
145		145
146	static inline void delayacct_thrashing_start(void)	146	static inline void delayacct_thrashing_start(bool *in_thrashing)
147	{	147	{
148	if (!static_branch_unlikely(&delayacct_key))	148	if (!static_branch_unlikely(&delayacct_key))
149	return;	149	return;
150		150
151	if (current->delays)	151	if (current->delays)
152	__delayacct_thrashing_start();	152	__delayacct_thrashing_start(in_thrashing);
153	}	153	}
154		154
155	static inline void delayacct_thrashing_end(void)	155	static inline void delayacct_thrashing_end(bool *in_thrashing)
156	{	156	{
157	if (!static_branch_unlikely(&delayacct_key))	157	if (!static_branch_unlikely(&delayacct_key))
158	return;	158	return;
159		159
160	if (current->delays)	160	if (current->delays)
161	__delayacct_thrashing_end();	161	__delayacct_thrashing_end(in_thrashing);
162	}	162	}
163		163
164	static inline void delayacct_swapin_start(void)	164	static inline void delayacct_swapin_start(void)
@@ -237,9 +237,9 @@ static inline void delayacct_freepages_start(void) · 72 unmodified lines
237	{}	237	{}
238	static inline void delayacct_freepages_end(void)	238	static inline void delayacct_freepages_end(void)
239	{}	239	{}
240	static inline void delayacct_thrashing_start(void)	240	static inline void delayacct_thrashing_start(bool *in_thrashing)
241	{}	241	{}
242	static inline void delayacct_thrashing_end(void)	242	static inline void delayacct_thrashing_end(bool *in_thrashing)
243	{}	243	{}
244	static inline void delayacct_swapin_start(void)	244	static inline void delayacct_swapin_start(void)
245	{}	245	{}

include/linux/fortify-string.h+2−0

  * __struct_size() vs __member_size() must be captured here to avoid
  * evaluating argument side-effects further into the macro layers.
  */
+#ifndef CONFIG_KMSAN
 #define memset(p, c, s) __fortify_memset_chk(p, c, s,			\
 		__struct_size(p), __member_size(p))
+#endif
 /*
  * To make sure the compiler can enforce protection against buffer overflows,

include/linux/gfp.h+3−23

@@ -18,6 +18,9 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) · 17 unmodified lines
18	VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) == GFP_MOVABLE_MASK);	18	VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) == GFP_MOVABLE_MASK);
19	BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) != ___GFP_MOVABLE);	19	BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) != ___GFP_MOVABLE);
20	BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_MOVABLE);	20	BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_MOVABLE);
		21	BUILD_BUG_ON((___GFP_RECLAIMABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_RECLAIMABLE);
		22	BUILD_BUG_ON(((___GFP_MOVABLE \| ___GFP_RECLAIMABLE) >>
		23	GFP_MOVABLE_SHIFT) != MIGRATE_HIGHATOMIC);
21		24
22	if (unlikely(page_group_by_mobility_disabled))	25	if (unlikely(page_group_by_mobility_disabled))
23	return MIGRATE_UNMOVABLE;	26	return MIGRATE_UNMOVABLE;
@@ -33,29 +36,6 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) · 9 unmodified lines
33	return !!(gfp_flags & __GFP_DIRECT_RECLAIM);	36	return !!(gfp_flags & __GFP_DIRECT_RECLAIM);
34	}	37	}
35		38
36	/**
37	* gfpflags_normal_context - is gfp_flags a normal sleepable context?
38	* @gfp_flags: gfp_flags to test
39	*
40	* Test whether @gfp_flags indicates that the allocation is from the
41	* %current context and allowed to sleep.
42	*
43	* An allocation being allowed to block doesn't mean it owns the %current
44	* context. When direct reclaim path tries to allocate memory, the
45	* allocation context is nested inside whatever %current was doing at the
46	* time of the original allocation. The nested allocation may be allowed
47	* to block but modifying anything %current owns can corrupt the outer
48	* context's expectations.
49	*
50	* %true result from this function indicates that the allocation context
51	* can sleep and use anything that's associated with %current.
52	*/
53	static inline bool gfpflags_normal_context(const gfp_t gfp_flags)
54	{
55	return (gfp_flags & (__GFP_DIRECT_RECLAIM \| __GFP_MEMALLOC)) ==
56	__GFP_DIRECT_RECLAIM;
57	}
58
59	#ifdef CONFIG_HIGHMEM	39	#ifdef CONFIG_HIGHMEM
60	#define OPT_ZONE_HIGHMEM ZONE_HIGHMEM	40	#define OPT_ZONE_HIGHMEM ZONE_HIGHMEM
61	#else	41	#else

include/linux/highmem.h+3−0

 #include <linux/kernel.h>
 #include <linux/bug.h>
 #include <linux/cacheflush.h>
+#include <linux/kmsan.h>
 #include <linux/mm.h>
 #include <linux/uaccess.h>
 #include <linux/hardirq.h>
 	vfrom = kmap_local_page(from);
 	vto = kmap_local_page(to);
 	copy_user_page(vto, vfrom, vaddr, to);
+	kmsan_unpoison_memory(page_address(to), PAGE_SIZE);
 	kunmap_local(vto);
 	kunmap_local(vfrom);
 }
 	vfrom = kmap_local_page(from);
 	vto = kmap_local_page(to);
 	copy_page(vto, vfrom);
+	kmsan_copy_page_meta(to, from);
 	kunmap_local(vto);
 	kunmap_local(vfrom);
 }

include/linux/huge_mm.h+21−7

 	       !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
-bool hugepage_vma_check(struct vm_area_struct *vma,
+bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-			unsigned long vm_flags,
+			bool smaps, bool in_pf, bool enforce_sysfs);
-			bool smaps, bool in_pf);
 #define transparent_hugepage_use_zero_page()				\
 	(transparent_hugepage_flags &					\
 int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags,
 		     int advice);
+int madvise_collapse(struct vm_area_struct *vma,
+		     struct vm_area_struct **prev,
+		     unsigned long start, unsigned long end);
 void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
 			   unsigned long end, long adjust_next);
 spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
 }
 static inline bool hugepage_vma_check(struct vm_area_struct *vma,
-				       unsigned long vm_flags,
+				      unsigned long vm_flags, bool smaps,
-				       bool smaps, bool in_pf)
+				      bool in_pf, bool enforce_sysfs)
 {
 	return false;
 }
 static inline int hugepage_madvise(struct vm_area_struct *vma,
 				   unsigned long *vm_flags, int advice)
 {
-	BUG();
+	return -EINVAL;
-	return 0;
+}
+static inline int madvise_collapse(struct vm_area_struct *vma,
+				   struct vm_area_struct **prev,
+				   unsigned long start, unsigned long end)
+{
+	return -EINVAL;
 }
 static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
 					 unsigned long start,
 					 unsigned long end,
 	return split_huge_page_to_list(&folio->page, list);
 }
+static inline int split_folio(struct folio *folio)
+{
+	return split_folio_to_list(folio, NULL);
+}
 /*
  * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to
  * limitations in the implementation like arm64 MTE can override this to

include/linux/hugetlb.h+56−7

 struct ctl_table;
 struct user_struct;
 struct mmu_gather;
+struct node;
 #ifndef CONFIG_ARCH_HAS_HUGEPD
 typedef struct { unsigned long pd; } hugepd_t;
 #endif
 };
+struct hugetlb_vma_lock {
+	struct kref refs;
+	struct rw_semaphore rw_sema;
+	struct vm_area_struct *vma;
+};
 extern struct resv_map *resv_map_alloc(void);
 void resv_map_release(struct kref *ref);
 						long min_hpages);
 void hugepage_put_subpool(struct hugepage_subpool *spool);
-void reset_vma_resv_huge_pages(struct vm_area_struct *vma);
+void hugetlb_dup_vma_private(struct vm_area_struct *vma);
 void clear_vma_resv_huge_pages(struct vm_area_struct *vma);
 int hugetlb_sysctl_handler(struct ctl_table *, int, void *, size_t *, loff_t *);
 int hugetlb_overcommit_handler(struct ctl_table *, int, void *, size_t *,
 struct page *follow_huge_pgd(struct mm_struct *mm, unsigned long address,
 			     pgd_t *pgd, int flags);
+void hugetlb_vma_lock_read(struct vm_area_struct *vma);
+void hugetlb_vma_unlock_read(struct vm_area_struct *vma);
+void hugetlb_vma_lock_write(struct vm_area_struct *vma);
+void hugetlb_vma_unlock_write(struct vm_area_struct *vma);
+int hugetlb_vma_trylock_write(struct vm_area_struct *vma);
+void hugetlb_vma_assert_locked(struct vm_area_struct *vma);
+void hugetlb_vma_lock_release(struct kref *kref);
 int pmd_huge(pmd_t pmd);
 int pud_huge(pud_t pud);
 unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 #else /* !CONFIG_HUGETLB_PAGE */
-static inline void reset_vma_resv_huge_pages(struct vm_area_struct *vma)
+static inline void hugetlb_dup_vma_private(struct vm_area_struct *vma)
 {
 }
 	return -EINVAL;
 }
+static inline void hugetlb_vma_lock_read(struct vm_area_struct *vma)
+{
+}
+static inline void hugetlb_vma_unlock_read(struct vm_area_struct *vma)
+{
+}
+static inline void hugetlb_vma_lock_write(struct vm_area_struct *vma)
+{
+}
+static inline void hugetlb_vma_unlock_write(struct vm_area_struct *vma)
+{
+}
+static inline int hugetlb_vma_trylock_write(struct vm_area_struct *vma)
+{
+	return 1;
+}
+static inline void hugetlb_vma_assert_locked(struct vm_area_struct *vma)
+{
+}
 static inline int pmd_huge(pmd_t pmd)
 {
 	return 0;
 				nodemask_t *nmask, gfp_t gfp_mask);
 struct page *alloc_huge_page_vma(struct hstate *h, struct vm_area_struct *vma,
 				unsigned long address);
-int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
+int hugetlb_add_to_page_cache(struct page *page, struct address_space *mapping,
 			pgoff_t idx);
 void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma,
 				unsigned long address, struct page *page);
 }
 #endif
+#ifdef CONFIG_NUMA
+void hugetlb_register_node(struct node *node);
+void hugetlb_unregister_node(struct node *node);
+#endif
 #else	/* CONFIG_HUGETLB_PAGE */
 struct hstate {};
 				   pte_t *ptep, pte_t pte)
 {
 }
+static inline void hugetlb_register_node(struct node *node)
+{
+}
+static inline void hugetlb_unregister_node(struct node *node)
+{
+}
 #endif	/* CONFIG_HUGETLB_PAGE */
 static inline spinlock_t *huge_pte_lock(struct hstate *h,
 #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA)
 extern void __init hugetlb_cma_reserve(int order);
-extern void __init hugetlb_cma_check(void);
 #else
 static inline __init void hugetlb_cma_reserve(int order)
 {
 }
-static inline __init void hugetlb_cma_check(void)
-{
-}
 #endif
 bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);

include/linux/hugetlb_cgroup.h+8−11

 	return __hugetlb_cgroup_from_page(page, true);
 }
-static inline int __set_hugetlb_cgroup(struct page *page,
+static inline void __set_hugetlb_cgroup(struct page *page,
 				       struct hugetlb_cgroup *h_cg, bool rsvd)
 {
 	VM_BUG_ON_PAGE(!PageHuge(page), page);
 	if (compound_order(page) < HUGETLB_CGROUP_MIN_ORDER)
-		return -1;
+		return;
 	if (rsvd)
 		set_page_private(page + SUBPAGE_INDEX_CGROUP_RSVD,
 				 (unsigned long)h_cg);
 	else
 		set_page_private(page + SUBPAGE_INDEX_CGROUP,
 				 (unsigned long)h_cg);
-	return 0;
 }
-static inline int set_hugetlb_cgroup(struct page *page,
+static inline void set_hugetlb_cgroup(struct page *page,
 				     struct hugetlb_cgroup *h_cg)
 {
-	return __set_hugetlb_cgroup(page, h_cg, false);
+	__set_hugetlb_cgroup(page, h_cg, false);
 }
-static inline int set_hugetlb_cgroup_rsvd(struct page *page,
+static inline void set_hugetlb_cgroup_rsvd(struct page *page,
 					  struct hugetlb_cgroup *h_cg)
 {
-	return __set_hugetlb_cgroup(page, h_cg, true);
+	__set_hugetlb_cgroup(page, h_cg, true);
 }
 static inline bool hugetlb_cgroup_disabled(void)
 	return NULL;
 }
-static inline int set_hugetlb_cgroup(struct page *page,
+static inline void set_hugetlb_cgroup(struct page *page,
 				     struct hugetlb_cgroup *h_cg)
 {
-	return 0;
 }
-static inline int set_hugetlb_cgroup_rsvd(struct page *page,
+static inline void set_hugetlb_cgroup_rsvd(struct page *page,
 					  struct hugetlb_cgroup *h_cg)
 {
-	return 0;
 }
 static inline bool hugetlb_cgroup_disabled(void)

include/linux/instrumented.h+56−3

@@ -2,7 +2,7 @@ · 1 unmodified line
2		2
3	/*	3	/*
4	* This header provides generic wrappers for memory access instrumentation that	4	* This header provides generic wrappers for memory access instrumentation that
5	* the compiler cannot emit for: KASAN, KCSAN.	5	* the compiler cannot emit for: KASAN, KCSAN, KMSAN.
6	*/	6	*/
7	#ifndef _LINUX_INSTRUMENTED_H	7	#ifndef _LINUX_INSTRUMENTED_H
8	#define _LINUX_INSTRUMENTED_H	8	#define _LINUX_INSTRUMENTED_H
@@ -10,6 +10,7 @@ · 1 unmodified line
10	#include <linux/compiler.h>	10	#include <linux/compiler.h>
11	#include <linux/kasan-checks.h>	11	#include <linux/kasan-checks.h>
12	#include <linux/kcsan-checks.h>	12	#include <linux/kcsan-checks.h>
		13	#include <linux/kmsan-checks.h>
13	#include <linux/types.h>	14	#include <linux/types.h>
14		15
15	/**	16	/**
@@ -117,10 +118,11 @@ instrument_copy_to_user(void __user to, const void from, unsigned long n) · 101 unmodified lines
117	{	118	{
118	kasan_check_read(from, n);	119	kasan_check_read(from, n);
119	kcsan_check_read(from, n);	120	kcsan_check_read(from, n);
		121	kmsan_copy_to_user(to, from, n, 0);
120	}	122	}
121		123
122	/**	124	/**
123	* instrument_copy_from_user - instrument writes of copy_from_user	125	* instrument_copy_from_user_before - add instrumentation before copy_from_user
124	*	126	*
125	* Instrument writes to kernel memory, that are due to copy_from_user (and	127	* Instrument writes to kernel memory, that are due to copy_from_user (and
126	* variants). The instrumentation should be inserted before the accesses.	128	* variants). The instrumentation should be inserted before the accesses.
@@ -130,10 +132,61 @@ instrument_copy_to_user(void __user to, const void from, unsigned long n) · 3 unmodified lines
130	* @n number of bytes to copy	132	* @n number of bytes to copy
131	*/	133	*/
132	static __always_inline void	134	static __always_inline void
133	instrument_copy_from_user(const void to, const void __user from, unsigned long n)	135	instrument_copy_from_user_before(const void to, const void __user from, unsigned long n)
134	{	136	{
135	kasan_check_write(to, n);	137	kasan_check_write(to, n);
136	kcsan_check_write(to, n);	138	kcsan_check_write(to, n);
137	}	139	}
138		140
		141	/**
		142	* instrument_copy_from_user_after - add instrumentation after copy_from_user
		143	*
		144	* Instrument writes to kernel memory, that are due to copy_from_user (and
		145	* variants). The instrumentation should be inserted after the accesses.
		146	*
		147	* @to destination address
		148	* @from source address
		149	* @n number of bytes to copy
		150	* @left number of bytes not copied (as returned by copy_from_user)
		151	*/
		152	static __always_inline void
		153	instrument_copy_from_user_after(const void to, const void __user from,
		154	unsigned long n, unsigned long left)
		155	{
		156	kmsan_unpoison_memory(to, n - left);
		157	}
		158
		159	/**
		160	* instrument_get_user() - add instrumentation to get_user()-like macros
		161	*
		162	* get_user() and friends are fragile, so it may depend on the implementation
		163	* whether the instrumentation happens before or after the data is copied from
		164	* the userspace.
		165	*
		166	* @to destination variable, may not be address-taken
		167	*/
		168	#define instrument_get_user(to) \
		169	({ \
		170	u64 __tmp = (u64)(to); \
		171	kmsan_unpoison_memory(&__tmp, sizeof(__tmp)); \
		172	to = __tmp; \
		173	})
		174
		175
		176	/**
		177	* instrument_put_user() - add instrumentation to put_user()-like macros
		178	*
		179	* put_user() and friends are fragile, so it may depend on the implementation
		180	* whether the instrumentation happens before or after the data is copied from
		181	* the userspace.
		182	*
		183	* @from source address
		184	* @ptr userspace pointer to copy to
		185	* @size number of bytes to copy
		186	*/
		187	#define instrument_put_user(from, ptr, size) \
		188	({ \
		189	kmsan_copy_to_user(ptr, &from, sizeof(from), 0); \
		190	})
		191
139	#endif /* _LINUX_INSTRUMENTED_H */	192	#endif /* _LINUX_INSTRUMENTED_H */

include/linux/kasan.h+22−33

 #ifdef CONFIG_KASAN
 struct kasan_cache {
+#ifdef CONFIG_KASAN_GENERIC
 	int alloc_meta_offset;
 	int free_meta_offset;
+#endif
 	bool is_kmalloc;
 };
-slab_flags_t __kasan_never_merge(void);
-static __always_inline slab_flags_t kasan_never_merge(void)
-{
-	if (kasan_enabled())
-		return __kasan_never_merge();
-	return 0;
-}
 void __kasan_unpoison_range(const void *addr, size_t size);
 static __always_inline void kasan_unpoison_range(const void *addr, size_t size)
 {
 		__kasan_unpoison_pages(page, order, init);
 }
-void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
-				slab_flags_t *flags);
-static __always_inline void kasan_cache_create(struct kmem_cache *cache,
-				unsigned int *size, slab_flags_t *flags)
-{
-	if (kasan_enabled())
-		__kasan_cache_create(cache, size, flags);
-}
 void __kasan_cache_create_kmalloc(struct kmem_cache *cache);
 static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache)
 {
 		__kasan_cache_create_kmalloc(cache);
 }
-size_t __kasan_metadata_size(struct kmem_cache *cache);
-static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache)
-{
-	if (kasan_enabled())
-		return __kasan_metadata_size(cache);
-	return 0;
-}
 void __kasan_poison_slab(struct slab *slab);
 static __always_inline void kasan_poison_slab(struct slab *slab)
 {
 #else /* CONFIG_KASAN */
-static inline slab_flags_t kasan_never_merge(void)
-{
-	return 0;
-}
 static inline void kasan_unpoison_range(const void *address, size_t size) {}
 static inline void kasan_poison_pages(struct page *page, unsigned int order,
 				      bool init) {}
 static inline void kasan_unpoison_pages(struct page *page, unsigned int order,
 					bool init) {}
-static inline void kasan_cache_create(struct kmem_cache *cache,
-				      unsigned int *size,
-				      slab_flags_t *flags) {}
 static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {}
-static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
 static inline void kasan_poison_slab(struct slab *slab) {}
 static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
 					void *object) {}
 #ifdef CONFIG_KASAN_GENERIC
+size_t kasan_metadata_size(struct kmem_cache *cache);
+slab_flags_t kasan_never_merge(void);
+void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+			slab_flags_t *flags);
 void kasan_cache_shrink(struct kmem_cache *cache);
 void kasan_cache_shutdown(struct kmem_cache *cache);
 void kasan_record_aux_stack(void *ptr);
 #else /* CONFIG_KASAN_GENERIC */
+/* Tag-based KASAN modes do not use per-object metadata. */
+static inline size_t kasan_metadata_size(struct kmem_cache *cache)
+{
+	return 0;
+}
+/* And thus nothing prevents cache merging. */
+static inline slab_flags_t kasan_never_merge(void)
+{
+	return 0;
+}
+/* And no cache-related metadata initialization is required. */
+static inline void kasan_cache_create(struct kmem_cache *cache,
+				      unsigned int *size,
+				      slab_flags_t *flags) {}
 static inline void kasan_cache_shrink(struct kmem_cache *cache) {}
 static inline void kasan_cache_shutdown(struct kmem_cache *cache) {}
 static inline void kasan_record_aux_stack(void *ptr) {}

include/linux/khugepaged.h+8−5

 				 unsigned long vm_flags);
 extern void khugepaged_min_free_kbytes_update(void);
 #ifdef CONFIG_SHMEM
-extern void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr);
+extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
+				   bool install_pmd);
 #else
-static inline void collapse_pte_mapped_thp(struct mm_struct *mm,
+static inline int collapse_pte_mapped_thp(struct mm_struct *mm,
-					   unsigned long addr)
+					  unsigned long addr, bool install_pmd)
 {
+	return 0;
 }
 #endif
 					unsigned long vm_flags)
 {
 }
-static inline void collapse_pte_mapped_thp(struct mm_struct *mm,
+static inline int collapse_pte_mapped_thp(struct mm_struct *mm,
-					   unsigned long addr)
+					  unsigned long addr, bool install_pmd)
 {
+	return 0;
 }
 static inline void khugepaged_min_free_kbytes_update(void)

include/linux/kmsan-checks.h+83−0

@@ -0,0 +1,83 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* KMSAN checks to be used for one-off annotations in subsystems.
	4	*
	5	* Copyright (C) 2017-2022 Google LLC
	6	* Author: Alexander Potapenko <glider@google.com>
	7	*
	8	*/
	9
	10	#ifndef _LINUX_KMSAN_CHECKS_H
	11	#define _LINUX_KMSAN_CHECKS_H
	12
	13	#include <linux/types.h>
	14
	15	#ifdef CONFIG_KMSAN
	16
	17	/**
	18	* kmsan_poison_memory() - Mark the memory range as uninitialized.
	19	* @address: address to start with.
	20	* @size: size of buffer to poison.
	21	* @flags: GFP flags for allocations done by this function.
	22	*
	23	* Until other data is written to this range, KMSAN will treat it as
	24	* uninitialized. Error reports for this memory will reference the call site of
	25	* kmsan_poison_memory() as origin.
	26	*/
	27	void kmsan_poison_memory(const void *address, size_t size, gfp_t flags);
	28
	29	/**
	30	* kmsan_unpoison_memory() - Mark the memory range as initialized.
	31	* @address: address to start with.
	32	* @size: size of buffer to unpoison.
	33	*
	34	* Until other data is written to this range, KMSAN will treat it as
	35	* initialized.
	36	*/
	37	void kmsan_unpoison_memory(const void *address, size_t size);
	38
	39	/**
	40	* kmsan_check_memory() - Check the memory range for being initialized.
	41	* @address: address to start with.
	42	* @size: size of buffer to check.
	43	*
	44	* If any piece of the given range is marked as uninitialized, KMSAN will report
	45	* an error.
	46	*/
	47	void kmsan_check_memory(const void *address, size_t size);
	48
	49	/**
	50	* kmsan_copy_to_user() - Notify KMSAN about a data transfer to userspace.
	51	* @to: destination address in the userspace.
	52	* @from: source address in the kernel.
	53	* @to_copy: number of bytes to copy.
	54	* @left: number of bytes not copied.
	55	*
	56	* If this is a real userspace data transfer, KMSAN checks the bytes that were
	57	* actually copied to ensure there was no information leak. If @to belongs to
	58	* the kernel space (which is possible for compat syscalls), KMSAN just copies
	59	* the metadata.
	60	*/
	61	void kmsan_copy_to_user(void __user to, const void from, size_t to_copy,
	62	size_t left);
	63
	64	#else
	65
	66	static inline void kmsan_poison_memory(const void *address, size_t size,
	67	gfp_t flags)
	68	{
	69	}
	70	static inline void kmsan_unpoison_memory(const void *address, size_t size)
	71	{
	72	}
	73	static inline void kmsan_check_memory(const void *address, size_t size)
	74	{
	75	}
	76	static inline void kmsan_copy_to_user(void __user to, const void from,
	77	size_t to_copy, size_t left)
	78	{
	79	}
	80
	81	#endif
	82
	83	#endif /* _LINUX_KMSAN_CHECKS_H */

include/linux/kmsan.h+330−0

@@ -0,0 +1,330 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* KMSAN API for subsystems.
	4	*
	5	* Copyright (C) 2017-2022 Google LLC
	6	* Author: Alexander Potapenko <glider@google.com>
	7	*
	8	*/
	9	#ifndef _LINUX_KMSAN_H
	10	#define _LINUX_KMSAN_H
	11
	12	#include <linux/dma-direction.h>
	13	#include <linux/gfp.h>
	14	#include <linux/kmsan-checks.h>
	15	#include <linux/types.h>
	16
	17	struct page;
	18	struct kmem_cache;
	19	struct task_struct;
	20	struct scatterlist;
	21	struct urb;
	22
	23	#ifdef CONFIG_KMSAN
	24
	25	/**
	26	* kmsan_task_create() - Initialize KMSAN state for the task.
	27	* @task: task to initialize.
	28	*/
	29	void kmsan_task_create(struct task_struct *task);
	30
	31	/**
	32	* kmsan_task_exit() - Notify KMSAN that a task has exited.
	33	* @task: task about to finish.
	34	*/
	35	void kmsan_task_exit(struct task_struct *task);
	36
	37	/**
	38	* kmsan_init_shadow() - Initialize KMSAN shadow at boot time.
	39	*
	40	* Allocate and initialize KMSAN metadata for early allocations.
	41	*/
	42	void __init kmsan_init_shadow(void);
	43
	44	/**
	45	* kmsan_init_runtime() - Initialize KMSAN state and enable KMSAN.
	46	*/
	47	void __init kmsan_init_runtime(void);
	48
	49	/**
	50	* kmsan_memblock_free_pages() - handle freeing of memblock pages.
	51	* @page: struct page to free.
	52	* @order: order of @page.
	53	*
	54	* Freed pages are either returned to buddy allocator or held back to be used
	55	* as metadata pages.
	56	*/
	57	bool __init kmsan_memblock_free_pages(struct page *page, unsigned int order);
	58
	59	/**
	60	* kmsan_alloc_page() - Notify KMSAN about an alloc_pages() call.
	61	* @page: struct page pointer returned by alloc_pages().
	62	* @order: order of allocated struct page.
	63	* @flags: GFP flags used by alloc_pages()
	64	*
	65	* KMSAN marks 1<<@order pages starting at @page as uninitialized, unless
	66	* @flags contain __GFP_ZERO.
	67	*/
	68	void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags);
	69
	70	/**
	71	* kmsan_free_page() - Notify KMSAN about a free_pages() call.
	72	* @page: struct page pointer passed to free_pages().
	73	* @order: order of deallocated struct page.
	74	*
	75	* KMSAN marks freed memory as uninitialized.
	76	*/
	77	void kmsan_free_page(struct page *page, unsigned int order);
	78
	79	/**
	80	* kmsan_copy_page_meta() - Copy KMSAN metadata between two pages.
	81	* @dst: destination page.
	82	* @src: source page.
	83	*
	84	* KMSAN copies the contents of metadata pages for @src into the metadata pages
	85	* for @dst. If @dst has no associated metadata pages, nothing happens.
	86	* If @src has no associated metadata pages, @dst metadata pages are unpoisoned.
	87	*/
	88	void kmsan_copy_page_meta(struct page dst, struct page src);
	89
	90	/**
	91	* kmsan_slab_alloc() - Notify KMSAN about a slab allocation.
	92	* @s: slab cache the object belongs to.
	93	* @object: object pointer.
	94	* @flags: GFP flags passed to the allocator.
	95	*
	96	* Depending on cache flags and GFP flags, KMSAN sets up the metadata of the
	97	* newly created object, marking it as initialized or uninitialized.
	98	*/
	99	void kmsan_slab_alloc(struct kmem_cache s, void object, gfp_t flags);
	100
	101	/**
	102	* kmsan_slab_free() - Notify KMSAN about a slab deallocation.
	103	* @s: slab cache the object belongs to.
	104	* @object: object pointer.
	105	*
	106	* KMSAN marks the freed object as uninitialized.
	107	*/
	108	void kmsan_slab_free(struct kmem_cache s, void object);
	109
	110	/**
	111	* kmsan_kmalloc_large() - Notify KMSAN about a large slab allocation.
	112	* @ptr: object pointer.
	113	* @size: object size.
	114	* @flags: GFP flags passed to the allocator.
	115	*
	116	* Similar to kmsan_slab_alloc(), but for large allocations.
	117	*/
	118	void kmsan_kmalloc_large(const void *ptr, size_t size, gfp_t flags);
	119
	120	/**
	121	* kmsan_kfree_large() - Notify KMSAN about a large slab deallocation.
	122	* @ptr: object pointer.
	123	*
	124	* Similar to kmsan_slab_free(), but for large allocations.
	125	*/
	126	void kmsan_kfree_large(const void *ptr);
	127
	128	/**
	129	* kmsan_map_kernel_range_noflush() - Notify KMSAN about a vmap.
	130	* @start: start of vmapped range.
	131	* @end: end of vmapped range.
	132	* @prot: page protection flags used for vmap.
	133	* @pages: array of pages.
	134	* @page_shift: page_shift passed to vmap_range_noflush().
	135	*
	136	* KMSAN maps shadow and origin pages of @pages into contiguous ranges in
	137	* vmalloc metadata address range.
	138	*/
	139	void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
	140	pgprot_t prot, struct page **pages,
	141	unsigned int page_shift);
	142
	143	/**
	144	* kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap.
	145	* @start: start of vunmapped range.
	146	* @end: end of vunmapped range.
	147	*
	148	* KMSAN unmaps the contiguous metadata ranges created by
	149	* kmsan_map_kernel_range_noflush().
	150	*/
	151	void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end);
	152
	153	/**
	154	* kmsan_ioremap_page_range() - Notify KMSAN about a ioremap_page_range() call.
	155	* @addr: range start.
	156	* @end: range end.
	157	* @phys_addr: physical range start.
	158	* @prot: page protection flags used for ioremap_page_range().
	159	* @page_shift: page_shift argument passed to vmap_range_noflush().
	160	*
	161	* KMSAN creates new metadata pages for the physical pages mapped into the
	162	* virtual memory.
	163	*/
	164	void kmsan_ioremap_page_range(unsigned long addr, unsigned long end,
	165	phys_addr_t phys_addr, pgprot_t prot,
	166	unsigned int page_shift);
	167
	168	/**
	169	* kmsan_iounmap_page_range() - Notify KMSAN about a iounmap_page_range() call.
	170	* @start: range start.
	171	* @end: range end.
	172	*
	173	* KMSAN unmaps the metadata pages for the given range and, unlike for
	174	* vunmap_page_range(), also deallocates them.
	175	*/
	176	void kmsan_iounmap_page_range(unsigned long start, unsigned long end);
	177
	178	/**
	179	* kmsan_handle_dma() - Handle a DMA data transfer.
	180	* @page: first page of the buffer.
	181	* @offset: offset of the buffer within the first page.
	182	* @size: buffer size.
	183	* @dir: one of possible dma_data_direction values.
	184	*
	185	* Depending on @direction, KMSAN:
	186	* * checks the buffer, if it is copied to device;
	187	* * initializes the buffer, if it is copied from device;
	188	* * does both, if this is a DMA_BIDIRECTIONAL transfer.
	189	*/
	190	void kmsan_handle_dma(struct page *page, size_t offset, size_t size,
	191	enum dma_data_direction dir);
	192
	193	/**
	194	* kmsan_handle_dma_sg() - Handle a DMA transfer using scatterlist.
	195	* @sg: scatterlist holding DMA buffers.
	196	* @nents: number of scatterlist entries.
	197	* @dir: one of possible dma_data_direction values.
	198	*
	199	* Depending on @direction, KMSAN:
	200	* * checks the buffers in the scatterlist, if they are copied to device;
	201	* * initializes the buffers, if they are copied from device;
	202	* * does both, if this is a DMA_BIDIRECTIONAL transfer.
	203	*/
	204	void kmsan_handle_dma_sg(struct scatterlist *sg, int nents,
	205	enum dma_data_direction dir);
	206
	207	/**
	208	* kmsan_handle_urb() - Handle a USB data transfer.
	209	* @urb: struct urb pointer.
	210	* @is_out: data transfer direction (true means output to hardware).
	211	*
	212	* If @is_out is true, KMSAN checks the transfer buffer of @urb. Otherwise,
	213	* KMSAN initializes the transfer buffer.
	214	*/
	215	void kmsan_handle_urb(const struct urb *urb, bool is_out);
	216
	217	/**
	218	* kmsan_unpoison_entry_regs() - Handle pt_regs in low-level entry code.
	219	* @regs: struct pt_regs pointer received from assembly code.
	220	*
	221	* KMSAN unpoisons the contents of the passed pt_regs, preventing potential
	222	* false positive reports. Unlike kmsan_unpoison_memory(),
	223	* kmsan_unpoison_entry_regs() can be called from the regions where
	224	* kmsan_in_runtime() returns true, which is the case in early entry code.
	225	*/
	226	void kmsan_unpoison_entry_regs(const struct pt_regs *regs);
	227
	228	#else
	229
	230	static inline void kmsan_init_shadow(void)
	231	{
	232	}
	233
	234	static inline void kmsan_init_runtime(void)
	235	{
	236	}
	237
	238	static inline bool kmsan_memblock_free_pages(struct page *page,
	239	unsigned int order)
	240	{
	241	return true;
	242	}
	243
	244	static inline void kmsan_task_create(struct task_struct *task)
	245	{
	246	}
	247
	248	static inline void kmsan_task_exit(struct task_struct *task)
	249	{
	250	}
	251
	252	static inline int kmsan_alloc_page(struct page *page, unsigned int order,
	253	gfp_t flags)
	254	{
	255	return 0;
	256	}
	257
	258	static inline void kmsan_free_page(struct page *page, unsigned int order)
	259	{
	260	}
	261
	262	static inline void kmsan_copy_page_meta(struct page dst, struct page src)
	263	{
	264	}
	265
	266	static inline void kmsan_slab_alloc(struct kmem_cache s, void object,
	267	gfp_t flags)
	268	{
	269	}
	270
	271	static inline void kmsan_slab_free(struct kmem_cache s, void object)
	272	{
	273	}
	274
	275	static inline void kmsan_kmalloc_large(const void *ptr, size_t size,
	276	gfp_t flags)
	277	{
	278	}
	279
	280	static inline void kmsan_kfree_large(const void *ptr)
	281	{
	282	}
	283
	284	static inline void kmsan_vmap_pages_range_noflush(unsigned long start,
	285	unsigned long end,
	286	pgprot_t prot,
	287	struct page **pages,
	288	unsigned int page_shift)
	289	{
	290	}
	291
	292	static inline void kmsan_vunmap_range_noflush(unsigned long start,
	293	unsigned long end)
	294	{
	295	}
	296
	297	static inline void kmsan_ioremap_page_range(unsigned long start,
	298	unsigned long end,
	299	phys_addr_t phys_addr,
	300	pgprot_t prot,
	301	unsigned int page_shift)
	302	{
	303	}
	304
	305	static inline void kmsan_iounmap_page_range(unsigned long start,
	306	unsigned long end)
	307	{
	308	}
	309
	310	static inline void kmsan_handle_dma(struct page *page, size_t offset,
	311	size_t size, enum dma_data_direction dir)
	312	{
	313	}
	314
	315	static inline void kmsan_handle_dma_sg(struct scatterlist *sg, int nents,
	316	enum dma_data_direction dir)
	317	{
	318	}
	319
	320	static inline void kmsan_handle_urb(const struct urb *urb, bool is_out)
	321	{
	322	}
	323
	324	static inline void kmsan_unpoison_entry_regs(const struct pt_regs *regs)
	325	{
	326	}
	327
	328	#endif
	329
	330	#endif /* _LINUX_KMSAN_H */

include/linux/kmsan_types.h+35−0

@@ -0,0 +1,35 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* A minimal header declaring types added by KMSAN to existing kernel structs.
	4	*
	5	* Copyright (C) 2017-2022 Google LLC
	6	* Author: Alexander Potapenko <glider@google.com>
	7	*
	8	*/
	9	#ifndef _LINUX_KMSAN_TYPES_H
	10	#define _LINUX_KMSAN_TYPES_H
	11
	12	/* These constants are defined in the MSan LLVM instrumentation pass. */
	13	#define KMSAN_RETVAL_SIZE 800
	14	#define KMSAN_PARAM_SIZE 800
	15
	16	struct kmsan_context_state {
	17	char param_tls[KMSAN_PARAM_SIZE];
	18	char retval_tls[KMSAN_RETVAL_SIZE];
	19	char va_arg_tls[KMSAN_PARAM_SIZE];
	20	char va_arg_origin_tls[KMSAN_PARAM_SIZE];
	21	u64 va_arg_overflow_size_tls;
	22	char param_origin_tls[KMSAN_PARAM_SIZE];
	23	u32 retval_origin_tls;
	24	};
	25
	26	#undef KMSAN_PARAM_SIZE
	27	#undef KMSAN_RETVAL_SIZE
	28
	29	struct kmsan_ctx {
	30	struct kmsan_context_state cstate;
	31	int kmsan_in_runtime;
	32	bool allow_reporting;
	33	};
	34
	35	#endif /* _LINUX_KMSAN_TYPES_H */

include/linux/ksm.h+0−3

@@ -15,9 +15,6 @@ · 14 unmodified lines
15	#include <linux/sched.h>	15	#include <linux/sched.h>
16	#include <linux/sched/coredump.h>	16	#include <linux/sched/coredump.h>
17		17
18	struct stable_node;
19	struct mem_cgroup;
20
21	#ifdef CONFIG_KSM	18	#ifdef CONFIG_KSM
22	int ksm_madvise(struct vm_area_struct *vma, unsigned long start,	19	int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
23	unsigned long end, int advice, unsigned long *vm_flags);	20	unsigned long end, int advice, unsigned long *vm_flags);

include/linux/maple_tree.h+685−0

@@ -0,0 +1,685 @@
	1	/* SPDX-License-Identifier: GPL-2.0+ */
	2	#ifndef _LINUX_MAPLE_TREE_H
	3	#define _LINUX_MAPLE_TREE_H
	4	/*
	5	* Maple Tree - An RCU-safe adaptive tree for storing ranges
	6	* Copyright (c) 2018-2022 Oracle
	7	* Authors: Liam R. Howlett <Liam.Howlett@Oracle.com>
	8	* Matthew Wilcox <willy@infradead.org>
	9	*/
	10
	11	#include <linux/kernel.h>
	12	#include <linux/rcupdate.h>
	13	#include <linux/spinlock.h>
	14	/* #define CONFIG_MAPLE_RCU_DISABLED */
	15	/* #define CONFIG_DEBUG_MAPLE_TREE_VERBOSE */
	16
	17	/*
	18	* Allocated nodes are mutable until they have been inserted into the tree,
	19	* at which time they cannot change their type until they have been removed
	20	* from the tree and an RCU grace period has passed.
	21	*
	22	* Removed nodes have their ->parent set to point to themselves. RCU readers
	23	* check ->parent before relying on the value that they loaded from the
	24	* slots array. This lets us reuse the slots array for the RCU head.
	25	*
	26	* Nodes in the tree point to their parent unless bit 0 is set.
	27	*/
	28	#if defined(CONFIG_64BIT) \|\| defined(BUILD_VDSO32_64)
	29	/* 64bit sizes */
	30	#define MAPLE_NODE_SLOTS 31 /* 256 bytes including ->parent */
	31	#define MAPLE_RANGE64_SLOTS 16 /* 256 bytes */
	32	#define MAPLE_ARANGE64_SLOTS 10 /* 240 bytes */
	33	#define MAPLE_ARANGE64_META_MAX 15 /* Out of range for metadata */
	34	#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 1)
	35	#else
	36	/* 32bit sizes */
	37	#define MAPLE_NODE_SLOTS 63 /* 256 bytes including ->parent */
	38	#define MAPLE_RANGE64_SLOTS 32 /* 256 bytes */
	39	#define MAPLE_ARANGE64_SLOTS 21 /* 240 bytes */
	40	#define MAPLE_ARANGE64_META_MAX 31 /* Out of range for metadata */
	41	#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 2)
	42	#endif /* defined(CONFIG_64BIT) \|\| defined(BUILD_VDSO32_64) */
	43
	44	#define MAPLE_NODE_MASK 255UL
	45
	46	/*
	47	* The node->parent of the root node has bit 0 set and the rest of the pointer
	48	* is a pointer to the tree itself. No more bits are available in this pointer
	49	* (on m68k, the data structure may only be 2-byte aligned).
	50	*
	51	* Internal non-root nodes can only have maple_range_* nodes as parents. The
	52	* parent pointer is 256B aligned like all other tree nodes. When storing a 32
	53	* or 64 bit values, the offset can fit into 4 bits. The 16 bit values need an
	54	* extra bit to store the offset. This extra bit comes from a reuse of the last
	55	* bit in the node type. This is possible by using bit 1 to indicate if bit 2
	56	* is part of the type or the slot.
	57	*
	58	* Once the type is decided, the decision of an allocation range type or a range
	59	* type is done by examining the immutable tree flag for the MAPLE_ALLOC_RANGE
	60	* flag.
	61	*
	62	* Node types:
	63	* 0x??1 = Root
	64	* 0x?00 = 16 bit nodes
	65	* 0x010 = 32 bit nodes
	66	* 0x110 = 64 bit nodes
	67	*
	68	* Slot size and location in the parent pointer:
	69	* type : slot location
	70	* 0x??1 : Root
	71	* 0x?00 : 16 bit values, type in 0-1, slot in 2-6
	72	* 0x010 : 32 bit values, type in 0-2, slot in 3-6
	73	* 0x110 : 64 bit values, type in 0-2, slot in 3-6
	74	*/
	75
	76	/*
	77	* This metadata is used to optimize the gap updating code and in reverse
	78	* searching for gaps or any other code that needs to find the end of the data.
	79	*/
	80	struct maple_metadata {
	81	unsigned char end;
	82	unsigned char gap;
	83	};
	84
	85	/*
	86	* Leaf nodes do not store pointers to nodes, they store user data. Users may
	87	* store almost any bit pattern. As noted above, the optimisation of storing an
	88	* entry at 0 in the root pointer cannot be done for data which have the bottom
	89	* two bits set to '10'. We also reserve values with the bottom two bits set to
	90	* '10' which are below 4096 (ie 2, 6, 10 .. 4094) for internal use. Some APIs
	91	* return errnos as a negative errno shifted right by two bits and the bottom
	92	* two bits set to '10', and while choosing to store these values in the array
	93	* is not an error, it may lead to confusion if you're testing for an error with
	94	* mas_is_err().
	95	*
	96	* Non-leaf nodes store the type of the node pointed to (enum maple_type in bits
	97	* 3-6), bit 2 is reserved. That leaves bits 0-1 unused for now.
	98	*
	99	* In regular B-Tree terms, pivots are called keys. The term pivot is used to
	100	* indicate that the tree is specifying ranges, Pivots may appear in the
	101	* subtree with an entry attached to the value whereas keys are unique to a
	102	* specific position of a B-tree. Pivot values are inclusive of the slot with
	103	* the same index.
	104	*/
	105
	106	struct maple_range_64 {
	107	struct maple_pnode *parent;
	108	unsigned long pivot[MAPLE_RANGE64_SLOTS - 1];
	109	union {
	110	void __rcu *slot[MAPLE_RANGE64_SLOTS];
	111	struct {
	112	void __rcu *pad[MAPLE_RANGE64_SLOTS - 1];
	113	struct maple_metadata meta;
	114	};
	115	};
	116	};
	117
	118	/*
	119	* At tree creation time, the user can specify that they're willing to trade off
	120	* storing fewer entries in a tree in return for storing more information in
	121	* each node.
	122	*
	123	* The maple tree supports recording the largest range of NULL entries available
	124	* in this node, also called gaps. This optimises the tree for allocating a
	125	* range.
	126	*/
	127	struct maple_arange_64 {
	128	struct maple_pnode *parent;
	129	unsigned long pivot[MAPLE_ARANGE64_SLOTS - 1];
	130	void __rcu *slot[MAPLE_ARANGE64_SLOTS];
	131	unsigned long gap[MAPLE_ARANGE64_SLOTS];
	132	struct maple_metadata meta;
	133	};
	134
	135	struct maple_alloc {
	136	unsigned long total;
	137	unsigned char node_count;
	138	unsigned int request_count;
	139	struct maple_alloc *slot[MAPLE_ALLOC_SLOTS];
	140	};
	141
	142	struct maple_topiary {
	143	struct maple_pnode *parent;
	144	struct maple_enode next; / Overlaps the pivot */
	145	};
	146
	147	enum maple_type {
	148	maple_dense,
	149	maple_leaf_64,
	150	maple_range_64,
	151	maple_arange_64,
	152	};
	153
	154
	155	/**
	156	* DOC: Maple tree flags
	157	*
	158	* * MT_FLAGS_ALLOC_RANGE - Track gaps in this tree
	159	* * MT_FLAGS_USE_RCU - Operate in RCU mode
	160	* * MT_FLAGS_HEIGHT_OFFSET - The position of the tree height in the flags
	161	* * MT_FLAGS_HEIGHT_MASK - The mask for the maple tree height value
	162	* * MT_FLAGS_LOCK_MASK - How the mt_lock is used
	163	* * MT_FLAGS_LOCK_IRQ - Acquired irq-safe
	164	* * MT_FLAGS_LOCK_BH - Acquired bh-safe
	165	* * MT_FLAGS_LOCK_EXTERN - mt_lock is not used
	166	*
	167	* MAPLE_HEIGHT_MAX The largest height that can be stored
	168	*/
	169	#define MT_FLAGS_ALLOC_RANGE 0x01
	170	#define MT_FLAGS_USE_RCU 0x02
	171	#define MT_FLAGS_HEIGHT_OFFSET 0x02
	172	#define MT_FLAGS_HEIGHT_MASK 0x7C
	173	#define MT_FLAGS_LOCK_MASK 0x300
	174	#define MT_FLAGS_LOCK_IRQ 0x100
	175	#define MT_FLAGS_LOCK_BH 0x200
	176	#define MT_FLAGS_LOCK_EXTERN 0x300
	177
	178	#define MAPLE_HEIGHT_MAX 31
	179
	180
	181	#define MAPLE_NODE_TYPE_MASK 0x0F
	182	#define MAPLE_NODE_TYPE_SHIFT 0x03
	183
	184	#define MAPLE_RESERVED_RANGE 4096
	185
	186	#ifdef CONFIG_LOCKDEP
	187	typedef struct lockdep_map *lockdep_map_p;
	188	#define mt_lock_is_held(mt) lock_is_held(mt->ma_external_lock)
	189	#define mt_set_external_lock(mt, lock) \
	190	(mt)->ma_external_lock = &(lock)->dep_map
	191	#else
	192	typedef struct { /* nothing */ } lockdep_map_p;
	193	#define mt_lock_is_held(mt) 1
	194	#define mt_set_external_lock(mt, lock) do { } while (0)
	195	#endif
	196
	197	/*
	198	* If the tree contains a single entry at index 0, it is usually stored in
	199	* tree->ma_root. To optimise for the page cache, an entry which ends in '00',
	200	* '01' or '11' is stored in the root, but an entry which ends in '10' will be
	201	* stored in a node. Bits 3-6 are used to store enum maple_type.
	202	*
	203	* The flags are used both to store some immutable information about this tree
	204	* (set at tree creation time) and dynamic information set under the spinlock.
	205	*
	206	* Another use of flags are to indicate global states of the tree. This is the
	207	* case with the MAPLE_USE_RCU flag, which indicates the tree is currently in
	208	* RCU mode. This mode was added to allow the tree to reuse nodes instead of
	209	* re-allocating and RCU freeing nodes when there is a single user.
	210	*/
	211	struct maple_tree {
	212	union {
	213	spinlock_t ma_lock;
	214	lockdep_map_p ma_external_lock;
	215	};
	216	void __rcu *ma_root;
	217	unsigned int ma_flags;
	218	};
	219
	220	/**
	221	* MTREE_INIT() - Initialize a maple tree
	222	* @name: The maple tree name
	223	* @__flags: The maple tree flags
	224	*
	225	*/
	226	#define MTREE_INIT(name, __flags) { \
	227	.ma_lock = __SPIN_LOCK_UNLOCKED((name).ma_lock), \
	228	.ma_flags = __flags, \
	229	.ma_root = NULL, \
	230	}
	231
	232	/**
	233	* MTREE_INIT_EXT() - Initialize a maple tree with an external lock.
	234	* @name: The tree name
	235	* @__flags: The maple tree flags
	236	* @__lock: The external lock
	237	*/
	238	#ifdef CONFIG_LOCKDEP
	239	#define MTREE_INIT_EXT(name, __flags, __lock) { \
	240	.ma_external_lock = &(__lock).dep_map, \
	241	.ma_flags = (__flags), \
	242	.ma_root = NULL, \
	243	}
	244	#else
	245	#define MTREE_INIT_EXT(name, __flags, __lock) MTREE_INIT(name, __flags)
	246	#endif
	247
	248	#define DEFINE_MTREE(name) \
	249	struct maple_tree name = MTREE_INIT(name, 0)
	250
	251	#define mtree_lock(mt) spin_lock((&(mt)->ma_lock))
	252	#define mtree_unlock(mt) spin_unlock((&(mt)->ma_lock))
	253
	254	/*
	255	* The Maple Tree squeezes various bits in at various points which aren't
	256	* necessarily obvious. Usually, this is done by observing that pointers are
	257	* N-byte aligned and thus the bottom log_2(N) bits are available for use. We
	258	* don't use the high bits of pointers to store additional information because
	259	* we don't know what bits are unused on any given architecture.
	260	*
	261	* Nodes are 256 bytes in size and are also aligned to 256 bytes, giving us 8
	262	* low bits for our own purposes. Nodes are currently of 4 types:
	263	* 1. Single pointer (Range is 0-0)
	264	* 2. Non-leaf Allocation Range nodes
	265	* 3. Non-leaf Range nodes
	266	* 4. Leaf Range nodes All nodes consist of a number of node slots,
	267	* pivots, and a parent pointer.
	268	*/
	269
	270	struct maple_node {
	271	union {
	272	struct {
	273	struct maple_pnode *parent;
	274	void __rcu *slot[MAPLE_NODE_SLOTS];
	275	};
	276	struct {
	277	void *pad;
	278	struct rcu_head rcu;
	279	struct maple_enode *piv_parent;
	280	unsigned char parent_slot;
	281	enum maple_type type;
	282	unsigned char slot_len;
	283	unsigned int ma_flags;
	284	};
	285	struct maple_range_64 mr64;
	286	struct maple_arange_64 ma64;
	287	struct maple_alloc alloc;
	288	};
	289	};
	290
	291	/*
	292	* More complicated stores can cause two nodes to become one or three and
	293	* potentially alter the height of the tree. Either half of the tree may need
	294	* to be rebalanced against the other. The ma_topiary struct is used to track
	295	* which nodes have been 'cut' from the tree so that the change can be done
	296	* safely at a later date. This is done to support RCU.
	297	*/
	298	struct ma_topiary {
	299	struct maple_enode *head;
	300	struct maple_enode *tail;
	301	struct maple_tree *mtree;
	302	};
	303
	304	void mtree_load(struct maple_tree mt, unsigned long index);
	305
	306	int mtree_insert(struct maple_tree *mt, unsigned long index,
	307	void *entry, gfp_t gfp);
	308	int mtree_insert_range(struct maple_tree *mt, unsigned long first,
	309	unsigned long last, void *entry, gfp_t gfp);
	310	int mtree_alloc_range(struct maple_tree mt, unsigned long startp,
	311	void *entry, unsigned long size, unsigned long min,
	312	unsigned long max, gfp_t gfp);
	313	int mtree_alloc_rrange(struct maple_tree mt, unsigned long startp,
	314	void *entry, unsigned long size, unsigned long min,
	315	unsigned long max, gfp_t gfp);
	316
	317	int mtree_store_range(struct maple_tree *mt, unsigned long first,
	318	unsigned long last, void *entry, gfp_t gfp);
	319	int mtree_store(struct maple_tree *mt, unsigned long index,
	320	void *entry, gfp_t gfp);
	321	void mtree_erase(struct maple_tree mt, unsigned long index);
	322
	323	void mtree_destroy(struct maple_tree *mt);
	324	void __mt_destroy(struct maple_tree *mt);
	325
	326	/**
	327	* mtree_empty() - Determine if a tree has any present entries.
	328	* @mt: Maple Tree.
	329	*
	330	* Context: Any context.
	331	* Return: %true if the tree contains only NULL pointers.
	332	*/
	333	static inline bool mtree_empty(const struct maple_tree *mt)
	334	{
	335	return mt->ma_root == NULL;
	336	}
	337
	338	/* Advanced API */
	339
	340	/*
	341	* The maple state is defined in the struct ma_state and is used to keep track
	342	* of information during operations, and even between operations when using the
	343	* advanced API.
	344	*
	345	* If state->node has bit 0 set then it references a tree location which is not
	346	* a node (eg the root). If bit 1 is set, the rest of the bits are a negative
	347	* errno. Bit 2 (the 'unallocated slots' bit) is clear. Bits 3-6 indicate the
	348	* node type.
	349	*
	350	* state->alloc either has a request number of nodes or an allocated node. If
	351	* stat->alloc has a requested number of nodes, the first bit will be set (0x1)
	352	* and the remaining bits are the value. If state->alloc is a node, then the
	353	* node will be of type maple_alloc. maple_alloc has MAPLE_NODE_SLOTS - 1 for
	354	* storing more allocated nodes, a total number of nodes allocated, and the
	355	* node_count in this node. node_count is the number of allocated nodes in this
	356	* node. The scaling beyond MAPLE_NODE_SLOTS - 1 is handled by storing further
	357	* nodes into state->alloc->slot[0]'s node. Nodes are taken from state->alloc
	358	* by removing a node from the state->alloc node until state->alloc->node_count
	359	* is 1, when state->alloc is returned and the state->alloc->slot[0] is promoted
	360	* to state->alloc. Nodes are pushed onto state->alloc by putting the current
	361	* state->alloc into the pushed node's slot[0].
	362	*
	363	* The state also contains the implied min/max of the state->node, the depth of
	364	* this search, and the offset. The implied min/max are either from the parent
	365	* node or are 0-oo for the root node. The depth is incremented or decremented
	366	* every time a node is walked down or up. The offset is the slot/pivot of
	367	* interest in the node - either for reading or writing.
	368	*
	369	* When returning a value the maple state index and last respectively contain
	370	* the start and end of the range for the entry. Ranges are inclusive in the
	371	* Maple Tree.
	372	*/
	373	struct ma_state {
	374	struct maple_tree tree; / The tree we're operating in */
	375	unsigned long index; /* The index we're operating on - range start */
	376	unsigned long last; /* The last index we're operating on - range end */
	377	struct maple_enode node; / The node containing this entry */
	378	unsigned long min; /* The minimum index of this node - implied pivot min */
	379	unsigned long max; /* The maximum index of this node - implied pivot max */
	380	struct maple_alloc alloc; / Allocated nodes for this operation */
	381	unsigned char depth; /* depth of tree descent during write */
	382	unsigned char offset;
	383	unsigned char mas_flags;
	384	};
	385
	386	struct ma_wr_state {
	387	struct ma_state *mas;
	388	struct maple_node node; / Decoded mas->node */
	389	unsigned long r_min; /* range min */
	390	unsigned long r_max; /* range max */
	391	enum maple_type type; /* mas->node type */
	392	unsigned char offset_end; /* The offset where the write ends */
	393	unsigned char node_end; /* mas->node end */
	394	unsigned long pivots; / mas->node->pivots pointer */
	395	unsigned long end_piv; /* The pivot at the offset end */
	396	void __rcu *slots; / mas->node->slots pointer */
	397	void entry; / The entry to write */
	398	void content; / The existing entry that is being overwritten */
	399	};
	400
	401	#define mas_lock(mas) spin_lock(&((mas)->tree->ma_lock))
	402	#define mas_unlock(mas) spin_unlock(&((mas)->tree->ma_lock))
	403
	404
	405	/*
	406	* Special values for ma_state.node.
	407	* MAS_START means we have not searched the tree.
	408	* MAS_ROOT means we have searched the tree and the entry we found lives in
	409	* the root of the tree (ie it has index 0, length 1 and is the only entry in
	410	* the tree).
	411	* MAS_NONE means we have searched the tree and there is no node in the
	412	* tree for this entry. For example, we searched for index 1 in an empty
	413	* tree. Or we have a tree which points to a full leaf node and we
	414	* searched for an entry which is larger than can be contained in that
	415	* leaf node.
	416	* MA_ERROR represents an errno. After dropping the lock and attempting
	417	* to resolve the error, the walk would have to be restarted from the
	418	* top of the tree as the tree may have been modified.
	419	*/
	420	#define MAS_START ((struct maple_enode *)1UL)
	421	#define MAS_ROOT ((struct maple_enode *)5UL)
	422	#define MAS_NONE ((struct maple_enode *)9UL)
	423	#define MAS_PAUSE ((struct maple_enode *)17UL)
	424	#define MA_ERROR(err) \
	425	((struct maple_enode *)(((unsigned long)err << 2) \| 2UL))
	426
	427	#define MA_STATE(name, mt, first, end) \
	428	struct ma_state name = { \
	429	.tree = mt, \
	430	.index = first, \
	431	.last = end, \
	432	.node = MAS_START, \
	433	.min = 0, \
	434	.max = ULONG_MAX, \
	435	.alloc = NULL, \
	436	}
	437
	438	#define MA_WR_STATE(name, ma_state, wr_entry) \
	439	struct ma_wr_state name = { \
	440	.mas = ma_state, \
	441	.content = NULL, \
	442	.entry = wr_entry, \
	443	}
	444
	445	#define MA_TOPIARY(name, tree) \
	446	struct ma_topiary name = { \
	447	.head = NULL, \
	448	.tail = NULL, \
	449	.mtree = tree, \
	450	}
	451
	452	void mas_walk(struct ma_state mas);
	453	void mas_store(struct ma_state mas, void *entry);
	454	void mas_erase(struct ma_state mas);
	455	int mas_store_gfp(struct ma_state mas, void entry, gfp_t gfp);
	456	void mas_store_prealloc(struct ma_state mas, void entry);
	457	void mas_find(struct ma_state mas, unsigned long max);
	458	void mas_find_rev(struct ma_state mas, unsigned long min);
	459	int mas_preallocate(struct ma_state mas, void entry, gfp_t gfp);
	460	bool mas_is_err(struct ma_state *mas);
	461
	462	bool mas_nomem(struct ma_state *mas, gfp_t gfp);
	463	void mas_pause(struct ma_state *mas);
	464	void maple_tree_init(void);
	465	void mas_destroy(struct ma_state *mas);
	466	int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries);
	467
	468	void mas_prev(struct ma_state mas, unsigned long min);
	469	void mas_next(struct ma_state mas, unsigned long max);
	470
	471	int mas_empty_area(struct ma_state *mas, unsigned long min, unsigned long max,
	472	unsigned long size);
	473
	474	/* Checks if a mas has not found anything */
	475	static inline bool mas_is_none(struct ma_state *mas)
	476	{
	477	return mas->node == MAS_NONE;
	478	}
	479
	480	/* Checks if a mas has been paused */
	481	static inline bool mas_is_paused(struct ma_state *mas)
	482	{
	483	return mas->node == MAS_PAUSE;
	484	}
	485
	486	void mas_dup_tree(struct ma_state oldmas, struct ma_state mas);
	487	void mas_dup_store(struct ma_state mas, void entry);
	488
	489	/*
	490	* This finds an empty area from the highest address to the lowest.
	491	* AKA "Topdown" version,
	492	*/
	493	int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
	494	unsigned long max, unsigned long size);
	495	/**
	496	* mas_reset() - Reset a Maple Tree operation state.
	497	* @mas: Maple Tree operation state.
	498	*
	499	* Resets the error or walk state of the @mas so future walks of the
	500	* array will start from the root. Use this if you have dropped the
	501	* lock and want to reuse the ma_state.
	502	*
	503	* Context: Any context.
	504	*/
	505	static inline void mas_reset(struct ma_state *mas)
	506	{
	507	mas->node = MAS_START;
	508	}
	509
	510	/**
	511	* mas_for_each() - Iterate over a range of the maple tree.
	512	* @__mas: Maple Tree operation state (maple_state)
	513	* @__entry: Entry retrieved from the tree
	514	* @__max: maximum index to retrieve from the tree
	515	*
	516	* When returned, mas->index and mas->last will hold the entire range for the
	517	* entry.
	518	*
	519	* Note: may return the zero entry.
	520	*
	521	*/
	522	#define mas_for_each(__mas, __entry, __max) \
	523	while (((__entry) = mas_find((__mas), (__max))) != NULL)
	524
	525
	526	/**
	527	* mas_set_range() - Set up Maple Tree operation state for a different index.
	528	* @mas: Maple Tree operation state.
	529	* @start: New start of range in the Maple Tree.
	530	* @last: New end of range in the Maple Tree.
	531	*
	532	* Move the operation state to refer to a different range. This will
	533	* have the effect of starting a walk from the top; see mas_next()
	534	* to move to an adjacent index.
	535	*/
	536	static inline
	537	void mas_set_range(struct ma_state *mas, unsigned long start, unsigned long last)
	538	{
	539	mas->index = start;
	540	mas->last = last;
	541	mas->node = MAS_START;
	542	}
	543
	544	/**
	545	* mas_set() - Set up Maple Tree operation state for a different index.
	546	* @mas: Maple Tree operation state.
	547	* @index: New index into the Maple Tree.
	548	*
	549	* Move the operation state to refer to a different index. This will
	550	* have the effect of starting a walk from the top; see mas_next()
	551	* to move to an adjacent index.
	552	*/
	553	static inline void mas_set(struct ma_state *mas, unsigned long index)
	554	{
	555
	556	mas_set_range(mas, index, index);
	557	}
	558
	559	static inline bool mt_external_lock(const struct maple_tree *mt)
	560	{
	561	return (mt->ma_flags & MT_FLAGS_LOCK_MASK) == MT_FLAGS_LOCK_EXTERN;
	562	}
	563
	564	/**
	565	* mt_init_flags() - Initialise an empty maple tree with flags.
	566	* @mt: Maple Tree
	567	* @flags: maple tree flags.
	568	*
	569	* If you need to initialise a Maple Tree with special flags (eg, an
	570	* allocation tree), use this function.
	571	*
	572	* Context: Any context.
	573	*/
	574	static inline void mt_init_flags(struct maple_tree *mt, unsigned int flags)
	575	{
	576	mt->ma_flags = flags;
	577	if (!mt_external_lock(mt))
	578	spin_lock_init(&mt->ma_lock);
	579	rcu_assign_pointer(mt->ma_root, NULL);
	580	}
	581
	582	/**
	583	* mt_init() - Initialise an empty maple tree.
	584	* @mt: Maple Tree
	585	*
	586	* An empty Maple Tree.
	587	*
	588	* Context: Any context.
	589	*/
	590	static inline void mt_init(struct maple_tree *mt)
	591	{
	592	mt_init_flags(mt, 0);
	593	}
	594
	595	static inline bool mt_in_rcu(struct maple_tree *mt)
	596	{
	597	#ifdef CONFIG_MAPLE_RCU_DISABLED
	598	return false;
	599	#endif
	600	return mt->ma_flags & MT_FLAGS_USE_RCU;
	601	}
	602
	603	/**
	604	* mt_clear_in_rcu() - Switch the tree to non-RCU mode.
	605	* @mt: The Maple Tree
	606	*/
	607	static inline void mt_clear_in_rcu(struct maple_tree *mt)
	608	{
	609	if (!mt_in_rcu(mt))
	610	return;
	611
	612	if (mt_external_lock(mt)) {
	613	BUG_ON(!mt_lock_is_held(mt));
	614	mt->ma_flags &= ~MT_FLAGS_USE_RCU;
	615	} else {
	616	mtree_lock(mt);
	617	mt->ma_flags &= ~MT_FLAGS_USE_RCU;
	618	mtree_unlock(mt);
	619	}
	620	}
	621
	622	/**
	623	* mt_set_in_rcu() - Switch the tree to RCU safe mode.
	624	* @mt: The Maple Tree
	625	*/
	626	static inline void mt_set_in_rcu(struct maple_tree *mt)
	627	{
	628	if (mt_in_rcu(mt))
	629	return;
	630
	631	if (mt_external_lock(mt)) {
	632	BUG_ON(!mt_lock_is_held(mt));
	633	mt->ma_flags \|= MT_FLAGS_USE_RCU;
	634	} else {
	635	mtree_lock(mt);
	636	mt->ma_flags \|= MT_FLAGS_USE_RCU;
	637	mtree_unlock(mt);
	638	}
	639	}
	640
	641	void mt_find(struct maple_tree mt, unsigned long *index, unsigned long max);
	642	void mt_find_after(struct maple_tree mt, unsigned long *index,
	643	unsigned long max);
	644	void mt_prev(struct maple_tree mt, unsigned long index, unsigned long min);
	645	void mt_next(struct maple_tree mt, unsigned long index, unsigned long max);
	646
	647	/**
	648	* mt_for_each - Iterate over each entry starting at index until max.
	649	* @__tree: The Maple Tree
	650	* @__entry: The current entry
	651	* @__index: The index to update to track the location in the tree
	652	* @__max: The maximum limit for @index
	653	*
	654	* Note: Will not return the zero entry.
	655	*/
	656	#define mt_for_each(__tree, __entry, __index, __max) \
	657	for (__entry = mt_find(__tree, &(__index), __max); \
	658	__entry; __entry = mt_find_after(__tree, &(__index), __max))
	659
	660
	661	#ifdef CONFIG_DEBUG_MAPLE_TREE
	662	extern atomic_t maple_tree_tests_run;
	663	extern atomic_t maple_tree_tests_passed;
	664
	665	void mt_dump(const struct maple_tree *mt);
	666	void mt_validate(struct maple_tree *mt);
	667	#define MT_BUG_ON(__tree, __x) do { \
	668	atomic_inc(&maple_tree_tests_run); \
	669	if (__x) { \
	670	pr_info("BUG at %s:%d (%u)\n", \
	671	__func__, __LINE__, __x); \
	672	mt_dump(__tree); \
	673	pr_info("Pass: %u Run:%u\n", \
	674	atomic_read(&maple_tree_tests_passed), \
	675	atomic_read(&maple_tree_tests_run)); \
	676	dump_stack(); \
	677	} else { \
	678	atomic_inc(&maple_tree_tests_passed); \
	679	} \
	680	} while (0)
	681	#else
	682	#define MT_BUG_ON(__tree, __x) BUG_ON(__x)
	683	#endif /* CONFIG_DEBUG_MAPLE_TREE */
	684
	685	#endif /_LINUX_MAPLE_TREE_H /

include/linux/memcontrol.h+49−50

 	MEM_CGROUP_NTARGETS,
 };
-struct memcg_vmstats_percpu {
+struct memcg_vmstats_percpu;
-	/* Local (CPU and cgroup) page state & events */
+struct memcg_vmstats;
-	long			state[MEMCG_NR_STAT];
-	unsigned long		events[NR_VM_EVENT_ITEMS];
-	/* Delta calculation for lockless upward propagation */
-	long			state_prev[MEMCG_NR_STAT];
-	unsigned long		events_prev[NR_VM_EVENT_ITEMS];
-	/* Cgroup1: threshold notifications & softlimit tree updates */
-	unsigned long		nr_page_events;
-	unsigned long		targets[MEM_CGROUP_NTARGETS];
-};
-struct memcg_vmstats {
-	/* Aggregated (CPU and subtree) page state & events */
-	long			state[MEMCG_NR_STAT];
-	unsigned long		events[NR_VM_EVENT_ITEMS];
-	/* Pending child counts during tree propagation */
-	long			state_pending[MEMCG_NR_STAT];
-	unsigned long		events_pending[NR_VM_EVENT_ITEMS];
-};
 struct mem_cgroup_reclaim_iter {
 	struct mem_cgroup *position;
 	struct mem_cgroup_threshold_ary *spare;
 };
-#if defined(CONFIG_SMP)
-struct memcg_padding {
-	char x[0];
-} ____cacheline_internodealigned_in_smp;
-#define MEMCG_PADDING(name)      struct memcg_padding name
-#else
-#define MEMCG_PADDING(name)
-#endif
 /*
  * Remember four most recent foreign writebacks with dirty pages in this
  * cgroup.  Inode sharing is expected to be uncommon and, even if we miss
 	spinlock_t		move_lock;
 	unsigned long		move_lock_flags;
-	MEMCG_PADDING(_pad1_);
+	CACHELINE_PADDING(_pad1_);
 	/* memory.stat */
-	struct memcg_vmstats	vmstats;
+	struct memcg_vmstats	*vmstats;
 	/* memory.events */
 	atomic_long_t		memory_events[MEMCG_NR_MEMORY_EVENTS];
 	struct list_head objcg_list;
 #endif
-	MEMCG_PADDING(_pad2_);
+	CACHELINE_PADDING(_pad2_);
 	/*
 	 * set > 0 if pages under this cgroup are moving to other cgroup.
 	struct deferred_split deferred_split_queue;
 #endif
+#ifdef CONFIG_LRU_GEN
+	/* per-memcg mm_struct list */
+	struct lru_gen_mm_list mm_list;
+#endif
 	struct mem_cgroup_per_node *nodeinfo[];
 };
 /*
- * size of first charge trial. "32" comes from vmscan.c's magic value.
+ * size of first charge trial.
- * TODO: maybe necessary to use big numbers in big irons.
+ * TODO: maybe necessary to use big numbers in big irons or dynamic based of the
+ * workload.
  */
-#define MEMCG_CHARGE_BATCH 32U
+#define MEMCG_CHARGE_BATCH 64U
 extern struct mem_cgroup *root_mem_cgroup;
  * - LRU isolation
  * - lock_page_memcg()
  * - exclusive reference
+ * - mem_cgroup_trylock_pages()
  *
  * For a kmem folio a caller should hold an rcu read lock to protect memcg
  * associated with a kmem folio from being released.
  * - LRU isolation
  * - lock_page_memcg()
  * - exclusive reference
+ * - mem_cgroup_trylock_pages()
  *
  * For a kmem page a caller should hold an rcu read lock to protect memcg
  * associated with a kmem page from being released.
 	return __mem_cgroup_charge(folio, mm, gfp);
 }
-int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm,
+int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
 				  gfp_t gfp, swp_entry_t entry);
 void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry);
 void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val);
+/* try to stablize folio_memcg() for all the pages in a memcg */
+static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
+{
+	rcu_read_lock();
+	if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account))
+		return true;
+	rcu_read_unlock();
+	return false;
+}
+static inline void mem_cgroup_unlock_pages(void)
+{
+	rcu_read_unlock();
+}
 /* idx can be of type enum memcg_stat_item or node_stat_item */
 static inline void mod_memcg_state(struct mem_cgroup *memcg,
 				   int idx, int val)
 	rcu_read_unlock();
 }
-static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx)
+unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx);
-{
-	long x = READ_ONCE(memcg->vmstats.state[idx]);
-#ifdef CONFIG_SMP
-	if (x < 0)
-		x = 0;
-#endif
-	return x;
-}
 static inline unsigned long lruvec_page_state(struct lruvec *lruvec,
 					      enum node_stat_item idx)
 	return 0;
 }
-static inline int mem_cgroup_swapin_charge_page(struct page *page,
+static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
 			struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
 {
 	return 0;
 {
 }
+static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg)
+{
+	/* to match folio_memcg_rcu() */
+	rcu_read_lock();
+	return true;
+}
+static inline void mem_cgroup_unlock_pages(void)
+{
+	rcu_read_unlock();
+}
 static inline void mem_cgroup_handle_over_high(void)
 {
 }
 {
 	struct mem_cgroup *memcg;
-	if (mem_cgroup_kmem_disabled())
+	if (!memcg_kmem_enabled())
 		return;
 	rcu_read_lock();

include/linux/memory-tiers.h+102−0

@@ -0,0 +1,102 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _LINUX_MEMORY_TIERS_H
	3	#define _LINUX_MEMORY_TIERS_H
	4
	5	#include <linux/types.h>
	6	#include <linux/nodemask.h>
	7	#include <linux/kref.h>
	8	#include <linux/mmzone.h>
	9	/*
	10	* Each tier cover a abstrace distance chunk size of 128
	11	*/
	12	#define MEMTIER_CHUNK_BITS 7
	13	#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS)
	14	/*
	15	* Smaller abstract distance values imply faster (higher) memory tiers. Offset
	16	* the DRAM adistance so that we can accommodate devices with a slightly lower
	17	* adistance value (slightly faster) than default DRAM adistance to be part of
	18	* the same memory tier.
	19	*/
	20	#define MEMTIER_ADISTANCE_DRAM ((4 * MEMTIER_CHUNK_SIZE) + (MEMTIER_CHUNK_SIZE >> 1))
	21	#define MEMTIER_HOTPLUG_PRIO 100
	22
	23	struct memory_tier;
	24	struct memory_dev_type {
	25	/* list of memory types that are part of same tier as this type */
	26	struct list_head tier_sibiling;
	27	/* abstract distance for this specific memory type */
	28	int adistance;
	29	/* Nodes of same abstract distance */
	30	nodemask_t nodes;
	31	struct kref kref;
	32	};
	33
	34	#ifdef CONFIG_NUMA
	35	extern bool numa_demotion_enabled;
	36	struct memory_dev_type *alloc_memory_type(int adistance);
	37	void destroy_memory_type(struct memory_dev_type *memtype);
	38	void init_node_memory_type(int node, struct memory_dev_type *default_type);
	39	void clear_node_memory_type(int node, struct memory_dev_type *memtype);
	40	#ifdef CONFIG_MIGRATION
	41	int next_demotion_node(int node);
	42	void node_get_allowed_targets(pg_data_t pgdat, nodemask_t targets);
	43	bool node_is_toptier(int node);
	44	#else
	45	static inline int next_demotion_node(int node)
	46	{
	47	return NUMA_NO_NODE;
	48	}
	49
	50	static inline void node_get_allowed_targets(pg_data_t pgdat, nodemask_t targets)
	51	{
	52	*targets = NODE_MASK_NONE;
	53	}
	54
	55	static inline bool node_is_toptier(int node)
	56	{
	57	return true;
	58	}
	59	#endif
	60
	61	#else
	62
	63	#define numa_demotion_enabled false
	64	/*
	65	* CONFIG_NUMA implementation returns non NULL error.
	66	*/
	67	static inline struct memory_dev_type *alloc_memory_type(int adistance)
	68	{
	69	return NULL;
	70	}
	71
	72	static inline void destroy_memory_type(struct memory_dev_type *memtype)
	73	{
	74
	75	}
	76
	77	static inline void init_node_memory_type(int node, struct memory_dev_type *default_type)
	78	{
	79
	80	}
	81
	82	static inline void clear_node_memory_type(int node, struct memory_dev_type *memtype)
	83	{
	84
	85	}
	86
	87	static inline int next_demotion_node(int node)
	88	{
	89	return NUMA_NO_NODE;
	90	}
	91
	92	static inline void node_get_allowed_targets(pg_data_t pgdat, nodemask_t targets)
	93	{
	94	*targets = NODE_MASK_NONE;
	95	}
	96
	97	static inline bool node_is_toptier(int node)
	98	{
	99	return true;
	100	}
	101	#endif /* CONFIG_NUMA */
	102	#endif /* _LINUX_MEMORY_TIERS_H */

include/linux/memory_hotplug.h+20−10

 struct zone;
 struct pglist_data;
 struct mem_section;
-struct memory_block;
 struct memory_group;
 struct resource;
 struct vmem_altmap;
 ({								\
 	memblock_alloc(sizeof(*pgdat), SMP_CACHE_BYTES);	\
 })
-/*
- * This definition is just for error path in node hotadd.
- * For node hotremove, we have to replace this.
- */
-#define generic_free_nodedata(pgdat)	kfree(pgdat)
 extern pg_data_t *node_data[];
 static inline void arch_refresh_nodedata(int nid, pg_data_t *pgdat)
 	BUG();
 	return NULL;
 }
-static inline void generic_free_nodedata(pg_data_t *pgdat)
-{
-}
 static inline void arch_refresh_nodedata(int nid, pg_data_t *pgdat)
 {
 }
 void mem_hotplug_begin(void);
 void mem_hotplug_done(void);
+/* See kswapd_is_running() */
+static inline void pgdat_kswapd_lock(pg_data_t *pgdat)
+{
+	mutex_lock(&pgdat->kswapd_lock);
+}
+static inline void pgdat_kswapd_unlock(pg_data_t *pgdat)
+{
+	mutex_unlock(&pgdat->kswapd_lock);
+}
+static inline void pgdat_kswapd_lock_init(pg_data_t *pgdat)
+{
+	mutex_init(&pgdat->kswapd_lock);
+}
 #else /* ! CONFIG_MEMORY_HOTPLUG */
 #define pfn_to_online_page(pfn)			\
 ({						\
 {
 	return false;
 }
+static inline void pgdat_kswapd_lock(pg_data_t *pgdat) {}
+static inline void pgdat_kswapd_unlock(pg_data_t *pgdat) {}
+static inline void pgdat_kswapd_lock_init(pg_data_t *pgdat) {}
 #endif /* ! CONFIG_MEMORY_HOTPLUG */
 /*
 extern void remove_pfn_range_from_zone(struct zone *zone,
 				       unsigned long start_pfn,
 				       unsigned long nr_pages);
-extern bool is_memblock_offlined(struct memory_block *mem);
 extern int sparse_add_section(int nid, unsigned long pfn,
 		unsigned long nr_pages, struct vmem_altmap *altmap,
 		struct dev_pagemap *pgmap);

include/linux/mempolicy.h+1−12

 				const nodemask_t *mask);
 extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy);
-static inline nodemask_t *policy_nodemask_current(gfp_t gfp)
-{
-	struct mempolicy *mpol = get_task_policy(current);
-	return policy_nodemask(gfp, mpol);
-}
 extern unsigned int mempolicy_slab_node(void);
 extern enum zone_type policy_zone;
 	return  (pol->mode == MPOL_PREFERRED_MANY);
 }
+extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone);
 #else
 {
 }
-static inline nodemask_t *policy_nodemask_current(gfp_t gfp)
-{
-	return NULL;
-}
 static inline bool mpol_is_preferred_many(struct mempolicy *pol)
 {
 	return  false;

include/linux/migrate.h+0−15

@@ -100,21 +100,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping, · 99 unmodified lines
100		100
101	#endif /* CONFIG_MIGRATION */	101	#endif /* CONFIG_MIGRATION */
102		102
103	#if defined(CONFIG_MIGRATION) && defined(CONFIG_NUMA)
104	extern void set_migration_target_nodes(void);
105	extern void migrate_on_reclaim_init(void);
106	extern bool numa_demotion_enabled;
107	extern int next_demotion_node(int node);
108	#else
109	static inline void set_migration_target_nodes(void) {}
110	static inline void migrate_on_reclaim_init(void) {}
111	static inline int next_demotion_node(int node)
112	{
113	return NUMA_NO_NODE;
114	}
115	#define numa_demotion_enabled false
116	#endif
117
118	#ifdef CONFIG_COMPACTION	103	#ifdef CONFIG_COMPACTION
119	bool PageMovable(struct page *page);	104	bool PageMovable(struct page *page);
120	void __SetPageMovable(struct page page, const struct movable_operations ops);	105	void __SetPageMovable(struct page page, const struct movable_operations ops);

include/linux/mm.h+121−40

@@ -661,6 +661,38 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) · 660 unmodified lines
661	return vma->vm_flags & VM_ACCESS_FLAGS;	661	return vma->vm_flags & VM_ACCESS_FLAGS;
662	}	662	}
663		663
		664	static inline
		665	struct vm_area_struct vma_find(struct vma_iterator vmi, unsigned long max)
		666	{
		667	return mas_find(&vmi->mas, max);
		668	}
		669
		670	static inline struct vm_area_struct vma_next(struct vma_iterator vmi)
		671	{
		672	/*
		673	* Uses vma_find() to get the first VMA when the iterator starts.
		674	* Calling mas_next() could skip the first entry.
		675	*/
		676	return vma_find(vmi, ULONG_MAX);
		677	}
		678
		679	static inline struct vm_area_struct vma_prev(struct vma_iterator vmi)
		680	{
		681	return mas_prev(&vmi->mas, 0);
		682	}
		683
		684	static inline unsigned long vma_iter_addr(struct vma_iterator *vmi)
		685	{
		686	return vmi->mas.index;
		687	}
		688
		689	#define for_each_vma(__vmi, __vma) \
		690	while (((__vma) = vma_next(&(__vmi))) != NULL)
		691
		692	/* The MM code likes to work with exclusive end addresses */
		693	#define for_each_vma_range(__vmi, __vma, __end) \
		694	while (((__vma) = vma_find(&(__vmi), (__end) - 1)) != NULL)
		695
664	#ifdef CONFIG_SHMEM	696	#ifdef CONFIG_SHMEM
665	/*	697	/*
666	* The vma_is_shmem is not inline because it is used only by slow	698	* The vma_is_shmem is not inline because it is used only by slow
@@ -697,7 +729,9 @@ static inline unsigned int compound_order(struct page *page) · 30 unmodified lines
697	*/	729	*/
698	static inline unsigned int folio_order(struct folio *folio)	730	static inline unsigned int folio_order(struct folio *folio)
699	{	731	{
700	return compound_order(&folio->page);	732	if (!folio_test_large(folio))
		733	return 0;
		734	return folio->_folio_order;
701	}	735	}
702		736
703	#include <linux/huge_mm.h>	737	#include <linux/huge_mm.h>
@@ -1255,6 +1289,18 @@ static inline int folio_nid(const struct folio *folio) · 551 unmodified lines
1255	}	1289	}
1256		1290
1257	#ifdef CONFIG_NUMA_BALANCING	1291	#ifdef CONFIG_NUMA_BALANCING
		1292	/* page access time bits needs to hold at least 4 seconds */
		1293	#define PAGE_ACCESS_TIME_MIN_BITS 12
		1294	#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS
		1295	#define PAGE_ACCESS_TIME_BUCKETS \
		1296	(PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT)
		1297	#else
		1298	#define PAGE_ACCESS_TIME_BUCKETS 0
		1299	#endif
		1300
		1301	#define PAGE_ACCESS_TIME_MASK \
		1302	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
		1303
1258	static inline int cpu_pid_to_cpupid(int cpu, int pid)	1304	static inline int cpu_pid_to_cpupid(int cpu, int pid)
1259	{	1305	{
1260	return ((cpu & LAST__CPU_MASK) << LAST__PID_SHIFT) \| (pid & LAST__PID_MASK);	1306	return ((cpu & LAST__CPU_MASK) << LAST__PID_SHIFT) \| (pid & LAST__PID_MASK);
@@ -1318,12 +1364,25 @@ static inline void page_cpupid_reset_last(struct page *page) · 57 unmodified lines
1318	page->flags \|= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT;	1364	page->flags \|= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT;
1319	}	1365	}
1320	#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */	1366	#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
		1367
		1368	static inline int xchg_page_access_time(struct page *page, int time)
		1369	{
		1370	int last_time;
		1371
		1372	last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS);
		1373	return last_time << PAGE_ACCESS_TIME_BUCKETS;
		1374	}
1321	#else /* !CONFIG_NUMA_BALANCING */	1375	#else /* !CONFIG_NUMA_BALANCING */
1322	static inline int page_cpupid_xchg_last(struct page *page, int cpupid)	1376	static inline int page_cpupid_xchg_last(struct page *page, int cpupid)
1323	{	1377	{
1324	return page_to_nid(page); /* XXX */	1378	return page_to_nid(page); /* XXX */
1325	}	1379	}
1326		1380
		1381	static inline int xchg_page_access_time(struct page *page, int time)
		1382	{
		1383	return 0;
		1384	}
		1385
1327	static inline int page_cpupid_last(struct page *page)	1386	static inline int page_cpupid_last(struct page *page)
1328	{	1387	{
1329	return page_to_nid(page); /* XXX */	1388	return page_to_nid(page); /* XXX */
@@ -1465,6 +1524,11 @@ static inline unsigned long folio_pfn(struct folio *folio) · 135 unmodified lines
1465	return page_to_pfn(&folio->page);	1524	return page_to_pfn(&folio->page);
1466	}	1525	}
1467		1526
		1527	static inline struct folio *pfn_folio(unsigned long pfn)
		1528	{
		1529	return page_folio(pfn_to_page(pfn));
		1530	}
		1531
1468	static inline atomic_t folio_pincount_ptr(struct folio folio)	1532	static inline atomic_t folio_pincount_ptr(struct folio folio)
1469	{	1533	{
1470	return &folio_page(folio, 1)->compound_pincount;	1534	return &folio_page(folio, 1)->compound_pincount;
@@ -1597,7 +1661,13 @@ static inline void set_page_links(struct page *page, enum zone_type zone, · 126 unmodified lines
1597	*/	1661	*/
1598	static inline long folio_nr_pages(struct folio *folio)	1662	static inline long folio_nr_pages(struct folio *folio)
1599	{	1663	{
1600	return compound_nr(&folio->page);	1664	if (!folio_test_large(folio))
		1665	return 1;
		1666	#ifdef CONFIG_64BIT
		1667	return folio->_folio_nr_pages;
		1668	#else
		1669	return 1L << folio->_folio_order;
		1670	#endif
1601	}	1671	}
1602		1672
1603	/**	1673	/**
@@ -1776,7 +1846,11 @@ extern void pagefault_out_of_memory(void); · 172 unmodified lines
1776	*/	1846	*/
1777	#define SHOW_MEM_FILTER_NODES (0x0001u) /* disallowed nodes */	1847	#define SHOW_MEM_FILTER_NODES (0x0001u) /* disallowed nodes */
1778		1848
1779	extern void show_free_areas(unsigned int flags, nodemask_t *nodemask);	1849	extern void __show_free_areas(unsigned int flags, nodemask_t *nodemask, int max_zone_idx);
		1850	static void __maybe_unused show_free_areas(unsigned int flags, nodemask_t *nodemask)
		1851	{
		1852	__show_free_areas(flags, nodemask, MAX_NR_ZONES - 1);
		1853	}
1780		1854
1781	#ifdef CONFIG_MMU	1855	#ifdef CONFIG_MMU
1782	extern bool can_do_mlock(void);	1856	extern bool can_do_mlock(void);
@@ -1795,8 +1869,9 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, · 12 unmodified lines
1795	unsigned long size);	1869	unsigned long size);
1796	void zap_page_range(struct vm_area_struct *vma, unsigned long address,	1870	void zap_page_range(struct vm_area_struct *vma, unsigned long address,
1797	unsigned long size);	1871	unsigned long size);
1798	void unmap_vmas(struct mmu_gather tlb, struct vm_area_struct start_vma,	1872	void unmap_vmas(struct mmu_gather tlb, struct maple_tree mt,
1799	unsigned long start, unsigned long end);	1873	struct vm_area_struct *start_vma, unsigned long start,
		1874	unsigned long end);
1800		1875
1801	struct mmu_notifier_range;	1876	struct mmu_notifier_range;
1802		1877
@@ -2495,7 +2570,6 @@ extern unsigned long absent_pages_in_range(unsigned long start_pfn, · 692 unmodified lines
2495	unsigned long end_pfn);	2570	unsigned long end_pfn);
2496	extern void get_pfn_range_for_nid(unsigned int nid,	2571	extern void get_pfn_range_for_nid(unsigned int nid,
2497	unsigned long start_pfn, unsigned long end_pfn);	2572	unsigned long start_pfn, unsigned long end_pfn);
2498	extern unsigned long find_min_pfn_with_active_regions(void);
2499		2573
2500	#ifndef CONFIG_NUMA	2574	#ifndef CONFIG_NUMA
2501	static inline int early_pfn_to_nid(unsigned long pfn)	2575	static inline int early_pfn_to_nid(unsigned long pfn)
@@ -2516,7 +2590,12 @@ extern void calculate_min_free_kbytes(void); · 14 unmodified lines
2516	extern int __meminit init_per_zone_wmark_min(void);	2590	extern int __meminit init_per_zone_wmark_min(void);
2517	extern void mem_init(void);	2591	extern void mem_init(void);
2518	extern void __init mmap_init(void);	2592	extern void __init mmap_init(void);
2519	extern void show_mem(unsigned int flags, nodemask_t *nodemask);	2593
		2594	extern void __show_mem(unsigned int flags, nodemask_t *nodemask, int max_zone_idx);
		2595	static inline void show_mem(unsigned int flags, nodemask_t *nodemask)
		2596	{
		2597	__show_mem(flags, nodemask, MAX_NR_ZONES - 1);
		2598	}
2520	extern long si_mem_available(void);	2599	extern long si_mem_available(void);
2521	extern void si_meminfo(struct sysinfo * val);	2600	extern void si_meminfo(struct sysinfo * val);
2522	extern void si_meminfo_node(struct sysinfo *val, int nid);	2601	extern void si_meminfo_node(struct sysinfo *val, int nid);
@@ -2593,14 +2672,15 @@ extern int __split_vma(struct mm_struct , struct vm_area_struct , · 70 unmodified lines
2593	extern int split_vma(struct mm_struct , struct vm_area_struct ,	2672	extern int split_vma(struct mm_struct , struct vm_area_struct ,
2594	unsigned long addr, int new_below);	2673	unsigned long addr, int new_below);
2595	extern int insert_vm_struct(struct mm_struct , struct vm_area_struct );	2674	extern int insert_vm_struct(struct mm_struct , struct vm_area_struct );
2596	extern void __vma_link_rb(struct mm_struct , struct vm_area_struct ,
2597	struct rb_node *, struct rb_node );
2598	extern void unlink_file_vma(struct vm_area_struct *);	2675	extern void unlink_file_vma(struct vm_area_struct *);
2599	extern struct vm_area_struct copy_vma(struct vm_area_struct *,	2676	extern struct vm_area_struct copy_vma(struct vm_area_struct *,
2600	unsigned long addr, unsigned long len, pgoff_t pgoff,	2677	unsigned long addr, unsigned long len, pgoff_t pgoff,
2601	bool *need_rmap_locks);	2678	bool *need_rmap_locks);
2602	extern void exit_mmap(struct mm_struct *);	2679	extern void exit_mmap(struct mm_struct *);
2603		2680
		2681	void vma_mas_store(struct vm_area_struct vma, struct ma_state mas);
		2682	void vma_mas_remove(struct vm_area_struct vma, struct ma_state mas);
		2683
2604	static inline int check_data_rlimit(unsigned long rlim,	2684	static inline int check_data_rlimit(unsigned long rlim,
2605	unsigned long new,	2685	unsigned long new,
2606	unsigned long start,	2686	unsigned long start,
@@ -2648,8 +2728,9 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, · 41 unmodified lines
2648	extern unsigned long do_mmap(struct file *file, unsigned long addr,	2728	extern unsigned long do_mmap(struct file *file, unsigned long addr,
2649	unsigned long len, unsigned long prot, unsigned long flags,	2729	unsigned long len, unsigned long prot, unsigned long flags,
2650	unsigned long pgoff, unsigned long populate, struct list_head uf);	2730	unsigned long pgoff, unsigned long populate, struct list_head uf);
2651	extern int __do_munmap(struct mm_struct *, unsigned long, size_t,	2731	extern int do_mas_munmap(struct ma_state mas, struct mm_struct mm,
2652	struct list_head *uf, bool downgrade);	2732	unsigned long start, size_t len, struct list_head *uf,
		2733	bool downgrade);
2653	extern int do_munmap(struct mm_struct *, unsigned long, size_t,	2734	extern int do_munmap(struct mm_struct *, unsigned long, size_t,
2654	struct list_head *uf);	2735	struct list_head *uf);
2655	extern int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior);	2736	extern int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior);
@@ -2716,26 +2797,12 @@ extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long add · 60 unmodified lines
2716	extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr,	2797	extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr,
2717	struct vm_area_struct **pprev);	2798	struct vm_area_struct **pprev);
2718		2799
2719	/**	2800	/*
2720	* find_vma_intersection() - Look up the first VMA which intersects the interval	2801	* Look up the first VMA which intersects the interval [start_addr, end_addr)
2721	* @mm: The process address space.	2802	* NULL if none. Assume start_addr < end_addr.
2722	* @start_addr: The inclusive start user address.
2723	* @end_addr: The exclusive end user address.
2724	*
2725	* Returns: The first VMA within the provided range, %NULL otherwise. Assumes
2726	* start_addr < end_addr.
2727	*/	2803	*/
2728	static inline
2729	struct vm_area_struct find_vma_intersection(struct mm_struct mm,	2804	struct vm_area_struct find_vma_intersection(struct mm_struct mm,
2730	unsigned long start_addr,	2805	unsigned long start_addr, unsigned long end_addr);
2731	unsigned long end_addr)
2732	{
2733	struct vm_area_struct *vma = find_vma(mm, start_addr);
2734
2735	if (vma && end_addr <= vma->vm_start)
2736	vma = NULL;
2737	return vma;
2738	}
2739		2806
2740	/**	2807	/**
2741	* vma_lookup() - Find a VMA at a specific address	2808	* vma_lookup() - Find a VMA at a specific address
@@ -2747,12 +2814,7 @@ struct vm_area_struct find_vma_intersection(struct mm_struct mm, · 5 unmodified lines
2747	static inline	2814	static inline
2748	struct vm_area_struct vma_lookup(struct mm_struct mm, unsigned long addr)	2815	struct vm_area_struct vma_lookup(struct mm_struct mm, unsigned long addr)
2749	{	2816	{
2750	struct vm_area_struct *vma = find_vma(mm, addr);	2817	return mtree_load(&mm->mm_mt, addr);
2751
2752	if (vma && addr < vma->vm_start)
2753	vma = NULL;
2754
2755	return vma;
2756	}	2818	}
2757		2819
2758	static inline unsigned long vm_start_gap(struct vm_area_struct *vma)	2820	static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
@@ -2788,7 +2850,7 @@ static inline unsigned long vma_pages(struct vm_area_struct *vma) · 29 unmodified lines
2788	static inline struct vm_area_struct find_exact_vma(struct mm_struct mm,	2850	static inline struct vm_area_struct find_exact_vma(struct mm_struct mm,
2789	unsigned long vm_start, unsigned long vm_end)	2851	unsigned long vm_start, unsigned long vm_end)
2790	{	2852	{
2791	struct vm_area_struct *vma = find_vma(mm, vm_start);	2853	struct vm_area_struct *vma = vma_lookup(mm, vm_start);
2792		2854
2793	if (vma && (vma->vm_start != vm_start \|\| vma->vm_end != vm_end))	2855	if (vma && (vma->vm_start != vm_start \|\| vma->vm_end != vm_end))
2794	vma = NULL;	2856	vma = NULL;
@@ -2888,7 +2950,6 @@ struct page follow_page(struct vm_area_struct vma, unsigned long address, · 93 unmodified lines
2888	* and return without waiting upon it */	2950	* and return without waiting upon it */
2889	#define FOLL_NOFAULT 0x80 /* do not fault in pages */	2951	#define FOLL_NOFAULT 0x80 /* do not fault in pages */
2890	#define FOLL_HWPOISON 0x100 /* check page is hwpoisoned */	2952	#define FOLL_HWPOISON 0x100 /* check page is hwpoisoned */
2891	#define FOLL_NUMA 0x200 /* force NUMA hinting page fault */
2892	#define FOLL_MIGRATION 0x400 /* wait for page to replace migration entry */	2953	#define FOLL_MIGRATION 0x400 /* wait for page to replace migration entry */
2893	#define FOLL_TRIED 0x800 /* a retry, previous pass started an IO */	2954	#define FOLL_TRIED 0x800 /* a retry, previous pass started an IO */
2894	#define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */	2955	#define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */
@@ -2975,8 +3036,8 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) · 80 unmodified lines
2975	* PageAnonExclusive() has to protect against concurrent GUP:	3036	* PageAnonExclusive() has to protect against concurrent GUP:
2976	* * Ordinary GUP: Using the PT lock	3037	* * Ordinary GUP: Using the PT lock
2977	* * GUP-fast and fork(): mm->write_protect_seq	3038	* * GUP-fast and fork(): mm->write_protect_seq
2978	* * GUP-fast and KSM or temporary unmapping (swap, migration):	3039	* * GUP-fast and KSM or temporary unmapping (swap, migration): see
2979	* clear/invalidate+flush of the page table entry	3040	* page_try_share_anon_rmap()
2980	*	3041	*
2981	* Must be called with the (sub)page that's actually referenced via the	3042	* Must be called with the (sub)page that's actually referenced via the
2982	* page table entry, which might not necessarily be the head page for a	3043	* page table entry, which might not necessarily be the head page for a
@@ -2997,6 +3058,11 @@ static inline bool gup_must_unshare(unsigned int flags, struct page *page) · 14 unmodified lines
2997	*/	3058	*/
2998	if (!PageAnon(page))	3059	if (!PageAnon(page))
2999	return false;	3060	return false;
		3061
		3062	/* Paired with a memory barrier in page_try_share_anon_rmap(). */
		3063	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
		3064	smp_rmb();
		3065
3000	/*	3066	/*
3001	* Note that PageKsm() pages cannot be exclusive, and consequently,	3067	* Note that PageKsm() pages cannot be exclusive, and consequently,
3002	* cannot get pinned.	3068	* cannot get pinned.
@@ -3004,6 +3070,21 @@ static inline bool gup_must_unshare(unsigned int flags, struct page *page) · 1 unmodified line
3004	return !PageAnonExclusive(page);	3070	return !PageAnonExclusive(page);
3005	}	3071	}
3006		3072
		3073	/*
		3074	* Indicates whether GUP can follow a PROT_NONE mapped page, or whether
		3075	* a (NUMA hinting) fault is required.
		3076	*/
		3077	static inline bool gup_can_follow_protnone(unsigned int flags)
		3078	{
		3079	/*
		3080	* FOLL_FORCE has to be able to make progress even if the VMA is
		3081	* inaccessible. Further, FOLL_FORCE access usually does not represent
		3082	* application behaviour and we should avoid triggering NUMA hinting
		3083	* faults.
		3084	*/
		3085	return flags & FOLL_FORCE;
		3086	}
		3087
3007	typedef int (pte_fn_t)(pte_t pte, unsigned long addr, void *data);	3088	typedef int (pte_fn_t)(pte_t pte, unsigned long addr, void *data);
3008	extern int apply_to_page_range(struct mm_struct *mm, unsigned long address,	3089	extern int apply_to_page_range(struct mm_struct *mm, unsigned long address,
3009	unsigned long size, pte_fn_t fn, void *data);	3090	unsigned long size, pte_fn_t fn, void *data);
@@ -3011,7 +3092,7 @@ extern int apply_to_existing_page_range(struct mm_struct *mm, · 1 unmodified line
3011	unsigned long address, unsigned long size,	3092	unsigned long address, unsigned long size,
3012	pte_fn_t fn, void *data);	3093	pte_fn_t fn, void *data);
3013		3094
3014	extern void init_mem_debugging_and_hardening(void);	3095	extern void __init init_mem_debugging_and_hardening(void);
3015	#ifdef CONFIG_PAGE_POISONING	3096	#ifdef CONFIG_PAGE_POISONING
3016	extern void __kernel_poison_pages(struct page *page, int numpages);	3097	extern void __kernel_poison_pages(struct page *page, int numpages);
3017	extern void __kernel_unpoison_pages(struct page *page, int numpages);	3098	extern void __kernel_unpoison_pages(struct page *page, int numpages);

include/linux/mm_inline.h+230−12

@@ -34,15 +34,25 @@ static inline int page_is_file_lru(struct page *page) · 33 unmodified lines
34	return folio_is_file_lru(page_folio(page));	34	return folio_is_file_lru(page_folio(page));
35	}	35	}
36		36
37	static __always_inline void update_lru_size(struct lruvec *lruvec,	37	static __always_inline void __update_lru_size(struct lruvec *lruvec,
38	enum lru_list lru, enum zone_type zid,	38	enum lru_list lru, enum zone_type zid,
39	long nr_pages)	39	long nr_pages)
40	{	40	{
41	struct pglist_data *pgdat = lruvec_pgdat(lruvec);	41	struct pglist_data *pgdat = lruvec_pgdat(lruvec);
42		42
		43	lockdep_assert_held(&lruvec->lru_lock);
		44	WARN_ON_ONCE(nr_pages != (int)nr_pages);
		45
43	__mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);	46	__mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
44	__mod_zone_page_state(&pgdat->node_zones[zid],	47	__mod_zone_page_state(&pgdat->node_zones[zid],
45	NR_ZONE_LRU_BASE + lru, nr_pages);	48	NR_ZONE_LRU_BASE + lru, nr_pages);
		49	}
		50
		51	static __always_inline void update_lru_size(struct lruvec *lruvec,
		52	enum lru_list lru, enum zone_type zid,
		53	long nr_pages)
		54	{
		55	__update_lru_size(lruvec, lru, zid, nr_pages);
46	#ifdef CONFIG_MEMCG	56	#ifdef CONFIG_MEMCG
47	mem_cgroup_update_lru_size(lruvec, lru, zid, nr_pages);	57	mem_cgroup_update_lru_size(lruvec, lru, zid, nr_pages);
48	#endif	58	#endif
@@ -66,11 +76,6 @@ static __always_inline void __folio_clear_lru_flags(struct folio *folio) · 17 unmodified lines
66	__folio_clear_unevictable(folio);	76	__folio_clear_unevictable(folio);
67	}	77	}
68		78
69	static __always_inline void __clear_page_lru_flags(struct page *page)
70	{
71	__folio_clear_lru_flags(page_folio(page));
72	}
73
74	/**	79	/**
75	* folio_lru_list - Which LRU list should a folio be on?	80	* folio_lru_list - Which LRU list should a folio be on?
76	* @folio: The folio to test.	81	* @folio: The folio to test.
@@ -94,11 +99,224 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio) · 17 unmodified lines
94	return lru;	99	return lru;
95	}	100	}
96		101
		102	#ifdef CONFIG_LRU_GEN
		103
		104	#ifdef CONFIG_LRU_GEN_ENABLED
		105	static inline bool lru_gen_enabled(void)
		106	{
		107	DECLARE_STATIC_KEY_TRUE(lru_gen_caps[NR_LRU_GEN_CAPS]);
		108
		109	return static_branch_likely(&lru_gen_caps[LRU_GEN_CORE]);
		110	}
		111	#else
		112	static inline bool lru_gen_enabled(void)
		113	{
		114	DECLARE_STATIC_KEY_FALSE(lru_gen_caps[NR_LRU_GEN_CAPS]);
		115
		116	return static_branch_unlikely(&lru_gen_caps[LRU_GEN_CORE]);
		117	}
		118	#endif
		119
		120	static inline bool lru_gen_in_fault(void)
		121	{
		122	return current->in_lru_fault;
		123	}
		124
		125	static inline int lru_gen_from_seq(unsigned long seq)
		126	{
		127	return seq % MAX_NR_GENS;
		128	}
		129
		130	static inline int lru_hist_from_seq(unsigned long seq)
		131	{
		132	return seq % NR_HIST_GENS;
		133	}
		134
		135	static inline int lru_tier_from_refs(int refs)
		136	{
		137	VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH));
		138
		139	/* see the comment in folio_lru_refs() */
		140	return order_base_2(refs + 1);
		141	}
		142
		143	static inline int folio_lru_refs(struct folio *folio)
		144	{
		145	unsigned long flags = READ_ONCE(folio->flags);
		146	bool workingset = flags & BIT(PG_workingset);
		147
		148	/*
		149	* Return the number of accesses beyond PG_referenced, i.e., N-1 if the
		150	* total number of accesses is N>1, since N=0,1 both map to the first
		151	* tier. lru_tier_from_refs() will account for this off-by-one. Also see
		152	* the comment on MAX_NR_TIERS.
		153	*/
		154	return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset;
		155	}
		156
		157	static inline int folio_lru_gen(struct folio *folio)
		158	{
		159	unsigned long flags = READ_ONCE(folio->flags);
		160
		161	return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
		162	}
		163
		164	static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen)
		165	{
		166	unsigned long max_seq = lruvec->lrugen.max_seq;
		167
		168	VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
		169
		170	/* see the comment on MIN_NR_GENS */
		171	return gen == lru_gen_from_seq(max_seq) \|\| gen == lru_gen_from_seq(max_seq - 1);
		172	}
		173
		174	static inline void lru_gen_update_size(struct lruvec lruvec, struct folio folio,
		175	int old_gen, int new_gen)
		176	{
		177	int type = folio_is_file_lru(folio);
		178	int zone = folio_zonenum(folio);
		179	int delta = folio_nr_pages(folio);
		180	enum lru_list lru = type * LRU_INACTIVE_FILE;
		181	struct lru_gen_struct *lrugen = &lruvec->lrugen;
		182
		183	VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
		184	VM_WARN_ON_ONCE(new_gen != -1 && new_gen >= MAX_NR_GENS);
		185	VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1);
		186
		187	if (old_gen >= 0)
		188	WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
		189	lrugen->nr_pages[old_gen][type][zone] - delta);
		190	if (new_gen >= 0)
		191	WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone],
		192	lrugen->nr_pages[new_gen][type][zone] + delta);
		193
		194	/* addition */
		195	if (old_gen < 0) {
		196	if (lru_gen_is_active(lruvec, new_gen))
		197	lru += LRU_ACTIVE;
		198	__update_lru_size(lruvec, lru, zone, delta);
		199	return;
		200	}
		201
		202	/* deletion */
		203	if (new_gen < 0) {
		204	if (lru_gen_is_active(lruvec, old_gen))
		205	lru += LRU_ACTIVE;
		206	__update_lru_size(lruvec, lru, zone, -delta);
		207	return;
		208	}
		209
		210	/* promotion */
		211	if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
		212	__update_lru_size(lruvec, lru, zone, -delta);
		213	__update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
		214	}
		215
		216	/* demotion requires isolation, e.g., lru_deactivate_fn() */
		217	VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
		218	}
		219
		220	static inline bool lru_gen_add_folio(struct lruvec lruvec, struct folio folio, bool reclaiming)
		221	{
		222	unsigned long seq;
		223	unsigned long flags;
		224	int gen = folio_lru_gen(folio);
		225	int type = folio_is_file_lru(folio);
		226	int zone = folio_zonenum(folio);
		227	struct lru_gen_struct *lrugen = &lruvec->lrugen;
		228
		229	VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
		230
		231	if (folio_test_unevictable(folio) \|\| !lrugen->enabled)
		232	return false;
		233	/*
		234	* There are three common cases for this page:
		235	* 1. If it's hot, e.g., freshly faulted in or previously hot and
		236	* migrated, add it to the youngest generation.
		237	* 2. If it's cold but can't be evicted immediately, i.e., an anon page
		238	* not in swapcache or a dirty page pending writeback, add it to the
		239	* second oldest generation.
		240	* 3. Everything else (clean, cold) is added to the oldest generation.
		241	*/
		242	if (folio_test_active(folio))
		243	seq = lrugen->max_seq;
		244	else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) \|\|
		245	(folio_test_reclaim(folio) &&
		246	(folio_test_dirty(folio) \|\| folio_test_writeback(folio))))
		247	seq = lrugen->min_seq[type] + 1;
		248	else
		249	seq = lrugen->min_seq[type];
		250
		251	gen = lru_gen_from_seq(seq);
		252	flags = (gen + 1UL) << LRU_GEN_PGOFF;
		253	/* see the comment on MIN_NR_GENS about PG_active */
		254	set_mask_bits(&folio->flags, LRU_GEN_MASK \| BIT(PG_active), flags);
		255
		256	lru_gen_update_size(lruvec, folio, -1, gen);
		257	/* for folio_rotate_reclaimable() */
		258	if (reclaiming)
		259	list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]);
		260	else
		261	list_add(&folio->lru, &lrugen->lists[gen][type][zone]);
		262
		263	return true;
		264	}
		265
		266	static inline bool lru_gen_del_folio(struct lruvec lruvec, struct folio folio, bool reclaiming)
		267	{
		268	unsigned long flags;
		269	int gen = folio_lru_gen(folio);
		270
		271	if (gen < 0)
		272	return false;
		273
		274	VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
		275	VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
		276
		277	/* for folio_migrate_flags() */
		278	flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
		279	flags = set_mask_bits(&folio->flags, LRU_GEN_MASK, flags);
		280	gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
		281
		282	lru_gen_update_size(lruvec, folio, gen, -1);
		283	list_del(&folio->lru);
		284
		285	return true;
		286	}
		287
		288	#else /* !CONFIG_LRU_GEN */
		289
		290	static inline bool lru_gen_enabled(void)
		291	{
		292	return false;
		293	}
		294
		295	static inline bool lru_gen_in_fault(void)
		296	{
		297	return false;
		298	}
		299
		300	static inline bool lru_gen_add_folio(struct lruvec lruvec, struct folio folio, bool reclaiming)
		301	{
		302	return false;
		303	}
		304
		305	static inline bool lru_gen_del_folio(struct lruvec lruvec, struct folio folio, bool reclaiming)
		306	{
		307	return false;
		308	}
		309
		310	#endif /* CONFIG_LRU_GEN */
		311
97	static __always_inline	312	static __always_inline
98	void lruvec_add_folio(struct lruvec lruvec, struct folio folio)	313	void lruvec_add_folio(struct lruvec lruvec, struct folio folio)
99	{	314	{
100	enum lru_list lru = folio_lru_list(folio);	315	enum lru_list lru = folio_lru_list(folio);
101		316
		317	if (lru_gen_add_folio(lruvec, folio, false))
		318	return;
		319
102	update_lru_size(lruvec, lru, folio_zonenum(folio),	320	update_lru_size(lruvec, lru, folio_zonenum(folio),
103	folio_nr_pages(folio));	321	folio_nr_pages(folio));
104	if (lru != LRU_UNEVICTABLE)	322	if (lru != LRU_UNEVICTABLE)
@@ -116,23 +334,23 @@ void lruvec_add_folio_tail(struct lruvec lruvec, struct folio folio) · 11 unmodified lines
116	{	334	{
117	enum lru_list lru = folio_lru_list(folio);	335	enum lru_list lru = folio_lru_list(folio);
118		336
		337	if (lru_gen_add_folio(lruvec, folio, true))
		338	return;
		339
119	update_lru_size(lruvec, lru, folio_zonenum(folio),	340	update_lru_size(lruvec, lru, folio_zonenum(folio),
120	folio_nr_pages(folio));	341	folio_nr_pages(folio));
121	/* This is not expected to be used on LRU_UNEVICTABLE */	342	/* This is not expected to be used on LRU_UNEVICTABLE */
122	list_add_tail(&folio->lru, &lruvec->lists[lru]);	343	list_add_tail(&folio->lru, &lruvec->lists[lru]);
123	}	344	}
124		345
125	static __always_inline void add_page_to_lru_list_tail(struct page *page,
126	struct lruvec *lruvec)
127	{
128	lruvec_add_folio_tail(lruvec, page_folio(page));
129	}
130
131	static __always_inline	346	static __always_inline
132	void lruvec_del_folio(struct lruvec lruvec, struct folio folio)	347	void lruvec_del_folio(struct lruvec lruvec, struct folio folio)
133	{	348	{
134	enum lru_list lru = folio_lru_list(folio);	349	enum lru_list lru = folio_lru_list(folio);
135		350
		351	if (lru_gen_del_folio(lruvec, folio, false))
		352	return;
		353
136	if (lru != LRU_UNEVICTABLE)	354	if (lru != LRU_UNEVICTABLE)
137	list_del(&folio->lru);	355	list_del(&folio->lru);
138	update_lru_size(lruvec, lru, folio_zonenum(folio),	356	update_lru_size(lruvec, lru, folio_zonenum(folio),

include/linux/mm_types.h+152−26

@@ -9,6 +9,7 @@ · 8 unmodified lines
9	#include <linux/list.h>	9	#include <linux/list.h>
10	#include <linux/spinlock.h>	10	#include <linux/spinlock.h>
11	#include <linux/rbtree.h>	11	#include <linux/rbtree.h>
		12	#include <linux/maple_tree.h>
12	#include <linux/rwsem.h>	13	#include <linux/rwsem.h>
13	#include <linux/completion.h>	14	#include <linux/completion.h>
14	#include <linux/cpumask.h>	15	#include <linux/cpumask.h>
@@ -223,6 +224,18 @@ struct page { · 208 unmodified lines
223	not kmapped, ie. highmem) */	224	not kmapped, ie. highmem) */
224	#endif /* WANT_PAGE_VIRTUAL */	225	#endif /* WANT_PAGE_VIRTUAL */
225		226
		227	#ifdef CONFIG_KMSAN
		228	/*
		229	* KMSAN metadata for this page:
		230	* - shadow page: every bit indicates whether the corresponding
		231	* bit of the original page is initialized (0) or not (1);
		232	* - origin page: every 4 bytes contain an id of the stack trace
		233	* where the uninitialized value was created.
		234	*/
		235	struct page *kmsan_shadow;
		236	struct page *kmsan_origin;
		237	#endif
		238
226	#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS	239	#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
227	int _last_cpupid;	240	int _last_cpupid;
228	#endif	241	#endif
@@ -244,6 +257,13 @@ struct page { · 15 unmodified lines
244	* @_refcount: Do not access this member directly. Use folio_ref_count()	257	* @_refcount: Do not access this member directly. Use folio_ref_count()
245	* to find how many references there are to this folio.	258	* to find how many references there are to this folio.
246	* @memcg_data: Memory Control Group data.	259	* @memcg_data: Memory Control Group data.
		260	* @_flags_1: For large folios, additional page flags.
		261	* @__head: Points to the folio. Do not use.
		262	* @_folio_dtor: Which destructor to use for this folio.
		263	* @_folio_order: Do not use directly, call folio_order().
		264	* @_total_mapcount: Do not use directly, call folio_entire_mapcount().
		265	* @_pincount: Do not use directly, call folio_maybe_dma_pinned().
		266	* @_folio_nr_pages: Do not use directly, call folio_nr_pages().
247	*	267	*
248	* A folio is a physically, virtually and logically contiguous set	268	* A folio is a physically, virtually and logically contiguous set
249	* of bytes. It is a power-of-two in size, and it is aligned to that	269	* of bytes. It is a power-of-two in size, and it is aligned to that
@@ -282,9 +302,17 @@ struct folio { · 32 unmodified lines
282	};	302	};
283	struct page page;	303	struct page page;
284	};	304	};
		305	unsigned long _flags_1;
		306	unsigned long __head;
		307	unsigned char _folio_dtor;
		308	unsigned char _folio_order;
		309	atomic_t _total_mapcount;
		310	atomic_t _pincount;
		311	#ifdef CONFIG_64BIT
		312	unsigned int _folio_nr_pages;
		313	#endif
285	};	314	};
286		315
287	static_assert(sizeof(struct page) == sizeof(struct folio));
288	#define FOLIO_MATCH(pg, fl) \	316	#define FOLIO_MATCH(pg, fl) \
289	static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl))	317	static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl))
290	FOLIO_MATCH(flags, flags);	318	FOLIO_MATCH(flags, flags);
@@ -299,6 +327,19 @@ FOLIO_MATCH(_refcount, _refcount); · 8 unmodified lines
299	FOLIO_MATCH(memcg_data, memcg_data);	327	FOLIO_MATCH(memcg_data, memcg_data);
300	#endif	328	#endif
301	#undef FOLIO_MATCH	329	#undef FOLIO_MATCH
		330	#define FOLIO_MATCH(pg, fl) \
		331	static_assert(offsetof(struct folio, fl) == \
		332	offsetof(struct page, pg) + sizeof(struct page))
		333	FOLIO_MATCH(flags, _flags_1);
		334	FOLIO_MATCH(compound_head, __head);
		335	FOLIO_MATCH(compound_dtor, _folio_dtor);
		336	FOLIO_MATCH(compound_order, _folio_order);
		337	FOLIO_MATCH(compound_mapcount, _total_mapcount);
		338	FOLIO_MATCH(compound_pincount, _pincount);
		339	#ifdef CONFIG_64BIT
		340	FOLIO_MATCH(compound_nr, _folio_nr_pages);
		341	#endif
		342	#undef FOLIO_MATCH
302		343
303	static inline atomic_t folio_mapcount_ptr(struct folio folio)	344	static inline atomic_t folio_mapcount_ptr(struct folio folio)
304	{	345	{
@@ -407,21 +448,6 @@ struct vm_area_struct { · 102 unmodified lines
407	unsigned long vm_end; /* The first byte after our end address	448	unsigned long vm_end; /* The first byte after our end address
408	within vm_mm. */	449	within vm_mm. */
409		450
410	/* linked list of VM areas per task, sorted by address */
411	struct vm_area_struct vm_next, vm_prev;
412
413	struct rb_node vm_rb;
414
415	/*
416	* Largest free memory gap in bytes to the left of this VMA.
417	* Either between this VMA and vma->vm_prev, or between one of the
418	* VMAs below us in the VMA rbtree and its ->vm_prev. This helps
419	* get_unmapped_area find a free area of the right size.
420	*/
421	unsigned long rb_subtree_gap;
422
423	/* Second cache line starts here. */
424
425	struct mm_struct vm_mm; / The address space we belong to. */	451	struct mm_struct vm_mm; / The address space we belong to. */
426		452
427	/*	453	/*
@@ -485,9 +511,7 @@ struct vm_area_struct { · 57 unmodified lines
485	struct kioctx_table;	511	struct kioctx_table;
486	struct mm_struct {	512	struct mm_struct {
487	struct {	513	struct {
488	struct vm_area_struct mmap; / list of VMAs */	514	struct maple_tree mm_mt;
489	struct rb_root mm_rb;
490	u64 vmacache_seqnum; /* per-thread vmacache */
491	#ifdef CONFIG_MMU	515	#ifdef CONFIG_MMU
492	unsigned long (get_unmapped_area) (struct file filp,	516	unsigned long (get_unmapped_area) (struct file filp,
493	unsigned long addr, unsigned long len,	517	unsigned long addr, unsigned long len,
@@ -501,7 +525,6 @@ struct mm_struct { · 7 unmodified lines
501	unsigned long mmap_compat_legacy_base;	525	unsigned long mmap_compat_legacy_base;
502	#endif	526	#endif
503	unsigned long task_size; /* size of task vm space */	527	unsigned long task_size; /* size of task vm space */
504	unsigned long highest_vm_end; /* highest vma end address */
505	pgd_t * pgd;	528	pgd_t * pgd;
506		529
507	#ifdef CONFIG_MEMBARRIER	530	#ifdef CONFIG_MEMBARRIER
@@ -631,22 +654,22 @@ struct mm_struct { · 123 unmodified lines
631	#endif	654	#endif
632	#ifdef CONFIG_NUMA_BALANCING	655	#ifdef CONFIG_NUMA_BALANCING
633	/*	656	/*
634	* numa_next_scan is the next time that the PTEs will be marked	657	* numa_next_scan is the next time that PTEs will be remapped
635	* pte_numa. NUMA hinting faults will gather statistics and	658	* PROT_NONE to trigger NUMA hinting faults; such faults gather
636	* migrate pages to new nodes if necessary.	659	* statistics and migrate pages to new nodes if necessary.
637	*/	660	*/
638	unsigned long numa_next_scan;	661	unsigned long numa_next_scan;
639		662
640	/* Restart point for scanning and setting pte_numa */	663	/* Restart point for scanning and remapping PTEs. */
641	unsigned long numa_scan_offset;	664	unsigned long numa_scan_offset;
642		665
643	/* numa_scan_seq prevents two threads setting pte_numa */	666	/* numa_scan_seq prevents two threads remapping PTEs. */
644	int numa_scan_seq;	667	int numa_scan_seq;
645	#endif	668	#endif
646	/*	669	/*
647	* An operation with batched TLB flushing is going on. Anything	670	* An operation with batched TLB flushing is going on. Anything
648	* that can move process memory needs to flush the TLB when	671	* that can move process memory needs to flush the TLB when
649	* moving a PROT_NONE or PROT_NUMA mapped page.	672	* moving a PROT_NONE mapped page.
650	*/	673	*/
651	atomic_t tlb_flush_pending;	674	atomic_t tlb_flush_pending;
652	#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH	675	#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
@@ -671,7 +694,28 @@ struct mm_struct { · 18 unmodified lines
671	* merging.	694	* merging.
672	*/	695	*/
673	unsigned long ksm_merging_pages;	696	unsigned long ksm_merging_pages;
		697	/*
		698	* Represent how many pages are checked for ksm merging
		699	* including merged and not merged.
		700	*/
		701	unsigned long ksm_rmap_items;
		702	#endif
		703	#ifdef CONFIG_LRU_GEN
		704	struct {
		705	/* this mm_struct is on lru_gen_mm_list */
		706	struct list_head list;
		707	/*
		708	* Set when switching to this mm_struct, as a hint of
		709	* whether it has been used since the last time per-node
		710	* page table walkers cleared the corresponding bits.
		711	*/
		712	unsigned long bitmap;
		713	#ifdef CONFIG_MEMCG
		714	/* points to the memcg of "owner" above */
		715	struct mem_cgroup *memcg;
674	#endif	716	#endif
		717	} lru_gen;
		718	#endif /* CONFIG_LRU_GEN */
675	} __randomize_layout;	719	} __randomize_layout;
676		720
677	/*	721	/*
@@ -681,6 +725,7 @@ struct mm_struct { · 3 unmodified lines
681	unsigned long cpu_bitmap[];	725	unsigned long cpu_bitmap[];
682	};	726	};
683		727
		728	#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE \| MT_FLAGS_LOCK_EXTERN)
684	extern struct mm_struct init_mm;	729	extern struct mm_struct init_mm;
685		730
686	/* Pointer magic because the dynamic array size confuses some compilers. */	731	/* Pointer magic because the dynamic array size confuses some compilers. */
@@ -698,6 +743,87 @@ static inline cpumask_t mm_cpumask(struct mm_struct mm) · 11 unmodified lines
698	return (struct cpumask *)&mm->cpu_bitmap;	743	return (struct cpumask *)&mm->cpu_bitmap;
699	}	744	}
700		745
		746	#ifdef CONFIG_LRU_GEN
		747
		748	struct lru_gen_mm_list {
		749	/* mm_struct list for page table walkers */
		750	struct list_head fifo;
		751	/* protects the list above */
		752	spinlock_t lock;
		753	};
		754
		755	void lru_gen_add_mm(struct mm_struct *mm);
		756	void lru_gen_del_mm(struct mm_struct *mm);
		757	#ifdef CONFIG_MEMCG
		758	void lru_gen_migrate_mm(struct mm_struct *mm);
		759	#endif
		760
		761	static inline void lru_gen_init_mm(struct mm_struct *mm)
		762	{
		763	INIT_LIST_HEAD(&mm->lru_gen.list);
		764	mm->lru_gen.bitmap = 0;
		765	#ifdef CONFIG_MEMCG
		766	mm->lru_gen.memcg = NULL;
		767	#endif
		768	}
		769
		770	static inline void lru_gen_use_mm(struct mm_struct *mm)
		771	{
		772	/*
		773	* When the bitmap is set, page reclaim knows this mm_struct has been
		774	* used since the last time it cleared the bitmap. So it might be worth
		775	* walking the page tables of this mm_struct to clear the accessed bit.
		776	*/
		777	WRITE_ONCE(mm->lru_gen.bitmap, -1);
		778	}
		779
		780	#else /* !CONFIG_LRU_GEN */
		781
		782	static inline void lru_gen_add_mm(struct mm_struct *mm)
		783	{
		784	}
		785
		786	static inline void lru_gen_del_mm(struct mm_struct *mm)
		787	{
		788	}
		789
		790	#ifdef CONFIG_MEMCG
		791	static inline void lru_gen_migrate_mm(struct mm_struct *mm)
		792	{
		793	}
		794	#endif
		795
		796	static inline void lru_gen_init_mm(struct mm_struct *mm)
		797	{
		798	}
		799
		800	static inline void lru_gen_use_mm(struct mm_struct *mm)
		801	{
		802	}
		803
		804	#endif /* CONFIG_LRU_GEN */
		805
		806	struct vma_iterator {
		807	struct ma_state mas;
		808	};
		809
		810	#define VMA_ITERATOR(name, __mm, __addr) \
		811	struct vma_iterator name = { \
		812	.mas = { \
		813	.tree = &(__mm)->mm_mt, \
		814	.index = __addr, \
		815	.node = MAS_START, \
		816	}, \
		817	}
		818
		819	static inline void vma_iter_init(struct vma_iterator *vmi,
		820	struct mm_struct *mm, unsigned long addr)
		821	{
		822	vmi->mas.tree = &mm->mm_mt;
		823	vmi->mas.index = addr;
		824	vmi->mas.node = MAS_START;
		825	}
		826
701	struct mmu_gather;	827	struct mmu_gather;
702	extern void tlb_gather_mmu(struct mmu_gather tlb, struct mm_struct mm);	828	extern void tlb_gather_mmu(struct mmu_gather tlb, struct mm_struct mm);
703	extern void tlb_gather_mmu_fullmm(struct mmu_gather tlb, struct mm_struct mm);	829	extern void tlb_gather_mmu_fullmm(struct mmu_gather tlb, struct mm_struct mm);

include/linux/mm_types_task.h+0−12

@@ -24,18 +24,6 @@ · 23 unmodified lines
24	IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK))	24	IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK))
25	#define ALLOC_SPLIT_PTLOCKS (SPINLOCK_SIZE > BITS_PER_LONG/8)	25	#define ALLOC_SPLIT_PTLOCKS (SPINLOCK_SIZE > BITS_PER_LONG/8)
26		26
27	/*
28	* The per task VMA cache array:
29	*/
30	#define VMACACHE_BITS 2
31	#define VMACACHE_SIZE (1U << VMACACHE_BITS)
32	#define VMACACHE_MASK (VMACACHE_SIZE - 1)
33
34	struct vmacache {
35	u64 seqnum;
36	struct vm_area_struct *vmas[VMACACHE_SIZE];
37	};
38
39	/*	27	/*
40	* When updating this, please also update struct resident_page_types[] in	28	* When updating this, please also update struct resident_page_types[] in
41	* kernel/fork.c	29	* kernel/fork.c

include/linux/mmzone.h+246−35

@@ -24,10 +24,10 @@ · 23 unmodified lines
24	#include <asm/page.h>	24	#include <asm/page.h>
25		25
26	/* Free memory management - zoned buddy allocator. */	26	/* Free memory management - zoned buddy allocator. */
27	#ifndef CONFIG_FORCE_MAX_ZONEORDER	27	#ifndef CONFIG_ARCH_FORCE_MAX_ORDER
28	#define MAX_ORDER 11	28	#define MAX_ORDER 11
29	#else	29	#else
30	#define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER	30	#define MAX_ORDER CONFIG_ARCH_FORCE_MAX_ORDER
31	#endif	31	#endif
32	#define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1))	32	#define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1))
33		33
@@ -121,20 +121,6 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) · 87 unmodified lines
121		121
122	struct pglist_data;	122	struct pglist_data;
123		123
124	/*
125	* Add a wild amount of padding here to ensure data fall into separate
126	* cachelines. There are very few zone structures in the machine, so space
127	* consumption is not a concern here.
128	*/
129	#if defined(CONFIG_SMP)
130	struct zone_padding {
131	char x[0];
132	} ____cacheline_internodealigned_in_smp;
133	#define ZONE_PADDING(name) struct zone_padding name;
134	#else
135	#define ZONE_PADDING(name)
136	#endif
137
138	#ifdef CONFIG_NUMA	124	#ifdef CONFIG_NUMA
139	enum numa_stat_item {	125	enum numa_stat_item {
140	NUMA_HIT, /* allocated in intended node */	126	NUMA_HIT, /* allocated in intended node */
@@ -222,6 +208,7 @@ enum node_stat_item { · 81 unmodified lines
222	#endif	208	#endif
223	#ifdef CONFIG_NUMA_BALANCING	209	#ifdef CONFIG_NUMA_BALANCING
224	PGPROMOTE_SUCCESS, /* promote successfully */	210	PGPROMOTE_SUCCESS, /* promote successfully */
		211	PGPROMOTE_CANDIDATE, /* candidate pages to promote */
225	#endif	212	#endif
226	NR_VM_NODE_STAT_ITEMS	213	NR_VM_NODE_STAT_ITEMS
227	};	214	};
@@ -307,6 +294,8 @@ static inline bool is_active_lru(enum lru_list lru) · 79 unmodified lines
307	return (lru == LRU_ACTIVE_ANON \|\| lru == LRU_ACTIVE_FILE);	294	return (lru == LRU_ACTIVE_ANON \|\| lru == LRU_ACTIVE_FILE);
308	}	295	}
309		296
		297	#define WORKINGSET_ANON 0
		298	#define WORKINGSET_FILE 1
310	#define ANON_AND_FILE 2	299	#define ANON_AND_FILE 2
311		300
312	enum lruvec_flags {	301	enum lruvec_flags {
@@ -315,6 +304,207 @@ enum lruvec_flags { · 2 unmodified lines
315	*/	304	*/
316	};	305	};
317		306
		307	#endif /* !__GENERATING_BOUNDS_H */
		308
		309	/*
		310	* Evictable pages are divided into multiple generations. The youngest and the
		311	* oldest generation numbers, max_seq and min_seq, are monotonically increasing.
		312	* They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An
		313	* offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the
		314	* corresponding generation. The gen counter in folio->flags stores gen+1 while
		315	* a page is on one of lrugen->lists[]. Otherwise it stores 0.
		316	*
		317	* A page is added to the youngest generation on faulting. The aging needs to
		318	* check the accessed bit at least twice before handing this page over to the
		319	* eviction. The first check takes care of the accessed bit set on the initial
		320	* fault; the second check makes sure this page hasn't been used since then.
		321	* This process, AKA second chance, requires a minimum of two generations,
		322	* hence MIN_NR_GENS. And to maintain ABI compatibility with the active/inactive
		323	* LRU, e.g., /proc/vmstat, these two generations are considered active; the
		324	* rest of generations, if they exist, are considered inactive. See
		325	* lru_gen_is_active().
		326	*
		327	* PG_active is always cleared while a page is on one of lrugen->lists[] so that
		328	* the aging needs not to worry about it. And it's set again when a page
		329	* considered active is isolated for non-reclaiming purposes, e.g., migration.
		330	* See lru_gen_add_folio() and lru_gen_del_folio().
		331	*
		332	* MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the
		333	* number of categories of the active/inactive LRU when keeping track of
		334	* accesses through page tables. This requires order_base_2(MAX_NR_GENS+1) bits
		335	* in folio->flags.
		336	*/
		337	#define MIN_NR_GENS 2U
		338	#define MAX_NR_GENS 4U
		339
		340	/*
		341	* Each generation is divided into multiple tiers. A page accessed N times
		342	* through file descriptors is in tier order_base_2(N). A page in the first tier
		343	* (N=0,1) is marked by PG_referenced unless it was faulted in through page
		344	* tables or read ahead. A page in any other tier (N>1) is marked by
		345	* PG_referenced and PG_workingset. This implies a minimum of two tiers is
		346	* supported without using additional bits in folio->flags.
		347	*
		348	* In contrast to moving across generations which requires the LRU lock, moving
		349	* across tiers only involves atomic operations on folio->flags and therefore
		350	* has a negligible cost in the buffered access path. In the eviction path,
		351	* comparisons of refaulted/(evicted+protected) from the first tier and the
		352	* rest infer whether pages accessed multiple times through file descriptors
		353	* are statistically hot and thus worth protecting.
		354	*
		355	* MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the
		356	* number of categories of the active/inactive LRU when keeping track of
		357	* accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in
		358	* folio->flags.
		359	*/
		360	#define MAX_NR_TIERS 4U
		361
		362	#ifndef __GENERATING_BOUNDS_H
		363
		364	struct lruvec;
		365	struct page_vma_mapped_walk;
		366
		367	#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
		368	#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
		369
		370	#ifdef CONFIG_LRU_GEN
		371
		372	enum {
		373	LRU_GEN_ANON,
		374	LRU_GEN_FILE,
		375	};
		376
		377	enum {
		378	LRU_GEN_CORE,
		379	LRU_GEN_MM_WALK,
		380	LRU_GEN_NONLEAF_YOUNG,
		381	NR_LRU_GEN_CAPS
		382	};
		383
		384	#define MIN_LRU_BATCH BITS_PER_LONG
		385	#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64)
		386
		387	/* whether to keep historical stats from evicted generations */
		388	#ifdef CONFIG_LRU_GEN_STATS
		389	#define NR_HIST_GENS MAX_NR_GENS
		390	#else
		391	#define NR_HIST_GENS 1U
		392	#endif
		393
		394	/*
		395	* The youngest generation number is stored in max_seq for both anon and file
		396	* types as they are aged on an equal footing. The oldest generation numbers are
		397	* stored in min_seq[] separately for anon and file types as clean file pages
		398	* can be evicted regardless of swap constraints.
		399	*
		400	* Normally anon and file min_seq are in sync. But if swapping is constrained,
		401	* e.g., out of swap space, file min_seq is allowed to advance and leave anon
		402	* min_seq behind.
		403	*
		404	* The number of pages in each generation is eventually consistent and therefore
		405	* can be transiently negative when reset_batch_size() is pending.
		406	*/
		407	struct lru_gen_struct {
		408	/* the aging increments the youngest generation number */
		409	unsigned long max_seq;
		410	/* the eviction increments the oldest generation numbers */
		411	unsigned long min_seq[ANON_AND_FILE];
		412	/* the birth time of each generation in jiffies */
		413	unsigned long timestamps[MAX_NR_GENS];
		414	/* the multi-gen LRU lists, lazily sorted on eviction */
		415	struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
		416	/* the multi-gen LRU sizes, eventually consistent */
		417	long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
		418	/* the exponential moving average of refaulted */
		419	unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS];
		420	/* the exponential moving average of evicted+protected */
		421	unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS];
		422	/* the first tier doesn't need protection, hence the minus one */
		423	unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS - 1];
		424	/* can be modified without holding the LRU lock */
		425	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
		426	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
		427	/* whether the multi-gen LRU is enabled */
		428	bool enabled;
		429	};
		430
		431	enum {
		432	MM_LEAF_TOTAL, /* total leaf entries */
		433	MM_LEAF_OLD, /* old leaf entries */
		434	MM_LEAF_YOUNG, /* young leaf entries */
		435	MM_NONLEAF_TOTAL, /* total non-leaf entries */
		436	MM_NONLEAF_FOUND, /* non-leaf entries found in Bloom filters */
		437	MM_NONLEAF_ADDED, /* non-leaf entries added to Bloom filters */
		438	NR_MM_STATS
		439	};
		440
		441	/* double-buffering Bloom filters */
		442	#define NR_BLOOM_FILTERS 2
		443
		444	struct lru_gen_mm_state {
		445	/* set to max_seq after each iteration */
		446	unsigned long seq;
		447	/* where the current iteration continues (inclusive) */
		448	struct list_head *head;
		449	/* where the last iteration ended (exclusive) */
		450	struct list_head *tail;
		451	/* to wait for the last page table walker to finish */
		452	struct wait_queue_head wait;
		453	/* Bloom filters flip after each iteration */
		454	unsigned long *filters[NR_BLOOM_FILTERS];
		455	/* the mm stats for debugging */
		456	unsigned long stats[NR_HIST_GENS][NR_MM_STATS];
		457	/* the number of concurrent page table walkers */
		458	int nr_walkers;
		459	};
		460
		461	struct lru_gen_mm_walk {
		462	/* the lruvec under reclaim */
		463	struct lruvec *lruvec;
		464	/* unstable max_seq from lru_gen_struct */
		465	unsigned long max_seq;
		466	/* the next address within an mm to scan */
		467	unsigned long next_addr;
		468	/* to batch promoted pages */
		469	int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
		470	/* to batch the mm stats */
		471	int mm_stats[NR_MM_STATS];
		472	/* total batched items */
		473	int batched;
		474	bool can_swap;
		475	bool force_scan;
		476	};
		477
		478	void lru_gen_init_lruvec(struct lruvec *lruvec);
		479	void lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
		480
		481	#ifdef CONFIG_MEMCG
		482	void lru_gen_init_memcg(struct mem_cgroup *memcg);
		483	void lru_gen_exit_memcg(struct mem_cgroup *memcg);
		484	#endif
		485
		486	#else /* !CONFIG_LRU_GEN */
		487
		488	static inline void lru_gen_init_lruvec(struct lruvec *lruvec)
		489	{
		490	}
		491
		492	static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
		493	{
		494	}
		495
		496	#ifdef CONFIG_MEMCG
		497	static inline void lru_gen_init_memcg(struct mem_cgroup *memcg)
		498	{
		499	}
		500
		501	static inline void lru_gen_exit_memcg(struct mem_cgroup *memcg)
		502	{
		503	}
		504	#endif
		505
		506	#endif /* CONFIG_LRU_GEN */
		507
318	struct lruvec {	508	struct lruvec {
319	struct list_head lists[NR_LRU_LISTS];	509	struct list_head lists[NR_LRU_LISTS];
320	/* per lruvec lru_lock for memcg */	510	/* per lruvec lru_lock for memcg */
@@ -332,6 +522,12 @@ struct lruvec { · 11 unmodified lines
332	unsigned long refaults[ANON_AND_FILE];	522	unsigned long refaults[ANON_AND_FILE];
333	/* Various lruvec state flags (enum lruvec_flags) */	523	/* Various lruvec state flags (enum lruvec_flags) */
334	unsigned long flags;	524	unsigned long flags;
		525	#ifdef CONFIG_LRU_GEN
		526	/* evictable pages divided into generations */
		527	struct lru_gen_struct lrugen;
		528	/* to concurrently iterate lru_gen_mm_list */
		529	struct lru_gen_mm_state mm_state;
		530	#endif
335	#ifdef CONFIG_MEMCG	531	#ifdef CONFIG_MEMCG
336	struct pglist_data *pgdat;	532	struct pglist_data *pgdat;
337	#endif	533	#endif
@@ -369,13 +565,6 @@ enum zone_watermarks { · 31 unmodified lines
369	#define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1))	565	#define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1))
370	#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP)	566	#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP)
371		567
372	/*
373	* Shift to encode migratetype and order in the same integer, with order
374	* in the least significant bits.
375	*/
376	#define NR_PCP_ORDER_WIDTH 8
377	#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_WIDTH) - 1)
378
379	#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost)	568	#define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost)
380	#define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost)	569	#define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost)
381	#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)	570	#define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
@@ -628,7 +817,7 @@ struct zone { · 246 unmodified lines
628	int initialized;	817	int initialized;
629		818
630	/* Write-intensive fields used from the page allocator */	819	/* Write-intensive fields used from the page allocator */
631	ZONE_PADDING(_pad1_)	820	CACHELINE_PADDING(_pad1_);
632		821
633	/* free areas of different sizes */	822	/* free areas of different sizes */
634	struct free_area free_area[MAX_ORDER];	823	struct free_area free_area[MAX_ORDER];
@@ -640,7 +829,7 @@ struct zone { · 5 unmodified lines
640	spinlock_t lock;	829	spinlock_t lock;
641		830
642	/* Write-intensive fields used by compaction and vmstats. */	831	/* Write-intensive fields used by compaction and vmstats. */
643	ZONE_PADDING(_pad2_)	832	CACHELINE_PADDING(_pad2_);
644		833
645	/*	834	/*
646	* When free pages are below this point, additional steps are taken	835	* When free pages are below this point, additional steps are taken
@@ -677,7 +866,7 @@ struct zone { · 30 unmodified lines
677		866
678	bool contiguous;	867	bool contiguous;
679		868
680	ZONE_PADDING(_pad3_)	869	CACHELINE_PADDING(_pad3_);
681	/* Zone statistics */	870	/* Zone statistics */
682	atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];	871	atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
683	atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];	872	atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];
@@ -747,6 +936,8 @@ static inline bool zone_is_empty(struct zone *zone) · 63 unmodified lines
747	#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)	936	#define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH)
748	#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)	937	#define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH)
749	#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH)	938	#define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH)
		939	#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH)
		940	#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH)
750		941
751	/*	942	/*
752	* Define the bit shifts to access each section. For non-existent	943	* Define the bit shifts to access each section. For non-existent
@@ -954,8 +1145,10 @@ typedef struct pglist_data { · 201 unmodified lines
954	atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */	1145	atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */
955	unsigned long nr_reclaim_start; /* nr pages written while throttled	1146	unsigned long nr_reclaim_start; /* nr pages written while throttled
956	* when throttling started. */	1147	* when throttling started. */
957	struct task_struct kswapd; / Protected by	1148	#ifdef CONFIG_MEMORY_HOTPLUG
958	mem_hotplug_begin/done() */	1149	struct mutex kswapd_lock;
		1150	#endif
		1151	struct task_struct kswapd; / Protected by kswapd_lock */
959	int kswapd_order;	1152	int kswapd_order;
960	enum zone_type kswapd_highest_zoneidx;	1153	enum zone_type kswapd_highest_zoneidx;
961		1154
@@ -983,7 +1176,7 @@ typedef struct pglist_data { · 21 unmodified lines
983	#endif /* CONFIG_NUMA */	1176	#endif /* CONFIG_NUMA */
984		1177
985	/* Write-intensive fields used by page reclaim */	1178	/* Write-intensive fields used by page reclaim */
986	ZONE_PADDING(_pad1_)	1179	CACHELINE_PADDING(_pad1_);
987		1180
988	#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT	1181	#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
989	/*	1182	/*
@@ -997,6 +1190,21 @@ typedef struct pglist_data { · 7 unmodified lines
997	struct deferred_split deferred_split_queue;	1190	struct deferred_split deferred_split_queue;
998	#endif	1191	#endif
999		1192
		1193	#ifdef CONFIG_NUMA_BALANCING
		1194	/* start time in ms of current promote rate limit period */
		1195	unsigned int nbp_rl_start;
		1196	/* number of promote candidate pages at start time of current rate limit period */
		1197	unsigned long nbp_rl_nr_cand;
		1198	/* promote threshold in ms */
		1199	unsigned int nbp_threshold;
		1200	/* start time in ms of current promote threshold adjustment period */
		1201	unsigned int nbp_th_start;
		1202	/*
		1203	* number of promote candidate pages at stat time of current promote
		1204	* threshold adjustment period
		1205	*/
		1206	unsigned long nbp_th_nr_cand;
		1207	#endif
1000	/* Fields commonly accessed by the page reclaim scanner */	1208	/* Fields commonly accessed by the page reclaim scanner */
1001		1209
1002	/*	1210	/*
@@ -1008,11 +1216,19 @@ typedef struct pglist_data { · 5 unmodified lines
1008		1216
1009	unsigned long flags;	1217	unsigned long flags;
1010		1218
1011	ZONE_PADDING(_pad2_)	1219	#ifdef CONFIG_LRU_GEN
		1220	/* kswap mm walk data */
		1221	struct lru_gen_mm_walk mm_walk;
		1222	#endif
		1223
		1224	CACHELINE_PADDING(_pad2_);
1012		1225
1013	/* Per-node vmstats */	1226	/* Per-node vmstats */
1014	struct per_cpu_nodestat __percpu *per_cpu_nodestats;	1227	struct per_cpu_nodestat __percpu *per_cpu_nodestats;
1015	atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];	1228	atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
		1229	#ifdef CONFIG_NUMA
		1230	struct memory_tier __rcu *memtier;
		1231	#endif
1016	} pg_data_t;	1232	} pg_data_t;
1017		1233
1018	#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)	1234	#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
@@ -1026,11 +1242,6 @@ static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat) · 7 unmodified lines
1026	return pgdat->node_start_pfn + pgdat->node_spanned_pages;	1242	return pgdat->node_start_pfn + pgdat->node_spanned_pages;
1027	}	1243	}
1028		1244
1029	static inline bool pgdat_is_empty(pg_data_t *pgdat)
1030	{
1031	return !pgdat->node_start_pfn && !pgdat->node_spanned_pages;
1032	}
1033
1034	#include <linux/memory_hotplug.h>	1245	#include <linux/memory_hotplug.h>
1035		1246
1036	void build_all_zonelists(pg_data_t *pgdat);	1247	void build_all_zonelists(pg_data_t *pgdat);

include/linux/node.h+4−25

 /*
  * include/linux/node.h - generic node definition
  *
  * This is mainly for topological representation. We define the
  * basic 'struct node' here, which can be embedded in per-arch
  * definitions of processors.
  *
  * Basic handling of the devices is done in drivers/base/node.c
  * and system devices are handled in drivers/base/sys.c.
  *
  * Nodes are exported via driverfs in the class/node/devices/
  * directory.
  */
 #ifndef _LINUX_NODE_H_
 #define _LINUX_NODE_H_
 #include <linux/device.h>
 #include <linux/cpumask.h>
 #include <linux/list.h>
-#include <linux/workqueue.h>
 /**
  * struct node_hmem_attrs - heterogeneous memory performance attributes
 struct node {
 	struct device	dev;
 	struct list_head access_list;
-#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS)
-	struct work_struct	node_work;
-#endif
 #ifdef CONFIG_HMEM_REPORTING
 	struct list_head cache_attrs;
 	struct device *cache_dev;
 struct memory_block;
 extern struct node *node_devices[];
-typedef  void (*node_registration_func_t)(struct node *);
 #if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_NUMA)
 void register_memory_blocks_under_node(int nid, unsigned long start_pfn,
 extern int register_memory_node_under_compute_node(unsigned int mem_nid,
 						   unsigned int cpu_nid,
 						   unsigned access);
-#ifdef CONFIG_HUGETLBFS
-extern void register_hugetlbfs_with_node(node_registration_func_t doregister,
-					 node_registration_func_t unregister);
-#endif
 #else
 static inline void node_dev_init(void)
 {
 static inline void unregister_memory_block_under_nodes(struct memory_block *mem_blk)
 {
 }
-static inline void register_hugetlbfs_with_node(node_registration_func_t reg,
-						node_registration_func_t unreg)
-{
-}
 #endif
 #define to_node(device) container_of(device, struct node, dev)
-static inline bool node_is_toptier(int node)
-{
-	return node_state(node, N_CPU);
-}
 #endif /* _LINUX_NODE_H_ */

include/linux/nodemask.h+12−2

 #define first_online_node	0
 #define first_memory_node	0
 #define next_online_node(nid)	(MAX_NUMNODES)
+#define next_memory_node(nid)	(MAX_NUMNODES)
 #define nr_node_ids		1U
 #define nr_online_nodes		1U
 static inline int node_random(const nodemask_t *maskp)
 {
 #if defined(CONFIG_NUMA) && (MAX_NUMNODES > 1)
-	int w, bit = NUMA_NO_NODE;
+	int w, bit;
 	w = nodes_weight(*maskp);
-	if (w)
+	switch (w) {
+	case 0:
+		bit = NUMA_NO_NODE;
+		break;
+	case 1:
+		bit = first_node(*maskp);
+		break;
+	default:
 		bit = find_nth_bit(maskp->bits, MAX_NUMNODES, get_random_int() % w);
+		break;
+	}
 	return bit;
 #else
 	return 0;

include/linux/oom.h+0−11

 	return tsk->signal->oom_mm;
 }
-/*
- * Use this helper if tsk->mm != mm and the victim mm needs a special
- * handling. This is guaranteed to stay true after once set.
- */
-static inline bool mm_is_oom_victim(struct mm_struct *mm)
-{
-	return test_bit(MMF_OOM_VICTIM, &mm->flags);
-}
 /*
  * Checks whether a page fault on the given mm is still reliable.
  * This is no longer true if the oom reaper started to reap the
 	return 0;
 }
-bool __oom_reap_task_mm(struct mm_struct *mm);
 long oom_badness(struct task_struct *p,
 		unsigned long totalpages);

include/linux/page-flags-layout.h+11−5

 #define SECTIONS_WIDTH		0
 #endif
-#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
+#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_SHIFT \
+	<= BITS_PER_LONG - NR_PAGEFLAGS
 #define NODES_WIDTH		NODES_SHIFT
 #elif defined(CONFIG_SPARSEMEM_VMEMMAP)
 #error "Vmemmap: No space for nodes field in page flags"
 #define LAST_CPUPID_SHIFT 0
 #endif
-#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT \
+#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \
-	<= BITS_PER_LONG - NR_PAGEFLAGS
+	KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS
 #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT
 #else
 #define LAST_CPUPID_WIDTH 0
 #define LAST_CPUPID_NOT_IN_PAGE_FLAGS
 #endif
-#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH \
+#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \
-	> BITS_PER_LONG - NR_PAGEFLAGS
+	KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS
 #error "Not enough bits in page flags"
 #endif
+/* see the comment on MAX_NR_TIERS */
+#define LRU_REFS_WIDTH	min(__LRU_REFS_WIDTH, BITS_PER_LONG - NR_PAGEFLAGS - \
+			    ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \
+			    NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH)
 #endif
 #endif /* _LINUX_PAGE_FLAGS_LAYOUT */

include/linux/page-flags.h+2−2

@@ -1058,7 +1058,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) · 1057 unmodified lines
1058	1UL << PG_private \| 1UL << PG_private_2 \| \	1058	1UL << PG_private \| 1UL << PG_private_2 \| \
1059	1UL << PG_writeback \| 1UL << PG_reserved \| \	1059	1UL << PG_writeback \| 1UL << PG_reserved \| \
1060	1UL << PG_slab \| 1UL << PG_active \| \	1060	1UL << PG_slab \| 1UL << PG_active \| \
1061	1UL << PG_unevictable \| __PG_MLOCKED)	1061	1UL << PG_unevictable \| __PG_MLOCKED \| LRU_GEN_MASK)
1062		1062
1063	/*	1063	/*
1064	* Flags checked when a page is prepped for return by the page allocator.	1064	* Flags checked when a page is prepped for return by the page allocator.
@@ -1069,7 +1069,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) · 4 unmodified lines
1069	* alloc-free cycle to prevent from reusing the page.	1069	* alloc-free cycle to prevent from reusing the page.
1070	*/	1070	*/
1071	#define PAGE_FLAGS_CHECK_AT_PREP \	1071	#define PAGE_FLAGS_CHECK_AT_PREP \
1072	(PAGEFLAGS_MASK & ~__PG_HWPOISON)	1072	((PAGEFLAGS_MASK & ~__PG_HWPOISON) \| LRU_GEN_MASK \| LRU_REFS_MASK)
1073		1073
1074	#define PAGE_FLAGS_PRIVATE \	1074	#define PAGE_FLAGS_PRIVATE \
1075	(1UL << PG_private \| 1UL << PG_private_2)	1075	(1UL << PG_private \| 1UL << PG_private_2)

include/linux/page_counter.h+14−12

@@ -3,15 +3,17 @@ · 2 unmodified lines
3	#define _LINUX_PAGE_COUNTER_H	3	#define _LINUX_PAGE_COUNTER_H
4		4
5	#include <linux/atomic.h>	5	#include <linux/atomic.h>
		6	#include <linux/cache.h>
6	#include <linux/kernel.h>	7	#include <linux/kernel.h>
7	#include <asm/page.h>	8	#include <asm/page.h>
8		9
9	struct page_counter {	10	struct page_counter {
		11	/*
		12	* Make sure 'usage' does not share cacheline with any other field. The
		13	* memcg->memory.usage is a hot member of struct mem_cgroup.
		14	*/
10	atomic_long_t usage;	15	atomic_long_t usage;
11	unsigned long min;	16	CACHELINE_PADDING(_pad1_);
12	unsigned long low;
13	unsigned long high;
14	unsigned long max;
15		17
16	/* effective memory.min and memory.min usage tracking */	18	/* effective memory.min and memory.min usage tracking */
17	unsigned long emin;	19	unsigned long emin;
@@ -23,18 +25,18 @@ struct page_counter { · 5 unmodified lines
23	atomic_long_t low_usage;	25	atomic_long_t low_usage;
24	atomic_long_t children_low_usage;	26	atomic_long_t children_low_usage;
25		27
26	/* legacy */
27	unsigned long watermark;	28	unsigned long watermark;
28	unsigned long failcnt;	29	unsigned long failcnt;
29		30
30	/*	31	/* Keep all the read most fields in a separete cacheline. */
31	* 'parent' is placed here to be far from 'usage' to reduce	32	CACHELINE_PADDING(_pad2_);
32	* cache false sharing, as 'usage' is written mostly while	33
33	* parent is frequently read for cgroup's hierarchical	34	unsigned long min;
34	* counting nature.	35	unsigned long low;
35	*/	36	unsigned long high;
		37	unsigned long max;
36	struct page_counter *parent;	38	struct page_counter *parent;
37	};	39	} ____cacheline_internodealigned_in_smp;
38		40
39	#if BITS_PER_LONG == 32	41	#if BITS_PER_LONG == 32
40	#define PAGE_COUNTER_MAX LONG_MAX	42	#define PAGE_COUNTER_MAX LONG_MAX

include/linux/page_ext.h+20−4

 	unsigned long flags;
 };
+extern bool early_page_ext;
 extern unsigned long page_ext_size;
 extern void pgdat_page_ext_init(struct pglist_data *pgdat);
+static inline bool early_page_ext_enabled(void)
+{
+	return early_page_ext;
+}
 #ifdef CONFIG_SPARSEMEM
 static inline void page_ext_init_flatmem(void)
 {
 }
 #endif
-struct page_ext *lookup_page_ext(const struct page *page);
+extern struct page_ext *page_ext_get(struct page *page);
+extern void page_ext_put(struct page_ext *page_ext);
 static inline struct page_ext *page_ext_next(struct page_ext *curr)
 {
 #else /* !CONFIG_PAGE_EXTENSION */
 struct page_ext;
-static inline void pgdat_page_ext_init(struct pglist_data *pgdat)
+static inline bool early_page_ext_enabled(void)
 {
+	return false;
 }
-static inline struct page_ext *lookup_page_ext(const struct page *page)
+static inline void pgdat_page_ext_init(struct pglist_data *pgdat)
 {
-	return NULL;
 }
 static inline void page_ext_init(void)
 static inline void page_ext_init_flatmem(void)
 {
 }
+static inline struct page_ext *page_ext_get(struct page *page)
+{
+	return NULL;
+}
+static inline void page_ext_put(struct page_ext *page_ext)
+{
+}
 #endif /* CONFIG_PAGE_EXTENSION */
 #endif /* __LINUX_PAGE_EXT_H */

include/linux/page_idle.h+24−10

  * If there is not enough space to store Idle and Young bits in page flags, use
  * page ext flags instead.
  */
 static inline bool folio_test_young(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
+	bool page_young;
 	if (unlikely(!page_ext))
 		return false;
-	return test_bit(PAGE_EXT_YOUNG, &page_ext->flags);
+	page_young = test_bit(PAGE_EXT_YOUNG, &page_ext->flags);
+	page_ext_put(page_ext);
+	return page_young;
 }
 static inline void folio_set_young(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
 	if (unlikely(!page_ext))
 		return;
 	set_bit(PAGE_EXT_YOUNG, &page_ext->flags);
+	page_ext_put(page_ext);
 }
 static inline bool folio_test_clear_young(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
+	bool page_young;
 	if (unlikely(!page_ext))
 		return false;
-	return test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags);
+	page_young = test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags);
+	page_ext_put(page_ext);
+	return page_young;
 }
 static inline bool folio_test_idle(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
+	bool page_idle;
 	if (unlikely(!page_ext))
 		return false;
-	return test_bit(PAGE_EXT_IDLE, &page_ext->flags);
+	page_idle =  test_bit(PAGE_EXT_IDLE, &page_ext->flags);
+	page_ext_put(page_ext);
+	return page_idle;
 }
 static inline void folio_set_idle(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
 	if (unlikely(!page_ext))
 		return;
 	set_bit(PAGE_EXT_IDLE, &page_ext->flags);
+	page_ext_put(page_ext);
 }
 static inline void folio_clear_idle(struct folio *folio)
 {
-	struct page_ext *page_ext = lookup_page_ext(&folio->page);
+	struct page_ext *page_ext = page_ext_get(&folio->page);
 	if (unlikely(!page_ext))
 		return;
 	clear_bit(PAGE_EXT_IDLE, &page_ext->flags);
+	page_ext_put(page_ext);
 }
 #endif /* !CONFIG_64BIT */

include/linux/pageblock-flags.h+4−0

 #endif /* CONFIG_HUGETLB_PAGE */
 #define pageblock_nr_pages	(1UL << pageblock_order)
+#define pageblock_align(pfn)	ALIGN((pfn), pageblock_nr_pages)
+#define pageblock_aligned(pfn)	IS_ALIGNED((pfn), pageblock_nr_pages)
+#define pageblock_start_pfn(pfn)	ALIGN_DOWN((pfn), pageblock_nr_pages)
+#define pageblock_end_pfn(pfn)		ALIGN((pfn) + 1, pageblock_nr_pages)
 /* Forward declaration */
 struct page;

include/linux/pagemap.h+5−9

 unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start,
 		pgoff_t end, struct folio_batch *fbatch);
-unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start,
+unsigned filemap_get_folios_contig(struct address_space *mapping,
-			       unsigned int nr_pages, struct page **pages);
+		pgoff_t *start, pgoff_t end, struct folio_batch *fbatch);
 unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
 			pgoff_t end, xa_mark_t tag, unsigned int nr_pages,
 			struct page **pages);
 }
 /*
- * lock_page_or_retry - Lock the page, unless this would block and the
+ * folio_lock_or_retry - Lock the folio, unless this would block and the
  * caller indicated that it can handle a retry.
  *
  * Return value and mmap_lock implications depend on flags; see
  * __folio_lock_or_retry().
  */
-static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm,
+static inline bool folio_lock_or_retry(struct folio *folio,
-				     unsigned int flags)
+		struct mm_struct *mm, unsigned int flags)
 {
-	struct folio *folio;
 	might_sleep();
-	folio = page_folio(page);
 	return folio_trylock(folio) || __folio_lock_or_retry(folio, mm, flags);
 }
 	return folio_wait_locked_killable(page_folio(page));
 }
-int folio_put_wait_locked(struct folio *folio, int state);
 void wait_on_page_writeback(struct page *page);
 void folio_wait_writeback(struct folio *folio);
 int folio_wait_writeback_killable(struct folio *folio);

include/linux/pagewalk.h+5−5

@@ -15,12 +15,12 @@ struct mm_walk; · 14 unmodified lines
15	* this handler is required to be able to handle	15	* this handler is required to be able to handle
16	* pmd_trans_huge() pmds. They may simply choose to	16	* pmd_trans_huge() pmds. They may simply choose to
17	* split_huge_page() instead of handling it explicitly.	17	* split_huge_page() instead of handling it explicitly.
18	* @pte_entry: if set, called for each non-empty PTE (lowest-level)	18	* @pte_entry: if set, called for each PTE (lowest-level) entry,
19	* entry	19	* including empty ones
20	* @pte_hole: if set, called for each hole at all levels,	20	* @pte_hole: if set, called for each hole at all levels,
21	* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD	21	* depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD.
22	* 4:PTE. Any folded depths (where PTRS_PER_P?D is equal	22	* Any folded depths (where PTRS_PER_P?D is equal to 1)
23	* to 1) are skipped.	23	* are skipped.
24	* @hugetlb_entry: if set, called for each hugetlb entry	24	* @hugetlb_entry: if set, called for each hugetlb entry
25	* @test_walk: caller specific callback function to determine whether	25	* @test_walk: caller specific callback function to determine whether
26	* we walk over the current vma or not. Returning 0 means	26	* we walk over the current vma or not. Returning 0 means

include/linux/pgtable.h+17−9

 #endif
 #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
 static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma,
 					    unsigned long address,
 					    pmd_t *pmdp)
 	BUILD_BUG();
 	return 0;
 }
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
 #endif
 #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 #endif
+#ifndef arch_has_hw_pte_young
+/*
+ * Return whether the accessed bit is supported on the local CPU.
+ *
+ * This stub assumes accessing through an old PTE triggers a page fault.
+ * Architectures that automatically set the access bit should overwrite it.
+ */
+static inline bool arch_has_hw_pte_young(void)
+{
+	return false;
+}
+#endif
 #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 				       unsigned long address,
 #endif
 #if !defined(CONFIG_TRANSPARENT_HUGEPAGE) || \
-	(defined(CONFIG_TRANSPARENT_HUGEPAGE) && \
+	!defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)
-	 !defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD))
 static inline int pud_trans_huge(pud_t pud)
 {
 	return 0;
 #endif
 #ifndef has_transparent_hugepage
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)
-#define has_transparent_hugepage() 1
-#else
-#define has_transparent_hugepage() 0
-#endif
 #endif
 /*

include/linux/rmap.h+62−11

@@ -166,7 +166,7 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, · 165 unmodified lines
166	unlink_anon_vmas(next);	166	unlink_anon_vmas(next);
167	}	167	}
168		168
169	struct anon_vma page_get_anon_vma(struct page page);	169	struct anon_vma folio_get_anon_vma(struct folio folio);
170		170
171	/* RMAP flags, currently only relevant for some anon rmap operations. */	171	/* RMAP flags, currently only relevant for some anon rmap operations. */
172	typedef int __bitwise rmap_t;	172	typedef int __bitwise rmap_t;
@@ -270,7 +270,7 @@ dup: · 97 unmodified lines
270	* @page: the exclusive anonymous page to try marking possibly shared	270	* @page: the exclusive anonymous page to try marking possibly shared
271	*	271	*
272	* The caller needs to hold the PT lock and has to have the page table entry	272	* The caller needs to hold the PT lock and has to have the page table entry
273	* cleared/invalidated+flushed, to properly sync against GUP-fast.	273	* cleared/invalidated.
274	*	274	*
275	* This is similar to page_try_dup_anon_rmap(), however, not used during fork()	275	* This is similar to page_try_dup_anon_rmap(), however, not used during fork()
276	* to duplicate a mapping, but instead to prepare for KSM or temporarily	276	* to duplicate a mapping, but instead to prepare for KSM or temporarily
@@ -286,12 +286,68 @@ static inline int page_try_share_anon_rmap(struct page *page) · 9 unmodified lines
286	{	286	{
287	VM_BUG_ON_PAGE(!PageAnon(page) \|\| !PageAnonExclusive(page), page);	287	VM_BUG_ON_PAGE(!PageAnon(page) \|\| !PageAnonExclusive(page), page);
288		288
289	/* See page_try_dup_anon_rmap(). */	289	/* device private pages cannot get pinned via GUP. */
290	if (likely(!is_device_private_page(page) &&	290	if (unlikely(is_device_private_page(page))) {
291	unlikely(page_maybe_dma_pinned(page))))	291	ClearPageAnonExclusive(page);
292	return -EBUSY;	292	return 0;
		293	}
		294
		295	/*
		296	* We have to make sure that when we clear PageAnonExclusive, that
		297	* the page is not pinned and that concurrent GUP-fast won't succeed in
		298	* concurrently pinning the page.
		299	*
		300	* Conceptually, PageAnonExclusive clearing consists of:
		301	* (A1) Clear PTE
		302	* (A2) Check if the page is pinned; back off if so.
		303	* (A3) Clear PageAnonExclusive
		304	* (A4) Restore PTE (optional, but certainly not writable)
		305	*
		306	* When clearing PageAnonExclusive, we cannot possibly map the page
		307	* writable again, because anon pages that may be shared must never
		308	* be writable. So in any case, if the PTE was writable it cannot
		309	* be writable anymore afterwards and there would be a PTE change. Only
		310	* if the PTE wasn't writable, there might not be a PTE change.
		311	*
		312	* Conceptually, GUP-fast pinning of an anon page consists of:
		313	* (B1) Read the PTE
		314	* (B2) FOLL_WRITE: check if the PTE is not writable; back off if so.
		315	* (B3) Pin the mapped page
		316	* (B4) Check if the PTE changed by re-reading it; back off if so.
		317	* (B5) If the original PTE is not writable, check if
		318	* PageAnonExclusive is not set; back off if so.
		319	*
		320	* If the PTE was writable, we only have to make sure that GUP-fast
		321	* observes a PTE change and properly backs off.
		322	*
		323	* If the PTE was not writable, we have to make sure that GUP-fast either
		324	* detects a (temporary) PTE change or that PageAnonExclusive is cleared
		325	* and properly backs off.
		326	*
		327	* Consequently, when clearing PageAnonExclusive(), we have to make
		328	* sure that (A1), (A2)/(A3) and (A4) happen in the right memory
		329	* order. In GUP-fast pinning code, we have to make sure that (B3),(B4)
		330	* and (B5) happen in the right memory order.
		331	*
		332	* We assume that there might not be a memory barrier after
		333	* clearing/invalidating the PTE (A1) and before restoring the PTE (A4),
		334	* so we use explicit ones here.
		335	*/
293		336
		337	/* Paired with the memory barrier in try_grab_folio(). */
		338	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
		339	smp_mb();
		340
		341	if (unlikely(page_maybe_dma_pinned(page)))
		342	return -EBUSY;
294	ClearPageAnonExclusive(page);	343	ClearPageAnonExclusive(page);
		344
		345	/*
		346	* This is conceptually a smp_wmb() paired with the smp_rmb() in
		347	* gup_must_unshare().
		348	*/
		349	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
		350	smp_mb__after_atomic();
295	return 0;	351	return 0;
296	}	352	}
297		353
@@ -405,13 +461,8 @@ struct rmap_walk_control { · 107 unmodified lines
405		461
406	void rmap_walk(struct folio folio, struct rmap_walk_control rwc);	462	void rmap_walk(struct folio folio, struct rmap_walk_control rwc);
407	void rmap_walk_locked(struct folio folio, struct rmap_walk_control rwc);	463	void rmap_walk_locked(struct folio folio, struct rmap_walk_control rwc);
408
409	/*
410	* Called by memory-failure.c to kill processes.
411	*/
412	struct anon_vma folio_lock_anon_vma_read(struct folio folio,	464	struct anon_vma folio_lock_anon_vma_read(struct folio folio,
413	struct rmap_walk_control *rwc);	465	struct rmap_walk_control *rwc);
414	void page_unlock_anon_vma_read(struct anon_vma *anon_vma);
415		466
416	#else /* !CONFIG_MMU */	467	#else /* !CONFIG_MMU */
417		468

include/linux/sched.h+13−1

 #include <linux/pid.h>
 #include <linux/sem.h>
 #include <linux/shm.h>
+#include <linux/kmsan_types.h>
 #include <linux/mutex.h>
 #include <linux/plist.h>
 #include <linux/hrtimer.h>
 	struct mm_struct		*active_mm;
 	/* Per-thread vma caching: */
-	struct vmacache			vmacache;
 #ifdef SPLIT_RSS_COUNTING
 	struct task_rss_stat		rss_stat;
 #ifdef CONFIG_MEMCG
 	unsigned			in_user_fault:1;
 #endif
+#ifdef CONFIG_LRU_GEN
+	/* whether the LRU algorithm may apply to this access */
+	unsigned			in_lru_fault:1;
+#endif
 #ifdef CONFIG_COMPAT_BRK
 	unsigned			brk_randomized:1;
 #endif
 #ifdef	CONFIG_CPU_SUP_INTEL
 	unsigned			reported_split_lock:1;
 #endif
+#ifdef CONFIG_TASK_DELAY_ACCT
+	/* delay due to memory thrashing */
+	unsigned                        in_thrashing:1;
+#endif
 	unsigned long			atomic_flags; /* Flags requiring atomic access. */
 #endif
 #endif
+#ifdef CONFIG_KMSAN
+	struct kmsan_ctx		kmsan_ctx;
+#endif
 #if IS_ENABLED(CONFIG_KUNIT)
 	struct kunit			*kunit_test;
 #endif

include/linux/sched/coredump.h+3−4

 #define MMF_UNSTABLE		22	/* mm is unstable for copy_from_user */
 #define MMF_HUGE_ZERO_PAGE	23      /* mm has ever used the global huge zero page */
 #define MMF_DISABLE_THP		24	/* disable THP for all VMAs */
-#define MMF_OOM_VICTIM		25	/* mm is the oom victim */
+#define MMF_OOM_REAP_QUEUED	25	/* mm was queued for oom_reaper */
-#define MMF_OOM_REAP_QUEUED	26	/* mm was queued for oom_reaper */
+#define MMF_MULTIPROCESS	26	/* mm is shared between processes */
-#define MMF_MULTIPROCESS	27	/* mm is shared between processes */
 /*
  * MMF_HAS_PINNED: Whether this mm has pinned any pages.  This can be either
  * replaced in the future by mm.pinned_vm when it becomes stable, or grow into
  * pinned pages were unpinned later on, we'll still keep this bit set for the
  * lifecycle of this mm, just for simplicity.
  */
-#define MMF_HAS_PINNED		28	/* FOLL_PIN has run, never cleared */
+#define MMF_HAS_PINNED		27	/* FOLL_PIN has run, never cleared */
 #define MMF_DISABLE_THP_MASK	(1 << MMF_DISABLE_THP)
 #define MMF_INIT_MASK		(MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\

include/linux/sched/sysctl.h+1−0

@@ -27,6 +27,7 @@ enum sched_tunable_scaling { · 26 unmodified lines
27		27
28	#ifdef CONFIG_NUMA_BALANCING	28	#ifdef CONFIG_NUMA_BALANCING
29	extern int sysctl_numa_balancing_mode;	29	extern int sysctl_numa_balancing_mode;
		30	extern unsigned int sysctl_numa_balancing_promote_rate_limit;
30	#else	31	#else
31	#define sysctl_numa_balancing_mode 0	32	#define sysctl_numa_balancing_mode 0
32	#endif	33	#endif

include/linux/shmem_fs.h+9−7

 extern void shmem_truncate_range(struct inode *inode, loff_t start, loff_t end);
 int shmem_unuse(unsigned int type);
-extern bool shmem_is_huge(struct vm_area_struct *vma,
+extern bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
-			  struct inode *inode, pgoff_t index);
+			  pgoff_t index, bool shmem_huge_force);
-static inline bool shmem_huge_enabled(struct vm_area_struct *vma)
+static inline bool shmem_huge_enabled(struct vm_area_struct *vma,
+				      bool shmem_huge_force)
 {
-	return shmem_is_huge(vma, file_inode(vma->vm_file), vma->vm_pgoff);
+	return shmem_is_huge(vma, file_inode(vma->vm_file), vma->vm_pgoff,
+			     shmem_huge_force);
 }
 extern unsigned long shmem_swap_usage(struct vm_area_struct *vma);
 extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,
 						pgoff_t start, pgoff_t end);
-/* Flag allocation requirements to shmem_getpage */
+/* Flag allocation requirements to shmem_get_folio */
 enum sgp_type {
 	SGP_READ,	/* don't exceed i_size, don't allocate page */
 	SGP_NOALLOC,	/* similar, but fail on hole or use fallocated page */
 	SGP_FALLOC,	/* like SGP_WRITE, but make existing page Uptodate */
 };
-extern int shmem_getpage(struct inode *inode, pgoff_t index,
+int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop,
-		struct page **pagep, enum sgp_type sgp);
+		enum sgp_type sgp);
 static inline struct page *shmem_read_mapping_page(
 				struct address_space *mapping, pgoff_t index)

include/linux/slab.h+7−1

 # define SLAB_ACCOUNT		0
 #endif
-#ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_GENERIC
 #define SLAB_KASAN		((slab_flags_t __force)0x08000000U)
 #else
 #define SLAB_KASAN		0
  */
 #define SLAB_NO_USER_FLAGS	((slab_flags_t __force)0x10000000U)
+#ifdef CONFIG_KFENCE
+#define SLAB_SKIP_KFENCE	((slab_flags_t __force)0x20000000U)
+#else
+#define SLAB_SKIP_KFENCE	0
+#endif
 /* The following flags affect the page allocator grouping pages by mobility */
 /* Objects are reclaimable */
 #define SLAB_RECLAIM_ACCOUNT	((slab_flags_t __force)0x00020000U)

include/linux/stackdepot.h+8−0

 #include <linux/gfp.h>
 typedef u32 depot_stack_handle_t;
+/*
+ * Number of bits in the handle that stack depot doesn't use. Users may store
+ * information in them.
+ */
+#define STACK_DEPOT_EXTRA_BITS 5
 depot_stack_handle_t __stack_depot_save(unsigned long *entries,
 					unsigned int nr_entries,
+					unsigned int extra_bits,
 					gfp_t gfp_flags, bool can_alloc);
 /*
 unsigned int stack_depot_fetch(depot_stack_handle_t handle,
 			       unsigned long **entries);
+unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle);
 int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
 		       int spaces);

include/linux/swap.h+25−16

  */
 struct reclaim_state {
 	unsigned long reclaimed_slab;
+#ifdef CONFIG_LRU_GEN
+	/* per-thread mm walk data */
+	struct lru_gen_mm_walk *mm_walk;
+#endif
 };
 #ifdef __KERNEL__
 	return entry;
 }
+static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry)
+{
+	folio->private = (void *)entry.val;
+}
 /* linux/mm/workingset.c */
 void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);
 void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg);
 /* linux/mm/swap.c */
-extern void lru_note_cost(struct lruvec *lruvec, bool file,
+void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages);
-			  unsigned int nr_pages);
+void lru_note_cost_folio(struct folio *);
-extern void lru_note_cost_folio(struct folio *);
+void folio_add_lru(struct folio *);
-extern void folio_add_lru(struct folio *);
+void folio_add_lru_vma(struct folio *, struct vm_area_struct *);
-extern void lru_cache_add(struct page *);
+void lru_cache_add(struct page *);
 void mark_page_accessed(struct page *);
 void folio_mark_accessed(struct folio *);
 extern void si_swapinfo(struct sysinfo *);
 swp_entry_t folio_alloc_swap(struct folio *folio);
-extern void put_swap_page(struct page *page, swp_entry_t entry);
+bool folio_free_swap(struct folio *folio);
+void put_swap_folio(struct folio *folio, swp_entry_t entry);
 extern swp_entry_t get_swap_page_of_type(int);
 extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size);
 extern int add_swap_count_continuation(swp_entry_t, gfp_t);
 extern int swp_swapcount(swp_entry_t entry);
 extern struct swap_info_struct *page_swap_info(struct page *);
 extern struct swap_info_struct *swp_swap_info(swp_entry_t entry);
-extern int try_to_free_swap(struct page *);
 struct backing_dev_info;
 extern int init_swap_address_space(unsigned int type, unsigned long nr_pages);
 extern void exit_swap_address_space(unsigned int type);
 {
 }
-static inline void put_swap_page(struct page *page, swp_entry_t swp)
+static inline void put_swap_folio(struct folio *folio, swp_entry_t swp)
 {
 }
 	return 0;
 }
-static inline int try_to_free_swap(struct page *page)
-{
-	return 0;
-}
 static inline swp_entry_t folio_alloc_swap(struct folio *folio)
 {
 	swp_entry_t entry;
 	return entry;
 }
+static inline bool folio_free_swap(struct folio *folio)
+{
+	return false;
+}
 static inline int add_swap_extent(struct swap_info_struct *sis,
 				  unsigned long start_page,
 				  unsigned long nr_pages, sector_t start_block)
 	cgroup_throttle_swaprate(&folio->page, gfp);
 }
-#ifdef CONFIG_MEMCG_SWAP
+#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
 void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry);
 int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);
 static inline int mem_cgroup_try_charge_swap(struct folio *folio,
 }
 extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
-extern bool mem_cgroup_swap_full(struct page *page);
+extern bool mem_cgroup_swap_full(struct folio *folio);
 #else
 static inline void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry)
 {
 	return get_nr_swap_pages();
 }
-static inline bool mem_cgroup_swap_full(struct page *page)
+static inline bool mem_cgroup_swap_full(struct folio *folio)
 {
 	return vm_swap_full();
 }

include/linux/swap_cgroup.h+2−2

@@ -4,7 +4,7 @@ · 3 unmodified lines
4		4
5	#include <linux/swap.h>	5	#include <linux/swap.h>
6		6
7	#ifdef CONFIG_MEMCG_SWAP	7	#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
8		8
9	extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,	9	extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent,
10	unsigned short old, unsigned short new);	10	unsigned short old, unsigned short new);
@@ -40,6 +40,6 @@ static inline void swap_cgroup_swapoff(int type) · 29 unmodified lines
40	return;	40	return;
41	}	41	}
42		42
43	#endif /* CONFIG_MEMCG_SWAP */	43	#endif
44		44
45	#endif /* __LINUX_SWAP_CGROUP_H */	45	#endif /* __LINUX_SWAP_CGROUP_H */

include/linux/swapfile.h+6−1

  */
 extern struct swap_info_struct *swap_info[];
 extern unsigned long generic_max_swapfile_size(void);
-extern unsigned long max_swapfile_size(void);
+unsigned long arch_max_swapfile_size(void);
+/* Maximum swapfile size supported for the arch (not inclusive). */
+extern unsigned long swapfile_maximum_size;
+/* Whether swap migration entry supports storing A/D bits for the arch */
+extern bool swap_migration_ad_supported;
 #endif /* _LINUX_SWAPFILE_H */

include/linux/swapops.h+131−19

@@ -8,6 +8,10 @@ · 7 unmodified lines
8		8
9	#ifdef CONFIG_MMU	9	#ifdef CONFIG_MMU
10		10
		11	#ifdef CONFIG_SWAP
		12	#include <linux/swapfile.h>
		13	#endif /* CONFIG_SWAP */
		14
11	/*	15	/*
12	* swapcache pages are stored in the swapper_space radix tree. We want to	16	* swapcache pages are stored in the swapper_space radix tree. We want to
13	* get good packing density in that tree, so the index should be dense in	17	* get good packing density in that tree, so the index should be dense in
@@ -23,6 +27,45 @@ · 9 unmodified lines
23	#define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT)	27	#define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT)
24	#define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1)	28	#define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1)
25		29
		30	/*
		31	* Definitions only for PFN swap entries (see is_pfn_swap_entry()). To
		32	* store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries
		33	* can use the extra bits to store other information besides PFN.
		34	*/
		35	#ifdef MAX_PHYSMEM_BITS
		36	#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
		37	#else /* MAX_PHYSMEM_BITS */
		38	#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
		39	#endif /* MAX_PHYSMEM_BITS */
		40	#define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1)
		41
		42	/**
		43	* Migration swap entry specific bitfield definitions. Layout:
		44	*
		45	* \|----------+--------------------\|
		46	* \| swp_type \| swp_offset \|
		47	* \|----------+--------+-+-+-------\|
		48	* \| \| resv \|D\|A\| PFN \|
		49	* \|----------+--------+-+-+-------\|
		50	*
		51	* @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set (bit A)
		52	* @SWP_MIG_DIRTY_BIT: Whether the page used to have dirty bit set (bit D)
		53	*
		54	* Note: A/D bits will be stored in migration entries iff there're enough
		55	* free bits in arch specific swp offset. By default we'll ignore A/D bits
		56	* when migrating a page. Please refer to migration_entry_supports_ad()
		57	* for more information. If there're more bits besides PFN and A/D bits,
		58	* they should be reserved and always be zeros.
		59	*/
		60	#define SWP_MIG_YOUNG_BIT (SWP_PFN_BITS)
		61	#define SWP_MIG_DIRTY_BIT (SWP_PFN_BITS + 1)
		62	#define SWP_MIG_TOTAL_BITS (SWP_PFN_BITS + 2)
		63
		64	#define SWP_MIG_YOUNG BIT(SWP_MIG_YOUNG_BIT)
		65	#define SWP_MIG_DIRTY BIT(SWP_MIG_DIRTY_BIT)
		66
		67	static inline bool is_pfn_swap_entry(swp_entry_t entry);
		68
26	/* Clear all flags but only keep swp_entry_t related information */	69	/* Clear all flags but only keep swp_entry_t related information */
27	static inline pte_t pte_swp_clear_flags(pte_t pte)	70	static inline pte_t pte_swp_clear_flags(pte_t pte)
28	{	71	{
@@ -64,6 +107,17 @@ static inline pgoff_t swp_offset(swp_entry_t entry) · 35 unmodified lines
64	return entry.val & SWP_OFFSET_MASK;	107	return entry.val & SWP_OFFSET_MASK;
65	}	108	}
66		109
		110	/*
		111	* This should only be called upon a pfn swap entry to get the PFN stored
		112	* in the swap entry. Please refers to is_pfn_swap_entry() for definition
		113	* of pfn swap entry.
		114	*/
		115	static inline unsigned long swp_offset_pfn(swp_entry_t entry)
		116	{
		117	VM_BUG_ON(!is_pfn_swap_entry(entry));
		118	return swp_offset(entry) & SWP_PFN_MASK;
		119	}
		120
67	/* check whether a pte points to a swap entry */	121	/* check whether a pte points to a swap entry */
68	static inline int is_swap_pte(pte_t pte)	122	static inline int is_swap_pte(pte_t pte)
69	{	123	{
@@ -240,6 +294,52 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) · 170 unmodified lines
240	return swp_entry(SWP_MIGRATION_WRITE, offset);	294	return swp_entry(SWP_MIGRATION_WRITE, offset);
241	}	295	}
242		296
		297	/*
		298	* Returns whether the host has large enough swap offset field to support
		299	* carrying over pgtable A/D bits for page migrations. The result is
		300	* pretty much arch specific.
		301	*/
		302	static inline bool migration_entry_supports_ad(void)
		303	{
		304	#ifdef CONFIG_SWAP
		305	return swap_migration_ad_supported;
		306	#else /* CONFIG_SWAP */
		307	return false;
		308	#endif /* CONFIG_SWAP */
		309	}
		310
		311	static inline swp_entry_t make_migration_entry_young(swp_entry_t entry)
		312	{
		313	if (migration_entry_supports_ad())
		314	return swp_entry(swp_type(entry),
		315	swp_offset(entry) \| SWP_MIG_YOUNG);
		316	return entry;
		317	}
		318
		319	static inline bool is_migration_entry_young(swp_entry_t entry)
		320	{
		321	if (migration_entry_supports_ad())
		322	return swp_offset(entry) & SWP_MIG_YOUNG;
		323	/* Keep the old behavior of aging page after migration */
		324	return false;
		325	}
		326
		327	static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry)
		328	{
		329	if (migration_entry_supports_ad())
		330	return swp_entry(swp_type(entry),
		331	swp_offset(entry) \| SWP_MIG_DIRTY);
		332	return entry;
		333	}
		334
		335	static inline bool is_migration_entry_dirty(swp_entry_t entry)
		336	{
		337	if (migration_entry_supports_ad())
		338	return swp_offset(entry) & SWP_MIG_DIRTY;
		339	/* Keep the old behavior of clean page after migration */
		340	return false;
		341	}
		342
243	extern void __migration_entry_wait(struct mm_struct mm, pte_t ptep,	343	extern void __migration_entry_wait(struct mm_struct mm, pte_t ptep,
244	spinlock_t *ptl);	344	spinlock_t *ptl);
245	extern void migration_entry_wait(struct mm_struct mm, pmd_t pmd,	345	extern void migration_entry_wait(struct mm_struct mm, pmd_t pmd,
@@ -247,8 +347,8 @@ extern void migration_entry_wait(struct mm_struct mm, pmd_t pmd, · 1 unmodified line
247	#ifdef CONFIG_HUGETLB_PAGE	347	#ifdef CONFIG_HUGETLB_PAGE
248	extern void __migration_entry_wait_huge(pte_t ptep, spinlock_t ptl);	348	extern void __migration_entry_wait_huge(pte_t ptep, spinlock_t ptl);
249	extern void migration_entry_wait_huge(struct vm_area_struct vma, pte_t pte);	349	extern void migration_entry_wait_huge(struct vm_area_struct vma, pte_t pte);
250	#endif	350	#endif /* CONFIG_HUGETLB_PAGE */
251	#else	351	#else /* CONFIG_MIGRATION */
252	static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)	352	static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)
253	{	353	{
254	return swp_entry(0, 0);	354	return swp_entry(0, 0);
@@ -276,7 +376,7 @@ static inline void migration_entry_wait(struct mm_struct mm, pmd_t pmd, · 21 unmodified lines
276	#ifdef CONFIG_HUGETLB_PAGE	376	#ifdef CONFIG_HUGETLB_PAGE
277	static inline void __migration_entry_wait_huge(pte_t ptep, spinlock_t ptl) { }	377	static inline void __migration_entry_wait_huge(pte_t ptep, spinlock_t ptl) { }
278	static inline void migration_entry_wait_huge(struct vm_area_struct vma, pte_t pte) { }	378	static inline void migration_entry_wait_huge(struct vm_area_struct vma, pte_t pte) { }
279	#endif	379	#endif /* CONFIG_HUGETLB_PAGE */
280	static inline int is_writable_migration_entry(swp_entry_t entry)	380	static inline int is_writable_migration_entry(swp_entry_t entry)
281	{	381	{
282	return 0;	382	return 0;
@@ -286,7 +386,26 @@ static inline int is_readable_migration_entry(swp_entry_t entry) · 3 unmodified lines
286	return 0;	386	return 0;
287	}	387	}
288		388
289	#endif	389	static inline swp_entry_t make_migration_entry_young(swp_entry_t entry)
		390	{
		391	return entry;
		392	}
		393
		394	static inline bool is_migration_entry_young(swp_entry_t entry)
		395	{
		396	return false;
		397	}
		398
		399	static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry)
		400	{
		401	return entry;
		402	}
		403
		404	static inline bool is_migration_entry_dirty(swp_entry_t entry)
		405	{
		406	return false;
		407	}
		408	#endif /* CONFIG_MIGRATION */
290		409
291	typedef unsigned long pte_marker;	410	typedef unsigned long pte_marker;
292		411
@@ -369,7 +488,7 @@ static inline int pte_none_mostly(pte_t pte) · 76 unmodified lines
369		488
370	static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)	489	static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
371	{	490	{
372	struct page *p = pfn_to_page(swp_offset(entry));	491	struct page *p = pfn_to_page(swp_offset_pfn(entry));
373		492
374	/*	493	/*
375	* Any use of migration entries may only occur while the	494	* Any use of migration entries may only occur while the
@@ -387,6 +506,9 @@ static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) · 11 unmodified lines
387	*/	506	*/
388	static inline bool is_pfn_swap_entry(swp_entry_t entry)	507	static inline bool is_pfn_swap_entry(swp_entry_t entry)
389	{	508	{
		509	/* Make sure the swp offset can always store the needed fields */
		510	BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS);
		511
390	return is_migration_entry(entry) \|\| is_device_private_entry(entry) \|\|	512	return is_migration_entry(entry) \|\| is_device_private_entry(entry) \|\|
391	is_device_exclusive_entry(entry);	513	is_device_exclusive_entry(entry);
392	}	514	}
@@ -426,7 +548,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) · 33 unmodified lines
426	{	548	{
427	return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd));	549	return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd));
428	}	550	}
429	#else	551	#else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
430	static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,	552	static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
431	struct page *page)	553	struct page *page)
432	{	554	{
@@ -455,7 +577,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) · 22 unmodified lines
455	{	577	{
456	return 0;	578	return 0;
457	}	579	}
458	#endif	580	#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
459		581
460	#ifdef CONFIG_MEMORY_FAILURE	582	#ifdef CONFIG_MEMORY_FAILURE
461		583
@@ -475,27 +597,17 @@ static inline int is_hwpoison_entry(swp_entry_t entry) · 13 unmodified lines
475	return swp_type(entry) == SWP_HWPOISON;	597	return swp_type(entry) == SWP_HWPOISON;
476	}	598	}
477		599
478	static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry)
479	{
480	return swp_offset(entry);
481	}
482
483	static inline void num_poisoned_pages_inc(void)	600	static inline void num_poisoned_pages_inc(void)
484	{	601	{
485	atomic_long_inc(&num_poisoned_pages);	602	atomic_long_inc(&num_poisoned_pages);
486	}	603	}
487		604
488	static inline void num_poisoned_pages_dec(void)
489	{
490	atomic_long_dec(&num_poisoned_pages);
491	}
492
493	static inline void num_poisoned_pages_sub(long i)	605	static inline void num_poisoned_pages_sub(long i)
494	{	606	{
495	atomic_long_sub(i, &num_poisoned_pages);	607	atomic_long_sub(i, &num_poisoned_pages);
496	}	608	}
497		609
498	#else	610	#else /* CONFIG_MEMORY_FAILURE */
499		611
500	static inline swp_entry_t make_hwpoison_entry(struct page *page)	612	static inline swp_entry_t make_hwpoison_entry(struct page *page)
501	{	613	{
@@ -514,7 +626,7 @@ static inline void num_poisoned_pages_inc(void) · 12 unmodified lines
514	static inline void num_poisoned_pages_sub(long i)	626	static inline void num_poisoned_pages_sub(long i)
515	{	627	{
516	}	628	}
517	#endif	629	#endif /* CONFIG_MEMORY_FAILURE */
518		630
519	static inline int non_swap_entry(swp_entry_t entry)	631	static inline int non_swap_entry(swp_entry_t entry)
520	{	632	{

include/linux/uaccess.h+14−5

 static __always_inline __must_check unsigned long
 __copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)
 {
-	instrument_copy_from_user(to, from, n);
+	unsigned long res;
+	instrument_copy_from_user_before(to, from, n);
 	check_object_size(to, n, false);
-	return raw_copy_from_user(to, from, n);
+	res = raw_copy_from_user(to, from, n);
+	instrument_copy_from_user_after(to, from, n, res);
+	return res;
 }
 static __always_inline __must_check unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
+	unsigned long res;
 	might_fault();
+	instrument_copy_from_user_before(to, from, n);
 	if (should_fail_usercopy())
 		return n;
-	instrument_copy_from_user(to, from, n);
 	check_object_size(to, n, false);
-	return raw_copy_from_user(to, from, n);
+	res = raw_copy_from_user(to, from, n);
+	instrument_copy_from_user_after(to, from, n, res);
+	return res;
 }
 /**
 	unsigned long res = n;
 	might_fault();
 	if (!should_fail_usercopy() && likely(access_ok(from, n))) {
-		instrument_copy_from_user(to, from, n);
+		instrument_copy_from_user_before(to, from, n);
 		res = raw_copy_from_user(to, from, n);
+		instrument_copy_from_user_after(to, from, n, res);
 	}
 	if (unlikely(res))
 		memset(to + (n - res), 0, res);

include/linux/userfaultfd_k.h+3−4

 			       unsigned long start,
 			       unsigned long end);
-extern int userfaultfd_unmap_prep(struct vm_area_struct *vma,
+extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
-				  unsigned long start, unsigned long end,
+				  unsigned long end, struct list_head *uf);
-				  struct list_head *uf);
 extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 				       struct list_head *uf);
 	return true;
 }
-static inline int userfaultfd_unmap_prep(struct vm_area_struct *vma,
+static inline int userfaultfd_unmap_prep(struct mm_struct *mm,
 					 unsigned long start, unsigned long end,
 					 struct list_head *uf)
 {

include/linux/vm_event_item.h+0−4

 		NR_TLB_LOCAL_FLUSH_ALL,
 		NR_TLB_LOCAL_FLUSH_ONE,
 #endif /* CONFIG_DEBUG_TLBFLUSH */
-#ifdef CONFIG_DEBUG_VM_VMACACHE
-		VMACACHE_FIND_CALLS,
-		VMACACHE_FIND_HITS,
-#endif
 #ifdef CONFIG_SWAP
 		SWAP_RA,
 		SWAP_RA_HIT,

include/linux/vmacache.h+0−28

@@ -1,28 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 */
2	#ifndef __LINUX_VMACACHE_H
3	#define __LINUX_VMACACHE_H
4
5	#include <linux/sched.h>
6	#include <linux/mm.h>
7
8	static inline void vmacache_flush(struct task_struct *tsk)
9	{
10	memset(tsk->vmacache.vmas, 0, sizeof(tsk->vmacache.vmas));
11	}
12
13	extern void vmacache_update(unsigned long addr, struct vm_area_struct *newvma);
14	extern struct vm_area_struct vmacache_find(struct mm_struct mm,
15	unsigned long addr);
16
17	#ifndef CONFIG_MMU
18	extern struct vm_area_struct vmacache_find_exact(struct mm_struct mm,
19	unsigned long start,
20	unsigned long end);
21	#endif
22
23	static inline void vmacache_invalidate(struct mm_struct *mm)
24	{
25	mm->vmacache_seqnum++;
26	}
27
28	#endif /* __LINUX_VMACACHE_H */

include/linux/vmstat.h+0−6

 #define count_vm_tlb_events(x, y) do { (void)(y); } while (0)
 #endif
-#ifdef CONFIG_DEBUG_VM_VMACACHE
-#define count_vm_vmacache_event(x) count_vm_event(x)
-#else
-#define count_vm_vmacache_event(x) do {} while (0)
-#endif
 #define __count_zid_vm_events(item, zid, delta) \
 	__count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta)

include/linux/writeback.h+0−8

 DECLARE_PER_CPU(int, dirty_throttle_leaks);
 /*
- * The 1/4 region under the global dirty thresh is for smooth dirty throttling:
- *
- *	(thresh - thresh/DIRTY_FULL_SCOPE, thresh)
- *
- * Further beyond, all dirtier tasks will enter a loop waiting (possibly long
- * time) for the dirty pages to drop, unless written enough pages.
- *
  * The global dirty threshold is normally equal to the global dirty limit,
  * except when the system suddenly allocates a lot of anonymous memory and
  * knocks down the global dirty threshold quickly, in which case the global
  * dirty limit will follow down slowly to prevent livelocking all dirtier tasks.
  */
 #define DIRTY_SCOPE		8
-#define DIRTY_FULL_SCOPE	(DIRTY_SCOPE / 2)
 struct backing_dev_info;

include/trace/events/huge_memory.h+37−0

@@ -11,11 +11,14 @@ · 10 unmodified lines
11	EM( SCAN_FAIL, "failed") \	11	EM( SCAN_FAIL, "failed") \
12	EM( SCAN_SUCCEED, "succeeded") \	12	EM( SCAN_SUCCEED, "succeeded") \
13	EM( SCAN_PMD_NULL, "pmd_null") \	13	EM( SCAN_PMD_NULL, "pmd_null") \
		14	EM( SCAN_PMD_NONE, "pmd_none") \
		15	EM( SCAN_PMD_MAPPED, "page_pmd_mapped") \
14	EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \	16	EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \
15	EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \	17	EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \
16	EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \	18	EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \
17	EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \	19	EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \
18	EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \	20	EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \
		21	EM( SCAN_PTE_MAPPED_HUGEPAGE, "pte_mapped_hugepage") \
19	EM( SCAN_PAGE_RO, "no_writable_page") \	22	EM( SCAN_PAGE_RO, "no_writable_page") \
20	EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \	23	EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \
21	EM( SCAN_PAGE_NULL, "page_null") \	24	EM( SCAN_PAGE_NULL, "page_null") \
@@ -166,5 +169,39 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, · 144 unmodified lines
166	__entry->ret)	169	__entry->ret)
167	);	170	);
168		171
		172	TRACE_EVENT(mm_khugepaged_scan_file,
		173
		174	TP_PROTO(struct mm_struct mm, struct page page, const char *filename,
		175	int present, int swap, int result),
		176
		177	TP_ARGS(mm, page, filename, present, swap, result),
		178
		179	TP_STRUCT__entry(
		180	__field(struct mm_struct *, mm)
		181	__field(unsigned long, pfn)
		182	__string(filename, filename)
		183	__field(int, present)
		184	__field(int, swap)
		185	__field(int, result)
		186	),
		187
		188	TP_fast_assign(
		189	__entry->mm = mm;
		190	__entry->pfn = page ? page_to_pfn(page) : -1;
		191	__assign_str(filename, filename);
		192	__entry->present = present;
		193	__entry->swap = swap;
		194	__entry->result = result;
		195	),
		196
		197	TP_printk("mm=%p, scan_pfn=0x%lx, filename=%s, present=%d, swap=%d, result=%s",
		198	__entry->mm,
		199	__entry->pfn,
		200	__get_str(filename),
		201	__entry->present,
		202	__entry->swap,
		203	__print_symbolic(__entry->result, SCAN_STATUS))
		204	);
		205
169	#endif /* __HUGE_MEMORY_H */	206	#endif /* __HUGE_MEMORY_H */
170	#include <trace/define_trace.h>	207	#include <trace/define_trace.h>

include/trace/events/maple_tree.h+123−0

@@ -0,0 +1,123 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#undef TRACE_SYSTEM
	3	#define TRACE_SYSTEM maple_tree
	4
	5	#if !defined(_TRACE_MM_H) \|\| defined(TRACE_HEADER_MULTI_READ)
	6	#define _TRACE_MM_H
	7
	8
	9	#include <linux/tracepoint.h>
	10
	11	struct ma_state;
	12
	13	TRACE_EVENT(ma_op,
	14
	15	TP_PROTO(const char fn, struct ma_state mas),
	16
	17	TP_ARGS(fn, mas),
	18
	19	TP_STRUCT__entry(
	20	__field(const char *, fn)
	21	__field(unsigned long, min)
	22	__field(unsigned long, max)
	23	__field(unsigned long, index)
	24	__field(unsigned long, last)
	25	__field(void *, node)
	26	),
	27
	28	TP_fast_assign(
	29	__entry->fn = fn;
	30	__entry->min = mas->min;
	31	__entry->max = mas->max;
	32	__entry->index = mas->index;
	33	__entry->last = mas->last;
	34	__entry->node = mas->node;
	35	),
	36
	37	TP_printk("%s\tNode: %p (%lu %lu) range: %lu-%lu",
	38	__entry->fn,
	39	(void *) __entry->node,
	40	(unsigned long) __entry->min,
	41	(unsigned long) __entry->max,
	42	(unsigned long) __entry->index,
	43	(unsigned long) __entry->last
	44	)
	45	)
	46	TRACE_EVENT(ma_read,
	47
	48	TP_PROTO(const char fn, struct ma_state mas),
	49
	50	TP_ARGS(fn, mas),
	51
	52	TP_STRUCT__entry(
	53	__field(const char *, fn)
	54	__field(unsigned long, min)
	55	__field(unsigned long, max)
	56	__field(unsigned long, index)
	57	__field(unsigned long, last)
	58	__field(void *, node)
	59	),
	60
	61	TP_fast_assign(
	62	__entry->fn = fn;
	63	__entry->min = mas->min;
	64	__entry->max = mas->max;
	65	__entry->index = mas->index;
	66	__entry->last = mas->last;
	67	__entry->node = mas->node;
	68	),
	69
	70	TP_printk("%s\tNode: %p (%lu %lu) range: %lu-%lu",
	71	__entry->fn,
	72	(void *) __entry->node,
	73	(unsigned long) __entry->min,
	74	(unsigned long) __entry->max,
	75	(unsigned long) __entry->index,
	76	(unsigned long) __entry->last
	77	)
	78	)
	79
	80	TRACE_EVENT(ma_write,
	81
	82	TP_PROTO(const char fn, struct ma_state mas, unsigned long piv,
	83	void *val),
	84
	85	TP_ARGS(fn, mas, piv, val),
	86
	87	TP_STRUCT__entry(
	88	__field(const char *, fn)
	89	__field(unsigned long, min)
	90	__field(unsigned long, max)
	91	__field(unsigned long, index)
	92	__field(unsigned long, last)
	93	__field(unsigned long, piv)
	94	__field(void *, val)
	95	__field(void *, node)
	96	),
	97
	98	TP_fast_assign(
	99	__entry->fn = fn;
	100	__entry->min = mas->min;
	101	__entry->max = mas->max;
	102	__entry->index = mas->index;
	103	__entry->last = mas->last;
	104	__entry->piv = piv;
	105	__entry->val = val;
	106	__entry->node = mas->node;
	107	),
	108
	109	TP_printk("%s\tNode %p (%lu %lu) range:%lu-%lu piv (%lu) val %p",
	110	__entry->fn,
	111	(void *) __entry->node,
	112	(unsigned long) __entry->min,
	113	(unsigned long) __entry->max,
	114	(unsigned long) __entry->index,
	115	(unsigned long) __entry->last,
	116	(unsigned long) __entry->piv,
	117	(void *) __entry->val
	118	)
	119	)
	120	#endif /* _TRACE_MM_H */
	121
	122	/* This part must be outside protection */
	123	#include <trace/define_trace.h>

include/trace/events/mmap.h+73−0

@@ -42,6 +42,79 @@ TRACE_EVENT(vm_unmapped_area, · 41 unmodified lines
42	__entry->low_limit, __entry->high_limit, __entry->align_mask,	42	__entry->low_limit, __entry->high_limit, __entry->align_mask,
43	__entry->align_offset)	43	__entry->align_offset)
44	);	44	);
		45
		46	TRACE_EVENT(vma_mas_szero,
		47	TP_PROTO(struct maple_tree *mt, unsigned long start,
		48	unsigned long end),
		49
		50	TP_ARGS(mt, start, end),
		51
		52	TP_STRUCT__entry(
		53	__field(struct maple_tree *, mt)
		54	__field(unsigned long, start)
		55	__field(unsigned long, end)
		56	),
		57
		58	TP_fast_assign(
		59	__entry->mt = mt;
		60	__entry->start = start;
		61	__entry->end = end;
		62	),
		63
		64	TP_printk("mt_mod %p, (NULL), SNULL, %lu, %lu,",
		65	__entry->mt,
		66	(unsigned long) __entry->start,
		67	(unsigned long) __entry->end
		68	)
		69	);
		70
		71	TRACE_EVENT(vma_store,
		72	TP_PROTO(struct maple_tree mt, struct vm_area_struct vma),
		73
		74	TP_ARGS(mt, vma),
		75
		76	TP_STRUCT__entry(
		77	__field(struct maple_tree *, mt)
		78	__field(struct vm_area_struct *, vma)
		79	__field(unsigned long, vm_start)
		80	__field(unsigned long, vm_end)
		81	),
		82
		83	TP_fast_assign(
		84	__entry->mt = mt;
		85	__entry->vma = vma;
		86	__entry->vm_start = vma->vm_start;
		87	__entry->vm_end = vma->vm_end - 1;
		88	),
		89
		90	TP_printk("mt_mod %p, (%p), STORE, %lu, %lu,",
		91	__entry->mt, __entry->vma,
		92	(unsigned long) __entry->vm_start,
		93	(unsigned long) __entry->vm_end
		94	)
		95	);
		96
		97
		98	TRACE_EVENT(exit_mmap,
		99	TP_PROTO(struct mm_struct *mm),
		100
		101	TP_ARGS(mm),
		102
		103	TP_STRUCT__entry(
		104	__field(struct mm_struct *, mm)
		105	__field(struct maple_tree *, mt)
		106	),
		107
		108	TP_fast_assign(
		109	__entry->mm = mm;
		110	__entry->mt = &mm->mm_mt;
		111	),
		112
		113	TP_printk("mt_mod %p, DESTROY\n",
		114	__entry->mt
		115	)
		116	);
		117
45	#endif	118	#endif
46		119
47	/* This part must be outside protection */	120	/* This part must be outside protection */

include/uapi/asm-generic/hugetlb_encode.h+13−13

@@ -20,18 +20,18 @@ · 19 unmodified lines
20	#define HUGETLB_FLAG_ENCODE_SHIFT 26	20	#define HUGETLB_FLAG_ENCODE_SHIFT 26
21	#define HUGETLB_FLAG_ENCODE_MASK 0x3f	21	#define HUGETLB_FLAG_ENCODE_MASK 0x3f
22		22
23	#define HUGETLB_FLAG_ENCODE_16KB (14 << HUGETLB_FLAG_ENCODE_SHIFT)	23	#define HUGETLB_FLAG_ENCODE_16KB (14U << HUGETLB_FLAG_ENCODE_SHIFT)
24	#define HUGETLB_FLAG_ENCODE_64KB (16 << HUGETLB_FLAG_ENCODE_SHIFT)	24	#define HUGETLB_FLAG_ENCODE_64KB (16U << HUGETLB_FLAG_ENCODE_SHIFT)
25	#define HUGETLB_FLAG_ENCODE_512KB (19 << HUGETLB_FLAG_ENCODE_SHIFT)	25	#define HUGETLB_FLAG_ENCODE_512KB (19U << HUGETLB_FLAG_ENCODE_SHIFT)
26	#define HUGETLB_FLAG_ENCODE_1MB (20 << HUGETLB_FLAG_ENCODE_SHIFT)	26	#define HUGETLB_FLAG_ENCODE_1MB (20U << HUGETLB_FLAG_ENCODE_SHIFT)
27	#define HUGETLB_FLAG_ENCODE_2MB (21 << HUGETLB_FLAG_ENCODE_SHIFT)	27	#define HUGETLB_FLAG_ENCODE_2MB (21U << HUGETLB_FLAG_ENCODE_SHIFT)
28	#define HUGETLB_FLAG_ENCODE_8MB (23 << HUGETLB_FLAG_ENCODE_SHIFT)	28	#define HUGETLB_FLAG_ENCODE_8MB (23U << HUGETLB_FLAG_ENCODE_SHIFT)
29	#define HUGETLB_FLAG_ENCODE_16MB (24 << HUGETLB_FLAG_ENCODE_SHIFT)	29	#define HUGETLB_FLAG_ENCODE_16MB (24U << HUGETLB_FLAG_ENCODE_SHIFT)
30	#define HUGETLB_FLAG_ENCODE_32MB (25 << HUGETLB_FLAG_ENCODE_SHIFT)	30	#define HUGETLB_FLAG_ENCODE_32MB (25U << HUGETLB_FLAG_ENCODE_SHIFT)
31	#define HUGETLB_FLAG_ENCODE_256MB (28 << HUGETLB_FLAG_ENCODE_SHIFT)	31	#define HUGETLB_FLAG_ENCODE_256MB (28U << HUGETLB_FLAG_ENCODE_SHIFT)
32	#define HUGETLB_FLAG_ENCODE_512MB (29 << HUGETLB_FLAG_ENCODE_SHIFT)	32	#define HUGETLB_FLAG_ENCODE_512MB (29U << HUGETLB_FLAG_ENCODE_SHIFT)
33	#define HUGETLB_FLAG_ENCODE_1GB (30 << HUGETLB_FLAG_ENCODE_SHIFT)	33	#define HUGETLB_FLAG_ENCODE_1GB (30U << HUGETLB_FLAG_ENCODE_SHIFT)
34	#define HUGETLB_FLAG_ENCODE_2GB (31 << HUGETLB_FLAG_ENCODE_SHIFT)	34	#define HUGETLB_FLAG_ENCODE_2GB (31U << HUGETLB_FLAG_ENCODE_SHIFT)
35	#define HUGETLB_FLAG_ENCODE_16GB (34 << HUGETLB_FLAG_ENCODE_SHIFT)	35	#define HUGETLB_FLAG_ENCODE_16GB (34U << HUGETLB_FLAG_ENCODE_SHIFT)
36		36
37	#endif /* _ASM_GENERIC_HUGETLB_ENCODE_H_ */	37	#endif /* _ASM_GENERIC_HUGETLB_ENCODE_H_ */

include/uapi/asm-generic/mman-common.h+2−0

@@ -77,6 +77,8 @@ · 76 unmodified lines
77		77
78	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */	78	#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
79		79
		80	#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
		81
80	/* compatibility flags */	82	/* compatibility flags */
81	#define MAP_FILE 0	83	#define MAP_FILE 0
82		84

include/uapi/linux/userfaultfd.h+4−0

@@ -12,6 +12,10 @@ · 11 unmodified lines
12		12
13	#include <linux/types.h>	13	#include <linux/types.h>
14		14
		15	/* ioctls for /dev/userfaultfd */
		16	#define USERFAULTFD_IOC 0xAA
		17	#define USERFAULTFD_IOC_NEW _IO(USERFAULTFD_IOC, 0x00)
		18
15	/*	19	/*
16	* If the UFFDIO_API is upgraded someday, the UFFDIO_UNREGISTER and	20	* If the UFFDIO_API is upgraded someday, the UFFDIO_UNREGISTER and
17	* UFFDIO_WAKE ioctls should be defined as _IOW and not as _IOR. In	21	* UFFDIO_WAKE ioctls should be defined as _IOW and not as _IOR. In

init/Kconfig+0−5

 	help
 	  Provides control over the memory footprint of tasks in a cgroup.
-config MEMCG_SWAP
-	bool
-	depends on MEMCG && SWAP
-	default y
 config MEMCG_KMEM
 	bool
 	depends on MEMCG && !SLOB

init/main.c+10−1

 #include <linux/percpu.h>
 #include <linux/kmod.h>
 #include <linux/kprobes.h>
+#include <linux/kmsan.h>
 #include <linux/vmalloc.h>
 #include <linux/kernel_stat.h>
 #include <linux/start_kernel.h>
 extern void init_IRQ(void);
 extern void radix_tree_init(void);
+extern void maple_tree_init(void);
 /*
  * Debug helper: via this flag we know that we are in 'early bootup code'
 	init_mem_debugging_and_hardening();
 	kfence_alloc_pool();
 	report_meminit();
+	kmsan_init_shadow();
 	stack_depot_early_init();
 	mem_init();
 	mem_init_print_info();
 	pgtable_init();
 	debug_objects_mem_init();
 	vmalloc_init();
+	/* Should be run after vmap initialization */
+	if (early_page_ext_enabled())
+		page_ext_init();
 	/* Should be run before the first non-init thread is created */
 	init_espfix_bsp();
 	/* Should be run after espfix64 is set up. */
 	pti_init();
+	kmsan_init_runtime();
 }
 #ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET
 		 "Interrupts were enabled *very* early, fixing it\n"))
 		local_irq_disable();
 	radix_tree_init();
+	maple_tree_init();
 	/*
 	 * Set up housekeeping before setting up workqueues to allow the unbound
 	padata_init();
 	page_alloc_init_late();
 	/* Initialize page ext after all struct pages are initialized. */
-	page_ext_init();
+	if (!early_page_ext_enabled())
+		page_ext_init();
 	do_basic_setup();

ipc/shm.c+10−11

 #ifdef CONFIG_MMU
 	loff_t size = 0;
 	struct file *file;
-	struct vm_area_struct *next;
+	VMA_ITERATOR(vmi, mm, addr);
 #endif
 	if (addr & ~PAGE_MASK)
 	 * match the usual checks anyway. So assume all vma's are
 	 * above the starting address given.
 	 */
-	vma = find_vma(mm, addr);
 #ifdef CONFIG_MMU
-	while (vma) {
+	for_each_vma(vmi, vma) {
-		next = vma->vm_next;
 		/*
 		 * Check if the starting address would match, i.e. it's
 		 * a fragment created by mprotect() and/or munmap(), or it
 			file = vma->vm_file;
 			size = i_size_read(file_inode(vma->vm_file));
 			do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start, NULL);
+			mas_pause(&vmi.mas);
 			/*
 			 * We discovered the size of the shm segment, so
 			 * break out of here and fall through to the next
 			 * searching for matching vma's.
 			 */
 			retval = 0;
-			vma = next;
+			vma = vma_next(&vmi);
 			break;
 		}
-		vma = next;
 	}
 	/*
 	 */
 	size = PAGE_ALIGN(size);
 	while (vma && (loff_t)(vma->vm_end - addr) <= size) {
-		next = vma->vm_next;
 		/* finding a matching vma now does not alter retval */
 		if ((vma->vm_ops == &shm_vm_ops) &&
 		    ((vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) &&
-		    (vma->vm_file == file))
+		    (vma->vm_file == file)) {
 			do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start, NULL);
-		vma = next;
+			mas_pause(&vmi.mas);
+		}
+		vma = vma_next(&vmi);
 	}
 #else	/* CONFIG_MMU */
+	vma = vma_lookup(mm, addr);
 	/* under NOMMU conditions, the exact address to be destroyed must be
 	 * given
 	 */

kernel/Makefile+1−0

@@ -38,6 +38,7 @@ KCOV_INSTRUMENT_kcov.o := n · 37 unmodified lines
38	KASAN_SANITIZE_kcov.o := n	38	KASAN_SANITIZE_kcov.o := n
39	KCSAN_SANITIZE_kcov.o := n	39	KCSAN_SANITIZE_kcov.o := n
40	UBSAN_SANITIZE_kcov.o := n	40	UBSAN_SANITIZE_kcov.o := n
		41	KMSAN_SANITIZE_kcov.o := n
41	CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) -fno-stack-protector	42	CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) -fno-stack-protector
42		43
43	# Don't instrument error handlers	44	# Don't instrument error handlers

kernel/acct.c+5−6

 	unsigned long vsize = 0;
 	if (group_dead && current->mm) {
+		struct mm_struct *mm = current->mm;
+		VMA_ITERATOR(vmi, mm, 0);
 		struct vm_area_struct *vma;
-		mmap_read_lock(current->mm);
+		mmap_read_lock(mm);
-		vma = current->mm->mmap;
+		for_each_vma(vmi, vma)
-		while (vma) {
 			vsize += vma->vm_end - vma->vm_start;
-			vma = vma->vm_next;
+		mmap_read_unlock(mm);
-		}
-		mmap_read_unlock(current->mm);
 	}
 	spin_lock_irq(&current->sighand->siglock);

kernel/bounds.c+7−0

@@ -22,6 +22,13 @@ int main(void) · 21 unmodified lines
22	DEFINE(NR_CPUS_BITS, ilog2(CONFIG_NR_CPUS));	22	DEFINE(NR_CPUS_BITS, ilog2(CONFIG_NR_CPUS));
23	#endif	23	#endif
24	DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));	24	DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
		25	#ifdef CONFIG_LRU_GEN
		26	DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1));
		27	DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2);
		28	#else
		29	DEFINE(LRU_GEN_WIDTH, 0);
		30	DEFINE(__LRU_REFS_WIDTH, 0);
		31	#endif
25	/* End of constants */	32	/* End of constants */
26		33
27	return 0;	34	return 0;

kernel/bpf/core.c+1−1

@@ -2007,7 +2007,7 @@ out: · 2006 unmodified lines
2007	static unsigned int PROG_NAME(stack_size)(const void ctx, const struct bpf_insn insn) \	2007	static unsigned int PROG_NAME(stack_size)(const void ctx, const struct bpf_insn insn) \
2008	{ \	2008	{ \
2009	u64 stack[stack_size / sizeof(u64)]; \	2009	u64 stack[stack_size / sizeof(u64)]; \
2010	u64 regs[MAX_BPF_EXT_REG]; \	2010	u64 regs[MAX_BPF_EXT_REG] = {}; \
2011	\	2011	\
2012	FP = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)]; \	2012	FP = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)]; \
2013	ARG1 = (u64) (unsigned long) ctx; \	2013	ARG1 = (u64) (unsigned long) ctx; \

kernel/bpf/task_iter.c+5−5

@@ -445,8 +445,8 @@ struct bpf_iter_seq_task_vma_info { · 444 unmodified lines
445	};	445	};
446		446
447	enum bpf_task_vma_iter_find_op {	447	enum bpf_task_vma_iter_find_op {
448	task_vma_iter_first_vma, /* use mm->mmap */	448	task_vma_iter_first_vma, /* use find_vma() with addr 0 */
449	task_vma_iter_next_vma, /* use curr_vma->vm_next */	449	task_vma_iter_next_vma, /* use vma_next() with curr_vma */
450	task_vma_iter_find_vma, /* use find_vma() to find next vma */	450	task_vma_iter_find_vma, /* use find_vma() to find next vma */
451	};	451	};
452		452
@@ -544,10 +544,10 @@ again: · 91 unmodified lines
544		544
545	switch (op) {	545	switch (op) {
546	case task_vma_iter_first_vma:	546	case task_vma_iter_first_vma:
547	curr_vma = curr_task->mm->mmap;	547	curr_vma = find_vma(curr_task->mm, 0);
548	break;	548	break;
549	case task_vma_iter_next_vma:	549	case task_vma_iter_next_vma:
550	curr_vma = curr_vma->vm_next;	550	curr_vma = find_vma(curr_task->mm, curr_vma->vm_end);
551	break;	551	break;
552	case task_vma_iter_find_vma:	552	case task_vma_iter_find_vma:
553	/* We dropped mmap_lock so it is necessary to use find_vma	553	/* We dropped mmap_lock so it is necessary to use find_vma
@@ -561,7 +561,7 @@ again: · 7 unmodified lines
561	if (curr_vma &&	561	if (curr_vma &&
562	curr_vma->vm_start == info->prev_vm_start &&	562	curr_vma->vm_start == info->prev_vm_start &&
563	curr_vma->vm_end == info->prev_vm_end)	563	curr_vma->vm_end == info->prev_vm_end)
564	curr_vma = curr_vma->vm_next;	564	curr_vma = find_vma(curr_task->mm, curr_vma->vm_end);
565	break;	565	break;
566	}	566	}
567	if (!curr_vma) {	567	if (!curr_vma) {

kernel/cgroup/cgroup-internal.h+0−1

@@ -164,7 +164,6 @@ struct cgroup_mgctx { · 163 unmodified lines
164	#define DEFINE_CGROUP_MGCTX(name) \	164	#define DEFINE_CGROUP_MGCTX(name) \
165	struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name)	165	struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name)
166		166
167	extern struct mutex cgroup_mutex;
168	extern spinlock_t css_set_lock;	167	extern spinlock_t css_set_lock;
169	extern struct cgroup_subsys *cgroup_subsys[];	168	extern struct cgroup_subsys *cgroup_subsys[];
170	extern struct list_head cgroup_roots;	169	extern struct list_head cgroup_roots;

kernel/debug/debug_core.c+0−12

 #include <linux/pid.h>
 #include <linux/smp.h>
 #include <linux/mm.h>
-#include <linux/vmacache.h>
 #include <linux/rcupdate.h>
 #include <linux/irq.h>
 #include <linux/security.h>
 	if (!CACHE_FLUSH_IS_SAFE)
 		return;
-	if (current->mm) {
-		int i;
-		for (i = 0; i < VMACACHE_SIZE; i++) {
-			if (!current->vmacache.vmas[i])
-				continue;
-			flush_cache_range(current->vmacache.vmas[i],
-					  addr, addr + BREAK_INSTR_SIZE);
-		}
-	}
 	/* Force flush instruction cache if it was outside the mm */
 	flush_icache_range(addr, addr + BREAK_INSTR_SIZE);
 }

kernel/delayacct.c+11−2

 		      &current->delays->freepages_count);
 }
-void __delayacct_thrashing_start(void)
+void __delayacct_thrashing_start(bool *in_thrashing)
 {
+	*in_thrashing = !!current->in_thrashing;
+	if (*in_thrashing)
+		return;
+	current->in_thrashing = 1;
 	current->delays->thrashing_start = local_clock();
 }
-void __delayacct_thrashing_end(void)
+void __delayacct_thrashing_end(bool *in_thrashing)
 {
+	if (*in_thrashing)
+		return;
+	current->in_thrashing = 0;
 	delayacct_end(&current->delays->lock,
 		      &current->delays->thrashing_start,
 		      &current->delays->thrashing_delay,

kernel/dma/mapping.c+7−3

 #include <linux/dma-map-ops.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/kmsan.h>
 #include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 		addr = dma_direct_map_page(dev, page, offset, size, dir, attrs);
 	else
 		addr = ops->map_page(dev, page, offset, size, dir, attrs);
+	kmsan_handle_dma(page, offset, size, dir);
 	debug_dma_map_page(dev, page, offset, size, dir, addr, attrs);
 	return addr;
 	else
 		ents = ops->map_sg(dev, sg, nents, dir, attrs);
-	if (ents > 0)
+	if (ents > 0) {
+		kmsan_handle_dma_sg(sg, nents, dir);
 		debug_dma_map_sg(dev, sg, nents, ents, dir, attrs);
-	else if (WARN_ON_ONCE(ents != -EINVAL && ents != -ENOMEM &&
+	} else if (WARN_ON_ONCE(ents != -EINVAL && ents != -ENOMEM &&
-			      ents != -EIO && ents != -EREMOTEIO))
+				ents != -EIO && ents != -EREMOTEIO)) {
 		return -EIO;
+	}
 	return ents;
 }

kernel/entry/common.c+5−0

 #include <linux/resume_user_mode.h>
 #include <linux/highmem.h>
 #include <linux/jump_label.h>
+#include <linux/kmsan.h>
 #include <linux/livepatch.h>
 #include <linux/audit.h>
 #include <linux/tick.h>
 	user_exit_irqoff();
 	instrumentation_begin();
+	kmsan_unpoison_entry_regs(regs);
 	trace_hardirqs_off_finish();
 	instrumentation_end();
 }
 		lockdep_hardirqs_off(CALLER_ADDR0);
 		ct_irq_enter();
 		instrumentation_begin();
+		kmsan_unpoison_entry_regs(regs);
 		trace_hardirqs_off_finish();
 		instrumentation_end();
 	 */
 	lockdep_hardirqs_off(CALLER_ADDR0);
 	instrumentation_begin();
+	kmsan_unpoison_entry_regs(regs);
 	rcu_irq_enter_check_tick();
 	trace_hardirqs_off_finish();
 	instrumentation_end();
 	ct_nmi_enter();
 	instrumentation_begin();
+	kmsan_unpoison_entry_regs(regs);
 	trace_hardirqs_off_finish();
 	ftrace_nmi_enter();
 	instrumentation_end();

kernel/events/core.c+2−1

@@ -10270,8 +10270,9 @@ static void perf_addr_filter_apply(struct perf_addr_filter *filter, · 10269 unmodified lines
10270	struct perf_addr_filter_range *fr)	10270	struct perf_addr_filter_range *fr)
10271	{	10271	{
10272	struct vm_area_struct *vma;	10272	struct vm_area_struct *vma;
		10273	VMA_ITERATOR(vmi, mm, 0);
10273		10274
10274	for (vma = mm->mmap; vma; vma = vma->vm_next) {	10275	for_each_vma(vmi, vma) {
10275	if (!vma->vm_file)	10276	if (!vma->vm_file)
10276	continue;	10277	continue;
10277		10278

kernel/events/uprobes.c+22−17

 #include <linux/export.h>
 #include <linux/rmap.h>		/* anon_vma_prepare */
 #include <linux/mmu_notifier.h>	/* set_pte_at_notify */
-#include <linux/swap.h>		/* try_to_free_swap */
+#include <linux/swap.h>		/* folio_free_swap */
 #include <linux/ptrace.h>	/* user_enable_single_step */
 #include <linux/kdebug.h>	/* notifier mechanism */
 #include "../../mm/internal.h"	/* munlock_vma_page */
 static int __replace_page(struct vm_area_struct *vma, unsigned long addr,
 				struct page *old_page, struct page *new_page)
 {
+	struct folio *old_folio = page_folio(old_page);
+	struct folio *new_folio;
 	struct mm_struct *mm = vma->vm_mm;
-	DEFINE_FOLIO_VMA_WALK(pvmw, page_folio(old_page), vma, addr, 0);
+	DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0);
 	int err;
 	struct mmu_notifier_range range;
 				addr + PAGE_SIZE);
 	if (new_page) {
-		err = mem_cgroup_charge(page_folio(new_page), vma->vm_mm,
+		new_folio = page_folio(new_page);
-					GFP_KERNEL);
+		err = mem_cgroup_charge(new_folio, vma->vm_mm, GFP_KERNEL);
 		if (err)
 			return err;
 	}
-	/* For try_to_free_swap() below */
+	/* For folio_free_swap() below */
-	lock_page(old_page);
+	folio_lock(old_folio);
 	mmu_notifier_invalidate_range_start(&range);
 	err = -EAGAIN;
 	VM_BUG_ON_PAGE(addr != pvmw.address, old_page);
 	if (new_page) {
-		get_page(new_page);
+		folio_get(new_folio);
 		page_add_new_anon_rmap(new_page, vma, addr);
-		lru_cache_add_inactive_or_unevictable(new_page, vma);
+		folio_add_lru_vma(new_folio, vma);
 	} else
 		/* no new page, just dec_mm_counter for old_page */
 		dec_mm_counter(mm, MM_ANONPAGES);
-	if (!PageAnon(old_page)) {
+	if (!folio_test_anon(old_folio)) {
 		dec_mm_counter(mm, mm_counter_file(old_page));
 		inc_mm_counter(mm, MM_ANONPAGES);
 	}
 				  mk_pte(new_page, vma->vm_page_prot));
 	page_remove_rmap(old_page, vma, false);
-	if (!page_mapped(old_page))
+	if (!folio_mapped(old_folio))
-		try_to_free_swap(old_page);
+		folio_free_swap(old_folio);
 	page_vma_mapped_walk_done(&pvmw);
-	put_page(old_page);
+	folio_put(old_folio);
 	err = 0;
  unlock:
 	mmu_notifier_invalidate_range_end(&range);
-	unlock_page(old_page);
+	folio_unlock(old_folio);
 	return err;
 }
 static struct vm_area_struct *
 find_ref_ctr_vma(struct uprobe *uprobe, struct mm_struct *mm)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *tmp;
-	for (tmp = mm->mmap; tmp; tmp = tmp->vm_next)
+	for_each_vma(vmi, tmp)
 		if (valid_ref_ctr_vma(uprobe, tmp))
 			return tmp;
 	/* try collapse pmd for compound page */
 	if (!ret && orig_page_huge)
-		collapse_pte_mapped_thp(mm, vaddr);
+		collapse_pte_mapped_thp(mm, vaddr, false);
 	return ret;
 }
 static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
 	int err = 0;
 	mmap_read_lock(mm);
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		unsigned long vaddr;
 		loff_t offset;
 static void mmf_recalc_uprobes(struct mm_struct *mm)
 {
+	VMA_ITERATOR(vmi, mm, 0);
 	struct vm_area_struct *vma;
-	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+	for_each_vma(vmi, vma) {
 		if (!valid_vma(vma, false))
 			continue;
 		/*

kernel/exit.c+3−0

 #include <linux/writeback.h>
 #include <linux/shm.h>
 #include <linux/kcov.h>
+#include <linux/kmsan.h>
 #include <linux/random.h>
 #include <linux/rcuwait.h>
 #include <linux/compat.h>
 		goto retry;
 	}
 	WRITE_ONCE(mm->owner, c);
+	lru_gen_migrate_mm(mm);
 	task_unlock(c);
 	put_task_struct(c);
 }
 	WARN_ON(tsk->plug);
 	kcov_task_exit(tsk);
+	kmsan_task_exit(tsk);
 	coredump_task_exit(tsk);
 	ptrace_event(PTRACE_EVENT_EXIT, code);

kernel/fork.c+43−36

 #include <linux/fdtable.h>
 #include <linux/iocontext.h>
 #include <linux/key.h>
+#include <linux/kmsan.h>
 #include <linux/binfmts.h>
 #include <linux/mman.h>
 #include <linux/mmu_notifier.h>
 #include <linux/fs.h>
 #include <linux/mm.h>
 #include <linux/mm_inline.h>
-#include <linux/vmacache.h>
 #include <linux/nsproxy.h>
 #include <linux/capability.h>
 #include <linux/cpu.h>
 		 */
 		*new = data_race(*orig);
 		INIT_LIST_HEAD(&new->anon_vma_chain);
-		new->vm_next = new->vm_prev = NULL;
 		dup_anon_vma_name(orig, new);
 	}
 	return new;
 static __latent_entropy int dup_mmap(struct mm_struct *mm,
 					struct mm_struct *oldmm)
 {
-	struct vm_area_struct *mpnt, *tmp, *prev, **pprev;
+	struct vm_area_struct *mpnt, *tmp;
-	struct rb_node **rb_link, *rb_parent;
 	int retval;
-	unsigned long charge;
+	unsigned long charge = 0;
 	LIST_HEAD(uf);
+	MA_STATE(old_mas, &oldmm->mm_mt, 0, 0);
+	MA_STATE(mas, &mm->mm_mt, 0, 0);
 	uprobe_start_dup_mmap();
 	if (mmap_write_lock_killable(oldmm)) {
 	mm->exec_vm = oldmm->exec_vm;
 	mm->stack_vm = oldmm->stack_vm;
-	rb_link = &mm->mm_rb.rb_node;
-	rb_parent = NULL;
-	pprev = &mm->mmap;
 	retval = ksm_fork(mm, oldmm);
 	if (retval)
 		goto out;
 	khugepaged_fork(mm, oldmm);
-	prev = NULL;
+	retval = mas_expected_entries(&mas, oldmm->map_count);
-	for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {
+	if (retval)
+		goto out;
+	mas_for_each(&old_mas, mpnt, ULONG_MAX) {
 		struct file *file;
 		if (mpnt->vm_flags & VM_DONTCOPY) {
 		 */
 		if (fatal_signal_pending(current)) {
 			retval = -EINTR;
-			goto out;
+			goto loop_out;
 		}
 		if (mpnt->vm_flags & VM_ACCOUNT) {
 			unsigned long len = vma_pages(mpnt);
 		}
 		/*
-		 * Clear hugetlb-related page reserves for children. This only
+		 * Copy/update hugetlb private vma information.
-		 * affects MAP_PRIVATE mappings. Faults generated by the child
-		 * are not guaranteed to succeed, even if read-only
 		 */
 		if (is_vm_hugetlb_page(tmp))
-			reset_vma_resv_huge_pages(tmp);
+			hugetlb_dup_vma_private(tmp);
-		/*
-		 * Link in the new vma and copy the page table entries.
-		 */
-		*pprev = tmp;
-		pprev = &tmp->vm_next;
-		tmp->vm_prev = prev;
-		prev = tmp;
-		__vma_link_rb(mm, tmp, rb_link, rb_parent);
+		/* Link the vma into the MT */
-		rb_link = &tmp->vm_rb.rb_right;
+		mas.index = tmp->vm_start;
-		rb_parent = &tmp->vm_rb;
+		mas.last = tmp->vm_end - 1;
+		mas_store(&mas, tmp);
+		if (mas_is_err(&mas))
+			goto fail_nomem_mas_store;
 		mm->map_count++;
 		if (!(tmp->vm_flags & VM_WIPEONFORK))
 			tmp->vm_ops->open(tmp);
 		if (retval)
-			goto out;
+			goto loop_out;
 	}
 	/* a new mm has just been created */
 	retval = arch_dup_mmap(oldmm, mm);
+loop_out:
+	mas_destroy(&mas);
 out:
 	mmap_write_unlock(mm);
 	flush_tlb_mm(oldmm);
 fail_uprobe_end:
 	uprobe_end_dup_mmap();
 	return retval;
+fail_nomem_mas_store:
+	unlink_anon_vmas(tmp);
 fail_nomem_anon_vma_fork:
 	mpol_put(vma_policy(tmp));
 fail_nomem_policy:
 fail_nomem:
 	retval = -ENOMEM;
 	vm_unacct_memory(charge);
-	goto out;
+	goto loop_out;
 }
 static inline int mm_alloc_pgd(struct mm_struct *mm)
 	tsk->worker_private = NULL;
 	kcov_task_init(tsk);
+	kmsan_task_create(tsk);
 	kmap_local_fork(tsk);
 #ifdef CONFIG_FAULT_INJECTION
 static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
 	struct user_namespace *user_ns)
 {
-	mm->mmap = NULL;
+	mt_init_flags(&mm->mm_mt, MM_MT_FLAGS);
-	mm->mm_rb = RB_ROOT;
+	mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock);
-	mm->vmacache_seqnum = 0;
 	atomic_set(&mm->mm_users, 1);
 	atomic_set(&mm->mm_count, 1);
 	seqcount_init(&mm->write_protect_seq);
 		goto fail_nocontext;
 	mm->user_ns = get_user_ns(user_ns);
+	lru_gen_init_mm(mm);
 	return mm;
 fail_nocontext:
 	}
 	if (mm->binfmt)
 		module_put(mm->binfmt->module);
+	lru_gen_del_mm(mm);
 	mmdrop(mm);
 }
 	/* Forbid mm->exe_file change if old file still mapped. */
 	old_exe_file = get_mm_exe_file(mm);
 	if (old_exe_file) {
+		VMA_ITERATOR(vmi, mm, 0);
 		mmap_read_lock(mm);
-		for (vma = mm->mmap; vma && !ret; vma = vma->vm_next) {
+		for_each_vma(vmi, vma) {
 			if (!vma->vm_file)
 				continue;
 			if (path_equal(&vma->vm_file->f_path,
-				       &old_exe_file->f_path))
+				       &old_exe_file->f_path)) {
 				ret = -EBUSY;
+				break;
+			}
 		}
 		mmap_read_unlock(mm);
 		fput(old_exe_file);
 	if (!oldmm)
 		return 0;
-	/* initialize the new vmacache entries */
-	vmacache_flush(tsk);
 	if (clone_flags & CLONE_VM) {
 		mmget(oldmm);
 		mm = oldmm;
 		get_task_struct(p);
 	}
+	if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) {
+		/* lock the task to synchronize with memcg migration */
+		task_lock(p);
+		lru_gen_add_mm(p->mm);
+		task_unlock(p);
+	}
 	wake_up_new_task(p);
 	/* forking complete and child started to run, tell ptracer */

kernel/kcov.c+7−0

 #include <linux/fs.h>
 #include <linux/hashtable.h>
 #include <linux/init.h>
+#include <linux/kmsan-checks.h>
 #include <linux/mm.h>
 #include <linux/preempt.h>
 #include <linux/printk.h>
 	INIT_LIST_HEAD(&area->list);
 	area->size = size;
 	list_add(&area->list, &kcov_remote_areas);
+	/*
+	 * KMSAN doesn't instrument this file, so it may not know area->list
+	 * is initialized. Unpoison it explicitly to avoid reports in
+	 * kcov_remote_area_get().
+	 */
+	kmsan_unpoison_memory(&area->list, sizeof(area->list));
 }
 static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t)

kernel/locking/Makefile+2−1

@@ -5,8 +5,9 @@ KCOV_INSTRUMENT := n · 4 unmodified lines
5		5
6	obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o	6	obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o
7		7
8	# Avoid recursion lockdep -> KCSAN -> ... -> lockdep.	8	# Avoid recursion lockdep -> sanitizer -> ... -> lockdep.
9	KCSAN_SANITIZE_lockdep.o := n	9	KCSAN_SANITIZE_lockdep.o := n
		10	KMSAN_SANITIZE_lockdep.o := n
10		11
11	ifdef CONFIG_FUNCTION_TRACER	12	ifdef CONFIG_FUNCTION_TRACER
12	CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)	13	CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)

kernel/sched/core.c+15−0

 }
 #ifdef CONFIG_PROC_SYSCTL
+static void reset_memory_tiering(void)
+{
+	struct pglist_data *pgdat;
+	for_each_online_pgdat(pgdat) {
+		pgdat->nbp_threshold = 0;
+		pgdat->nbp_th_nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
+		pgdat->nbp_th_start = jiffies_to_msecs(jiffies);
+	}
+}
 int sysctl_numa_balancing(struct ctl_table *table, int write,
 			  void *buffer, size_t *lenp, loff_t *ppos)
 {
 	if (err < 0)
 		return err;
 	if (write) {
+		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		    (state & NUMA_BALANCING_MEMORY_TIERING))
+			reset_memory_tiering();
 		sysctl_numa_balancing_mode = state;
 		__set_numabalancing_state(state);
 	}
 		 * finish_task_switch()'s mmdrop().
 		 */
 		switch_mm_irqs_off(prev->active_mm, next->mm, next);
+		lru_gen_use_mm(next->mm);
 		if (!prev->mm) {                        // from kernel
 			/* will mmdrop() in finish_task_switch(). */

kernel/sched/debug.c+1−0

@@ -333,6 +333,7 @@ static __init int sched_init_debug(void) · 332 unmodified lines
333	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);	333	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
334	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);	334	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
335	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);	335	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
		336	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
336	#endif	337	#endif
337		338
338	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);	339	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);

kernel/sched/fair.c+172−3

@@ -40,6 +40,7 @@ · 39 unmodified lines
40		40
41	#include <linux/cpuidle.h>	41	#include <linux/cpuidle.h>
42	#include <linux/interrupt.h>	42	#include <linux/interrupt.h>
		43	#include <linux/memory-tiers.h>
43	#include <linux/mempolicy.h>	44	#include <linux/mempolicy.h>
44	#include <linux/mutex_api.h>	45	#include <linux/mutex_api.h>
45	#include <linux/profile.h>	46	#include <linux/profile.h>
@@ -1090,6 +1091,12 @@ unsigned int sysctl_numa_balancing_scan_size = 256; · 1044 unmodified lines
1090	/* Scan @scan_size MB every @scan_period after an initial @scan_delay in ms */	1091	/* Scan @scan_size MB every @scan_period after an initial @scan_delay in ms */
1091	unsigned int sysctl_numa_balancing_scan_delay = 1000;	1092	unsigned int sysctl_numa_balancing_scan_delay = 1000;
1092		1093
		1094	/* The page with hint page fault latency < threshold in ms is considered hot */
		1095	unsigned int sysctl_numa_balancing_hot_threshold = MSEC_PER_SEC;
		1096
		1097	/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
		1098	unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
		1099
1093	struct numa_group {	1100	struct numa_group {
1094	refcount_t refcount;	1101	refcount_t refcount;
1095		1102
@@ -1432,6 +1439,120 @@ static inline unsigned long group_weight(struct task_struct *p, int nid, · 336 unmodified lines
1432	return 1000 * faults / total_faults;	1439	return 1000 * faults / total_faults;
1433	}	1440	}
1434		1441
		1442	/*
		1443	* If memory tiering mode is enabled, cpupid of slow memory page is
		1444	* used to record scan time instead of CPU and PID. When tiering mode
		1445	* is disabled at run time, the scan time (in cpupid) will be
		1446	* interpreted as CPU and PID. So CPU needs to be checked to avoid to
		1447	* access out of array bound.
		1448	*/
		1449	static inline bool cpupid_valid(int cpupid)
		1450	{
		1451	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
		1452	}
		1453
		1454	/*
		1455	* For memory tiering mode, if there are enough free pages (more than
		1456	* enough watermark defined here) in fast memory node, to take full
		1457	* advantage of fast memory capacity, all recently accessed slow
		1458	* memory pages will be migrated to fast memory node without
		1459	* considering hot threshold.
		1460	*/
		1461	static bool pgdat_free_space_enough(struct pglist_data *pgdat)
		1462	{
		1463	int z;
		1464	unsigned long enough_wmark;
		1465
		1466	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,
		1467	pgdat->node_present_pages >> 4);
		1468	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
		1469	struct zone *zone = pgdat->node_zones + z;
		1470
		1471	if (!populated_zone(zone))
		1472	continue;
		1473
		1474	if (zone_watermark_ok(zone, 0,
		1475	wmark_pages(zone, WMARK_PROMO) + enough_wmark,
		1476	ZONE_MOVABLE, 0))
		1477	return true;
		1478	}
		1479	return false;
		1480	}
		1481
		1482	/*
		1483	* For memory tiering mode, when page tables are scanned, the scan
		1484	* time will be recorded in struct page in addition to make page
		1485	* PROT_NONE for slow memory page. So when the page is accessed, in
		1486	* hint page fault handler, the hint page fault latency is calculated
		1487	* via,
		1488	*
		1489	* hint page fault latency = hint page fault time - scan time
		1490	*
		1491	* The smaller the hint page fault latency, the higher the possibility
		1492	* for the page to be hot.
		1493	*/
		1494	static int numa_hint_fault_latency(struct page *page)
		1495	{
		1496	int last_time, time;
		1497
		1498	time = jiffies_to_msecs(jiffies);
		1499	last_time = xchg_page_access_time(page, time);
		1500
		1501	return (time - last_time) & PAGE_ACCESS_TIME_MASK;
		1502	}
		1503
		1504	/*
		1505	* For memory tiering mode, too high promotion/demotion throughput may
		1506	* hurt application latency. So we provide a mechanism to rate limit
		1507	* the number of pages that are tried to be promoted.
		1508	*/
		1509	static bool numa_promotion_rate_limit(struct pglist_data *pgdat,
		1510	unsigned long rate_limit, int nr)
		1511	{
		1512	unsigned long nr_cand;
		1513	unsigned int now, start;
		1514
		1515	now = jiffies_to_msecs(jiffies);
		1516	mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE, nr);
		1517	nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
		1518	start = pgdat->nbp_rl_start;
		1519	if (now - start > MSEC_PER_SEC &&
		1520	cmpxchg(&pgdat->nbp_rl_start, start, now) == start)
		1521	pgdat->nbp_rl_nr_cand = nr_cand;
		1522	if (nr_cand - pgdat->nbp_rl_nr_cand >= rate_limit)
		1523	return true;
		1524	return false;
		1525	}
		1526
		1527	#define NUMA_MIGRATION_ADJUST_STEPS 16
		1528
		1529	static void numa_promotion_adjust_threshold(struct pglist_data *pgdat,
		1530	unsigned long rate_limit,
		1531	unsigned int ref_th)
		1532	{
		1533	unsigned int now, start, th_period, unit_th, th;
		1534	unsigned long nr_cand, ref_cand, diff_cand;
		1535
		1536	now = jiffies_to_msecs(jiffies);
		1537	th_period = sysctl_numa_balancing_scan_period_max;
		1538	start = pgdat->nbp_th_start;
		1539	if (now - start > th_period &&
		1540	cmpxchg(&pgdat->nbp_th_start, start, now) == start) {
		1541	ref_cand = rate_limit *
		1542	sysctl_numa_balancing_scan_period_max / MSEC_PER_SEC;
		1543	nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
		1544	diff_cand = nr_cand - pgdat->nbp_th_nr_cand;
		1545	unit_th = ref_th * 2 / NUMA_MIGRATION_ADJUST_STEPS;
		1546	th = pgdat->nbp_threshold ? : ref_th;
		1547	if (diff_cand > ref_cand * 11 / 10)
		1548	th = max(th - unit_th, unit_th);
		1549	else if (diff_cand < ref_cand * 9 / 10)
		1550	th = min(th + unit_th, ref_th * 2);
		1551	pgdat->nbp_th_nr_cand = nr_cand;
		1552	pgdat->nbp_threshold = th;
		1553	}
		1554	}
		1555
1435	bool should_numa_migrate_memory(struct task_struct p, struct page page,	1556	bool should_numa_migrate_memory(struct task_struct p, struct page page,
1436	int src_nid, int dst_cpu)	1557	int src_nid, int dst_cpu)
1437	{	1558	{
@@ -1439,9 +1560,44 @@ bool should_numa_migrate_memory(struct task_struct p, struct page page, · 1 unmodified line
1439	int dst_nid = cpu_to_node(dst_cpu);	1560	int dst_nid = cpu_to_node(dst_cpu);
1440	int last_cpupid, this_cpupid;	1561	int last_cpupid, this_cpupid;
1441		1562
		1563	/*
		1564	* The pages in slow memory node should be migrated according
		1565	* to hot/cold instead of private/shared.
		1566	*/
		1567	if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
		1568	!node_is_toptier(src_nid)) {
		1569	struct pglist_data *pgdat;
		1570	unsigned long rate_limit;
		1571	unsigned int latency, th, def_th;
		1572
		1573	pgdat = NODE_DATA(dst_nid);
		1574	if (pgdat_free_space_enough(pgdat)) {
		1575	/* workload changed, reset hot threshold */
		1576	pgdat->nbp_threshold = 0;
		1577	return true;
		1578	}
		1579
		1580	def_th = sysctl_numa_balancing_hot_threshold;
		1581	rate_limit = sysctl_numa_balancing_promote_rate_limit << \
		1582	(20 - PAGE_SHIFT);
		1583	numa_promotion_adjust_threshold(pgdat, rate_limit, def_th);
		1584
		1585	th = pgdat->nbp_threshold ? : def_th;
		1586	latency = numa_hint_fault_latency(page);
		1587	if (latency >= th)
		1588	return false;
		1589
		1590	return !numa_promotion_rate_limit(pgdat, rate_limit,
		1591	thp_nr_pages(page));
		1592	}
		1593
1442	this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);	1594	this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
1443	last_cpupid = page_cpupid_xchg_last(page, this_cpupid);	1595	last_cpupid = page_cpupid_xchg_last(page, this_cpupid);
1444		1596
		1597	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
		1598	!node_is_toptier(src_nid) && !cpupid_valid(last_cpupid))
		1599	return false;
		1600
1445	/*	1601	/*
1446	* Allow first faults or private faults to migrate immediately early in	1602	* Allow first faults or private faults to migrate immediately early in
1447	* the lifetime of a task. The magic number 4 is based on waiting for	1603	* the lifetime of a task. The magic number 4 is based on waiting for
@@ -2681,6 +2837,15 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · 1233 unmodified lines
2681	if (!p->mm)	2837	if (!p->mm)
2682	return;	2838	return;
2683		2839
		2840	/*
		2841	* NUMA faults statistics are unnecessary for the slow memory
		2842	* node for memory tiering mode.
		2843	*/
		2844	if (!node_is_toptier(mem_node) &&
		2845	(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING \|\|
		2846	!cpupid_valid(last_cpupid)))
		2847	return;
		2848
2684	/* Allocate buffer to track faults on a per-node basis */	2849	/* Allocate buffer to track faults on a per-node basis */
2685	if (unlikely(!p->numa_faults)) {	2850	if (unlikely(!p->numa_faults)) {
2686	int size = sizeof(p->numa_faults)	2851	int size = sizeof(p->numa_faults)
@@ -2761,6 +2926,7 @@ static void task_numa_work(struct callback_head *work) · 74 unmodified lines
2761	struct task_struct *p = current;	2926	struct task_struct *p = current;
2762	struct mm_struct *mm = p->mm;	2927	struct mm_struct *mm = p->mm;
2763	u64 runtime = p->se.sum_exec_runtime;	2928	u64 runtime = p->se.sum_exec_runtime;
		2929	MA_STATE(mas, &mm->mm_mt, 0, 0);
2764	struct vm_area_struct *vma;	2930	struct vm_area_struct *vma;
2765	unsigned long start, end;	2931	unsigned long start, end;
2766	unsigned long nr_pte_updates = 0;	2932	unsigned long nr_pte_updates = 0;
@@ -2817,13 +2983,16 @@ static void task_numa_work(struct callback_head *work) · 50 unmodified lines
2817		2983
2818	if (!mmap_read_trylock(mm))	2984	if (!mmap_read_trylock(mm))
2819	return;	2985	return;
2820	vma = find_vma(mm, start);	2986	mas_set(&mas, start);
		2987	vma = mas_find(&mas, ULONG_MAX);
2821	if (!vma) {	2988	if (!vma) {
2822	reset_ptenuma_scan(p);	2989	reset_ptenuma_scan(p);
2823	start = 0;	2990	start = 0;
2824	vma = mm->mmap;	2991	mas_set(&mas, start);
		2992	vma = mas_find(&mas, ULONG_MAX);
2825	}	2993	}
2826	for (; vma; vma = vma->vm_next) {	2994
		2995	for (; vma; vma = mas_find(&mas, ULONG_MAX)) {
2827	if (!vma_migratable(vma) \|\| !vma_policy_mof(vma) \|\|	2996	if (!vma_migratable(vma) \|\| !vma_policy_mof(vma) \|\|
2828	is_vm_hugetlb_page(vma) \|\| (vma->vm_flags & VM_MIXEDMAP)) {	2997	is_vm_hugetlb_page(vma) \|\| (vma->vm_flags & VM_MIXEDMAP)) {
2829	continue;	2998	continue;

kernel/sched/sched.h+1−0

@@ -2446,6 +2446,7 @@ extern unsigned int sysctl_numa_balancing_scan_delay; · 2445 unmodified lines
2446	extern unsigned int sysctl_numa_balancing_scan_period_min;	2446	extern unsigned int sysctl_numa_balancing_scan_period_min;
2447	extern unsigned int sysctl_numa_balancing_scan_period_max;	2447	extern unsigned int sysctl_numa_balancing_scan_period_max;
2448	extern unsigned int sysctl_numa_balancing_scan_size;	2448	extern unsigned int sysctl_numa_balancing_scan_size;
		2449	extern unsigned int sysctl_numa_balancing_hot_threshold;
2449	#endif	2450	#endif
2450		2451
2451	#ifdef CONFIG_SCHED_HRTICK	2452	#ifdef CONFIG_SCHED_HRTICK

kernel/sysctl.c+8−0

@@ -1643,6 +1643,14 @@ static struct ctl_table kern_table[] = { · 1642 unmodified lines
1643	.extra1 = SYSCTL_ZERO,	1643	.extra1 = SYSCTL_ZERO,
1644	.extra2 = SYSCTL_FOUR,	1644	.extra2 = SYSCTL_FOUR,
1645	},	1645	},
		1646	{
		1647	.procname = "numa_balancing_promote_rate_limit_MBps",
		1648	.data = &sysctl_numa_balancing_promote_rate_limit,
		1649	.maxlen = sizeof(unsigned int),
		1650	.mode = 0644,
		1651	.proc_handler = proc_dointvec_minmax,
		1652	.extra1 = SYSCTL_ZERO,
		1653	},
1646	#endif /* CONFIG_NUMA_BALANCING */	1654	#endif /* CONFIG_NUMA_BALANCING */
1647	{	1655	{
1648	.procname = "panic",	1656	.procname = "panic",

lib/Kconfig.debug+13−5

 	  If unsure, say N.
-config DEBUG_VM_VMACACHE
+config DEBUG_VM_MAPLE_TREE
-	bool "Debug VMA caching"
+	bool "Debug VM maple trees"
 	depends on DEBUG_VM
+	select DEBUG_MAPLE_TREE
 	help
-	  Enable this to turn on VMA caching debug information. Doing so
+	  Enable VM maple tree debugging information and extra validations.
-	  can cause significant overhead, so only enable it in non-production
-	  environments.
 	  If unsure, say N.
 source "lib/Kconfig.kasan"
 source "lib/Kconfig.kfence"
+source "lib/Kconfig.kmsan"
 endmenu # "Memory Debugging"
 	  If unsure, say N.
+config DEBUG_MAPLE_TREE
+	bool "Debug maple trees"
+	depends on DEBUG_KERNEL
+	help
+	  Enable maple tree debugging information and extra validations.
+	  If unsure, say N.
 endmenu
 config DEBUG_CREDENTIALS

lib/Kconfig.kasan+0−8

@@ -167,14 +167,6 @@ config KASAN_STACK · 166 unmodified lines
167	as well, as it adds inline-style instrumentation that is run	167	as well, as it adds inline-style instrumentation that is run
168	unconditionally.	168	unconditionally.
169		169
170	config KASAN_TAGS_IDENTIFY
171	bool "Memory corruption type identification"
172	depends on KASAN_SW_TAGS \|\| KASAN_HW_TAGS
173	help
174	Enables best-effort identification of the bug types (use-after-free
175	or out-of-bounds) at the cost of increased memory consumption.
176	Only applicable for the tag-based KASAN modes.
177
178	config KASAN_VMALLOC	170	config KASAN_VMALLOC
179	bool "Check accesses to vmalloc allocations"	171	bool "Check accesses to vmalloc allocations"
180	depends on HAVE_ARCH_KASAN_VMALLOC	172	depends on HAVE_ARCH_KASAN_VMALLOC

lib/Kconfig.kmsan+62−0

@@ -0,0 +1,62 @@
	1	# SPDX-License-Identifier: GPL-2.0-only
	2	config HAVE_ARCH_KMSAN
	3	bool
	4
	5	config HAVE_KMSAN_COMPILER
	6	# Clang versions <14.0.0 also support -fsanitize=kernel-memory, but not
	7	# all the features necessary to build the kernel with KMSAN.
	8	depends on CC_IS_CLANG && CLANG_VERSION >= 140000
	9	def_bool $(cc-option,-fsanitize=kernel-memory -mllvm -msan-disable-checks=1)
	10
	11	config KMSAN
	12	bool "KMSAN: detector of uninitialized values use"
	13	depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
	14	depends on SLUB && DEBUG_KERNEL && !KASAN && !KCSAN
	15	select STACKDEPOT
	16	select STACKDEPOT_ALWAYS_INIT
	17	help
	18	KernelMemorySanitizer (KMSAN) is a dynamic detector of uses of
	19	uninitialized values in the kernel. It is based on compiler
	20	instrumentation provided by Clang and thus requires Clang to build.
	21
	22	An important note is that KMSAN is not intended for production use,
	23	because it drastically increases kernel memory footprint and slows
	24	the whole system down.
	25
	26	See <file:Documentation/dev-tools/kmsan.rst> for more details.
	27
	28	if KMSAN
	29
	30	config HAVE_KMSAN_PARAM_RETVAL
	31	# -fsanitize-memory-param-retval is supported only by Clang >= 14.
	32	depends on HAVE_KMSAN_COMPILER
	33	def_bool $(cc-option,-fsanitize=kernel-memory -fsanitize-memory-param-retval)
	34
	35	config KMSAN_CHECK_PARAM_RETVAL
	36	bool "Check for uninitialized values passed to and returned from functions"
	37	default y
	38	depends on HAVE_KMSAN_PARAM_RETVAL
	39	help
	40	If the compiler supports -fsanitize-memory-param-retval, KMSAN will
	41	eagerly check every function parameter passed by value and every
	42	function return value.
	43
	44	Disabling KMSAN_CHECK_PARAM_RETVAL will result in tracking shadow for
	45	function parameters and return values across function borders. This
	46	is a more relaxed mode, but it generates more instrumentation code and
	47	may potentially report errors in corner cases when non-instrumented
	48	functions call instrumented ones.
	49
	50	config KMSAN_KUNIT_TEST
	51	tristate "KMSAN integration test suite" if !KUNIT_ALL_TESTS
	52	default KUNIT_ALL_TESTS
	53	depends on TRACEPOINTS && KUNIT
	54	help
	55	Test suite for KMSAN, testing various error detection scenarios,
	56	and checking that reports are correctly output to console.
	57
	58	Say Y here if you want the test to be built into the kernel and run
	59	during boot; say M if you want the test to build as a module; say N
	60	if you are unsure.
	61
	62	endif

lib/Makefile+4−6

 lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 rbtree.o radix-tree.o timerqueue.o xarray.o \
-	 idr.o extable.o irq_regs.o argv_split.o \
+	 maple_tree.o idr.o extable.o irq_regs.o argv_split.o \
 	 flex_proportions.o ratelimit.o show_mem.o \
 	 is_single_threaded.o plist.o decompress.o kobject_uevent.o \
 	 earlycpio.o seq_buf.o siphash.o dec_and_lock.o \
 obj-$(CONFIG_TEST_SIPHASH) += test_siphash.o
 obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o
 obj-$(CONFIG_TEST_IDA) += test_ida.o
-obj-$(CONFIG_KASAN_KUNIT_TEST) += test_kasan.o
-CFLAGS_test_kasan.o += -fno-builtin
-CFLAGS_test_kasan.o += $(call cc-disable-warning, vla)
-obj-$(CONFIG_KASAN_MODULE_TEST) += test_kasan_module.o
-CFLAGS_test_kasan_module.o += -fno-builtin
 obj-$(CONFIG_TEST_UBSAN) += test_ubsan.o
 CFLAGS_test_ubsan.o += $(call cc-disable-warning, vla)
 UBSAN_SANITIZE_test_ubsan.o := y
 CFLAGS_stackdepot.o += -fno-builtin
 obj-$(CONFIG_STACKDEPOT) += stackdepot.o
 KASAN_SANITIZE_stackdepot.o := n
+# In particular, instrumenting stackdepot.c with KMSAN will result in infinite
+# recursion.
+KMSAN_SANITIZE_stackdepot.o := n
 KCOV_INSTRUMENT_stackdepot.o := n
 obj-$(CONFIG_REF_TRACKER) += ref_tracker.o

lib/iomap.c+44−0

  */
 #include <linux/pci.h>
 #include <linux/io.h>
+#include <linux/kmsan-checks.h>
 #include <linux/export.h>
 #define mmio_read64be(addr) swab64(readq(addr))
 #endif
+/*
+ * Here and below, we apply __no_kmsan_checks to functions reading data from
+ * hardware, to ensure that KMSAN marks their return values as initialized.
+ */
+__no_kmsan_checks
 unsigned int ioread8(const void __iomem *addr)
 {
 	IO_COND(addr, return inb(port), return readb(addr));
 	return 0xff;
 }
+__no_kmsan_checks
 unsigned int ioread16(const void __iomem *addr)
 {
 	IO_COND(addr, return inw(port), return readw(addr));
 	return 0xffff;
 }
+__no_kmsan_checks
 unsigned int ioread16be(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read16be(port), return mmio_read16be(addr));
 	return 0xffff;
 }
+__no_kmsan_checks
 unsigned int ioread32(const void __iomem *addr)
 {
 	IO_COND(addr, return inl(port), return readl(addr));
 	return 0xffffffff;
 }
+__no_kmsan_checks
 unsigned int ioread32be(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read32be(port), return mmio_read32be(addr));
 	return lo | (hi << 32);
 }
+__no_kmsan_checks
 u64 ioread64_lo_hi(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read64_lo_hi(port), return readq(addr));
 	return 0xffffffffffffffffULL;
 }
+__no_kmsan_checks
 u64 ioread64_hi_lo(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read64_hi_lo(port), return readq(addr));
 	return 0xffffffffffffffffULL;
 }
+__no_kmsan_checks
 u64 ioread64be_lo_hi(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read64be_lo_hi(port),
 	return 0xffffffffffffffffULL;
 }
+__no_kmsan_checks
 u64 ioread64be_hi_lo(const void __iomem *addr)
 {
 	IO_COND(addr, return pio_read64be_hi_lo(port),
 void iowrite8(u8 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, outb(val,port), writeb(val, addr));
 }
 void iowrite16(u16 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, outw(val,port), writew(val, addr));
 }
 void iowrite16be(u16 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write16be(val,port), mmio_write16be(val, addr));
 }
 void iowrite32(u32 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, outl(val,port), writel(val, addr));
 }
 void iowrite32be(u32 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write32be(val,port), mmio_write32be(val, addr));
 }
 EXPORT_SYMBOL(iowrite8);
 void iowrite64_lo_hi(u64 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write64_lo_hi(val, port),
 		writeq(val, addr));
 }
 void iowrite64_hi_lo(u64 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write64_hi_lo(val, port),
 		writeq(val, addr));
 }
 void iowrite64be_lo_hi(u64 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write64be_lo_hi(val, port),
 		mmio_write64be(val, addr));
 }
 void iowrite64be_hi_lo(u64 val, void __iomem *addr)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(&val, sizeof(val));
 	IO_COND(addr, pio_write64be_hi_lo(val, port),
 		mmio_write64be(val, addr));
 }
 void ioread8_rep(const void __iomem *addr, void *dst, unsigned long count)
 {
 	IO_COND(addr, insb(port,dst,count), mmio_insb(addr, dst, count));
+	/* KMSAN must treat values read from devices as initialized. */
+	kmsan_unpoison_memory(dst, count);
 }
 void ioread16_rep(const void __iomem *addr, void *dst, unsigned long count)
 {
 	IO_COND(addr, insw(port,dst,count), mmio_insw(addr, dst, count));
+	/* KMSAN must treat values read from devices as initialized. */
+	kmsan_unpoison_memory(dst, count * 2);
 }
 void ioread32_rep(const void __iomem *addr, void *dst, unsigned long count)
 {
 	IO_COND(addr, insl(port,dst,count), mmio_insl(addr, dst, count));
+	/* KMSAN must treat values read from devices as initialized. */
+	kmsan_unpoison_memory(dst, count * 4);
 }
 EXPORT_SYMBOL(ioread8_rep);
 EXPORT_SYMBOL(ioread16_rep);
 void iowrite8_rep(void __iomem *addr, const void *src, unsigned long count)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(src, count);
 	IO_COND(addr, outsb(port, src, count), mmio_outsb(addr, src, count));
 }
 void iowrite16_rep(void __iomem *addr, const void *src, unsigned long count)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(src, count * 2);
 	IO_COND(addr, outsw(port, src, count), mmio_outsw(addr, src, count));
 }
 void iowrite32_rep(void __iomem *addr, const void *src, unsigned long count)
 {
+	/* Make sure uninitialized memory isn't copied to devices. */
+	kmsan_check_memory(src, count * 4);
 	IO_COND(addr, outsl(port, src,count), mmio_outsl(addr, src, count));
 }
 EXPORT_SYMBOL(iowrite8_rep);

lib/iov_iter.c+6−3

 static int copyin(void *to, const void __user *from, size_t n)
 {
+	size_t res = n;
 	if (should_fail_usercopy())
 		return n;
 	if (access_ok(from, n)) {
-		instrument_copy_from_user(to, from, n);
+		instrument_copy_from_user_before(to, from, n);
-		n = raw_copy_from_user(to, from, n);
+		res = raw_copy_from_user(to, from, n);
+		instrument_copy_from_user_after(to, from, n, res);
 	}
-	return n;
+	return res;
 }
 static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe,

lib/maple_tree.c+7130−0

@@ -0,0 +1,7130 @@
	1	// SPDX-License-Identifier: GPL-2.0+
	2	/*
	3	* Maple Tree implementation
	4	* Copyright (c) 2018-2022 Oracle Corporation
	5	* Authors: Liam R. Howlett <Liam.Howlett@oracle.com>
	6	* Matthew Wilcox <willy@infradead.org>
	7	*/
	8
	9	/*
	10	* DOC: Interesting implementation details of the Maple Tree
	11	*
	12	* Each node type has a number of slots for entries and a number of slots for
	13	* pivots. In the case of dense nodes, the pivots are implied by the position
	14	* and are simply the slot index + the minimum of the node.
	15	*
	16	* In regular B-Tree terms, pivots are called keys. The term pivot is used to
	17	* indicate that the tree is specifying ranges, Pivots may appear in the
	18	* subtree with an entry attached to the value where as keys are unique to a
	19	* specific position of a B-tree. Pivot values are inclusive of the slot with
	20	* the same index.
	21	*
	22	*
	23	* The following illustrates the layout of a range64 nodes slots and pivots.
	24	*
	25	*
	26	* Slots -> \| 0 \| 1 \| 2 \| ... \| 12 \| 13 \| 14 \| 15 \|
	27	* ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬
	28	* │ │ │ │ │ │ │ │ └─ Implied maximum
	29	* │ │ │ │ │ │ │ └─ Pivot 14
	30	* │ │ │ │ │ │ └─ Pivot 13
	31	* │ │ │ │ │ └─ Pivot 12
	32	* │ │ │ │ └─ Pivot 11
	33	* │ │ │ └─ Pivot 2
	34	* │ │ └─ Pivot 1
	35	* │ └─ Pivot 0
	36	* └─ Implied minimum
	37	*
	38	* Slot contents:
	39	* Internal (non-leaf) nodes contain pointers to other nodes.
	40	* Leaf nodes contain entries.
	41	*
	42	* The location of interest is often referred to as an offset. All offsets have
	43	* a slot, but the last offset has an implied pivot from the node above (or
	44	* UINT_MAX for the root node.
	45	*
	46	* Ranges complicate certain write activities. When modifying any of
	47	* the B-tree variants, it is known that one entry will either be added or
	48	* deleted. When modifying the Maple Tree, one store operation may overwrite
	49	* the entire data set, or one half of the tree, or the middle half of the tree.
	50	*
	51	*/
	52
	53
	54	#include <linux/maple_tree.h>
	55	#include <linux/xarray.h>
	56	#include <linux/types.h>
	57	#include <linux/export.h>
	58	#include <linux/slab.h>
	59	#include <linux/limits.h>
	60	#include <asm/barrier.h>
	61
	62	#define CREATE_TRACE_POINTS
	63	#include <trace/events/maple_tree.h>
	64
	65	#define MA_ROOT_PARENT 1
	66
	67	/*
	68	* Maple state flags
	69	* * MA_STATE_BULK - Bulk insert mode
	70	* * MA_STATE_REBALANCE - Indicate a rebalance during bulk insert
	71	* * MA_STATE_PREALLOC - Preallocated nodes, WARN_ON allocation
	72	*/
	73	#define MA_STATE_BULK 1
	74	#define MA_STATE_REBALANCE 2
	75	#define MA_STATE_PREALLOC 4
	76
	77	#define ma_parent_ptr(x) ((struct maple_pnode *)(x))
	78	#define ma_mnode_ptr(x) ((struct maple_node *)(x))
	79	#define ma_enode_ptr(x) ((struct maple_enode *)(x))
	80	static struct kmem_cache *maple_node_cache;
	81
	82	#ifdef CONFIG_DEBUG_MAPLE_TREE
	83	static const unsigned long mt_max[] = {
	84	[maple_dense] = MAPLE_NODE_SLOTS,
	85	[maple_leaf_64] = ULONG_MAX,
	86	[maple_range_64] = ULONG_MAX,
	87	[maple_arange_64] = ULONG_MAX,
	88	};
	89	#define mt_node_max(x) mt_max[mte_node_type(x)]
	90	#endif
	91
	92	static const unsigned char mt_slots[] = {
	93	[maple_dense] = MAPLE_NODE_SLOTS,
	94	[maple_leaf_64] = MAPLE_RANGE64_SLOTS,
	95	[maple_range_64] = MAPLE_RANGE64_SLOTS,
	96	[maple_arange_64] = MAPLE_ARANGE64_SLOTS,
	97	};
	98	#define mt_slot_count(x) mt_slots[mte_node_type(x)]
	99
	100	static const unsigned char mt_pivots[] = {
	101	[maple_dense] = 0,
	102	[maple_leaf_64] = MAPLE_RANGE64_SLOTS - 1,
	103	[maple_range_64] = MAPLE_RANGE64_SLOTS - 1,
	104	[maple_arange_64] = MAPLE_ARANGE64_SLOTS - 1,
	105	};
	106	#define mt_pivot_count(x) mt_pivots[mte_node_type(x)]
	107
	108	static const unsigned char mt_min_slots[] = {
	109	[maple_dense] = MAPLE_NODE_SLOTS / 2,
	110	[maple_leaf_64] = (MAPLE_RANGE64_SLOTS / 2) - 2,
	111	[maple_range_64] = (MAPLE_RANGE64_SLOTS / 2) - 2,
	112	[maple_arange_64] = (MAPLE_ARANGE64_SLOTS / 2) - 1,
	113	};
	114	#define mt_min_slot_count(x) mt_min_slots[mte_node_type(x)]
	115
	116	#define MAPLE_BIG_NODE_SLOTS (MAPLE_RANGE64_SLOTS * 2 + 2)
	117	#define MAPLE_BIG_NODE_GAPS (MAPLE_ARANGE64_SLOTS * 2 + 1)
	118
	119	struct maple_big_node {
	120	struct maple_pnode *parent;
	121	unsigned long pivot[MAPLE_BIG_NODE_SLOTS - 1];
	122	union {
	123	struct maple_enode *slot[MAPLE_BIG_NODE_SLOTS];
	124	struct {
	125	unsigned long padding[MAPLE_BIG_NODE_GAPS];
	126	unsigned long gap[MAPLE_BIG_NODE_GAPS];
	127	};
	128	};
	129	unsigned char b_end;
	130	enum maple_type type;
	131	};
	132
	133	/*
	134	* The maple_subtree_state is used to build a tree to replace a segment of an
	135	* existing tree in a more atomic way. Any walkers of the older tree will hit a
	136	* dead node and restart on updates.
	137	*/
	138	struct maple_subtree_state {
	139	struct ma_state orig_l; / Original left side of subtree */
	140	struct ma_state orig_r; / Original right side of subtree */
	141	struct ma_state l; / New left side of subtree */
	142	struct ma_state m; / New middle of subtree (rare) */
	143	struct ma_state r; / New right side of subtree */
	144	struct ma_topiary free; / nodes to be freed */
	145	struct ma_topiary destroy; / Nodes to be destroyed (walked and freed) */
	146	struct maple_big_node *bn;
	147	};
	148
	149	/* Functions */
	150	static inline struct maple_node *mt_alloc_one(gfp_t gfp)
	151	{
	152	return kmem_cache_alloc(maple_node_cache, gfp \| __GFP_ZERO);
	153	}
	154
	155	static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
	156	{
	157	return kmem_cache_alloc_bulk(maple_node_cache, gfp \| __GFP_ZERO, size,
	158	nodes);
	159	}
	160
	161	static inline void mt_free_bulk(size_t size, void __rcu **nodes)
	162	{
	163	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
	164	}
	165
	166	static void mt_free_rcu(struct rcu_head *head)
	167	{
	168	struct maple_node *node = container_of(head, struct maple_node, rcu);
	169
	170	kmem_cache_free(maple_node_cache, node);
	171	}
	172
	173	/*
	174	* ma_free_rcu() - Use rcu callback to free a maple node
	175	* @node: The node to free
	176	*
	177	* The maple tree uses the parent pointer to indicate this node is no longer in
	178	* use and will be freed.
	179	*/
	180	static void ma_free_rcu(struct maple_node *node)
	181	{
	182	node->parent = ma_parent_ptr(node);
	183	call_rcu(&node->rcu, mt_free_rcu);
	184	}
	185
	186	static unsigned int mt_height(const struct maple_tree *mt)
	187	{
	188	return (mt->ma_flags & MT_FLAGS_HEIGHT_MASK) >> MT_FLAGS_HEIGHT_OFFSET;
	189	}
	190
	191	static void mas_set_height(struct ma_state *mas)
	192	{
	193	unsigned int new_flags = mas->tree->ma_flags;
	194
	195	new_flags &= ~MT_FLAGS_HEIGHT_MASK;
	196	BUG_ON(mas->depth > MAPLE_HEIGHT_MAX);
	197	new_flags \|= mas->depth << MT_FLAGS_HEIGHT_OFFSET;
	198	mas->tree->ma_flags = new_flags;
	199	}
	200
	201	static unsigned int mas_mt_height(struct ma_state *mas)
	202	{
	203	return mt_height(mas->tree);
	204	}
	205
	206	static inline enum maple_type mte_node_type(const struct maple_enode *entry)
	207	{
	208	return ((unsigned long)entry >> MAPLE_NODE_TYPE_SHIFT) &
	209	MAPLE_NODE_TYPE_MASK;
	210	}
	211
	212	static inline bool ma_is_dense(const enum maple_type type)
	213	{
	214	return type < maple_leaf_64;
	215	}
	216
	217	static inline bool ma_is_leaf(const enum maple_type type)
	218	{
	219	return type < maple_range_64;
	220	}
	221
	222	static inline bool mte_is_leaf(const struct maple_enode *entry)
	223	{
	224	return ma_is_leaf(mte_node_type(entry));
	225	}
	226
	227	/*
	228	* We also reserve values with the bottom two bits set to '10' which are
	229	* below 4096
	230	*/
	231	static inline bool mt_is_reserved(const void *entry)
	232	{
	233	return ((unsigned long)entry < MAPLE_RESERVED_RANGE) &&
	234	xa_is_internal(entry);
	235	}
	236
	237	static inline void mas_set_err(struct ma_state *mas, long err)
	238	{
	239	mas->node = MA_ERROR(err);
	240	}
	241
	242	static inline bool mas_is_ptr(struct ma_state *mas)
	243	{
	244	return mas->node == MAS_ROOT;
	245	}
	246
	247	static inline bool mas_is_start(struct ma_state *mas)
	248	{
	249	return mas->node == MAS_START;
	250	}
	251
	252	bool mas_is_err(struct ma_state *mas)
	253	{
	254	return xa_is_err(mas->node);
	255	}
	256
	257	static inline bool mas_searchable(struct ma_state *mas)
	258	{
	259	if (mas_is_none(mas))
	260	return false;
	261
	262	if (mas_is_ptr(mas))
	263	return false;
	264
	265	return true;
	266	}
	267
	268	static inline struct maple_node mte_to_node(const struct maple_enode entry)
	269	{
	270	return (struct maple_node *)((unsigned long)entry & ~MAPLE_NODE_MASK);
	271	}
	272
	273	/*
	274	* mte_to_mat() - Convert a maple encoded node to a maple topiary node.
	275	* @entry: The maple encoded node
	276	*
	277	* Return: a maple topiary pointer
	278	*/
	279	static inline struct maple_topiary mte_to_mat(const struct maple_enode entry)
	280	{
	281	return (struct maple_topiary *)
	282	((unsigned long)entry & ~MAPLE_NODE_MASK);
	283	}
	284
	285	/*
	286	* mas_mn() - Get the maple state node.
	287	* @mas: The maple state
	288	*
	289	* Return: the maple node (not encoded - bare pointer).
	290	*/
	291	static inline struct maple_node mas_mn(const struct ma_state mas)
	292	{
	293	return mte_to_node(mas->node);
	294	}
	295
	296	/*
	297	* mte_set_node_dead() - Set a maple encoded node as dead.
	298	* @mn: The maple encoded node.
	299	*/
	300	static inline void mte_set_node_dead(struct maple_enode *mn)
	301	{
	302	mte_to_node(mn)->parent = ma_parent_ptr(mte_to_node(mn));
	303	smp_wmb(); /* Needed for RCU */
	304	}
	305
	306	/* Bit 1 indicates the root is a node */
	307	#define MAPLE_ROOT_NODE 0x02
	308	/* maple_type stored bit 3-6 */
	309	#define MAPLE_ENODE_TYPE_SHIFT 0x03
	310	/* Bit 2 means a NULL somewhere below */
	311	#define MAPLE_ENODE_NULL 0x04
	312
	313	static inline struct maple_enode mt_mk_node(const struct maple_node node,
	314	enum maple_type type)
	315	{
	316	return (void *)((unsigned long)node \|
	317	(type << MAPLE_ENODE_TYPE_SHIFT) \| MAPLE_ENODE_NULL);
	318	}
	319
	320	static inline void mte_mk_root(const struct maple_enode node)
	321	{
	322	return (void *)((unsigned long)node \| MAPLE_ROOT_NODE);
	323	}
	324
	325	static inline void mte_safe_root(const struct maple_enode node)
	326	{
	327	return (void *)((unsigned long)node & ~MAPLE_ROOT_NODE);
	328	}
	329
	330	static inline void mte_set_full(const struct maple_enode *node)
	331	{
	332	node = (void *)((unsigned long)node & ~MAPLE_ENODE_NULL);
	333	}
	334
	335	static inline void mte_clear_full(const struct maple_enode *node)
	336	{
	337	node = (void *)((unsigned long)node \| MAPLE_ENODE_NULL);
	338	}
	339
	340	static inline bool ma_is_root(struct maple_node *node)
	341	{
	342	return ((unsigned long)node->parent & MA_ROOT_PARENT);
	343	}
	344
	345	static inline bool mte_is_root(const struct maple_enode *node)
	346	{
	347	return ma_is_root(mte_to_node(node));
	348	}
	349
	350	static inline bool mas_is_root_limits(const struct ma_state *mas)
	351	{
	352	return !mas->min && mas->max == ULONG_MAX;
	353	}
	354
	355	static inline bool mt_is_alloc(struct maple_tree *mt)
	356	{
	357	return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE);
	358	}
	359
	360	/*
	361	* The Parent Pointer
	362	* Excluding root, the parent pointer is 256B aligned like all other tree nodes.
	363	* When storing a 32 or 64 bit values, the offset can fit into 5 bits. The 16
	364	* bit values need an extra bit to store the offset. This extra bit comes from
	365	* a reuse of the last bit in the node type. This is possible by using bit 1 to
	366	* indicate if bit 2 is part of the type or the slot.
	367	*
	368	* Note types:
	369	* 0x??1 = Root
	370	* 0x?00 = 16 bit nodes
	371	* 0x010 = 32 bit nodes
	372	* 0x110 = 64 bit nodes
	373	*
	374	* Slot size and alignment
	375	* 0b??1 : Root
	376	* 0b?00 : 16 bit values, type in 0-1, slot in 2-7
	377	* 0b010 : 32 bit values, type in 0-2, slot in 3-7
	378	* 0b110 : 64 bit values, type in 0-2, slot in 3-7
	379	*/
	380
	381	#define MAPLE_PARENT_ROOT 0x01
	382
	383	#define MAPLE_PARENT_SLOT_SHIFT 0x03
	384	#define MAPLE_PARENT_SLOT_MASK 0xF8
	385
	386	#define MAPLE_PARENT_16B_SLOT_SHIFT 0x02
	387	#define MAPLE_PARENT_16B_SLOT_MASK 0xFC
	388
	389	#define MAPLE_PARENT_RANGE64 0x06
	390	#define MAPLE_PARENT_RANGE32 0x04
	391	#define MAPLE_PARENT_NOT_RANGE16 0x02
	392
	393	/*
	394	* mte_parent_shift() - Get the parent shift for the slot storage.
	395	* @parent: The parent pointer cast as an unsigned long
	396	* Return: The shift into that pointer to the star to of the slot
	397	*/
	398	static inline unsigned long mte_parent_shift(unsigned long parent)
	399	{
	400	/* Note bit 1 == 0 means 16B */
	401	if (likely(parent & MAPLE_PARENT_NOT_RANGE16))
	402	return MAPLE_PARENT_SLOT_SHIFT;
	403
	404	return MAPLE_PARENT_16B_SLOT_SHIFT;
	405	}
	406
	407	/*
	408	* mte_parent_slot_mask() - Get the slot mask for the parent.
	409	* @parent: The parent pointer cast as an unsigned long.
	410	* Return: The slot mask for that parent.
	411	*/
	412	static inline unsigned long mte_parent_slot_mask(unsigned long parent)
	413	{
	414	/* Note bit 1 == 0 means 16B */
	415	if (likely(parent & MAPLE_PARENT_NOT_RANGE16))
	416	return MAPLE_PARENT_SLOT_MASK;
	417
	418	return MAPLE_PARENT_16B_SLOT_MASK;
	419	}
	420
	421	/*
	422	* mas_parent_enum() - Return the maple_type of the parent from the stored
	423	* parent type.
	424	* @mas: The maple state
	425	* @node: The maple_enode to extract the parent's enum
	426	* Return: The node->parent maple_type
	427	*/
	428	static inline
	429	enum maple_type mte_parent_enum(struct maple_enode *p_enode,
	430	struct maple_tree *mt)
	431	{
	432	unsigned long p_type;
	433
	434	p_type = (unsigned long)p_enode;
	435	if (p_type & MAPLE_PARENT_ROOT)
	436	return 0; /* Validated in the caller. */
	437
	438	p_type &= MAPLE_NODE_MASK;
	439	p_type = p_type & ~(MAPLE_PARENT_ROOT \| mte_parent_slot_mask(p_type));
	440
	441	switch (p_type) {
	442	case MAPLE_PARENT_RANGE64: /* or MAPLE_PARENT_ARANGE64 */
	443	if (mt_is_alloc(mt))
	444	return maple_arange_64;
	445	return maple_range_64;
	446	}
	447
	448	return 0;
	449	}
	450
	451	static inline
	452	enum maple_type mas_parent_enum(struct ma_state mas, struct maple_enode enode)
	453	{
	454	return mte_parent_enum(ma_enode_ptr(mte_to_node(enode)->parent), mas->tree);
	455	}
	456
	457	/*
	458	* mte_set_parent() - Set the parent node and encode the slot
	459	* @enode: The encoded maple node.
	460	* @parent: The encoded maple node that is the parent of @enode.
	461	* @slot: The slot that @enode resides in @parent.
	462	*
	463	* Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the
	464	* parent type.
	465	*/
	466	static inline
	467	void mte_set_parent(struct maple_enode enode, const struct maple_enode parent,
	468	unsigned char slot)
	469	{
	470	unsigned long val = (unsigned long) parent;
	471	unsigned long shift;
	472	unsigned long type;
	473	enum maple_type p_type = mte_node_type(parent);
	474
	475	BUG_ON(p_type == maple_dense);
	476	BUG_ON(p_type == maple_leaf_64);
	477
	478	switch (p_type) {
	479	case maple_range_64:
	480	case maple_arange_64:
	481	shift = MAPLE_PARENT_SLOT_SHIFT;
	482	type = MAPLE_PARENT_RANGE64;
	483	break;
	484	default:
	485	case maple_dense:
	486	case maple_leaf_64:
	487	shift = type = 0;
	488	break;
	489	}
	490
	491	val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */
	492	val \|= (slot << shift) \| type;
	493	mte_to_node(enode)->parent = ma_parent_ptr(val);
	494	}
	495
	496	/*
	497	* mte_parent_slot() - get the parent slot of @enode.
	498	* @enode: The encoded maple node.
	499	*
	500	* Return: The slot in the parent node where @enode resides.
	501	*/
	502	static inline unsigned int mte_parent_slot(const struct maple_enode *enode)
	503	{
	504	unsigned long val = (unsigned long) mte_to_node(enode)->parent;
	505
	506	/* Root. */
	507	if (val & 1)
	508	return 0;
	509
	510	/*
	511	* Okay to use MAPLE_PARENT_16B_SLOT_MASK as the last bit will be lost
	512	* by shift if the parent shift is MAPLE_PARENT_SLOT_SHIFT
	513	*/
	514	return (val & MAPLE_PARENT_16B_SLOT_MASK) >> mte_parent_shift(val);
	515	}
	516
	517	/*
	518	* mte_parent() - Get the parent of @node.
	519	* @node: The encoded maple node.
	520	*
	521	* Return: The parent maple node.
	522	*/
	523	static inline struct maple_node mte_parent(const struct maple_enode enode)
	524	{
	525	return (void *)((unsigned long)
	526	(mte_to_node(enode)->parent) & ~MAPLE_NODE_MASK);
	527	}
	528
	529	/*
	530	* ma_dead_node() - check if the @enode is dead.
	531	* @enode: The encoded maple node
	532	*
	533	* Return: true if dead, false otherwise.
	534	*/
	535	static inline bool ma_dead_node(const struct maple_node *node)
	536	{
	537	struct maple_node parent = (void )((unsigned long)
	538	node->parent & ~MAPLE_NODE_MASK);
	539
	540	return (parent == node);
	541	}
	542	/*
	543	* mte_dead_node() - check if the @enode is dead.
	544	* @enode: The encoded maple node
	545	*
	546	* Return: true if dead, false otherwise.
	547	*/
	548	static inline bool mte_dead_node(const struct maple_enode *enode)
	549	{
	550	struct maple_node parent, node;
	551
	552	node = mte_to_node(enode);
	553	parent = mte_parent(enode);
	554	return (parent == node);
	555	}
	556
	557	/*
	558	* mas_allocated() - Get the number of nodes allocated in a maple state.
	559	* @mas: The maple state
	560	*
	561	* The ma_state alloc member is overloaded to hold a pointer to the first
	562	* allocated node or to the number of requested nodes to allocate. If bit 0 is
	563	* set, then the alloc contains the number of requested nodes. If there is an
	564	* allocated node, then the total allocated nodes is in that node.
	565	*
	566	* Return: The total number of nodes allocated
	567	*/
	568	static inline unsigned long mas_allocated(const struct ma_state *mas)
	569	{
	570	if (!mas->alloc \|\| ((unsigned long)mas->alloc & 0x1))
	571	return 0;
	572
	573	return mas->alloc->total;
	574	}
	575
	576	/*
	577	* mas_set_alloc_req() - Set the requested number of allocations.
	578	* @mas: the maple state
	579	* @count: the number of allocations.
	580	*
	581	* The requested number of allocations is either in the first allocated node,
	582	* located in @mas->alloc->request_count, or directly in @mas->alloc if there is
	583	* no allocated node. Set the request either in the node or do the necessary
	584	* encoding to store in @mas->alloc directly.
	585	*/
	586	static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
	587	{
	588	if (!mas->alloc \|\| ((unsigned long)mas->alloc & 0x1)) {
	589	if (!count)
	590	mas->alloc = NULL;
	591	else
	592	mas->alloc = (struct maple_alloc *)(((count) << 1U) \| 1U);
	593	return;
	594	}
	595
	596	mas->alloc->request_count = count;
	597	}
	598
	599	/*
	600	* mas_alloc_req() - get the requested number of allocations.
	601	* @mas: The maple state
	602	*
	603	* The alloc count is either stored directly in @mas, or in
	604	* @mas->alloc->request_count if there is at least one node allocated. Decode
	605	* the request count if it's stored directly in @mas->alloc.
	606	*
	607	* Return: The allocation request count.
	608	*/
	609	static inline unsigned int mas_alloc_req(const struct ma_state *mas)
	610	{
	611	if ((unsigned long)mas->alloc & 0x1)
	612	return (unsigned long)(mas->alloc) >> 1;
	613	else if (mas->alloc)
	614	return mas->alloc->request_count;
	615	return 0;
	616	}
	617
	618	/*
	619	* ma_pivots() - Get a pointer to the maple node pivots.
	620	* @node - the maple node
	621	* @type - the node type
	622	*
	623	* Return: A pointer to the maple node pivots
	624	*/
	625	static inline unsigned long ma_pivots(struct maple_node node,
	626	enum maple_type type)
	627	{
	628	switch (type) {
	629	case maple_arange_64:
	630	return node->ma64.pivot;
	631	case maple_range_64:
	632	case maple_leaf_64:
	633	return node->mr64.pivot;
	634	case maple_dense:
	635	return NULL;
	636	}
	637	return NULL;
	638	}
	639
	640	/*
	641	* ma_gaps() - Get a pointer to the maple node gaps.
	642	* @node - the maple node
	643	* @type - the node type
	644	*
	645	* Return: A pointer to the maple node gaps
	646	*/
	647	static inline unsigned long ma_gaps(struct maple_node node,
	648	enum maple_type type)
	649	{
	650	switch (type) {
	651	case maple_arange_64:
	652	return node->ma64.gap;
	653	case maple_range_64:
	654	case maple_leaf_64:
	655	case maple_dense:
	656	return NULL;
	657	}
	658	return NULL;
	659	}
	660
	661	/*
	662	* mte_pivot() - Get the pivot at @piv of the maple encoded node.
	663	* @mn: The maple encoded node.
	664	* @piv: The pivot.
	665	*
	666	* Return: the pivot at @piv of @mn.
	667	*/
	668	static inline unsigned long mte_pivot(const struct maple_enode *mn,
	669	unsigned char piv)
	670	{
	671	struct maple_node *node = mte_to_node(mn);
	672
	673	if (piv >= mt_pivots[piv]) {
	674	WARN_ON(1);
	675	return 0;
	676	}
	677	switch (mte_node_type(mn)) {
	678	case maple_arange_64:
	679	return node->ma64.pivot[piv];
	680	case maple_range_64:
	681	case maple_leaf_64:
	682	return node->mr64.pivot[piv];
	683	case maple_dense:
	684	return 0;
	685	}
	686	return 0;
	687	}
	688
	689	/*
	690	* mas_safe_pivot() - get the pivot at @piv or mas->max.
	691	* @mas: The maple state
	692	* @pivots: The pointer to the maple node pivots
	693	* @piv: The pivot to fetch
	694	* @type: The maple node type
	695	*
	696	* Return: The pivot at @piv within the limit of the @pivots array, @mas->max
	697	* otherwise.
	698	*/
	699	static inline unsigned long
	700	mas_safe_pivot(const struct ma_state mas, unsigned long pivots,
	701	unsigned char piv, enum maple_type type)
	702	{
	703	if (piv >= mt_pivots[type])
	704	return mas->max;
	705
	706	return pivots[piv];
	707	}
	708
	709	/*
	710	* mas_safe_min() - Return the minimum for a given offset.
	711	* @mas: The maple state
	712	* @pivots: The pointer to the maple node pivots
	713	* @offset: The offset into the pivot array
	714	*
	715	* Return: The minimum range value that is contained in @offset.
	716	*/
	717	static inline unsigned long
	718	mas_safe_min(struct ma_state mas, unsigned long pivots, unsigned char offset)
	719	{
	720	if (likely(offset))
	721	return pivots[offset - 1] + 1;
	722
	723	return mas->min;
	724	}
	725
	726	/*
	727	* mas_logical_pivot() - Get the logical pivot of a given offset.
	728	* @mas: The maple state
	729	* @pivots: The pointer to the maple node pivots
	730	* @offset: The offset into the pivot array
	731	* @type: The maple node type
	732	*
	733	* When there is no value at a pivot (beyond the end of the data), then the
	734	* pivot is actually @mas->max.
	735	*
	736	* Return: the logical pivot of a given @offset.
	737	*/
	738	static inline unsigned long
	739	mas_logical_pivot(struct ma_state mas, unsigned long pivots,
	740	unsigned char offset, enum maple_type type)
	741	{
	742	unsigned long lpiv = mas_safe_pivot(mas, pivots, offset, type);
	743
	744	if (likely(lpiv))
	745	return lpiv;
	746
	747	if (likely(offset))
	748	return mas->max;
	749
	750	return lpiv;
	751	}
	752
	753	/*
	754	* mte_set_pivot() - Set a pivot to a value in an encoded maple node.
	755	* @mn: The encoded maple node
	756	* @piv: The pivot offset
	757	* @val: The value of the pivot
	758	*/
	759	static inline void mte_set_pivot(struct maple_enode *mn, unsigned char piv,
	760	unsigned long val)
	761	{
	762	struct maple_node *node = mte_to_node(mn);
	763	enum maple_type type = mte_node_type(mn);
	764
	765	BUG_ON(piv >= mt_pivots[type]);
	766	switch (type) {
	767	default:
	768	case maple_range_64:
	769	case maple_leaf_64:
	770	node->mr64.pivot[piv] = val;
	771	break;
	772	case maple_arange_64:
	773	node->ma64.pivot[piv] = val;
	774	break;
	775	case maple_dense:
	776	break;
	777	}
	778
	779	}
	780
	781	/*
	782	* ma_slots() - Get a pointer to the maple node slots.
	783	* @mn: The maple node
	784	* @mt: The maple node type
	785	*
	786	* Return: A pointer to the maple node slots
	787	*/
	788	static inline void __rcu *ma_slots(struct maple_node mn, enum maple_type mt)
	789	{
	790	switch (mt) {
	791	default:
	792	case maple_arange_64:
	793	return mn->ma64.slot;
	794	case maple_range_64:
	795	case maple_leaf_64:
	796	return mn->mr64.slot;
	797	case maple_dense:
	798	return mn->slot;
	799	}
	800	}
	801
	802	static inline bool mt_locked(const struct maple_tree *mt)
	803	{
	804	return mt_external_lock(mt) ? mt_lock_is_held(mt) :
	805	lockdep_is_held(&mt->ma_lock);
	806	}
	807
	808	static inline void mt_slot(const struct maple_tree mt,
	809	void __rcu **slots, unsigned char offset)
	810	{
	811	return rcu_dereference_check(slots[offset], mt_locked(mt));
	812	}
	813
	814	/*
	815	* mas_slot_locked() - Get the slot value when holding the maple tree lock.
	816	* @mas: The maple state
	817	* @slots: The pointer to the slots
	818	* @offset: The offset into the slots array to fetch
	819	*
	820	* Return: The entry stored in @slots at the @offset.
	821	*/
	822	static inline void mas_slot_locked(struct ma_state mas, void __rcu **slots,
	823	unsigned char offset)
	824	{
	825	return rcu_dereference_protected(slots[offset], mt_locked(mas->tree));
	826	}
	827
	828	/*
	829	* mas_slot() - Get the slot value when not holding the maple tree lock.
	830	* @mas: The maple state
	831	* @slots: The pointer to the slots
	832	* @offset: The offset into the slots array to fetch
	833	*
	834	* Return: The entry stored in @slots at the @offset
	835	*/
	836	static inline void mas_slot(struct ma_state mas, void __rcu **slots,
	837	unsigned char offset)
	838	{
	839	return mt_slot(mas->tree, slots, offset);
	840	}
	841
	842	/*
	843	* mas_root() - Get the maple tree root.
	844	* @mas: The maple state.
	845	*
	846	* Return: The pointer to the root of the tree
	847	*/
	848	static inline void mas_root(struct ma_state mas)
	849	{
	850	return rcu_dereference_check(mas->tree->ma_root, mt_locked(mas->tree));
	851	}
	852
	853	static inline void mt_root_locked(struct maple_tree mt)
	854	{
	855	return rcu_dereference_protected(mt->ma_root, mt_locked(mt));
	856	}
	857
	858	/*
	859	* mas_root_locked() - Get the maple tree root when holding the maple tree lock.
	860	* @mas: The maple state.
	861	*
	862	* Return: The pointer to the root of the tree
	863	*/
	864	static inline void mas_root_locked(struct ma_state mas)
	865	{
	866	return mt_root_locked(mas->tree);
	867	}
	868
	869	static inline struct maple_metadata ma_meta(struct maple_node mn,
	870	enum maple_type mt)
	871	{
	872	switch (mt) {
	873	case maple_arange_64:
	874	return &mn->ma64.meta;
	875	default:
	876	return &mn->mr64.meta;
	877	}
	878	}
	879
	880	/*
	881	* ma_set_meta() - Set the metadata information of a node.
	882	* @mn: The maple node
	883	* @mt: The maple node type
	884	* @offset: The offset of the highest sub-gap in this node.
	885	* @end: The end of the data in this node.
	886	*/
	887	static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt,
	888	unsigned char offset, unsigned char end)
	889	{
	890	struct maple_metadata *meta = ma_meta(mn, mt);
	891
	892	meta->gap = offset;
	893	meta->end = end;
	894	}
	895
	896	/*
	897	* ma_meta_end() - Get the data end of a node from the metadata
	898	* @mn: The maple node
	899	* @mt: The maple node type
	900	*/
	901	static inline unsigned char ma_meta_end(struct maple_node *mn,
	902	enum maple_type mt)
	903	{
	904	struct maple_metadata *meta = ma_meta(mn, mt);
	905
	906	return meta->end;
	907	}
	908
	909	/*
	910	* ma_meta_gap() - Get the largest gap location of a node from the metadata
	911	* @mn: The maple node
	912	* @mt: The maple node type
	913	*/
	914	static inline unsigned char ma_meta_gap(struct maple_node *mn,
	915	enum maple_type mt)
	916	{
	917	BUG_ON(mt != maple_arange_64);
	918
	919	return mn->ma64.meta.gap;
	920	}
	921
	922	/*
	923	* ma_set_meta_gap() - Set the largest gap location in a nodes metadata
	924	* @mn: The maple node
	925	* @mn: The maple node type
	926	* @offset: The location of the largest gap.
	927	*/
	928	static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt,
	929	unsigned char offset)
	930	{
	931
	932	struct maple_metadata *meta = ma_meta(mn, mt);
	933
	934	meta->gap = offset;
	935	}
	936
	937	/*
	938	* mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes.
	939	* @mat - the ma_topiary, a linked list of dead nodes.
	940	* @dead_enode - the node to be marked as dead and added to the tail of the list
	941	*
	942	* Add the @dead_enode to the linked list in @mat.
	943	*/
	944	static inline void mat_add(struct ma_topiary *mat,
	945	struct maple_enode *dead_enode)
	946	{
	947	mte_set_node_dead(dead_enode);
	948	mte_to_mat(dead_enode)->next = NULL;
	949	if (!mat->tail) {
	950	mat->tail = mat->head = dead_enode;
	951	return;
	952	}
	953
	954	mte_to_mat(mat->tail)->next = dead_enode;
	955	mat->tail = dead_enode;
	956	}
	957
	958	static void mte_destroy_walk(struct maple_enode , struct maple_tree );
	959	static inline void mas_free(struct ma_state mas, struct maple_enode used);
	960
	961	/*
	962	* mas_mat_free() - Free all nodes in a dead list.
	963	* @mas - the maple state
	964	* @mat - the ma_topiary linked list of dead nodes to free.
	965	*
	966	* Free walk a dead list.
	967	*/
	968	static void mas_mat_free(struct ma_state mas, struct ma_topiary mat)
	969	{
	970	struct maple_enode *next;
	971
	972	while (mat->head) {
	973	next = mte_to_mat(mat->head)->next;
	974	mas_free(mas, mat->head);
	975	mat->head = next;
	976	}
	977	}
	978
	979	/*
	980	* mas_mat_destroy() - Free all nodes and subtrees in a dead list.
	981	* @mas - the maple state
	982	* @mat - the ma_topiary linked list of dead nodes to free.
	983	*
	984	* Destroy walk a dead list.
	985	*/
	986	static void mas_mat_destroy(struct ma_state mas, struct ma_topiary mat)
	987	{
	988	struct maple_enode *next;
	989
	990	while (mat->head) {
	991	next = mte_to_mat(mat->head)->next;
	992	mte_destroy_walk(mat->head, mat->mtree);
	993	mat->head = next;
	994	}
	995	}
	996	/*
	997	* mas_descend() - Descend into the slot stored in the ma_state.
	998	* @mas - the maple state.
	999	*
	1000	* Note: Not RCU safe, only use in write side or debug code.
	1001	*/
	1002	static inline void mas_descend(struct ma_state *mas)
	1003	{
	1004	enum maple_type type;
	1005	unsigned long *pivots;
	1006	struct maple_node *node;
	1007	void __rcu **slots;
	1008
	1009	node = mas_mn(mas);
	1010	type = mte_node_type(mas->node);
	1011	pivots = ma_pivots(node, type);
	1012	slots = ma_slots(node, type);
	1013
	1014	if (mas->offset)
	1015	mas->min = pivots[mas->offset - 1] + 1;
	1016	mas->max = mas_safe_pivot(mas, pivots, mas->offset, type);
	1017	mas->node = mas_slot(mas, slots, mas->offset);
	1018	}
	1019
	1020	/*
	1021	* mte_set_gap() - Set a maple node gap.
	1022	* @mn: The encoded maple node
	1023	* @gap: The offset of the gap to set
	1024	* @val: The gap value
	1025	*/
	1026	static inline void mte_set_gap(const struct maple_enode *mn,
	1027	unsigned char gap, unsigned long val)
	1028	{
	1029	switch (mte_node_type(mn)) {
	1030	default:
	1031	break;
	1032	case maple_arange_64:
	1033	mte_to_node(mn)->ma64.gap[gap] = val;
	1034	break;
	1035	}
	1036	}
	1037
	1038	/*
	1039	* mas_ascend() - Walk up a level of the tree.
	1040	* @mas: The maple state
	1041	*
	1042	* Sets the @mas->max and @mas->min to the correct values when walking up. This
	1043	* may cause several levels of walking up to find the correct min and max.
	1044	* May find a dead node which will cause a premature return.
	1045	* Return: 1 on dead node, 0 otherwise
	1046	*/
	1047	static int mas_ascend(struct ma_state *mas)
	1048	{
	1049	struct maple_enode p_enode; / parent enode. */
	1050	struct maple_enode a_enode; / ancestor enode. */
	1051	struct maple_node a_node; / ancestor node. */
	1052	struct maple_node p_node; / parent node. */
	1053	unsigned char a_slot;
	1054	enum maple_type a_type;
	1055	unsigned long min, max;
	1056	unsigned long *pivots;
	1057	unsigned char offset;
	1058	bool set_max = false, set_min = false;
	1059
	1060	a_node = mas_mn(mas);
	1061	if (ma_is_root(a_node)) {
	1062	mas->offset = 0;
	1063	return 0;
	1064	}
	1065
	1066	p_node = mte_parent(mas->node);
	1067	if (unlikely(a_node == p_node))
	1068	return 1;
	1069	a_type = mas_parent_enum(mas, mas->node);
	1070	offset = mte_parent_slot(mas->node);
	1071	a_enode = mt_mk_node(p_node, a_type);
	1072
	1073	/* Check to make sure all parent information is still accurate */
	1074	if (p_node != mte_parent(mas->node))
	1075	return 1;
	1076
	1077	mas->node = a_enode;
	1078	mas->offset = offset;
	1079
	1080	if (mte_is_root(a_enode)) {
	1081	mas->max = ULONG_MAX;
	1082	mas->min = 0;
	1083	return 0;
	1084	}
	1085
	1086	min = 0;
	1087	max = ULONG_MAX;
	1088	do {
	1089	p_enode = a_enode;
	1090	a_type = mas_parent_enum(mas, p_enode);
	1091	a_node = mte_parent(p_enode);
	1092	a_slot = mte_parent_slot(p_enode);
	1093	pivots = ma_pivots(a_node, a_type);
	1094	a_enode = mt_mk_node(a_node, a_type);
	1095
	1096	if (!set_min && a_slot) {
	1097	set_min = true;
	1098	min = pivots[a_slot - 1] + 1;
	1099	}
	1100
	1101	if (!set_max && a_slot < mt_pivots[a_type]) {
	1102	set_max = true;
	1103	max = pivots[a_slot];
	1104	}
	1105
	1106	if (unlikely(ma_dead_node(a_node)))
	1107	return 1;
	1108
	1109	if (unlikely(ma_is_root(a_node)))
	1110	break;
	1111
	1112	} while (!set_min \|\| !set_max);
	1113
	1114	mas->max = max;
	1115	mas->min = min;
	1116	return 0;
	1117	}
	1118
	1119	/*
	1120	* mas_pop_node() - Get a previously allocated maple node from the maple state.
	1121	* @mas: The maple state
	1122	*
	1123	* Return: A pointer to a maple node.
	1124	*/
	1125	static inline struct maple_node mas_pop_node(struct ma_state mas)
	1126	{
	1127	struct maple_alloc ret, node = mas->alloc;
	1128	unsigned long total = mas_allocated(mas);
	1129
	1130	/* nothing or a request pending. */
	1131	if (unlikely(!total))
	1132	return NULL;
	1133
	1134	if (total == 1) {
	1135	/* single allocation in this ma_state */
	1136	mas->alloc = NULL;
	1137	ret = node;
	1138	goto single_node;
	1139	}
	1140
	1141	if (!node->node_count) {
	1142	/* Single allocation in this node. */
	1143	mas->alloc = node->slot[0];
	1144	node->slot[0] = NULL;
	1145	mas->alloc->total = node->total - 1;
	1146	ret = node;
	1147	goto new_head;
	1148	}
	1149
	1150	node->total--;
	1151	ret = node->slot[node->node_count];
	1152	node->slot[node->node_count--] = NULL;
	1153
	1154	single_node:
	1155	new_head:
	1156	ret->total = 0;
	1157	ret->node_count = 0;
	1158	if (ret->request_count) {
	1159	mas_set_alloc_req(mas, ret->request_count + 1);
	1160	ret->request_count = 0;
	1161	}
	1162	return (struct maple_node *)ret;
	1163	}
	1164
	1165	/*
	1166	* mas_push_node() - Push a node back on the maple state allocation.
	1167	* @mas: The maple state
	1168	* @used: The used maple node
	1169	*
	1170	* Stores the maple node back into @mas->alloc for reuse. Updates allocated and
	1171	* requested node count as necessary.
	1172	*/
	1173	static inline void mas_push_node(struct ma_state mas, struct maple_node used)
	1174	{
	1175	struct maple_alloc reuse = (struct maple_alloc )used;
	1176	struct maple_alloc *head = mas->alloc;
	1177	unsigned long count;
	1178	unsigned int requested = mas_alloc_req(mas);
	1179
	1180	memset(reuse, 0, sizeof(*reuse));
	1181	count = mas_allocated(mas);
	1182
	1183	if (count && (head->node_count < MAPLE_ALLOC_SLOTS - 1)) {
	1184	if (head->slot[0])
	1185	head->node_count++;
	1186	head->slot[head->node_count] = reuse;
	1187	head->total++;
	1188	goto done;
	1189	}
	1190
	1191	reuse->total = 1;
	1192	if ((head) && !((unsigned long)head & 0x1)) {
	1193	head->request_count = 0;
	1194	reuse->slot[0] = head;
	1195	reuse->total += head->total;
	1196	}
	1197
	1198	mas->alloc = reuse;
	1199	done:
	1200	if (requested > 1)
	1201	mas_set_alloc_req(mas, requested - 1);
	1202	}
	1203
	1204	/*
	1205	* mas_alloc_nodes() - Allocate nodes into a maple state
	1206	* @mas: The maple state
	1207	* @gfp: The GFP Flags
	1208	*/
	1209	static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
	1210	{
	1211	struct maple_alloc *node;
	1212	struct maple_alloc **nodep = &mas->alloc;
	1213	unsigned long allocated = mas_allocated(mas);
	1214	unsigned long success = allocated;
	1215	unsigned int requested = mas_alloc_req(mas);
	1216	unsigned int count;
	1217	void **slots = NULL;
	1218	unsigned int max_req = 0;
	1219
	1220	if (!requested)
	1221	return;
	1222
	1223	mas_set_alloc_req(mas, 0);
	1224	if (mas->mas_flags & MA_STATE_PREALLOC) {
	1225	if (allocated)
	1226	return;
	1227	WARN_ON(!allocated);
	1228	}
	1229
	1230	if (!allocated \|\| mas->alloc->node_count == MAPLE_ALLOC_SLOTS - 1) {
	1231	node = (struct maple_alloc *)mt_alloc_one(gfp);
	1232	if (!node)
	1233	goto nomem_one;
	1234
	1235	if (allocated)
	1236	node->slot[0] = mas->alloc;
	1237
	1238	success++;
	1239	mas->alloc = node;
	1240	requested--;
	1241	}
	1242
	1243	node = mas->alloc;
	1244	while (requested) {
	1245	max_req = MAPLE_ALLOC_SLOTS;
	1246	if (node->slot[0]) {
	1247	unsigned int offset = node->node_count + 1;
	1248
	1249	slots = (void **)&node->slot[offset];
	1250	max_req -= offset;
	1251	} else {
	1252	slots = (void **)&node->slot;
	1253	}
	1254
	1255	max_req = min(requested, max_req);
	1256	count = mt_alloc_bulk(gfp, max_req, slots);
	1257	if (!count)
	1258	goto nomem_bulk;
	1259
	1260	node->node_count += count;
	1261	/* zero indexed. */
	1262	if (slots == (void **)&node->slot)
	1263	node->node_count--;
	1264
	1265	success += count;
	1266	nodep = &node->slot[0];
	1267	node = *nodep;
	1268	requested -= count;
	1269	}
	1270	mas->alloc->total = success;
	1271	return;
	1272
	1273	nomem_bulk:
	1274	/* Clean up potential freed allocations on bulk failure */
	1275	memset(slots, 0, max_req * sizeof(unsigned long));
	1276	nomem_one:
	1277	mas_set_alloc_req(mas, requested);
	1278	if (mas->alloc && !(((unsigned long)mas->alloc & 0x1)))
	1279	mas->alloc->total = success;
	1280	mas_set_err(mas, -ENOMEM);
	1281	return;
	1282
	1283	}
	1284
	1285	/*
	1286	* mas_free() - Free an encoded maple node
	1287	* @mas: The maple state
	1288	* @used: The encoded maple node to free.
	1289	*
	1290	* Uses rcu free if necessary, pushes @used back on the maple state allocations
	1291	* otherwise.
	1292	*/
	1293	static inline void mas_free(struct ma_state mas, struct maple_enode used)
	1294	{
	1295	struct maple_node *tmp = mte_to_node(used);
	1296
	1297	if (mt_in_rcu(mas->tree))
	1298	ma_free_rcu(tmp);
	1299	else
	1300	mas_push_node(mas, tmp);
	1301	}
	1302
	1303	/*
	1304	* mas_node_count() - Check if enough nodes are allocated and request more if
	1305	* there is not enough nodes.
	1306	* @mas: The maple state
	1307	* @count: The number of nodes needed
	1308	* @gfp: the gfp flags
	1309	*/
	1310	static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
	1311	{
	1312	unsigned long allocated = mas_allocated(mas);
	1313
	1314	if (allocated < count) {
	1315	mas_set_alloc_req(mas, count - allocated);
	1316	mas_alloc_nodes(mas, gfp);
	1317	}
	1318	}
	1319
	1320	/*
	1321	* mas_node_count() - Check if enough nodes are allocated and request more if
	1322	* there is not enough nodes.
	1323	* @mas: The maple state
	1324	* @count: The number of nodes needed
	1325	*
	1326	* Note: Uses GFP_NOWAIT \| __GFP_NOWARN for gfp flags.
	1327	*/
	1328	static void mas_node_count(struct ma_state *mas, int count)
	1329	{
	1330	return mas_node_count_gfp(mas, count, GFP_NOWAIT \| __GFP_NOWARN);
	1331	}
	1332
	1333	/*
	1334	* mas_start() - Sets up maple state for operations.
	1335	* @mas: The maple state.
	1336	*
	1337	* If mas->node == MAS_START, then set the min, max, depth, and offset to
	1338	* defaults.
	1339	*
	1340	* Return:
	1341	* - If mas->node is an error or not MAS_START, return NULL.
	1342	* - If it's an empty tree: NULL & mas->node == MAS_NONE
	1343	* - If it's a single entry: The entry & mas->node == MAS_ROOT
	1344	* - If it's a tree: NULL & mas->node == safe root node.
	1345	*/
	1346	static inline struct maple_enode mas_start(struct ma_state mas)
	1347	{
	1348	if (likely(mas_is_start(mas))) {
	1349	struct maple_enode *root;
	1350
	1351	mas->node = MAS_NONE;
	1352	mas->min = 0;
	1353	mas->max = ULONG_MAX;
	1354	mas->depth = 0;
	1355	mas->offset = 0;
	1356
	1357	root = mas_root(mas);
	1358	/* Tree with nodes */
	1359	if (likely(xa_is_node(root))) {
	1360	mas->node = mte_safe_root(root);
	1361	return NULL;
	1362	}
	1363
	1364	/* empty tree */
	1365	if (unlikely(!root)) {
	1366	mas->offset = MAPLE_NODE_SLOTS;
	1367	return NULL;
	1368	}
	1369
	1370	/* Single entry tree */
	1371	mas->node = MAS_ROOT;
	1372	mas->offset = MAPLE_NODE_SLOTS;
	1373
	1374	/* Single entry tree. */
	1375	if (mas->index > 0)
	1376	return NULL;
	1377
	1378	return root;
	1379	}
	1380
	1381	return NULL;
	1382	}
	1383
	1384	/*
	1385	* ma_data_end() - Find the end of the data in a node.
	1386	* @node: The maple node
	1387	* @type: The maple node type
	1388	* @pivots: The array of pivots in the node
	1389	* @max: The maximum value in the node
	1390	*
	1391	* Uses metadata to find the end of the data when possible.
	1392	* Return: The zero indexed last slot with data (may be null).
	1393	*/
	1394	static inline unsigned char ma_data_end(struct maple_node *node,
	1395	enum maple_type type,
	1396	unsigned long *pivots,
	1397	unsigned long max)
	1398	{
	1399	unsigned char offset;
	1400
	1401	if (type == maple_arange_64)
	1402	return ma_meta_end(node, type);
	1403
	1404	offset = mt_pivots[type] - 1;
	1405	if (likely(!pivots[offset]))
	1406	return ma_meta_end(node, type);
	1407
	1408	if (likely(pivots[offset] == max))
	1409	return offset;
	1410
	1411	return mt_pivots[type];
	1412	}
	1413
	1414	/*
	1415	* mas_data_end() - Find the end of the data (slot).
	1416	* @mas: the maple state
	1417	*
	1418	* This method is optimized to check the metadata of a node if the node type
	1419	* supports data end metadata.
	1420	*
	1421	* Return: The zero indexed last slot with data (may be null).
	1422	*/
	1423	static inline unsigned char mas_data_end(struct ma_state *mas)
	1424	{
	1425	enum maple_type type;
	1426	struct maple_node *node;
	1427	unsigned char offset;
	1428	unsigned long *pivots;
	1429
	1430	type = mte_node_type(mas->node);
	1431	node = mas_mn(mas);
	1432	if (type == maple_arange_64)
	1433	return ma_meta_end(node, type);
	1434
	1435	pivots = ma_pivots(node, type);
	1436	offset = mt_pivots[type] - 1;
	1437	if (likely(!pivots[offset]))
	1438	return ma_meta_end(node, type);
	1439
	1440	if (likely(pivots[offset] == mas->max))
	1441	return offset;
	1442
	1443	return mt_pivots[type];
	1444	}
	1445
	1446	/*
	1447	* mas_leaf_max_gap() - Returns the largest gap in a leaf node
	1448	* @mas - the maple state
	1449	*
	1450	* Return: The maximum gap in the leaf.
	1451	*/
	1452	static unsigned long mas_leaf_max_gap(struct ma_state *mas)
	1453	{
	1454	enum maple_type mt;
	1455	unsigned long pstart, gap, max_gap;
	1456	struct maple_node *mn;
	1457	unsigned long *pivots;
	1458	void __rcu **slots;
	1459	unsigned char i;
	1460	unsigned char max_piv;
	1461
	1462	mt = mte_node_type(mas->node);
	1463	mn = mas_mn(mas);
	1464	slots = ma_slots(mn, mt);
	1465	max_gap = 0;
	1466	if (unlikely(ma_is_dense(mt))) {
	1467	gap = 0;
	1468	for (i = 0; i < mt_slots[mt]; i++) {
	1469	if (slots[i]) {
	1470	if (gap > max_gap)
	1471	max_gap = gap;
	1472	gap = 0;
	1473	} else {
	1474	gap++;
	1475	}
	1476	}
	1477	if (gap > max_gap)
	1478	max_gap = gap;
	1479	return max_gap;
	1480	}
	1481
	1482	/*
	1483	* Check the first implied pivot optimizes the loop below and slot 1 may
	1484	* be skipped if there is a gap in slot 0.
	1485	*/
	1486	pivots = ma_pivots(mn, mt);
	1487	if (likely(!slots[0])) {
	1488	max_gap = pivots[0] - mas->min + 1;
	1489	i = 2;
	1490	} else {
	1491	i = 1;
	1492	}
	1493
	1494	/* reduce max_piv as the special case is checked before the loop */
	1495	max_piv = ma_data_end(mn, mt, pivots, mas->max) - 1;
	1496	/*
	1497	* Check end implied pivot which can only be a gap on the right most
	1498	* node.
	1499	*/
	1500	if (unlikely(mas->max == ULONG_MAX) && !slots[max_piv + 1]) {
	1501	gap = ULONG_MAX - pivots[max_piv];
	1502	if (gap > max_gap)
	1503	max_gap = gap;
	1504	}
	1505
	1506	for (; i <= max_piv; i++) {
	1507	/* data == no gap. */
	1508	if (likely(slots[i]))
	1509	continue;
	1510
	1511	pstart = pivots[i - 1];
	1512	gap = pivots[i] - pstart;
	1513	if (gap > max_gap)
	1514	max_gap = gap;
	1515
	1516	/* There cannot be two gaps in a row. */
	1517	i++;
	1518	}
	1519	return max_gap;
	1520	}
	1521
	1522	/*
	1523	* ma_max_gap() - Get the maximum gap in a maple node (non-leaf)
	1524	* @node: The maple node
	1525	* @gaps: The pointer to the gaps
	1526	* @mt: The maple node type
	1527	* @*off: Pointer to store the offset location of the gap.
	1528	*
	1529	* Uses the metadata data end to scan backwards across set gaps.
	1530	*
	1531	* Return: The maximum gap value
	1532	*/
	1533	static inline unsigned long
	1534	ma_max_gap(struct maple_node node, unsigned long gaps, enum maple_type mt,
	1535	unsigned char *off)
	1536	{
	1537	unsigned char offset, i;
	1538	unsigned long max_gap = 0;
	1539
	1540	i = offset = ma_meta_end(node, mt);
	1541	do {
	1542	if (gaps[i] > max_gap) {
	1543	max_gap = gaps[i];
	1544	offset = i;
	1545	}
	1546	} while (i--);
	1547
	1548	*off = offset;
	1549	return max_gap;
	1550	}
	1551
	1552	/*
	1553	* mas_max_gap() - find the largest gap in a non-leaf node and set the slot.
	1554	* @mas: The maple state.
	1555	*
	1556	* If the metadata gap is set to MAPLE_ARANGE64_META_MAX, there is no gap.
	1557	*
	1558	* Return: The gap value.
	1559	*/
	1560	static inline unsigned long mas_max_gap(struct ma_state *mas)
	1561	{
	1562	unsigned long *gaps;
	1563	unsigned char offset;
	1564	enum maple_type mt;
	1565	struct maple_node *node;
	1566
	1567	mt = mte_node_type(mas->node);
	1568	if (ma_is_leaf(mt))
	1569	return mas_leaf_max_gap(mas);
	1570
	1571	node = mas_mn(mas);
	1572	offset = ma_meta_gap(node, mt);
	1573	if (offset == MAPLE_ARANGE64_META_MAX)
	1574	return 0;
	1575
	1576	gaps = ma_gaps(node, mt);
	1577	return gaps[offset];
	1578	}
	1579
	1580	/*
	1581	* mas_parent_gap() - Set the parent gap and any gaps above, as needed
	1582	* @mas: The maple state
	1583	* @offset: The gap offset in the parent to set
	1584	* @new: The new gap value.
	1585	*
	1586	* Set the parent gap then continue to set the gap upwards, using the metadata
	1587	* of the parent to see if it is necessary to check the node above.
	1588	*/
	1589	static inline void mas_parent_gap(struct ma_state *mas, unsigned char offset,
	1590	unsigned long new)
	1591	{
	1592	unsigned long meta_gap = 0;
	1593	struct maple_node *pnode;
	1594	struct maple_enode *penode;
	1595	unsigned long *pgaps;
	1596	unsigned char meta_offset;
	1597	enum maple_type pmt;
	1598
	1599	pnode = mte_parent(mas->node);
	1600	pmt = mas_parent_enum(mas, mas->node);
	1601	penode = mt_mk_node(pnode, pmt);
	1602	pgaps = ma_gaps(pnode, pmt);
	1603
	1604	ascend:
	1605	meta_offset = ma_meta_gap(pnode, pmt);
	1606	if (meta_offset == MAPLE_ARANGE64_META_MAX)
	1607	meta_gap = 0;
	1608	else
	1609	meta_gap = pgaps[meta_offset];
	1610
	1611	pgaps[offset] = new;
	1612
	1613	if (meta_gap == new)
	1614	return;
	1615
	1616	if (offset != meta_offset) {
	1617	if (meta_gap > new)
	1618	return;
	1619
	1620	ma_set_meta_gap(pnode, pmt, offset);
	1621	} else if (new < meta_gap) {
	1622	meta_offset = 15;
	1623	new = ma_max_gap(pnode, pgaps, pmt, &meta_offset);
	1624	ma_set_meta_gap(pnode, pmt, meta_offset);
	1625	}
	1626
	1627	if (ma_is_root(pnode))
	1628	return;
	1629
	1630	/* Go to the parent node. */
	1631	pnode = mte_parent(penode);
	1632	pmt = mas_parent_enum(mas, penode);
	1633	pgaps = ma_gaps(pnode, pmt);
	1634	offset = mte_parent_slot(penode);
	1635	penode = mt_mk_node(pnode, pmt);
	1636	goto ascend;
	1637	}
	1638
	1639	/*
	1640	* mas_update_gap() - Update a nodes gaps and propagate up if necessary.
	1641	* @mas - the maple state.
	1642	*/
	1643	static inline void mas_update_gap(struct ma_state *mas)
	1644	{
	1645	unsigned char pslot;
	1646	unsigned long p_gap;
	1647	unsigned long max_gap;
	1648
	1649	if (!mt_is_alloc(mas->tree))
	1650	return;
	1651
	1652	if (mte_is_root(mas->node))
	1653	return;
	1654
	1655	max_gap = mas_max_gap(mas);
	1656
	1657	pslot = mte_parent_slot(mas->node);
	1658	p_gap = ma_gaps(mte_parent(mas->node),
	1659	mas_parent_enum(mas, mas->node))[pslot];
	1660
	1661	if (p_gap != max_gap)
	1662	mas_parent_gap(mas, pslot, max_gap);
	1663	}
	1664
	1665	/*
	1666	* mas_adopt_children() - Set the parent pointer of all nodes in @parent to
	1667	* @parent with the slot encoded.
	1668	* @mas - the maple state (for the tree)
	1669	* @parent - the maple encoded node containing the children.
	1670	*/
	1671	static inline void mas_adopt_children(struct ma_state *mas,
	1672	struct maple_enode *parent)
	1673	{
	1674	enum maple_type type = mte_node_type(parent);
	1675	struct maple_node *node = mas_mn(mas);
	1676	void __rcu **slots = ma_slots(node, type);
	1677	unsigned long *pivots = ma_pivots(node, type);
	1678	struct maple_enode *child;
	1679	unsigned char offset;
	1680
	1681	offset = ma_data_end(node, type, pivots, mas->max);
	1682	do {
	1683	child = mas_slot_locked(mas, slots, offset);
	1684	mte_set_parent(child, parent, offset);
	1685	} while (offset--);
	1686	}
	1687
	1688	/*
	1689	* mas_replace() - Replace a maple node in the tree with mas->node. Uses the
	1690	* parent encoding to locate the maple node in the tree.
	1691	* @mas - the ma_state to use for operations.
	1692	* @advanced - boolean to adopt the child nodes and free the old node (false) or
	1693	* leave the node (true) and handle the adoption and free elsewhere.
	1694	*/
	1695	static inline void mas_replace(struct ma_state *mas, bool advanced)
	1696	__must_hold(mas->tree->lock)
	1697	{
	1698	struct maple_node *mn = mas_mn(mas);
	1699	struct maple_enode *old_enode;
	1700	unsigned char offset = 0;
	1701	void __rcu **slots = NULL;
	1702
	1703	if (ma_is_root(mn)) {
	1704	old_enode = mas_root_locked(mas);
	1705	} else {
	1706	offset = mte_parent_slot(mas->node);
	1707	slots = ma_slots(mte_parent(mas->node),
	1708	mas_parent_enum(mas, mas->node));
	1709	old_enode = mas_slot_locked(mas, slots, offset);
	1710	}
	1711
	1712	if (!advanced && !mte_is_leaf(mas->node))
	1713	mas_adopt_children(mas, mas->node);
	1714
	1715	if (mte_is_root(mas->node)) {
	1716	mn->parent = ma_parent_ptr(
	1717	((unsigned long)mas->tree \| MA_ROOT_PARENT));
	1718	rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
	1719	mas_set_height(mas);
	1720	} else {
	1721	rcu_assign_pointer(slots[offset], mas->node);
	1722	}
	1723
	1724	if (!advanced)
	1725	mas_free(mas, old_enode);
	1726	}
	1727
	1728	/*
	1729	* mas_new_child() - Find the new child of a node.
	1730	* @mas: the maple state
	1731	* @child: the maple state to store the child.
	1732	*/
	1733	static inline bool mas_new_child(struct ma_state mas, struct ma_state child)
	1734	__must_hold(mas->tree->lock)
	1735	{
	1736	enum maple_type mt;
	1737	unsigned char offset;
	1738	unsigned char end;
	1739	unsigned long *pivots;
	1740	struct maple_enode *entry;
	1741	struct maple_node *node;
	1742	void __rcu **slots;
	1743
	1744	mt = mte_node_type(mas->node);
	1745	node = mas_mn(mas);
	1746	slots = ma_slots(node, mt);
	1747	pivots = ma_pivots(node, mt);
	1748	end = ma_data_end(node, mt, pivots, mas->max);
	1749	for (offset = mas->offset; offset <= end; offset++) {
	1750	entry = mas_slot_locked(mas, slots, offset);
	1751	if (mte_parent(entry) == node) {
	1752	child = mas;
	1753	mas->offset = offset + 1;
	1754	child->offset = offset;
	1755	mas_descend(child);
	1756	child->offset = 0;
	1757	return true;
	1758	}
	1759	}
	1760	return false;
	1761	}
	1762
	1763	/*
	1764	* mab_shift_right() - Shift the data in mab right. Note, does not clean out the
	1765	* old data or set b_node->b_end.
	1766	* @b_node: the maple_big_node
	1767	* @shift: the shift count
	1768	*/
	1769	static inline void mab_shift_right(struct maple_big_node *b_node,
	1770	unsigned char shift)
	1771	{
	1772	unsigned long size = b_node->b_end * sizeof(unsigned long);
	1773
	1774	memmove(b_node->pivot + shift, b_node->pivot, size);
	1775	memmove(b_node->slot + shift, b_node->slot, size);
	1776	if (b_node->type == maple_arange_64)
	1777	memmove(b_node->gap + shift, b_node->gap, size);
	1778	}
	1779
	1780	/*
	1781	* mab_middle_node() - Check if a middle node is needed (unlikely)
	1782	* @b_node: the maple_big_node that contains the data.
	1783	* @size: the amount of data in the b_node
	1784	* @split: the potential split location
	1785	* @slot_count: the size that can be stored in a single node being considered.
	1786	*
	1787	* Return: true if a middle node is required.
	1788	*/
	1789	static inline bool mab_middle_node(struct maple_big_node *b_node, int split,
	1790	unsigned char slot_count)
	1791	{
	1792	unsigned char size = b_node->b_end;
	1793
	1794	if (size >= 2 * slot_count)
	1795	return true;
	1796
	1797	if (!b_node->slot[split] && (size >= 2 * slot_count - 1))
	1798	return true;
	1799
	1800	return false;
	1801	}
	1802
	1803	/*
	1804	* mab_no_null_split() - ensure the split doesn't fall on a NULL
	1805	* @b_node: the maple_big_node with the data
	1806	* @split: the suggested split location
	1807	* @slot_count: the number of slots in the node being considered.
	1808	*
	1809	* Return: the split location.
	1810	*/
	1811	static inline int mab_no_null_split(struct maple_big_node *b_node,
	1812	unsigned char split, unsigned char slot_count)
	1813	{
	1814	if (!b_node->slot[split]) {
	1815	/*
	1816	* If the split is less than the max slot && the right side will
	1817	* still be sufficient, then increment the split on NULL.
	1818	*/
	1819	if ((split < slot_count - 1) &&
	1820	(b_node->b_end - split) > (mt_min_slots[b_node->type]))
	1821	split++;
	1822	else
	1823	split--;
	1824	}
	1825	return split;
	1826	}
	1827
	1828	/*
	1829	* mab_calc_split() - Calculate the split location and if there needs to be two
	1830	* splits.
	1831	* @bn: The maple_big_node with the data
	1832	* @mid_split: The second split, if required. 0 otherwise.
	1833	*
	1834	* Return: The first split location. The middle split is set in @mid_split.
	1835	*/
	1836	static inline int mab_calc_split(struct ma_state *mas,
	1837	struct maple_big_node bn, unsigned char mid_split, unsigned long min)
	1838	{
	1839	unsigned char b_end = bn->b_end;
	1840	int split = b_end / 2; /* Assume equal split. */
	1841	unsigned char slot_min, slot_count = mt_slots[bn->type];
	1842
	1843	/*
	1844	* To support gap tracking, all NULL entries are kept together and a node cannot
	1845	* end on a NULL entry, with the exception of the left-most leaf. The
	1846	* limitation means that the split of a node must be checked for this condition
	1847	* and be able to put more data in one direction or the other.
	1848	*/
	1849	if (unlikely((mas->mas_flags & MA_STATE_BULK))) {
	1850	*mid_split = 0;
	1851	split = b_end - mt_min_slots[bn->type];
	1852
	1853	if (!ma_is_leaf(bn->type))
	1854	return split;
	1855
	1856	mas->mas_flags \|= MA_STATE_REBALANCE;
	1857	if (!bn->slot[split])
	1858	split--;
	1859	return split;
	1860	}
	1861
	1862	/*
	1863	* Although extremely rare, it is possible to enter what is known as the 3-way
	1864	* split scenario. The 3-way split comes about by means of a store of a range
	1865	* that overwrites the end and beginning of two full nodes. The result is a set
	1866	* of entries that cannot be stored in 2 nodes. Sometimes, these two nodes can
	1867	* also be located in different parent nodes which are also full. This can
	1868	* carry upwards all the way to the root in the worst case.
	1869	*/
	1870	if (unlikely(mab_middle_node(bn, split, slot_count))) {
	1871	split = b_end / 3;
	1872	mid_split = split 2;
	1873	} else {
	1874	slot_min = mt_min_slots[bn->type];
	1875
	1876	*mid_split = 0;
	1877	/*
	1878	* Avoid having a range less than the slot count unless it
	1879	* causes one node to be deficient.
	1880	* NOTE: mt_min_slots is 1 based, b_end and split are zero.
	1881	*/
	1882	while (((bn->pivot[split] - min) < slot_count - 1) &&
	1883	(split < slot_count - 1) && (b_end - split > slot_min))
	1884	split++;
	1885	}
	1886
	1887	/* Avoid ending a node on a NULL entry */
	1888	split = mab_no_null_split(bn, split, slot_count);
	1889	if (!(*mid_split))
	1890	return split;
	1891
	1892	mid_split = mab_no_null_split(bn, mid_split, slot_count);
	1893
	1894	return split;
	1895	}
	1896
	1897	/*
	1898	* mas_mab_cp() - Copy data from a maple state inclusively to a maple_big_node
	1899	* and set @b_node->b_end to the next free slot.
	1900	* @mas: The maple state
	1901	* @mas_start: The starting slot to copy
	1902	* @mas_end: The end slot to copy (inclusively)
	1903	* @b_node: The maple_big_node to place the data
	1904	* @mab_start: The starting location in maple_big_node to store the data.
	1905	*/
	1906	static inline void mas_mab_cp(struct ma_state *mas, unsigned char mas_start,
	1907	unsigned char mas_end, struct maple_big_node *b_node,
	1908	unsigned char mab_start)
	1909	{
	1910	enum maple_type mt;
	1911	struct maple_node *node;
	1912	void __rcu **slots;
	1913	unsigned long pivots, gaps;
	1914	int i = mas_start, j = mab_start;
	1915	unsigned char piv_end;
	1916
	1917	node = mas_mn(mas);
	1918	mt = mte_node_type(mas->node);
	1919	pivots = ma_pivots(node, mt);
	1920	if (!i) {
	1921	b_node->pivot[j] = pivots[i++];
	1922	if (unlikely(i > mas_end))
	1923	goto complete;
	1924	j++;
	1925	}
	1926
	1927	piv_end = min(mas_end, mt_pivots[mt]);
	1928	for (; i < piv_end; i++, j++) {
	1929	b_node->pivot[j] = pivots[i];
	1930	if (unlikely(!b_node->pivot[j]))
	1931	break;
	1932
	1933	if (unlikely(mas->max == b_node->pivot[j]))
	1934	goto complete;
	1935	}
	1936
	1937	if (likely(i <= mas_end))
	1938	b_node->pivot[j] = mas_safe_pivot(mas, pivots, i, mt);
	1939
	1940	complete:
	1941	b_node->b_end = ++j;
	1942	j -= mab_start;
	1943	slots = ma_slots(node, mt);
	1944	memcpy(b_node->slot + mab_start, slots + mas_start, sizeof(void ) j);
	1945	if (!ma_is_leaf(mt) && mt_is_alloc(mas->tree)) {
	1946	gaps = ma_gaps(node, mt);
	1947	memcpy(b_node->gap + mab_start, gaps + mas_start,
	1948	sizeof(unsigned long) * j);
	1949	}
	1950	}
	1951
	1952	/*
	1953	* mas_leaf_set_meta() - Set the metadata of a leaf if possible.
	1954	* @mas: The maple state
	1955	* @node: The maple node
	1956	* @pivots: pointer to the maple node pivots
	1957	* @mt: The maple type
	1958	* @end: The assumed end
	1959	*
	1960	* Note, end may be incremented within this function but not modified at the
	1961	* source. This is fine since the metadata is the last thing to be stored in a
	1962	* node during a write.
	1963	*/
	1964	static inline void mas_leaf_set_meta(struct ma_state *mas,
	1965	struct maple_node node, unsigned long pivots,
	1966	enum maple_type mt, unsigned char end)
	1967	{
	1968	/* There is no room for metadata already */
	1969	if (mt_pivots[mt] <= end)
	1970	return;
	1971
	1972	if (pivots[end] && pivots[end] < mas->max)
	1973	end++;
	1974
	1975	if (end < mt_slots[mt] - 1)
	1976	ma_set_meta(node, mt, 0, end);
	1977	}
	1978
	1979	/*
	1980	* mab_mas_cp() - Copy data from maple_big_node to a maple encoded node.
	1981	* @b_node: the maple_big_node that has the data
	1982	* @mab_start: the start location in @b_node.
	1983	* @mab_end: The end location in @b_node (inclusively)
	1984	* @mas: The maple state with the maple encoded node.
	1985	*/
	1986	static inline void mab_mas_cp(struct maple_big_node *b_node,
	1987	unsigned char mab_start, unsigned char mab_end,
	1988	struct ma_state *mas, bool new_max)
	1989	{
	1990	int i, j = 0;
	1991	enum maple_type mt = mte_node_type(mas->node);
	1992	struct maple_node *node = mte_to_node(mas->node);
	1993	void __rcu **slots = ma_slots(node, mt);
	1994	unsigned long *pivots = ma_pivots(node, mt);
	1995	unsigned long *gaps = NULL;
	1996	unsigned char end;
	1997
	1998	if (mab_end - mab_start > mt_pivots[mt])
	1999	mab_end--;
	2000
	2001	if (!pivots[mt_pivots[mt] - 1])
	2002	slots[mt_pivots[mt]] = NULL;
	2003
	2004	i = mab_start;
	2005	do {
	2006	pivots[j++] = b_node->pivot[i++];
	2007	} while (i <= mab_end && likely(b_node->pivot[i]));
	2008
	2009	memcpy(slots, b_node->slot + mab_start,
	2010	sizeof(void ) (i - mab_start));
	2011
	2012	if (new_max)
	2013	mas->max = b_node->pivot[i - 1];
	2014
	2015	end = j - 1;
	2016	if (likely(!ma_is_leaf(mt) && mt_is_alloc(mas->tree))) {
	2017	unsigned long max_gap = 0;
	2018	unsigned char offset = 15;
	2019
	2020	gaps = ma_gaps(node, mt);
	2021	do {
	2022	gaps[--j] = b_node->gap[--i];
	2023	if (gaps[j] > max_gap) {
	2024	offset = j;
	2025	max_gap = gaps[j];
	2026	}
	2027	} while (j);
	2028
	2029	ma_set_meta(node, mt, offset, end);
	2030	} else {
	2031	mas_leaf_set_meta(mas, node, pivots, mt, end);
	2032	}
	2033	}
	2034
	2035	/*
	2036	* mas_descend_adopt() - Descend through a sub-tree and adopt children.
	2037	* @mas: the maple state with the maple encoded node of the sub-tree.
	2038	*
	2039	* Descend through a sub-tree and adopt children who do not have the correct
	2040	* parents set. Follow the parents which have the correct parents as they are
	2041	* the new entries which need to be followed to find other incorrectly set
	2042	* parents.
	2043	*/
	2044	static inline void mas_descend_adopt(struct ma_state *mas)
	2045	{
	2046	struct ma_state list[3], next[3];
	2047	int i, n;
	2048
	2049	/*
	2050	* At each level there may be up to 3 correct parent pointers which indicates
	2051	* the new nodes which need to be walked to find any new nodes at a lower level.
	2052	*/
	2053
	2054	for (i = 0; i < 3; i++) {
	2055	list[i] = *mas;
	2056	list[i].offset = 0;
	2057	next[i].offset = 0;
	2058	}
	2059	next[0] = *mas;
	2060
	2061	while (!mte_is_leaf(list[0].node)) {
	2062	n = 0;
	2063	for (i = 0; i < 3; i++) {
	2064	if (mas_is_none(&list[i]))
	2065	continue;
	2066
	2067	if (i && list[i-1].node == list[i].node)
	2068	continue;
	2069
	2070	while ((n < 3) && (mas_new_child(&list[i], &next[n])))
	2071	n++;
	2072
	2073	mas_adopt_children(&list[i], list[i].node);
	2074	}
	2075
	2076	while (n < 3)
	2077	next[n++].node = MAS_NONE;
	2078
	2079	/* descend by setting the list to the children */
	2080	for (i = 0; i < 3; i++)
	2081	list[i] = next[i];
	2082	}
	2083	}
	2084
	2085	/*
	2086	* mas_bulk_rebalance() - Rebalance the end of a tree after a bulk insert.
	2087	* @mas: The maple state
	2088	* @end: The maple node end
	2089	* @mt: The maple node type
	2090	*/
	2091	static inline void mas_bulk_rebalance(struct ma_state *mas, unsigned char end,
	2092	enum maple_type mt)
	2093	{
	2094	if (!(mas->mas_flags & MA_STATE_BULK))
	2095	return;
	2096
	2097	if (mte_is_root(mas->node))
	2098	return;
	2099
	2100	if (end > mt_min_slots[mt]) {
	2101	mas->mas_flags &= ~MA_STATE_REBALANCE;
	2102	return;
	2103	}
	2104	}
	2105
	2106	/*
	2107	* mas_store_b_node() - Store an @entry into the b_node while also copying the
	2108	* data from a maple encoded node.
	2109	* @wr_mas: the maple write state
	2110	* @b_node: the maple_big_node to fill with data
	2111	* @offset_end: the offset to end copying
	2112	*
	2113	* Return: The actual end of the data stored in @b_node
	2114	*/
	2115	static inline void mas_store_b_node(struct ma_wr_state *wr_mas,
	2116	struct maple_big_node *b_node, unsigned char offset_end)
	2117	{
	2118	unsigned char slot;
	2119	unsigned char b_end;
	2120	/* Possible underflow of piv will wrap back to 0 before use. */
	2121	unsigned long piv;
	2122	struct ma_state *mas = wr_mas->mas;
	2123
	2124	b_node->type = wr_mas->type;
	2125	b_end = 0;
	2126	slot = mas->offset;
	2127	if (slot) {
	2128	/* Copy start data up to insert. */
	2129	mas_mab_cp(mas, 0, slot - 1, b_node, 0);
	2130	b_end = b_node->b_end;
	2131	piv = b_node->pivot[b_end - 1];
	2132	} else
	2133	piv = mas->min - 1;
	2134
	2135	if (piv + 1 < mas->index) {
	2136	/* Handle range starting after old range */
	2137	b_node->slot[b_end] = wr_mas->content;
	2138	if (!wr_mas->content)
	2139	b_node->gap[b_end] = mas->index - 1 - piv;
	2140	b_node->pivot[b_end++] = mas->index - 1;
	2141	}
	2142
	2143	/* Store the new entry. */
	2144	mas->offset = b_end;
	2145	b_node->slot[b_end] = wr_mas->entry;
	2146	b_node->pivot[b_end] = mas->last;
	2147
	2148	/* Appended. */
	2149	if (mas->last >= mas->max)
	2150	goto b_end;
	2151
	2152	/* Handle new range ending before old range ends */
	2153	piv = mas_logical_pivot(mas, wr_mas->pivots, offset_end, wr_mas->type);
	2154	if (piv > mas->last) {
	2155	if (piv == ULONG_MAX)
	2156	mas_bulk_rebalance(mas, b_node->b_end, wr_mas->type);
	2157
	2158	if (offset_end != slot)
	2159	wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
	2160	offset_end);
	2161
	2162	b_node->slot[++b_end] = wr_mas->content;
	2163	if (!wr_mas->content)
	2164	b_node->gap[b_end] = piv - mas->last + 1;
	2165	b_node->pivot[b_end] = piv;
	2166	}
	2167
	2168	slot = offset_end + 1;
	2169	if (slot > wr_mas->node_end)
	2170	goto b_end;
	2171
	2172	/* Copy end data to the end of the node. */
	2173	mas_mab_cp(mas, slot, wr_mas->node_end + 1, b_node, ++b_end);
	2174	b_node->b_end--;
	2175	return;
	2176
	2177	b_end:
	2178	b_node->b_end = b_end;
	2179	}
	2180
	2181	/*
	2182	* mas_prev_sibling() - Find the previous node with the same parent.
	2183	* @mas: the maple state
	2184	*
	2185	* Return: True if there is a previous sibling, false otherwise.
	2186	*/
	2187	static inline bool mas_prev_sibling(struct ma_state *mas)
	2188	{
	2189	unsigned int p_slot = mte_parent_slot(mas->node);
	2190
	2191	if (mte_is_root(mas->node))
	2192	return false;
	2193
	2194	if (!p_slot)
	2195	return false;
	2196
	2197	mas_ascend(mas);
	2198	mas->offset = p_slot - 1;
	2199	mas_descend(mas);
	2200	return true;
	2201	}
	2202
	2203	/*
	2204	* mas_next_sibling() - Find the next node with the same parent.
	2205	* @mas: the maple state
	2206	*
	2207	* Return: true if there is a next sibling, false otherwise.
	2208	*/
	2209	static inline bool mas_next_sibling(struct ma_state *mas)
	2210	{
	2211	MA_STATE(parent, mas->tree, mas->index, mas->last);
	2212
	2213	if (mte_is_root(mas->node))
	2214	return false;
	2215
	2216	parent = *mas;
	2217	mas_ascend(&parent);
	2218	parent.offset = mte_parent_slot(mas->node) + 1;
	2219	if (parent.offset > mas_data_end(&parent))
	2220	return false;
	2221
	2222	*mas = parent;
	2223	mas_descend(mas);
	2224	return true;
	2225	}
	2226
	2227	/*
	2228	* mte_node_or_node() - Return the encoded node or MAS_NONE.
	2229	* @enode: The encoded maple node.
	2230	*
	2231	* Shorthand to avoid setting %NULLs in the tree or maple_subtree_state.
	2232	*
	2233	* Return: @enode or MAS_NONE
	2234	*/
	2235	static inline struct maple_enode mte_node_or_none(struct maple_enode enode)
	2236	{
	2237	if (enode)
	2238	return enode;
	2239
	2240	return ma_enode_ptr(MAS_NONE);
	2241	}
	2242
	2243	/*
	2244	* mas_wr_node_walk() - Find the correct offset for the index in the @mas.
	2245	* @wr_mas: The maple write state
	2246	*
	2247	* Uses mas_slot_locked() and does not need to worry about dead nodes.
	2248	*/
	2249	static inline void mas_wr_node_walk(struct ma_wr_state *wr_mas)
	2250	{
	2251	struct ma_state *mas = wr_mas->mas;
	2252	unsigned char count;
	2253	unsigned char offset;
	2254	unsigned long index, min, max;
	2255
	2256	if (unlikely(ma_is_dense(wr_mas->type))) {
	2257	wr_mas->r_max = wr_mas->r_min = mas->index;
	2258	mas->offset = mas->index = mas->min;
	2259	return;
	2260	}
	2261
	2262	wr_mas->node = mas_mn(wr_mas->mas);
	2263	wr_mas->pivots = ma_pivots(wr_mas->node, wr_mas->type);
	2264	count = wr_mas->node_end = ma_data_end(wr_mas->node, wr_mas->type,
	2265	wr_mas->pivots, mas->max);
	2266	offset = mas->offset;
	2267	min = mas_safe_min(mas, wr_mas->pivots, offset);
	2268	if (unlikely(offset == count))
	2269	goto max;
	2270
	2271	max = wr_mas->pivots[offset];
	2272	index = mas->index;
	2273	if (unlikely(index <= max))
	2274	goto done;
	2275
	2276	if (unlikely(!max && offset))
	2277	goto max;
	2278
	2279	min = max + 1;
	2280	while (++offset < count) {
	2281	max = wr_mas->pivots[offset];
	2282	if (index <= max)
	2283	goto done;
	2284	else if (unlikely(!max))
	2285	break;
	2286
	2287	min = max + 1;
	2288	}
	2289
	2290	max:
	2291	max = mas->max;
	2292	done:
	2293	wr_mas->r_max = max;
	2294	wr_mas->r_min = min;
	2295	wr_mas->offset_end = mas->offset = offset;
	2296	}
	2297
	2298	/*
	2299	* mas_topiary_range() - Add a range of slots to the topiary.
	2300	* @mas: The maple state
	2301	* @destroy: The topiary to add the slots (usually destroy)
	2302	* @start: The starting slot inclusively
	2303	* @end: The end slot inclusively
	2304	*/
	2305	static inline void mas_topiary_range(struct ma_state *mas,
	2306	struct ma_topiary *destroy, unsigned char start, unsigned char end)
	2307	{
	2308	void __rcu **slots;
	2309	unsigned char offset;
	2310
	2311	MT_BUG_ON(mas->tree, mte_is_leaf(mas->node));
	2312	slots = ma_slots(mas_mn(mas), mte_node_type(mas->node));
	2313	for (offset = start; offset <= end; offset++) {
	2314	struct maple_enode *enode = mas_slot_locked(mas, slots, offset);
	2315
	2316	if (mte_dead_node(enode))
	2317	continue;
	2318
	2319	mat_add(destroy, enode);
	2320	}
	2321	}
	2322
	2323	/*
	2324	* mast_topiary() - Add the portions of the tree to the removal list; either to
	2325	* be freed or discarded (destroy walk).
	2326	* @mast: The maple_subtree_state.
	2327	*/
	2328	static inline void mast_topiary(struct maple_subtree_state *mast)
	2329	{
	2330	MA_WR_STATE(wr_mas, mast->orig_l, NULL);
	2331	unsigned char r_start, r_end;
	2332	unsigned char l_start, l_end;
	2333	void __rcu l_slots, r_slots;
	2334
	2335	wr_mas.type = mte_node_type(mast->orig_l->node);
	2336	mast->orig_l->index = mast->orig_l->last;
	2337	mas_wr_node_walk(&wr_mas);
	2338	l_start = mast->orig_l->offset + 1;
	2339	l_end = mas_data_end(mast->orig_l);
	2340	r_start = 0;
	2341	r_end = mast->orig_r->offset;
	2342
	2343	if (r_end)
	2344	r_end--;
	2345
	2346	l_slots = ma_slots(mas_mn(mast->orig_l),
	2347	mte_node_type(mast->orig_l->node));
	2348
	2349	r_slots = ma_slots(mas_mn(mast->orig_r),
	2350	mte_node_type(mast->orig_r->node));
	2351
	2352	if ((l_start < l_end) &&
	2353	mte_dead_node(mas_slot_locked(mast->orig_l, l_slots, l_start))) {
	2354	l_start++;
	2355	}
	2356
	2357	if (mte_dead_node(mas_slot_locked(mast->orig_r, r_slots, r_end))) {
	2358	if (r_end)
	2359	r_end--;
	2360	}
	2361
	2362	if ((l_start > r_end) && (mast->orig_l->node == mast->orig_r->node))
	2363	return;
	2364
	2365	/* At the node where left and right sides meet, add the parts between */
	2366	if (mast->orig_l->node == mast->orig_r->node) {
	2367	return mas_topiary_range(mast->orig_l, mast->destroy,
	2368	l_start, r_end);
	2369	}
	2370
	2371	/* mast->orig_r is different and consumed. */
	2372	if (mte_is_leaf(mast->orig_r->node))
	2373	return;
	2374
	2375	if (mte_dead_node(mas_slot_locked(mast->orig_l, l_slots, l_end)))
	2376	l_end--;
	2377
	2378
	2379	if (l_start <= l_end)
	2380	mas_topiary_range(mast->orig_l, mast->destroy, l_start, l_end);
	2381
	2382	if (mte_dead_node(mas_slot_locked(mast->orig_r, r_slots, r_start)))
	2383	r_start++;
	2384
	2385	if (r_start <= r_end)
	2386	mas_topiary_range(mast->orig_r, mast->destroy, 0, r_end);
	2387	}
	2388
	2389	/*
	2390	* mast_rebalance_next() - Rebalance against the next node
	2391	* @mast: The maple subtree state
	2392	* @old_r: The encoded maple node to the right (next node).
	2393	*/
	2394	static inline void mast_rebalance_next(struct maple_subtree_state *mast)
	2395	{
	2396	unsigned char b_end = mast->bn->b_end;
	2397
	2398	mas_mab_cp(mast->orig_r, 0, mt_slot_count(mast->orig_r->node),
	2399	mast->bn, b_end);
	2400	mast->orig_r->last = mast->orig_r->max;
	2401	}
	2402
	2403	/*
	2404	* mast_rebalance_prev() - Rebalance against the previous node
	2405	* @mast: The maple subtree state
	2406	* @old_l: The encoded maple node to the left (previous node)
	2407	*/
	2408	static inline void mast_rebalance_prev(struct maple_subtree_state *mast)
	2409	{
	2410	unsigned char end = mas_data_end(mast->orig_l) + 1;
	2411	unsigned char b_end = mast->bn->b_end;
	2412
	2413	mab_shift_right(mast->bn, end);
	2414	mas_mab_cp(mast->orig_l, 0, end - 1, mast->bn, 0);
	2415	mast->l->min = mast->orig_l->min;
	2416	mast->orig_l->index = mast->orig_l->min;
	2417	mast->bn->b_end = end + b_end;
	2418	mast->l->offset += end;
	2419	}
	2420
	2421	/*
	2422	* mast_spanning_rebalance() - Rebalance nodes with nearest neighbour favouring
	2423	* the node to the right. Checking the nodes to the right then the left at each
	2424	* level upwards until root is reached. Free and destroy as needed.
	2425	* Data is copied into the @mast->bn.
	2426	* @mast: The maple_subtree_state.
	2427	*/
	2428	static inline
	2429	bool mast_spanning_rebalance(struct maple_subtree_state *mast)
	2430	{
	2431	struct ma_state r_tmp = *mast->orig_r;
	2432	struct ma_state l_tmp = *mast->orig_l;
	2433	struct maple_enode *ancestor = NULL;
	2434	unsigned char start, end;
	2435	unsigned char depth = 0;
	2436
	2437	r_tmp = *mast->orig_r;
	2438	l_tmp = *mast->orig_l;
	2439	do {
	2440	mas_ascend(mast->orig_r);
	2441	mas_ascend(mast->orig_l);
	2442	depth++;
	2443	if (!ancestor &&
	2444	(mast->orig_r->node == mast->orig_l->node)) {
	2445	ancestor = mast->orig_r->node;
	2446	end = mast->orig_r->offset - 1;
	2447	start = mast->orig_l->offset + 1;
	2448	}
	2449
	2450	if (mast->orig_r->offset < mas_data_end(mast->orig_r)) {
	2451	if (!ancestor) {
	2452	ancestor = mast->orig_r->node;
	2453	start = 0;
	2454	}
	2455
	2456	mast->orig_r->offset++;
	2457	do {
	2458	mas_descend(mast->orig_r);
	2459	mast->orig_r->offset = 0;
	2460	depth--;
	2461	} while (depth);
	2462
	2463	mast_rebalance_next(mast);
	2464	do {
	2465	unsigned char l_off = 0;
	2466	struct maple_enode *child = r_tmp.node;
	2467
	2468	mas_ascend(&r_tmp);
	2469	if (ancestor == r_tmp.node)
	2470	l_off = start;
	2471
	2472	if (r_tmp.offset)
	2473	r_tmp.offset--;
	2474
	2475	if (l_off < r_tmp.offset)
	2476	mas_topiary_range(&r_tmp, mast->destroy,
	2477	l_off, r_tmp.offset);
	2478
	2479	if (l_tmp.node != child)
	2480	mat_add(mast->free, child);
	2481
	2482	} while (r_tmp.node != ancestor);
	2483
	2484	*mast->orig_l = l_tmp;
	2485	return true;
	2486
	2487	} else if (mast->orig_l->offset != 0) {
	2488	if (!ancestor) {
	2489	ancestor = mast->orig_l->node;
	2490	end = mas_data_end(mast->orig_l);
	2491	}
	2492
	2493	mast->orig_l->offset--;
	2494	do {
	2495	mas_descend(mast->orig_l);
	2496	mast->orig_l->offset =
	2497	mas_data_end(mast->orig_l);
	2498	depth--;
	2499	} while (depth);
	2500
	2501	mast_rebalance_prev(mast);
	2502	do {
	2503	unsigned char r_off;
	2504	struct maple_enode *child = l_tmp.node;
	2505
	2506	mas_ascend(&l_tmp);
	2507	if (ancestor == l_tmp.node)
	2508	r_off = end;
	2509	else
	2510	r_off = mas_data_end(&l_tmp);
	2511
	2512	if (l_tmp.offset < r_off)
	2513	l_tmp.offset++;
	2514
	2515	if (l_tmp.offset < r_off)
	2516	mas_topiary_range(&l_tmp, mast->destroy,
	2517	l_tmp.offset, r_off);
	2518
	2519	if (r_tmp.node != child)
	2520	mat_add(mast->free, child);
	2521
	2522	} while (l_tmp.node != ancestor);
	2523
	2524	*mast->orig_r = r_tmp;
	2525	return true;
	2526	}
	2527	} while (!mte_is_root(mast->orig_r->node));
	2528
	2529	*mast->orig_r = r_tmp;
	2530	*mast->orig_l = l_tmp;
	2531	return false;
	2532	}
	2533
	2534	/*
	2535	* mast_ascend_free() - Add current original maple state nodes to the free list
	2536	* and ascend.
	2537	* @mast: the maple subtree state.
	2538	*
	2539	* Ascend the original left and right sides and add the previous nodes to the
	2540	* free list. Set the slots to point to the correct location in the new nodes.
	2541	*/
	2542	static inline void
	2543	mast_ascend_free(struct maple_subtree_state *mast)
	2544	{
	2545	MA_WR_STATE(wr_mas, mast->orig_r, NULL);
	2546	struct maple_enode *left = mast->orig_l->node;
	2547	struct maple_enode *right = mast->orig_r->node;
	2548
	2549	mas_ascend(mast->orig_l);
	2550	mas_ascend(mast->orig_r);
	2551	mat_add(mast->free, left);
	2552
	2553	if (left != right)
	2554	mat_add(mast->free, right);
	2555
	2556	mast->orig_r->offset = 0;
	2557	mast->orig_r->index = mast->r->max;
	2558	/* last should be larger than or equal to index */
	2559	if (mast->orig_r->last < mast->orig_r->index)
	2560	mast->orig_r->last = mast->orig_r->index;
	2561	/*
	2562	* The node may not contain the value so set slot to ensure all
	2563	* of the nodes contents are freed or destroyed.
	2564	*/
	2565	wr_mas.type = mte_node_type(mast->orig_r->node);
	2566	mas_wr_node_walk(&wr_mas);
	2567	/* Set up the left side of things */
	2568	mast->orig_l->offset = 0;
	2569	mast->orig_l->index = mast->l->min;
	2570	wr_mas.mas = mast->orig_l;
	2571	wr_mas.type = mte_node_type(mast->orig_l->node);
	2572	mas_wr_node_walk(&wr_mas);
	2573
	2574	mast->bn->type = wr_mas.type;
	2575	}
	2576
	2577	/*
	2578	* mas_new_ma_node() - Create and return a new maple node. Helper function.
	2579	* @mas: the maple state with the allocations.
	2580	* @b_node: the maple_big_node with the type encoding.
	2581	*
	2582	* Use the node type from the maple_big_node to allocate a new node from the
	2583	* ma_state. This function exists mainly for code readability.
	2584	*
	2585	* Return: A new maple encoded node
	2586	*/
	2587	static inline struct maple_enode
	2588	mas_new_ma_node(struct ma_state mas, struct maple_big_node *b_node)
	2589	{
	2590	return mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)), b_node->type);
	2591	}
	2592
	2593	/*
	2594	* mas_mab_to_node() - Set up right and middle nodes
	2595	*
	2596	* @mas: the maple state that contains the allocations.
	2597	* @b_node: the node which contains the data.
	2598	* @left: The pointer which will have the left node
	2599	* @right: The pointer which may have the right node
	2600	* @middle: the pointer which may have the middle node (rare)
	2601	* @mid_split: the split location for the middle node
	2602	*
	2603	* Return: the split of left.
	2604	*/
	2605	static inline unsigned char mas_mab_to_node(struct ma_state *mas,
	2606	struct maple_big_node b_node, struct maple_enode *left,
	2607	struct maple_enode right, struct maple_enode middle,
	2608	unsigned char *mid_split, unsigned long min)
	2609	{
	2610	unsigned char split = 0;
	2611	unsigned char slot_count = mt_slots[b_node->type];
	2612
	2613	*left = mas_new_ma_node(mas, b_node);
	2614	*right = NULL;
	2615	*middle = NULL;
	2616	*mid_split = 0;
	2617
	2618	if (b_node->b_end < slot_count) {
	2619	split = b_node->b_end;
	2620	} else {
	2621	split = mab_calc_split(mas, b_node, mid_split, min);
	2622	*right = mas_new_ma_node(mas, b_node);
	2623	}
	2624
	2625	if (*mid_split)
	2626	*middle = mas_new_ma_node(mas, b_node);
	2627
	2628	return split;
	2629
	2630	}
	2631
	2632	/*
	2633	* mab_set_b_end() - Add entry to b_node at b_node->b_end and increment the end
	2634	* pointer.
	2635	* @b_node - the big node to add the entry
	2636	* @mas - the maple state to get the pivot (mas->max)
	2637	* @entry - the entry to add, if NULL nothing happens.
	2638	*/
	2639	static inline void mab_set_b_end(struct maple_big_node *b_node,
	2640	struct ma_state *mas,
	2641	void *entry)
	2642	{
	2643	if (!entry)
	2644	return;
	2645
	2646	b_node->slot[b_node->b_end] = entry;
	2647	if (mt_is_alloc(mas->tree))
	2648	b_node->gap[b_node->b_end] = mas_max_gap(mas);
	2649	b_node->pivot[b_node->b_end++] = mas->max;
	2650	}
	2651
	2652	/*
	2653	* mas_set_split_parent() - combine_then_separate helper function. Sets the parent
	2654	* of @mas->node to either @left or @right, depending on @slot and @split
	2655	*
	2656	* @mas - the maple state with the node that needs a parent
	2657	* @left - possible parent 1
	2658	* @right - possible parent 2
	2659	* @slot - the slot the mas->node was placed
	2660	* @split - the split location between @left and @right
	2661	*/
	2662	static inline void mas_set_split_parent(struct ma_state *mas,
	2663	struct maple_enode *left,
	2664	struct maple_enode *right,
	2665	unsigned char *slot, unsigned char split)
	2666	{
	2667	if (mas_is_none(mas))
	2668	return;
	2669
	2670	if ((*slot) <= split)
	2671	mte_set_parent(mas->node, left, *slot);
	2672	else if (right)
	2673	mte_set_parent(mas->node, right, (*slot) - split - 1);
	2674
	2675	(*slot)++;
	2676	}
	2677
	2678	/*
	2679	* mte_mid_split_check() - Check if the next node passes the mid-split
	2680	* @**l: Pointer to left encoded maple node.
	2681	* @**m: Pointer to middle encoded maple node.
	2682	* @**r: Pointer to right encoded maple node.
	2683	* @slot: The offset
	2684	* @*split: The split location.
	2685	* @mid_split: The middle split.
	2686	*/
	2687	static inline void mte_mid_split_check(struct maple_enode **l,
	2688	struct maple_enode **r,
	2689	struct maple_enode *right,
	2690	unsigned char slot,
	2691	unsigned char *split,
	2692	unsigned char mid_split)
	2693	{
	2694	if (*r == right)
	2695	return;
	2696
	2697	if (slot < mid_split)
	2698	return;
	2699
	2700	l = r;
	2701	*r = right;
	2702	*split = mid_split;
	2703	}
	2704
	2705	/*
	2706	* mast_set_split_parents() - Helper function to set three nodes parents. Slot
	2707	* is taken from @mast->l.
	2708	* @mast - the maple subtree state
	2709	* @left - the left node
	2710	* @right - the right node
	2711	* @split - the split location.
	2712	*/
	2713	static inline void mast_set_split_parents(struct maple_subtree_state *mast,
	2714	struct maple_enode *left,
	2715	struct maple_enode *middle,
	2716	struct maple_enode *right,
	2717	unsigned char split,
	2718	unsigned char mid_split)
	2719	{
	2720	unsigned char slot;
	2721	struct maple_enode *l = left;
	2722	struct maple_enode *r = right;
	2723
	2724	if (mas_is_none(mast->l))
	2725	return;
	2726
	2727	if (middle)
	2728	r = middle;
	2729
	2730	slot = mast->l->offset;
	2731
	2732	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	2733	mas_set_split_parent(mast->l, l, r, &slot, split);
	2734
	2735	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	2736	mas_set_split_parent(mast->m, l, r, &slot, split);
	2737
	2738	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	2739	mas_set_split_parent(mast->r, l, r, &slot, split);
	2740	}
	2741
	2742	/*
	2743	* mas_wmb_replace() - Write memory barrier and replace
	2744	* @mas: The maple state
	2745	* @free: the maple topiary list of nodes to free
	2746	* @destroy: The maple topiary list of nodes to destroy (walk and free)
	2747	*
	2748	* Updates gap as necessary.
	2749	*/
	2750	static inline void mas_wmb_replace(struct ma_state *mas,
	2751	struct ma_topiary *free,
	2752	struct ma_topiary *destroy)
	2753	{
	2754	/* All nodes must see old data as dead prior to replacing that data */
	2755	smp_wmb(); /* Needed for RCU */
	2756
	2757	/* Insert the new data in the tree */
	2758	mas_replace(mas, true);
	2759
	2760	if (!mte_is_leaf(mas->node))
	2761	mas_descend_adopt(mas);
	2762
	2763	mas_mat_free(mas, free);
	2764
	2765	if (destroy)
	2766	mas_mat_destroy(mas, destroy);
	2767
	2768	if (mte_is_leaf(mas->node))
	2769	return;
	2770
	2771	mas_update_gap(mas);
	2772	}
	2773
	2774	/*
	2775	* mast_new_root() - Set a new tree root during subtree creation
	2776	* @mast: The maple subtree state
	2777	* @mas: The maple state
	2778	*/
	2779	static inline void mast_new_root(struct maple_subtree_state *mast,
	2780	struct ma_state *mas)
	2781	{
	2782	mas_mn(mast->l)->parent =
	2783	ma_parent_ptr(((unsigned long)mas->tree \| MA_ROOT_PARENT));
	2784	if (!mte_dead_node(mast->orig_l->node) &&
	2785	!mte_is_root(mast->orig_l->node)) {
	2786	do {
	2787	mast_ascend_free(mast);
	2788	mast_topiary(mast);
	2789	} while (!mte_is_root(mast->orig_l->node));
	2790	}
	2791	if ((mast->orig_l->node != mas->node) &&
	2792	(mast->l->depth > mas_mt_height(mas))) {
	2793	mat_add(mast->free, mas->node);
	2794	}
	2795	}
	2796
	2797	/*
	2798	* mast_cp_to_nodes() - Copy data out to nodes.
	2799	* @mast: The maple subtree state
	2800	* @left: The left encoded maple node
	2801	* @middle: The middle encoded maple node
	2802	* @right: The right encoded maple node
	2803	* @split: The location to split between left and (middle ? middle : right)
	2804	* @mid_split: The location to split between middle and right.
	2805	*/
	2806	static inline void mast_cp_to_nodes(struct maple_subtree_state *mast,
	2807	struct maple_enode left, struct maple_enode middle,
	2808	struct maple_enode *right, unsigned char split, unsigned char mid_split)
	2809	{
	2810	bool new_lmax = true;
	2811
	2812	mast->l->node = mte_node_or_none(left);
	2813	mast->m->node = mte_node_or_none(middle);
	2814	mast->r->node = mte_node_or_none(right);
	2815
	2816	mast->l->min = mast->orig_l->min;
	2817	if (split == mast->bn->b_end) {
	2818	mast->l->max = mast->orig_r->max;
	2819	new_lmax = false;
	2820	}
	2821
	2822	mab_mas_cp(mast->bn, 0, split, mast->l, new_lmax);
	2823
	2824	if (middle) {
	2825	mab_mas_cp(mast->bn, 1 + split, mid_split, mast->m, true);
	2826	mast->m->min = mast->bn->pivot[split] + 1;
	2827	split = mid_split;
	2828	}
	2829
	2830	mast->r->max = mast->orig_r->max;
	2831	if (right) {
	2832	mab_mas_cp(mast->bn, 1 + split, mast->bn->b_end, mast->r, false);
	2833	mast->r->min = mast->bn->pivot[split] + 1;
	2834	}
	2835	}
	2836
	2837	/*
	2838	* mast_combine_cp_left - Copy in the original left side of the tree into the
	2839	* combined data set in the maple subtree state big node.
	2840	* @mast: The maple subtree state
	2841	*/
	2842	static inline void mast_combine_cp_left(struct maple_subtree_state *mast)
	2843	{
	2844	unsigned char l_slot = mast->orig_l->offset;
	2845
	2846	if (!l_slot)
	2847	return;
	2848
	2849	mas_mab_cp(mast->orig_l, 0, l_slot - 1, mast->bn, 0);
	2850	}
	2851
	2852	/*
	2853	* mast_combine_cp_right: Copy in the original right side of the tree into the
	2854	* combined data set in the maple subtree state big node.
	2855	* @mast: The maple subtree state
	2856	*/
	2857	static inline void mast_combine_cp_right(struct maple_subtree_state *mast)
	2858	{
	2859	if (mast->bn->pivot[mast->bn->b_end - 1] >= mast->orig_r->max)
	2860	return;
	2861
	2862	mas_mab_cp(mast->orig_r, mast->orig_r->offset + 1,
	2863	mt_slot_count(mast->orig_r->node), mast->bn,
	2864	mast->bn->b_end);
	2865	mast->orig_r->last = mast->orig_r->max;
	2866	}
	2867
	2868	/*
	2869	* mast_sufficient: Check if the maple subtree state has enough data in the big
	2870	* node to create at least one sufficient node
	2871	* @mast: the maple subtree state
	2872	*/
	2873	static inline bool mast_sufficient(struct maple_subtree_state *mast)
	2874	{
	2875	if (mast->bn->b_end > mt_min_slot_count(mast->orig_l->node))
	2876	return true;
	2877
	2878	return false;
	2879	}
	2880
	2881	/*
	2882	* mast_overflow: Check if there is too much data in the subtree state for a
	2883	* single node.
	2884	* @mast: The maple subtree state
	2885	*/
	2886	static inline bool mast_overflow(struct maple_subtree_state *mast)
	2887	{
	2888	if (mast->bn->b_end >= mt_slot_count(mast->orig_l->node))
	2889	return true;
	2890
	2891	return false;
	2892	}
	2893
	2894	static inline void mtree_range_walk(struct ma_state mas)
	2895	{
	2896	unsigned long *pivots;
	2897	unsigned char offset;
	2898	struct maple_node *node;
	2899	struct maple_enode next, last;
	2900	enum maple_type type;
	2901	void __rcu **slots;
	2902	unsigned char end;
	2903	unsigned long max, min;
	2904	unsigned long prev_max, prev_min;
	2905
	2906	last = next = mas->node;
	2907	prev_min = min = mas->min;
	2908	max = mas->max;
	2909	do {
	2910	offset = 0;
	2911	last = next;
	2912	node = mte_to_node(next);
	2913	type = mte_node_type(next);
	2914	pivots = ma_pivots(node, type);
	2915	end = ma_data_end(node, type, pivots, max);
	2916	if (unlikely(ma_dead_node(node)))
	2917	goto dead_node;
	2918
	2919	if (pivots[offset] >= mas->index) {
	2920	prev_max = max;
	2921	prev_min = min;
	2922	max = pivots[offset];
	2923	goto next;
	2924	}
	2925
	2926	do {
	2927	offset++;
	2928	} while ((offset < end) && (pivots[offset] < mas->index));
	2929
	2930	prev_min = min;
	2931	min = pivots[offset - 1] + 1;
	2932	prev_max = max;
	2933	if (likely(offset < end && pivots[offset]))
	2934	max = pivots[offset];
	2935
	2936	next:
	2937	slots = ma_slots(node, type);
	2938	next = mt_slot(mas->tree, slots, offset);
	2939	if (unlikely(ma_dead_node(node)))
	2940	goto dead_node;
	2941	} while (!ma_is_leaf(type));
	2942
	2943	mas->offset = offset;
	2944	mas->index = min;
	2945	mas->last = max;
	2946	mas->min = prev_min;
	2947	mas->max = prev_max;
	2948	mas->node = last;
	2949	return (void *) next;
	2950
	2951	dead_node:
	2952	mas_reset(mas);
	2953	return NULL;
	2954	}
	2955
	2956	/*
	2957	* mas_spanning_rebalance() - Rebalance across two nodes which may not be peers.
	2958	* @mas: The starting maple state
	2959	* @mast: The maple_subtree_state, keeps track of 4 maple states.
	2960	* @count: The estimated count of iterations needed.
	2961	*
	2962	* Follow the tree upwards from @l_mas and @r_mas for @count, or until the root
	2963	* is hit. First @b_node is split into two entries which are inserted into the
	2964	* next iteration of the loop. @b_node is returned populated with the final
	2965	* iteration. @mas is used to obtain allocations. orig_l_mas keeps track of the
	2966	* nodes that will remain active by using orig_l_mas->index and orig_l_mas->last
	2967	* to account of what has been copied into the new sub-tree. The update of
	2968	* orig_l_mas->last is used in mas_consume to find the slots that will need to
	2969	* be either freed or destroyed. orig_l_mas->depth keeps track of the height of
	2970	* the new sub-tree in case the sub-tree becomes the full tree.
	2971	*
	2972	* Return: the number of elements in b_node during the last loop.
	2973	*/
	2974	static int mas_spanning_rebalance(struct ma_state *mas,
	2975	struct maple_subtree_state *mast, unsigned char count)
	2976	{
	2977	unsigned char split, mid_split;
	2978	unsigned char slot = 0;
	2979	struct maple_enode left = NULL, middle = NULL, *right = NULL;
	2980
	2981	MA_STATE(l_mas, mas->tree, mas->index, mas->index);
	2982	MA_STATE(r_mas, mas->tree, mas->index, mas->last);
	2983	MA_STATE(m_mas, mas->tree, mas->index, mas->index);
	2984	MA_TOPIARY(free, mas->tree);
	2985	MA_TOPIARY(destroy, mas->tree);
	2986
	2987	/*
	2988	* The tree needs to be rebalanced and leaves need to be kept at the same level.
	2989	* Rebalancing is done by use of the ``struct maple_topiary``.
	2990	*/
	2991	mast->l = &l_mas;
	2992	mast->m = &m_mas;
	2993	mast->r = &r_mas;
	2994	mast->free = &free;
	2995	mast->destroy = &destroy;
	2996	l_mas.node = r_mas.node = m_mas.node = MAS_NONE;
	2997	if (!(mast->orig_l->min && mast->orig_r->max == ULONG_MAX) &&
	2998	unlikely(mast->bn->b_end <= mt_min_slots[mast->bn->type]))
	2999	mast_spanning_rebalance(mast);
	3000
	3001	mast->orig_l->depth = 0;
	3002
	3003	/*
	3004	* Each level of the tree is examined and balanced, pushing data to the left or
	3005	* right, or rebalancing against left or right nodes is employed to avoid
	3006	* rippling up the tree to limit the amount of churn. Once a new sub-section of
	3007	* the tree is created, there may be a mix of new and old nodes. The old nodes
	3008	* will have the incorrect parent pointers and currently be in two trees: the
	3009	* original tree and the partially new tree. To remedy the parent pointers in
	3010	* the old tree, the new data is swapped into the active tree and a walk down
	3011	* the tree is performed and the parent pointers are updated.
	3012	* See mas_descend_adopt() for more information..
	3013	*/
	3014	while (count--) {
	3015	mast->bn->b_end--;
	3016	mast->bn->type = mte_node_type(mast->orig_l->node);
	3017	split = mas_mab_to_node(mas, mast->bn, &left, &right, &middle,
	3018	&mid_split, mast->orig_l->min);
	3019	mast_set_split_parents(mast, left, middle, right, split,
	3020	mid_split);
	3021	mast_cp_to_nodes(mast, left, middle, right, split, mid_split);
	3022
	3023	/*
	3024	* Copy data from next level in the tree to mast->bn from next
	3025	* iteration
	3026	*/
	3027	memset(mast->bn, 0, sizeof(struct maple_big_node));
	3028	mast->bn->type = mte_node_type(left);
	3029	mast->orig_l->depth++;
	3030
	3031	/* Root already stored in l->node. */
	3032	if (mas_is_root_limits(mast->l))
	3033	goto new_root;
	3034
	3035	mast_ascend_free(mast);
	3036	mast_combine_cp_left(mast);
	3037	l_mas.offset = mast->bn->b_end;
	3038	mab_set_b_end(mast->bn, &l_mas, left);
	3039	mab_set_b_end(mast->bn, &m_mas, middle);
	3040	mab_set_b_end(mast->bn, &r_mas, right);
	3041
	3042	/* Copy anything necessary out of the right node. */
	3043	mast_combine_cp_right(mast);
	3044	mast_topiary(mast);
	3045	mast->orig_l->last = mast->orig_l->max;
	3046
	3047	if (mast_sufficient(mast))
	3048	continue;
	3049
	3050	if (mast_overflow(mast))
	3051	continue;
	3052
	3053	/* May be a new root stored in mast->bn */
	3054	if (mas_is_root_limits(mast->orig_l))
	3055	break;
	3056
	3057	mast_spanning_rebalance(mast);
	3058
	3059	/* rebalancing from other nodes may require another loop. */
	3060	if (!count)
	3061	count++;
	3062	}
	3063
	3064	l_mas.node = mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)),
	3065	mte_node_type(mast->orig_l->node));
	3066	mast->orig_l->depth++;
	3067	mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, &l_mas, true);
	3068	mte_set_parent(left, l_mas.node, slot);
	3069	if (middle)
	3070	mte_set_parent(middle, l_mas.node, ++slot);
	3071
	3072	if (right)
	3073	mte_set_parent(right, l_mas.node, ++slot);
	3074
	3075	if (mas_is_root_limits(mast->l)) {
	3076	new_root:
	3077	mast_new_root(mast, mas);
	3078	} else {
	3079	mas_mn(&l_mas)->parent = mas_mn(mast->orig_l)->parent;
	3080	}
	3081
	3082	if (!mte_dead_node(mast->orig_l->node))
	3083	mat_add(&free, mast->orig_l->node);
	3084
	3085	mas->depth = mast->orig_l->depth;
	3086	*mast->orig_l = l_mas;
	3087	mte_set_node_dead(mas->node);
	3088
	3089	/* Set up mas for insertion. */
	3090	mast->orig_l->depth = mas->depth;
	3091	mast->orig_l->alloc = mas->alloc;
	3092	mas = mast->orig_l;
	3093	mas_wmb_replace(mas, &free, &destroy);
	3094	mtree_range_walk(mas);
	3095	return mast->bn->b_end;
	3096	}
	3097
	3098	/*
	3099	* mas_rebalance() - Rebalance a given node.
	3100	* @mas: The maple state
	3101	* @b_node: The big maple node.
	3102	*
	3103	* Rebalance two nodes into a single node or two new nodes that are sufficient.
	3104	* Continue upwards until tree is sufficient.
	3105	*
	3106	* Return: the number of elements in b_node during the last loop.
	3107	*/
	3108	static inline int mas_rebalance(struct ma_state *mas,
	3109	struct maple_big_node *b_node)
	3110	{
	3111	char empty_count = mas_mt_height(mas);
	3112	struct maple_subtree_state mast;
	3113	unsigned char shift, b_end = ++b_node->b_end;
	3114
	3115	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
	3116	MA_STATE(r_mas, mas->tree, mas->index, mas->last);
	3117
	3118	trace_ma_op(__func__, mas);
	3119
	3120	/*
	3121	* Rebalancing occurs if a node is insufficient. Data is rebalanced
	3122	* against the node to the right if it exists, otherwise the node to the
	3123	* left of this node is rebalanced against this node. If rebalancing
	3124	* causes just one node to be produced instead of two, then the parent
	3125	* is also examined and rebalanced if it is insufficient. Every level
	3126	* tries to combine the data in the same way. If one node contains the
	3127	* entire range of the tree, then that node is used as a new root node.
	3128	*/
	3129	mas_node_count(mas, 1 + empty_count * 3);
	3130	if (mas_is_err(mas))
	3131	return 0;
	3132
	3133	mast.orig_l = &l_mas;
	3134	mast.orig_r = &r_mas;
	3135	mast.bn = b_node;
	3136	mast.bn->type = mte_node_type(mas->node);
	3137
	3138	l_mas = r_mas = *mas;
	3139
	3140	if (mas_next_sibling(&r_mas)) {
	3141	mas_mab_cp(&r_mas, 0, mt_slot_count(r_mas.node), b_node, b_end);
	3142	r_mas.last = r_mas.index = r_mas.max;
	3143	} else {
	3144	mas_prev_sibling(&l_mas);
	3145	shift = mas_data_end(&l_mas) + 1;
	3146	mab_shift_right(b_node, shift);
	3147	mas->offset += shift;
	3148	mas_mab_cp(&l_mas, 0, shift - 1, b_node, 0);
	3149	b_node->b_end = shift + b_end;
	3150	l_mas.index = l_mas.last = l_mas.min;
	3151	}
	3152
	3153	return mas_spanning_rebalance(mas, &mast, empty_count);
	3154	}
	3155
	3156	/*
	3157	* mas_destroy_rebalance() - Rebalance left-most node while destroying the maple
	3158	* state.
	3159	* @mas: The maple state
	3160	* @end: The end of the left-most node.
	3161	*
	3162	* During a mass-insert event (such as forking), it may be necessary to
	3163	* rebalance the left-most node when it is not sufficient.
	3164	*/
	3165	static inline void mas_destroy_rebalance(struct ma_state *mas, unsigned char end)
	3166	{
	3167	enum maple_type mt = mte_node_type(mas->node);
	3168	struct maple_node reuse, newnode, parent, new_left, left, *node;
	3169	struct maple_enode *eparent;
	3170	unsigned char offset, tmp, split = mt_slots[mt] / 2;
	3171	void __rcu l_slots, slots;
	3172	unsigned long l_pivs, pivs, gap;
	3173	bool in_rcu = mt_in_rcu(mas->tree);
	3174
	3175	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
	3176
	3177	l_mas = *mas;
	3178	mas_prev_sibling(&l_mas);
	3179
	3180	/* set up node. */
	3181	if (in_rcu) {
	3182	/* Allocate for both left and right as well as parent. */
	3183	mas_node_count(mas, 3);
	3184	if (mas_is_err(mas))
	3185	return;
	3186
	3187	newnode = mas_pop_node(mas);
	3188	} else {
	3189	newnode = &reuse;
	3190	}
	3191
	3192	node = mas_mn(mas);
	3193	newnode->parent = node->parent;
	3194	slots = ma_slots(newnode, mt);
	3195	pivs = ma_pivots(newnode, mt);
	3196	left = mas_mn(&l_mas);
	3197	l_slots = ma_slots(left, mt);
	3198	l_pivs = ma_pivots(left, mt);
	3199	if (!l_slots[split])
	3200	split++;
	3201	tmp = mas_data_end(&l_mas) - split;
	3202
	3203	memcpy(slots, l_slots + split + 1, sizeof(void ) tmp);
	3204	memcpy(pivs, l_pivs + split + 1, sizeof(unsigned long) * tmp);
	3205	pivs[tmp] = l_mas.max;
	3206	memcpy(slots + tmp, ma_slots(node, mt), sizeof(void ) end);
	3207	memcpy(pivs + tmp, ma_pivots(node, mt), sizeof(unsigned long) * end);
	3208
	3209	l_mas.max = l_pivs[split];
	3210	mas->min = l_mas.max + 1;
	3211	eparent = mt_mk_node(mte_parent(l_mas.node),
	3212	mas_parent_enum(&l_mas, l_mas.node));
	3213	tmp += end;
	3214	if (!in_rcu) {
	3215	unsigned char max_p = mt_pivots[mt];
	3216	unsigned char max_s = mt_slots[mt];
	3217
	3218	if (tmp < max_p)
	3219	memset(pivs + tmp, 0,
	3220	sizeof(unsigned long ) (max_p - tmp));
	3221
	3222	if (tmp < mt_slots[mt])
	3223	memset(slots + tmp, 0, sizeof(void ) (max_s - tmp));
	3224
	3225	memcpy(node, newnode, sizeof(struct maple_node));
	3226	ma_set_meta(node, mt, 0, tmp - 1);
	3227	mte_set_pivot(eparent, mte_parent_slot(l_mas.node),
	3228	l_pivs[split]);
	3229
	3230	/* Remove data from l_pivs. */
	3231	tmp = split + 1;
	3232	memset(l_pivs + tmp, 0, sizeof(unsigned long) * (max_p - tmp));
	3233	memset(l_slots + tmp, 0, sizeof(void ) (max_s - tmp));
	3234	ma_set_meta(left, mt, 0, split);
	3235
	3236	goto done;
	3237	}
	3238
	3239	/* RCU requires replacing both l_mas, mas, and parent. */
	3240	mas->node = mt_mk_node(newnode, mt);
	3241	ma_set_meta(newnode, mt, 0, tmp);
	3242
	3243	new_left = mas_pop_node(mas);
	3244	new_left->parent = left->parent;
	3245	mt = mte_node_type(l_mas.node);
	3246	slots = ma_slots(new_left, mt);
	3247	pivs = ma_pivots(new_left, mt);
	3248	memcpy(slots, l_slots, sizeof(void ) split);
	3249	memcpy(pivs, l_pivs, sizeof(unsigned long) * split);
	3250	ma_set_meta(new_left, mt, 0, split);
	3251	l_mas.node = mt_mk_node(new_left, mt);
	3252
	3253	/* replace parent. */
	3254	offset = mte_parent_slot(mas->node);
	3255	mt = mas_parent_enum(&l_mas, l_mas.node);
	3256	parent = mas_pop_node(mas);
	3257	slots = ma_slots(parent, mt);
	3258	pivs = ma_pivots(parent, mt);
	3259	memcpy(parent, mte_to_node(eparent), sizeof(struct maple_node));
	3260	rcu_assign_pointer(slots[offset], mas->node);
	3261	rcu_assign_pointer(slots[offset - 1], l_mas.node);
	3262	pivs[offset - 1] = l_mas.max;
	3263	eparent = mt_mk_node(parent, mt);
	3264	done:
	3265	gap = mas_leaf_max_gap(mas);
	3266	mte_set_gap(eparent, mte_parent_slot(mas->node), gap);
	3267	gap = mas_leaf_max_gap(&l_mas);
	3268	mte_set_gap(eparent, mte_parent_slot(l_mas.node), gap);
	3269	mas_ascend(mas);
	3270
	3271	if (in_rcu)
	3272	mas_replace(mas, false);
	3273
	3274	mas_update_gap(mas);
	3275	}
	3276
	3277	/*
	3278	* mas_split_final_node() - Split the final node in a subtree operation.
	3279	* @mast: the maple subtree state
	3280	* @mas: The maple state
	3281	* @height: The height of the tree in case it's a new root.
	3282	*/
	3283	static inline bool mas_split_final_node(struct maple_subtree_state *mast,
	3284	struct ma_state *mas, int height)
	3285	{
	3286	struct maple_enode *ancestor;
	3287
	3288	if (mte_is_root(mas->node)) {
	3289	if (mt_is_alloc(mas->tree))
	3290	mast->bn->type = maple_arange_64;
	3291	else
	3292	mast->bn->type = maple_range_64;
	3293	mas->depth = height;
	3294	}
	3295	/*
	3296	* Only a single node is used here, could be root.
	3297	* The Big_node data should just fit in a single node.
	3298	*/
	3299	ancestor = mas_new_ma_node(mas, mast->bn);
	3300	mte_set_parent(mast->l->node, ancestor, mast->l->offset);
	3301	mte_set_parent(mast->r->node, ancestor, mast->r->offset);
	3302	mte_to_node(ancestor)->parent = mas_mn(mas)->parent;
	3303
	3304	mast->l->node = ancestor;
	3305	mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, mast->l, true);
	3306	mas->offset = mast->bn->b_end - 1;
	3307	return true;
	3308	}
	3309
	3310	/*
	3311	* mast_fill_bnode() - Copy data into the big node in the subtree state
	3312	* @mast: The maple subtree state
	3313	* @mas: the maple state
	3314	* @skip: The number of entries to skip for new nodes insertion.
	3315	*/
	3316	static inline void mast_fill_bnode(struct maple_subtree_state *mast,
	3317	struct ma_state *mas,
	3318	unsigned char skip)
	3319	{
	3320	bool cp = true;
	3321	struct maple_enode *old = mas->node;
	3322	unsigned char split;
	3323
	3324	memset(mast->bn->gap, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->gap));
	3325	memset(mast->bn->slot, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->slot));
	3326	memset(mast->bn->pivot, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->pivot));
	3327	mast->bn->b_end = 0;
	3328
	3329	if (mte_is_root(mas->node)) {
	3330	cp = false;
	3331	} else {
	3332	mas_ascend(mas);
	3333	mat_add(mast->free, old);
	3334	mas->offset = mte_parent_slot(mas->node);
	3335	}
	3336
	3337	if (cp && mast->l->offset)
	3338	mas_mab_cp(mas, 0, mast->l->offset - 1, mast->bn, 0);
	3339
	3340	split = mast->bn->b_end;
	3341	mab_set_b_end(mast->bn, mast->l, mast->l->node);
	3342	mast->r->offset = mast->bn->b_end;
	3343	mab_set_b_end(mast->bn, mast->r, mast->r->node);
	3344	if (mast->bn->pivot[mast->bn->b_end - 1] == mas->max)
	3345	cp = false;
	3346
	3347	if (cp)
	3348	mas_mab_cp(mas, split + skip, mt_slot_count(mas->node) - 1,
	3349	mast->bn, mast->bn->b_end);
	3350
	3351	mast->bn->b_end--;
	3352	mast->bn->type = mte_node_type(mas->node);
	3353	}
	3354
	3355	/*
	3356	* mast_split_data() - Split the data in the subtree state big node into regular
	3357	* nodes.
	3358	* @mast: The maple subtree state
	3359	* @mas: The maple state
	3360	* @split: The location to split the big node
	3361	*/
	3362	static inline void mast_split_data(struct maple_subtree_state *mast,
	3363	struct ma_state *mas, unsigned char split)
	3364	{
	3365	unsigned char p_slot;
	3366
	3367	mab_mas_cp(mast->bn, 0, split, mast->l, true);
	3368	mte_set_pivot(mast->r->node, 0, mast->r->max);
	3369	mab_mas_cp(mast->bn, split + 1, mast->bn->b_end, mast->r, false);
	3370	mast->l->offset = mte_parent_slot(mas->node);
	3371	mast->l->max = mast->bn->pivot[split];
	3372	mast->r->min = mast->l->max + 1;
	3373	if (mte_is_leaf(mas->node))
	3374	return;
	3375
	3376	p_slot = mast->orig_l->offset;
	3377	mas_set_split_parent(mast->orig_l, mast->l->node, mast->r->node,
	3378	&p_slot, split);
	3379	mas_set_split_parent(mast->orig_r, mast->l->node, mast->r->node,
	3380	&p_slot, split);
	3381	}
	3382
	3383	/*
	3384	* mas_push_data() - Instead of splitting a node, it is beneficial to push the
	3385	* data to the right or left node if there is room.
	3386	* @mas: The maple state
	3387	* @height: The current height of the maple state
	3388	* @mast: The maple subtree state
	3389	* @left: Push left or not.
	3390	*
	3391	* Keeping the height of the tree low means faster lookups.
	3392	*
	3393	* Return: True if pushed, false otherwise.
	3394	*/
	3395	static inline bool mas_push_data(struct ma_state *mas, int height,
	3396	struct maple_subtree_state *mast, bool left)
	3397	{
	3398	unsigned char slot_total = mast->bn->b_end;
	3399	unsigned char end, space, split;
	3400
	3401	MA_STATE(tmp_mas, mas->tree, mas->index, mas->last);
	3402	tmp_mas = *mas;
	3403	tmp_mas.depth = mast->l->depth;
	3404
	3405	if (left && !mas_prev_sibling(&tmp_mas))
	3406	return false;
	3407	else if (!left && !mas_next_sibling(&tmp_mas))
	3408	return false;
	3409
	3410	end = mas_data_end(&tmp_mas);
	3411	slot_total += end;
	3412	space = 2 * mt_slot_count(mas->node) - 2;
	3413	/* -2 instead of -1 to ensure there isn't a triple split */
	3414	if (ma_is_leaf(mast->bn->type))
	3415	space--;
	3416
	3417	if (mas->max == ULONG_MAX)
	3418	space--;
	3419
	3420	if (slot_total >= space)
	3421	return false;
	3422
	3423	/* Get the data; Fill mast->bn */
	3424	mast->bn->b_end++;
	3425	if (left) {
	3426	mab_shift_right(mast->bn, end + 1);
	3427	mas_mab_cp(&tmp_mas, 0, end, mast->bn, 0);
	3428	mast->bn->b_end = slot_total + 1;
	3429	} else {
	3430	mas_mab_cp(&tmp_mas, 0, end, mast->bn, mast->bn->b_end);
	3431	}
	3432
	3433	/* Configure mast for splitting of mast->bn */
	3434	split = mt_slots[mast->bn->type] - 2;
	3435	if (left) {
	3436	/* Switch mas to prev node */
	3437	mat_add(mast->free, mas->node);
	3438	*mas = tmp_mas;
	3439	/* Start using mast->l for the left side. */
	3440	tmp_mas.node = mast->l->node;
	3441	*mast->l = tmp_mas;
	3442	} else {
	3443	mat_add(mast->free, tmp_mas.node);
	3444	tmp_mas.node = mast->r->node;
	3445	*mast->r = tmp_mas;
	3446	split = slot_total - split;
	3447	}
	3448	split = mab_no_null_split(mast->bn, split, mt_slots[mast->bn->type]);
	3449	/* Update parent slot for split calculation. */
	3450	if (left)
	3451	mast->orig_l->offset += end + 1;
	3452
	3453	mast_split_data(mast, mas, split);
	3454	mast_fill_bnode(mast, mas, 2);
	3455	mas_split_final_node(mast, mas, height + 1);
	3456	return true;
	3457	}
	3458
	3459	/*
	3460	* mas_split() - Split data that is too big for one node into two.
	3461	* @mas: The maple state
	3462	* @b_node: The maple big node
	3463	* Return: 1 on success, 0 on failure.
	3464	*/
	3465	static int mas_split(struct ma_state mas, struct maple_big_node b_node)
	3466	{
	3467
	3468	struct maple_subtree_state mast;
	3469	int height = 0;
	3470	unsigned char mid_split, split = 0;
	3471
	3472	/*
	3473	* Splitting is handled differently from any other B-tree; the Maple
	3474	* Tree splits upwards. Splitting up means that the split operation
	3475	* occurs when the walk of the tree hits the leaves and not on the way
	3476	* down. The reason for splitting up is that it is impossible to know
	3477	* how much space will be needed until the leaf is (or leaves are)
	3478	* reached. Since overwriting data is allowed and a range could
	3479	* overwrite more than one range or result in changing one entry into 3
	3480	* entries, it is impossible to know if a split is required until the
	3481	* data is examined.
	3482	*
	3483	* Splitting is a balancing act between keeping allocations to a minimum
	3484	* and avoiding a 'jitter' event where a tree is expanded to make room
	3485	* for an entry followed by a contraction when the entry is removed. To
	3486	* accomplish the balance, there are empty slots remaining in both left
	3487	* and right nodes after a split.
	3488	*/
	3489	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
	3490	MA_STATE(r_mas, mas->tree, mas->index, mas->last);
	3491	MA_STATE(prev_l_mas, mas->tree, mas->index, mas->last);
	3492	MA_STATE(prev_r_mas, mas->tree, mas->index, mas->last);
	3493	MA_TOPIARY(mat, mas->tree);
	3494
	3495	trace_ma_op(__func__, mas);
	3496	mas->depth = mas_mt_height(mas);
	3497	/* Allocation failures will happen early. */
	3498	mas_node_count(mas, 1 + mas->depth * 2);
	3499	if (mas_is_err(mas))
	3500	return 0;
	3501
	3502	mast.l = &l_mas;
	3503	mast.r = &r_mas;
	3504	mast.orig_l = &prev_l_mas;
	3505	mast.orig_r = &prev_r_mas;
	3506	mast.free = &mat;
	3507	mast.bn = b_node;
	3508
	3509	while (height++ <= mas->depth) {
	3510	if (mt_slots[b_node->type] > b_node->b_end) {
	3511	mas_split_final_node(&mast, mas, height);
	3512	break;
	3513	}
	3514
	3515	l_mas = r_mas = *mas;
	3516	l_mas.node = mas_new_ma_node(mas, b_node);
	3517	r_mas.node = mas_new_ma_node(mas, b_node);
	3518	/*
	3519	* Another way that 'jitter' is avoided is to terminate a split up early if the
	3520	* left or right node has space to spare. This is referred to as "pushing left"
	3521	* or "pushing right" and is similar to the B* tree, except the nodes left or
	3522	* right can rarely be reused due to RCU, but the ripple upwards is halted which
	3523	* is a significant savings.
	3524	*/
	3525	/* Try to push left. */
	3526	if (mas_push_data(mas, height, &mast, true))
	3527	break;
	3528
	3529	/* Try to push right. */
	3530	if (mas_push_data(mas, height, &mast, false))
	3531	break;
	3532
	3533	split = mab_calc_split(mas, b_node, &mid_split, prev_l_mas.min);
	3534	mast_split_data(&mast, mas, split);
	3535	/*
	3536	* Usually correct, mab_mas_cp in the above call overwrites
	3537	* r->max.
	3538	*/
	3539	mast.r->max = mas->max;
	3540	mast_fill_bnode(&mast, mas, 1);
	3541	prev_l_mas = *mast.l;
	3542	prev_r_mas = *mast.r;
	3543	}
	3544
	3545	/* Set the original node as dead */
	3546	mat_add(mast.free, mas->node);
	3547	mas->node = l_mas.node;
	3548	mas_wmb_replace(mas, mast.free, NULL);
	3549	mtree_range_walk(mas);
	3550	return 1;
	3551	}
	3552
	3553	/*
	3554	* mas_reuse_node() - Reuse the node to store the data.
	3555	* @wr_mas: The maple write state
	3556	* @bn: The maple big node
	3557	* @end: The end of the data.
	3558	*
	3559	* Will always return false in RCU mode.
	3560	*
	3561	* Return: True if node was reused, false otherwise.
	3562	*/
	3563	static inline bool mas_reuse_node(struct ma_wr_state *wr_mas,
	3564	struct maple_big_node *bn, unsigned char end)
	3565	{
	3566	/* Need to be rcu safe. */
	3567	if (mt_in_rcu(wr_mas->mas->tree))
	3568	return false;
	3569
	3570	if (end > bn->b_end) {
	3571	int clear = mt_slots[wr_mas->type] - bn->b_end;
	3572
	3573	memset(wr_mas->slots + bn->b_end, 0, sizeof(void ) clear--);
	3574	memset(wr_mas->pivots + bn->b_end, 0, sizeof(void ) clear);
	3575	}
	3576	mab_mas_cp(bn, 0, bn->b_end, wr_mas->mas, false);
	3577	return true;
	3578	}
	3579
	3580	/*
	3581	* mas_commit_b_node() - Commit the big node into the tree.
	3582	* @wr_mas: The maple write state
	3583	* @b_node: The maple big node
	3584	* @end: The end of the data.
	3585	*/
	3586	static inline int mas_commit_b_node(struct ma_wr_state *wr_mas,
	3587	struct maple_big_node *b_node, unsigned char end)
	3588	{
	3589	struct maple_node *node;
	3590	unsigned char b_end = b_node->b_end;
	3591	enum maple_type b_type = b_node->type;
	3592
	3593	if ((b_end < mt_min_slots[b_type]) &&
	3594	(!mte_is_root(wr_mas->mas->node)) &&
	3595	(mas_mt_height(wr_mas->mas) > 1))
	3596	return mas_rebalance(wr_mas->mas, b_node);
	3597
	3598	if (b_end >= mt_slots[b_type])
	3599	return mas_split(wr_mas->mas, b_node);
	3600
	3601	if (mas_reuse_node(wr_mas, b_node, end))
	3602	goto reuse_node;
	3603
	3604	mas_node_count(wr_mas->mas, 1);
	3605	if (mas_is_err(wr_mas->mas))
	3606	return 0;
	3607
	3608	node = mas_pop_node(wr_mas->mas);
	3609	node->parent = mas_mn(wr_mas->mas)->parent;
	3610	wr_mas->mas->node = mt_mk_node(node, b_type);
	3611	mab_mas_cp(b_node, 0, b_end, wr_mas->mas, true);
	3612
	3613	mas_replace(wr_mas->mas, false);
	3614	reuse_node:
	3615	mas_update_gap(wr_mas->mas);
	3616	return 1;
	3617	}
	3618
	3619	/*
	3620	* mas_root_expand() - Expand a root to a node
	3621	* @mas: The maple state
	3622	* @entry: The entry to store into the tree
	3623	*/
	3624	static inline int mas_root_expand(struct ma_state mas, void entry)
	3625	{
	3626	void *contents = mas_root_locked(mas);
	3627	enum maple_type type = maple_leaf_64;
	3628	struct maple_node *node;
	3629	void __rcu **slots;
	3630	unsigned long *pivots;
	3631	int slot = 0;
	3632
	3633	mas_node_count(mas, 1);
	3634	if (unlikely(mas_is_err(mas)))
	3635	return 0;
	3636
	3637	node = mas_pop_node(mas);
	3638	pivots = ma_pivots(node, type);
	3639	slots = ma_slots(node, type);
	3640	node->parent = ma_parent_ptr(
	3641	((unsigned long)mas->tree \| MA_ROOT_PARENT));
	3642	mas->node = mt_mk_node(node, type);
	3643
	3644	if (mas->index) {
	3645	if (contents) {
	3646	rcu_assign_pointer(slots[slot], contents);
	3647	if (likely(mas->index > 1))
	3648	slot++;
	3649	}
	3650	pivots[slot++] = mas->index - 1;
	3651	}
	3652
	3653	rcu_assign_pointer(slots[slot], entry);
	3654	mas->offset = slot;
	3655	pivots[slot] = mas->last;
	3656	if (mas->last != ULONG_MAX)
	3657	slot++;
	3658	mas->depth = 1;
	3659	mas_set_height(mas);
	3660
	3661	/* swap the new root into the tree */
	3662	rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
	3663	ma_set_meta(node, maple_leaf_64, 0, slot);
	3664	return slot;
	3665	}
	3666
	3667	static inline void mas_store_root(struct ma_state mas, void entry)
	3668	{
	3669	if (likely((mas->last != 0) \|\| (mas->index != 0)))
	3670	mas_root_expand(mas, entry);
	3671	else if (((unsigned long) (entry) & 3) == 2)
	3672	mas_root_expand(mas, entry);
	3673	else {
	3674	rcu_assign_pointer(mas->tree->ma_root, entry);
	3675	mas->node = MAS_START;
	3676	}
	3677	}
	3678
	3679	/*
	3680	* mas_is_span_wr() - Check if the write needs to be treated as a write that
	3681	* spans the node.
	3682	* @mas: The maple state
	3683	* @piv: The pivot value being written
	3684	* @type: The maple node type
	3685	* @entry: The data to write
	3686	*
	3687	* Spanning writes are writes that start in one node and end in another OR if
	3688	* the write of a %NULL will cause the node to end with a %NULL.
	3689	*
	3690	* Return: True if this is a spanning write, false otherwise.
	3691	*/
	3692	static bool mas_is_span_wr(struct ma_wr_state *wr_mas)
	3693	{
	3694	unsigned long max;
	3695	unsigned long last = wr_mas->mas->last;
	3696	unsigned long piv = wr_mas->r_max;
	3697	enum maple_type type = wr_mas->type;
	3698	void *entry = wr_mas->entry;
	3699
	3700	/* Contained in this pivot */
	3701	if (piv > last)
	3702	return false;
	3703
	3704	max = wr_mas->mas->max;
	3705	if (unlikely(ma_is_leaf(type))) {
	3706	/* Fits in the node, but may span slots. */
	3707	if (last < max)
	3708	return false;
	3709
	3710	/* Writes to the end of the node but not null. */
	3711	if ((last == max) && entry)
	3712	return false;
	3713
	3714	/*
	3715	* Writing ULONG_MAX is not a spanning write regardless of the
	3716	* value being written as long as the range fits in the node.
	3717	*/
	3718	if ((last == ULONG_MAX) && (last == max))
	3719	return false;
	3720	} else if (piv == last) {
	3721	if (entry)
	3722	return false;
	3723
	3724	/* Detect spanning store wr walk */
	3725	if (last == ULONG_MAX)
	3726	return false;
	3727	}
	3728
	3729	trace_ma_write(__func__, wr_mas->mas, piv, entry);
	3730
	3731	return true;
	3732	}
	3733
	3734	static inline void mas_wr_walk_descend(struct ma_wr_state *wr_mas)
	3735	{
	3736	wr_mas->mas->depth++;
	3737	wr_mas->type = mte_node_type(wr_mas->mas->node);
	3738	mas_wr_node_walk(wr_mas);
	3739	wr_mas->slots = ma_slots(wr_mas->node, wr_mas->type);
	3740	}
	3741
	3742	static inline void mas_wr_walk_traverse(struct ma_wr_state *wr_mas)
	3743	{
	3744	wr_mas->mas->max = wr_mas->r_max;
	3745	wr_mas->mas->min = wr_mas->r_min;
	3746	wr_mas->mas->node = wr_mas->content;
	3747	wr_mas->mas->offset = 0;
	3748	}
	3749	/*
	3750	* mas_wr_walk() - Walk the tree for a write.
	3751	* @wr_mas: The maple write state
	3752	*
	3753	* Uses mas_slot_locked() and does not need to worry about dead nodes.
	3754	*
	3755	* Return: True if it's contained in a node, false on spanning write.
	3756	*/
	3757	static bool mas_wr_walk(struct ma_wr_state *wr_mas)
	3758	{
	3759	struct ma_state *mas = wr_mas->mas;
	3760
	3761	while (true) {
	3762	mas_wr_walk_descend(wr_mas);
	3763	if (unlikely(mas_is_span_wr(wr_mas)))
	3764	return false;
	3765
	3766	wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
	3767	mas->offset);
	3768	if (ma_is_leaf(wr_mas->type))
	3769	return true;
	3770
	3771	mas_wr_walk_traverse(wr_mas);
	3772	}
	3773
	3774	return true;
	3775	}
	3776
	3777	static bool mas_wr_walk_index(struct ma_wr_state *wr_mas)
	3778	{
	3779	struct ma_state *mas = wr_mas->mas;
	3780
	3781	while (true) {
	3782	mas_wr_walk_descend(wr_mas);
	3783	wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
	3784	mas->offset);
	3785	if (ma_is_leaf(wr_mas->type))
	3786	return true;
	3787	mas_wr_walk_traverse(wr_mas);
	3788
	3789	}
	3790	return true;
	3791	}
	3792	/*
	3793	* mas_extend_spanning_null() - Extend a store of a %NULL to include surrounding %NULLs.
	3794	* @l_wr_mas: The left maple write state
	3795	* @r_wr_mas: The right maple write state
	3796	*/
	3797	static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas,
	3798	struct ma_wr_state *r_wr_mas)
	3799	{
	3800	struct ma_state *r_mas = r_wr_mas->mas;
	3801	struct ma_state *l_mas = l_wr_mas->mas;
	3802	unsigned char l_slot;
	3803
	3804	l_slot = l_mas->offset;
	3805	if (!l_wr_mas->content)
	3806	l_mas->index = l_wr_mas->r_min;
	3807
	3808	if ((l_mas->index == l_wr_mas->r_min) &&
	3809	(l_slot &&
	3810	!mas_slot_locked(l_mas, l_wr_mas->slots, l_slot - 1))) {
	3811	if (l_slot > 1)
	3812	l_mas->index = l_wr_mas->pivots[l_slot - 2] + 1;
	3813	else
	3814	l_mas->index = l_mas->min;
	3815
	3816	l_mas->offset = l_slot - 1;
	3817	}
	3818
	3819	if (!r_wr_mas->content) {
	3820	if (r_mas->last < r_wr_mas->r_max)
	3821	r_mas->last = r_wr_mas->r_max;
	3822	r_mas->offset++;
	3823	} else if ((r_mas->last == r_wr_mas->r_max) &&
	3824	(r_mas->last < r_mas->max) &&
	3825	!mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) {
	3826	r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots,
	3827	r_wr_mas->type, r_mas->offset + 1);
	3828	r_mas->offset++;
	3829	}
	3830	}
	3831
	3832	static inline void mas_state_walk(struct ma_state mas)
	3833	{
	3834	void *entry;
	3835
	3836	entry = mas_start(mas);
	3837	if (mas_is_none(mas))
	3838	return NULL;
	3839
	3840	if (mas_is_ptr(mas))
	3841	return entry;
	3842
	3843	return mtree_range_walk(mas);
	3844	}
	3845
	3846	/*
	3847	* mtree_lookup_walk() - Internal quick lookup that does not keep maple state up
	3848	* to date.
	3849	*
	3850	* @mas: The maple state.
	3851	*
	3852	* Note: Leaves mas in undesirable state.
	3853	* Return: The entry for @mas->index or %NULL on dead node.
	3854	*/
	3855	static inline void mtree_lookup_walk(struct ma_state mas)
	3856	{
	3857	unsigned long *pivots;
	3858	unsigned char offset;
	3859	struct maple_node *node;
	3860	struct maple_enode *next;
	3861	enum maple_type type;
	3862	void __rcu **slots;
	3863	unsigned char end;
	3864	unsigned long max;
	3865
	3866	next = mas->node;
	3867	max = ULONG_MAX;
	3868	do {
	3869	offset = 0;
	3870	node = mte_to_node(next);
	3871	type = mte_node_type(next);
	3872	pivots = ma_pivots(node, type);
	3873	end = ma_data_end(node, type, pivots, max);
	3874	if (unlikely(ma_dead_node(node)))
	3875	goto dead_node;
	3876
	3877	if (pivots[offset] >= mas->index)
	3878	goto next;
	3879
	3880	do {
	3881	offset++;
	3882	} while ((offset < end) && (pivots[offset] < mas->index));
	3883
	3884	if (likely(offset > end))
	3885	max = pivots[offset];
	3886
	3887	next:
	3888	slots = ma_slots(node, type);
	3889	next = mt_slot(mas->tree, slots, offset);
	3890	if (unlikely(ma_dead_node(node)))
	3891	goto dead_node;
	3892	} while (!ma_is_leaf(type));
	3893
	3894	return (void *) next;
	3895
	3896	dead_node:
	3897	mas_reset(mas);
	3898	return NULL;
	3899	}
	3900
	3901	/*
	3902	* mas_new_root() - Create a new root node that only contains the entry passed
	3903	* in.
	3904	* @mas: The maple state
	3905	* @entry: The entry to store.
	3906	*
	3907	* Only valid when the index == 0 and the last == ULONG_MAX
	3908	*
	3909	* Return 0 on error, 1 on success.
	3910	*/
	3911	static inline int mas_new_root(struct ma_state mas, void entry)
	3912	{
	3913	struct maple_enode *root = mas_root_locked(mas);
	3914	enum maple_type type = maple_leaf_64;
	3915	struct maple_node *node;
	3916	void __rcu **slots;
	3917	unsigned long *pivots;
	3918
	3919	if (!entry && !mas->index && mas->last == ULONG_MAX) {
	3920	mas->depth = 0;
	3921	mas_set_height(mas);
	3922	rcu_assign_pointer(mas->tree->ma_root, entry);
	3923	mas->node = MAS_START;
	3924	goto done;
	3925	}
	3926
	3927	mas_node_count(mas, 1);
	3928	if (mas_is_err(mas))
	3929	return 0;
	3930
	3931	node = mas_pop_node(mas);
	3932	pivots = ma_pivots(node, type);
	3933	slots = ma_slots(node, type);
	3934	node->parent = ma_parent_ptr(
	3935	((unsigned long)mas->tree \| MA_ROOT_PARENT));
	3936	mas->node = mt_mk_node(node, type);
	3937	rcu_assign_pointer(slots[0], entry);
	3938	pivots[0] = mas->last;
	3939	mas->depth = 1;
	3940	mas_set_height(mas);
	3941	rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
	3942
	3943	done:
	3944	if (xa_is_node(root))
	3945	mte_destroy_walk(root, mas->tree);
	3946
	3947	return 1;
	3948	}
	3949	/*
	3950	* mas_wr_spanning_store() - Create a subtree with the store operation completed
	3951	* and new nodes where necessary, then place the sub-tree in the actual tree.
	3952	* Note that mas is expected to point to the node which caused the store to
	3953	* span.
	3954	* @wr_mas: The maple write state
	3955	*
	3956	* Return: 0 on error, positive on success.
	3957	*/
	3958	static inline int mas_wr_spanning_store(struct ma_wr_state *wr_mas)
	3959	{
	3960	struct maple_subtree_state mast;
	3961	struct maple_big_node b_node;
	3962	struct ma_state *mas;
	3963	unsigned char height;
	3964
	3965	/* Left and Right side of spanning store */
	3966	MA_STATE(l_mas, NULL, 0, 0);
	3967	MA_STATE(r_mas, NULL, 0, 0);
	3968
	3969	MA_WR_STATE(r_wr_mas, &r_mas, wr_mas->entry);
	3970	MA_WR_STATE(l_wr_mas, &l_mas, wr_mas->entry);
	3971
	3972	/*
	3973	* A store operation that spans multiple nodes is called a spanning
	3974	* store and is handled early in the store call stack by the function
	3975	* mas_is_span_wr(). When a spanning store is identified, the maple
	3976	* state is duplicated. The first maple state walks the left tree path
	3977	* to ``index``, the duplicate walks the right tree path to ``last``.
	3978	* The data in the two nodes are combined into a single node, two nodes,
	3979	* or possibly three nodes (see the 3-way split above). A ``NULL``
	3980	* written to the last entry of a node is considered a spanning store as
	3981	* a rebalance is required for the operation to complete and an overflow
	3982	* of data may happen.
	3983	*/
	3984	mas = wr_mas->mas;
	3985	trace_ma_op(__func__, mas);
	3986
	3987	if (unlikely(!mas->index && mas->last == ULONG_MAX))
	3988	return mas_new_root(mas, wr_mas->entry);
	3989	/*
	3990	* Node rebalancing may occur due to this store, so there may be three new
	3991	* entries per level plus a new root.
	3992	*/
	3993	height = mas_mt_height(mas);
	3994	mas_node_count(mas, 1 + height * 3);
	3995	if (mas_is_err(mas))
	3996	return 0;
	3997
	3998	/*
	3999	* Set up right side. Need to get to the next offset after the spanning
	4000	* store to ensure it's not NULL and to combine both the next node and
	4001	* the node with the start together.
	4002	*/
	4003	r_mas = *mas;
	4004	/* Avoid overflow, walk to next slot in the tree. */
	4005	if (r_mas.last + 1)
	4006	r_mas.last++;
	4007
	4008	r_mas.index = r_mas.last;
	4009	mas_wr_walk_index(&r_wr_mas);
	4010	r_mas.last = r_mas.index = mas->last;
	4011
	4012	/* Set up left side. */
	4013	l_mas = *mas;
	4014	mas_wr_walk_index(&l_wr_mas);
	4015
	4016	if (!wr_mas->entry) {
	4017	mas_extend_spanning_null(&l_wr_mas, &r_wr_mas);
	4018	mas->offset = l_mas.offset;
	4019	mas->index = l_mas.index;
	4020	mas->last = l_mas.last = r_mas.last;
	4021	}
	4022
	4023	/* expanding NULLs may make this cover the entire range */
	4024	if (!l_mas.index && r_mas.last == ULONG_MAX) {
	4025	mas_set_range(mas, 0, ULONG_MAX);
	4026	return mas_new_root(mas, wr_mas->entry);
	4027	}
	4028
	4029	memset(&b_node, 0, sizeof(struct maple_big_node));
	4030	/* Copy l_mas and store the value in b_node. */
	4031	mas_store_b_node(&l_wr_mas, &b_node, l_wr_mas.node_end);
	4032	/* Copy r_mas into b_node. */
	4033	if (r_mas.offset <= r_wr_mas.node_end)
	4034	mas_mab_cp(&r_mas, r_mas.offset, r_wr_mas.node_end,
	4035	&b_node, b_node.b_end + 1);
	4036	else
	4037	b_node.b_end++;
	4038
	4039	/* Stop spanning searches by searching for just index. */
	4040	l_mas.index = l_mas.last = mas->index;
	4041
	4042	mast.bn = &b_node;
	4043	mast.orig_l = &l_mas;
	4044	mast.orig_r = &r_mas;
	4045	/* Combine l_mas and r_mas and split them up evenly again. */
	4046	return mas_spanning_rebalance(mas, &mast, height + 1);
	4047	}
	4048
	4049	/*
	4050	* mas_wr_node_store() - Attempt to store the value in a node
	4051	* @wr_mas: The maple write state
	4052	*
	4053	* Attempts to reuse the node, but may allocate.
	4054	*
	4055	* Return: True if stored, false otherwise
	4056	*/
	4057	static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas)
	4058	{
	4059	struct ma_state *mas = wr_mas->mas;
	4060	void __rcu **dst_slots;
	4061	unsigned long *dst_pivots;
	4062	unsigned char dst_offset;
	4063	unsigned char new_end = wr_mas->node_end;
	4064	unsigned char offset;
	4065	unsigned char node_slots = mt_slots[wr_mas->type];
	4066	struct maple_node reuse, *newnode;
	4067	unsigned char copy_size, max_piv = mt_pivots[wr_mas->type];
	4068	bool in_rcu = mt_in_rcu(mas->tree);
	4069
	4070	offset = mas->offset;
	4071	if (mas->last == wr_mas->r_max) {
	4072	/* runs right to the end of the node */
	4073	if (mas->last == mas->max)
	4074	new_end = offset;
	4075	/* don't copy this offset */
	4076	wr_mas->offset_end++;
	4077	} else if (mas->last < wr_mas->r_max) {
	4078	/* new range ends in this range */
	4079	if (unlikely(wr_mas->r_max == ULONG_MAX))
	4080	mas_bulk_rebalance(mas, wr_mas->node_end, wr_mas->type);
	4081
	4082	new_end++;
	4083	} else {
	4084	if (wr_mas->end_piv == mas->last)
	4085	wr_mas->offset_end++;
	4086
	4087	new_end -= wr_mas->offset_end - offset - 1;
	4088	}
	4089
	4090	/* new range starts within a range */
	4091	if (wr_mas->r_min < mas->index)
	4092	new_end++;
	4093
	4094	/* Not enough room */
	4095	if (new_end >= node_slots)
	4096	return false;
	4097
	4098	/* Not enough data. */
	4099	if (!mte_is_root(mas->node) && (new_end <= mt_min_slots[wr_mas->type]) &&
	4100	!(mas->mas_flags & MA_STATE_BULK))
	4101	return false;
	4102
	4103	/* set up node. */
	4104	if (in_rcu) {
	4105	mas_node_count(mas, 1);
	4106	if (mas_is_err(mas))
	4107	return false;
	4108
	4109	newnode = mas_pop_node(mas);
	4110	} else {
	4111	memset(&reuse, 0, sizeof(struct maple_node));
	4112	newnode = &reuse;
	4113	}
	4114
	4115	newnode->parent = mas_mn(mas)->parent;
	4116	dst_pivots = ma_pivots(newnode, wr_mas->type);
	4117	dst_slots = ma_slots(newnode, wr_mas->type);
	4118	/* Copy from start to insert point */
	4119	memcpy(dst_pivots, wr_mas->pivots, sizeof(unsigned long) * (offset + 1));
	4120	memcpy(dst_slots, wr_mas->slots, sizeof(void ) (offset + 1));
	4121	dst_offset = offset;
	4122
	4123	/* Handle insert of new range starting after old range */
	4124	if (wr_mas->r_min < mas->index) {
	4125	mas->offset++;
	4126	rcu_assign_pointer(dst_slots[dst_offset], wr_mas->content);
	4127	dst_pivots[dst_offset++] = mas->index - 1;
	4128	}
	4129
	4130	/* Store the new entry and range end. */
	4131	if (dst_offset < max_piv)
	4132	dst_pivots[dst_offset] = mas->last;
	4133	mas->offset = dst_offset;
	4134	rcu_assign_pointer(dst_slots[dst_offset], wr_mas->entry);
	4135
	4136	/*
	4137	* this range wrote to the end of the node or it overwrote the rest of
	4138	* the data
	4139	*/
	4140	if (wr_mas->offset_end > wr_mas->node_end \|\| mas->last >= mas->max) {
	4141	new_end = dst_offset;
	4142	goto done;
	4143	}
	4144
	4145	dst_offset++;
	4146	/* Copy to the end of node if necessary. */
	4147	copy_size = wr_mas->node_end - wr_mas->offset_end + 1;
	4148	memcpy(dst_slots + dst_offset, wr_mas->slots + wr_mas->offset_end,
	4149	sizeof(void ) copy_size);
	4150	if (dst_offset < max_piv) {
	4151	if (copy_size > max_piv - dst_offset)
	4152	copy_size = max_piv - dst_offset;
	4153
	4154	memcpy(dst_pivots + dst_offset,
	4155	wr_mas->pivots + wr_mas->offset_end,
	4156	sizeof(unsigned long) * copy_size);
	4157	}
	4158
	4159	if ((wr_mas->node_end == node_slots - 1) && (new_end < node_slots - 1))
	4160	dst_pivots[new_end] = mas->max;
	4161
	4162	done:
	4163	mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end);
	4164	if (in_rcu) {
	4165	mas->node = mt_mk_node(newnode, wr_mas->type);
	4166	mas_replace(mas, false);
	4167	} else {
	4168	memcpy(wr_mas->node, newnode, sizeof(struct maple_node));
	4169	}
	4170	trace_ma_write(__func__, mas, 0, wr_mas->entry);
	4171	mas_update_gap(mas);
	4172	return true;
	4173	}
	4174
	4175	/*
	4176	* mas_wr_slot_store: Attempt to store a value in a slot.
	4177	* @wr_mas: the maple write state
	4178	*
	4179	* Return: True if stored, false otherwise
	4180	*/
	4181	static inline bool mas_wr_slot_store(struct ma_wr_state *wr_mas)
	4182	{
	4183	struct ma_state *mas = wr_mas->mas;
	4184	unsigned long lmax; /* Logical max. */
	4185	unsigned char offset = mas->offset;
	4186
	4187	if ((wr_mas->r_max > mas->last) && ((wr_mas->r_min != mas->index) \|\|
	4188	(offset != wr_mas->node_end)))
	4189	return false;
	4190
	4191	if (offset == wr_mas->node_end - 1)
	4192	lmax = mas->max;
	4193	else
	4194	lmax = wr_mas->pivots[offset + 1];
	4195
	4196	/* going to overwrite too many slots. */
	4197	if (lmax < mas->last)
	4198	return false;
	4199
	4200	if (wr_mas->r_min == mas->index) {
	4201	/* overwriting two or more ranges with one. */
	4202	if (lmax == mas->last)
	4203	return false;
	4204
	4205	/* Overwriting all of offset and a portion of offset + 1. */
	4206	rcu_assign_pointer(wr_mas->slots[offset], wr_mas->entry);
	4207	wr_mas->pivots[offset] = mas->last;
	4208	goto done;
	4209	}
	4210
	4211	/* Doesn't end on the next range end. */
	4212	if (lmax != mas->last)
	4213	return false;
	4214
	4215	/* Overwriting a portion of offset and all of offset + 1 */
	4216	if ((offset + 1 < mt_pivots[wr_mas->type]) &&
	4217	(wr_mas->entry \|\| wr_mas->pivots[offset + 1]))
	4218	wr_mas->pivots[offset + 1] = mas->last;
	4219
	4220	rcu_assign_pointer(wr_mas->slots[offset + 1], wr_mas->entry);
	4221	wr_mas->pivots[offset] = mas->index - 1;
	4222	mas->offset++; /* Keep mas accurate. */
	4223
	4224	done:
	4225	trace_ma_write(__func__, mas, 0, wr_mas->entry);
	4226	mas_update_gap(mas);
	4227	return true;
	4228	}
	4229
	4230	static inline void mas_wr_end_piv(struct ma_wr_state *wr_mas)
	4231	{
	4232	while ((wr_mas->mas->last > wr_mas->end_piv) &&
	4233	(wr_mas->offset_end < wr_mas->node_end))
	4234	wr_mas->end_piv = wr_mas->pivots[++wr_mas->offset_end];
	4235
	4236	if (wr_mas->mas->last > wr_mas->end_piv)
	4237	wr_mas->end_piv = wr_mas->mas->max;
	4238	}
	4239
	4240	static inline void mas_wr_extend_null(struct ma_wr_state *wr_mas)
	4241	{
	4242	struct ma_state *mas = wr_mas->mas;
	4243
	4244	if (mas->last < wr_mas->end_piv && !wr_mas->slots[wr_mas->offset_end])
	4245	mas->last = wr_mas->end_piv;
	4246
	4247	/* Check next slot(s) if we are overwriting the end */
	4248	if ((mas->last == wr_mas->end_piv) &&
	4249	(wr_mas->node_end != wr_mas->offset_end) &&
	4250	!wr_mas->slots[wr_mas->offset_end + 1]) {
	4251	wr_mas->offset_end++;
	4252	if (wr_mas->offset_end == wr_mas->node_end)
	4253	mas->last = mas->max;
	4254	else
	4255	mas->last = wr_mas->pivots[wr_mas->offset_end];
	4256	wr_mas->end_piv = mas->last;
	4257	}
	4258
	4259	if (!wr_mas->content) {
	4260	/* If this one is null, the next and prev are not */
	4261	mas->index = wr_mas->r_min;
	4262	} else {
	4263	/* Check prev slot if we are overwriting the start */
	4264	if (mas->index == wr_mas->r_min && mas->offset &&
	4265	!wr_mas->slots[mas->offset - 1]) {
	4266	mas->offset--;
	4267	wr_mas->r_min = mas->index =
	4268	mas_safe_min(mas, wr_mas->pivots, mas->offset);
	4269	wr_mas->r_max = wr_mas->pivots[mas->offset];
	4270	}
	4271	}
	4272	}
	4273
	4274	static inline bool mas_wr_append(struct ma_wr_state *wr_mas)
	4275	{
	4276	unsigned char end = wr_mas->node_end;
	4277	unsigned char new_end = end + 1;
	4278	struct ma_state *mas = wr_mas->mas;
	4279	unsigned char node_pivots = mt_pivots[wr_mas->type];
	4280
	4281	if ((mas->index != wr_mas->r_min) && (mas->last == wr_mas->r_max)) {
	4282	if (new_end < node_pivots)
	4283	wr_mas->pivots[new_end] = wr_mas->pivots[end];
	4284
	4285	if (new_end < node_pivots)
	4286	ma_set_meta(wr_mas->node, maple_leaf_64, 0, new_end);
	4287
	4288	rcu_assign_pointer(wr_mas->slots[new_end], wr_mas->entry);
	4289	mas->offset = new_end;
	4290	wr_mas->pivots[end] = mas->index - 1;
	4291
	4292	return true;
	4293	}
	4294
	4295	if ((mas->index == wr_mas->r_min) && (mas->last < wr_mas->r_max)) {
	4296	if (new_end < node_pivots)
	4297	wr_mas->pivots[new_end] = wr_mas->pivots[end];
	4298
	4299	rcu_assign_pointer(wr_mas->slots[new_end], wr_mas->content);
	4300	if (new_end < node_pivots)
	4301	ma_set_meta(wr_mas->node, maple_leaf_64, 0, new_end);
	4302
	4303	wr_mas->pivots[end] = mas->last;
	4304	rcu_assign_pointer(wr_mas->slots[end], wr_mas->entry);
	4305	return true;
	4306	}
	4307
	4308	return false;
	4309	}
	4310
	4311	/*
	4312	* mas_wr_bnode() - Slow path for a modification.
	4313	* @wr_mas: The write maple state
	4314	*
	4315	* This is where split, rebalance end up.
	4316	*/
	4317	static void mas_wr_bnode(struct ma_wr_state *wr_mas)
	4318	{
	4319	struct maple_big_node b_node;
	4320
	4321	trace_ma_write(__func__, wr_mas->mas, 0, wr_mas->entry);
	4322	memset(&b_node, 0, sizeof(struct maple_big_node));
	4323	mas_store_b_node(wr_mas, &b_node, wr_mas->offset_end);
	4324	mas_commit_b_node(wr_mas, &b_node, wr_mas->node_end);
	4325	}
	4326
	4327	static inline void mas_wr_modify(struct ma_wr_state *wr_mas)
	4328	{
	4329	unsigned char node_slots;
	4330	unsigned char node_size;
	4331	struct ma_state *mas = wr_mas->mas;
	4332
	4333	/* Direct replacement */
	4334	if (wr_mas->r_min == mas->index && wr_mas->r_max == mas->last) {
	4335	rcu_assign_pointer(wr_mas->slots[mas->offset], wr_mas->entry);
	4336	if (!!wr_mas->entry ^ !!wr_mas->content)
	4337	mas_update_gap(mas);
	4338	return;
	4339	}
	4340
	4341	/* Attempt to append */
	4342	node_slots = mt_slots[wr_mas->type];
	4343	node_size = wr_mas->node_end - wr_mas->offset_end + mas->offset + 2;
	4344	if (mas->max == ULONG_MAX)
	4345	node_size++;
	4346
	4347	/* slot and node store will not fit, go to the slow path */
	4348	if (unlikely(node_size >= node_slots))
	4349	goto slow_path;
	4350
	4351	if (wr_mas->entry && (wr_mas->node_end < node_slots - 1) &&
	4352	(mas->offset == wr_mas->node_end) && mas_wr_append(wr_mas)) {
	4353	if (!wr_mas->content \|\| !wr_mas->entry)
	4354	mas_update_gap(mas);
	4355	return;
	4356	}
	4357
	4358	if ((wr_mas->offset_end - mas->offset <= 1) && mas_wr_slot_store(wr_mas))
	4359	return;
	4360	else if (mas_wr_node_store(wr_mas))
	4361	return;
	4362
	4363	if (mas_is_err(mas))
	4364	return;
	4365
	4366	slow_path:
	4367	mas_wr_bnode(wr_mas);
	4368	}
	4369
	4370	/*
	4371	* mas_wr_store_entry() - Internal call to store a value
	4372	* @mas: The maple state
	4373	* @entry: The entry to store.
	4374	*
	4375	* Return: The contents that was stored at the index.
	4376	*/
	4377	static inline void mas_wr_store_entry(struct ma_wr_state wr_mas)
	4378	{
	4379	struct ma_state *mas = wr_mas->mas;
	4380
	4381	wr_mas->content = mas_start(mas);
	4382	if (mas_is_none(mas) \|\| mas_is_ptr(mas)) {
	4383	mas_store_root(mas, wr_mas->entry);
	4384	return wr_mas->content;
	4385	}
	4386
	4387	if (unlikely(!mas_wr_walk(wr_mas))) {
	4388	mas_wr_spanning_store(wr_mas);
	4389	return wr_mas->content;
	4390	}
	4391
	4392	/* At this point, we are at the leaf node that needs to be altered. */
	4393	wr_mas->end_piv = wr_mas->r_max;
	4394	mas_wr_end_piv(wr_mas);
	4395
	4396	if (!wr_mas->entry)
	4397	mas_wr_extend_null(wr_mas);
	4398
	4399	/* New root for a single pointer */
	4400	if (unlikely(!mas->index && mas->last == ULONG_MAX)) {
	4401	mas_new_root(mas, wr_mas->entry);
	4402	return wr_mas->content;
	4403	}
	4404
	4405	mas_wr_modify(wr_mas);
	4406	return wr_mas->content;
	4407	}
	4408
	4409	/**
	4410	* mas_insert() - Internal call to insert a value
	4411	* @mas: The maple state
	4412	* @entry: The entry to store
	4413	*
	4414	* Return: %NULL or the contents that already exists at the requested index
	4415	* otherwise. The maple state needs to be checked for error conditions.
	4416	*/
	4417	static inline void mas_insert(struct ma_state mas, void *entry)
	4418	{
	4419	MA_WR_STATE(wr_mas, mas, entry);
	4420
	4421	/*
	4422	* Inserting a new range inserts either 0, 1, or 2 pivots within the
	4423	* tree. If the insert fits exactly into an existing gap with a value
	4424	* of NULL, then the slot only needs to be written with the new value.
	4425	* If the range being inserted is adjacent to another range, then only a
	4426	* single pivot needs to be inserted (as well as writing the entry). If
	4427	* the new range is within a gap but does not touch any other ranges,
	4428	* then two pivots need to be inserted: the start - 1, and the end. As
	4429	* usual, the entry must be written. Most operations require a new node
	4430	* to be allocated and replace an existing node to ensure RCU safety,
	4431	* when in RCU mode. The exception to requiring a newly allocated node
	4432	* is when inserting at the end of a node (appending). When done
	4433	* carefully, appending can reuse the node in place.
	4434	*/
	4435	wr_mas.content = mas_start(mas);
	4436	if (wr_mas.content)
	4437	goto exists;
	4438
	4439	if (mas_is_none(mas) \|\| mas_is_ptr(mas)) {
	4440	mas_store_root(mas, entry);
	4441	return NULL;
	4442	}
	4443
	4444	/* spanning writes always overwrite something */
	4445	if (!mas_wr_walk(&wr_mas))
	4446	goto exists;
	4447
	4448	/* At this point, we are at the leaf node that needs to be altered. */
	4449	wr_mas.offset_end = mas->offset;
	4450	wr_mas.end_piv = wr_mas.r_max;
	4451
	4452	if (wr_mas.content \|\| (mas->last > wr_mas.r_max))
	4453	goto exists;
	4454
	4455	if (!entry)
	4456	return NULL;
	4457
	4458	mas_wr_modify(&wr_mas);
	4459	return wr_mas.content;
	4460
	4461	exists:
	4462	mas_set_err(mas, -EEXIST);
	4463	return wr_mas.content;
	4464
	4465	}
	4466
	4467	/*
	4468	* mas_prev_node() - Find the prev non-null entry at the same level in the
	4469	* tree. The prev value will be mas->node[mas->offset] or MAS_NONE.
	4470	* @mas: The maple state
	4471	* @min: The lower limit to search
	4472	*
	4473	* The prev node value will be mas->node[mas->offset] or MAS_NONE.
	4474	* Return: 1 if the node is dead, 0 otherwise.
	4475	*/
	4476	static inline int mas_prev_node(struct ma_state *mas, unsigned long min)
	4477	{
	4478	enum maple_type mt;
	4479	int offset, level;
	4480	void __rcu **slots;
	4481	struct maple_node *node;
	4482	struct maple_enode *enode;
	4483	unsigned long *pivots;
	4484
	4485	if (mas_is_none(mas))
	4486	return 0;
	4487
	4488	level = 0;
	4489	do {
	4490	node = mas_mn(mas);
	4491	if (ma_is_root(node))
	4492	goto no_entry;
	4493
	4494	/* Walk up. */
	4495	if (unlikely(mas_ascend(mas)))
	4496	return 1;
	4497	offset = mas->offset;
	4498	level++;
	4499	} while (!offset);
	4500
	4501	offset--;
	4502	mt = mte_node_type(mas->node);
	4503	node = mas_mn(mas);
	4504	slots = ma_slots(node, mt);
	4505	pivots = ma_pivots(node, mt);
	4506	mas->max = pivots[offset];
	4507	if (offset)
	4508	mas->min = pivots[offset - 1] + 1;
	4509	if (unlikely(ma_dead_node(node)))
	4510	return 1;
	4511
	4512	if (mas->max < min)
	4513	goto no_entry_min;
	4514
	4515	while (level > 1) {
	4516	level--;
	4517	enode = mas_slot(mas, slots, offset);
	4518	if (unlikely(ma_dead_node(node)))
	4519	return 1;
	4520
	4521	mas->node = enode;
	4522	mt = mte_node_type(mas->node);
	4523	node = mas_mn(mas);
	4524	slots = ma_slots(node, mt);
	4525	pivots = ma_pivots(node, mt);
	4526	offset = ma_data_end(node, mt, pivots, mas->max);
	4527	if (offset)
	4528	mas->min = pivots[offset - 1] + 1;
	4529
	4530	if (offset < mt_pivots[mt])
	4531	mas->max = pivots[offset];
	4532
	4533	if (mas->max < min)
	4534	goto no_entry;
	4535	}
	4536
	4537	mas->node = mas_slot(mas, slots, offset);
	4538	if (unlikely(ma_dead_node(node)))
	4539	return 1;
	4540
	4541	mas->offset = mas_data_end(mas);
	4542	if (unlikely(mte_dead_node(mas->node)))
	4543	return 1;
	4544
	4545	return 0;
	4546
	4547	no_entry_min:
	4548	mas->offset = offset;
	4549	if (offset)
	4550	mas->min = pivots[offset - 1] + 1;
	4551	no_entry:
	4552	if (unlikely(ma_dead_node(node)))
	4553	return 1;
	4554
	4555	mas->node = MAS_NONE;
	4556	return 0;
	4557	}
	4558
	4559	/*
	4560	* mas_next_node() - Get the next node at the same level in the tree.
	4561	* @mas: The maple state
	4562	* @max: The maximum pivot value to check.
	4563	*
	4564	* The next value will be mas->node[mas->offset] or MAS_NONE.
	4565	* Return: 1 on dead node, 0 otherwise.
	4566	*/
	4567	static inline int mas_next_node(struct ma_state mas, struct maple_node node,
	4568	unsigned long max)
	4569	{
	4570	unsigned long min, pivot;
	4571	unsigned long *pivots;
	4572	struct maple_enode *enode;
	4573	int level = 0;
	4574	unsigned char offset;
	4575	enum maple_type mt;
	4576	void __rcu **slots;
	4577
	4578	if (mas->max >= max)
	4579	goto no_entry;
	4580
	4581	level = 0;
	4582	do {
	4583	if (ma_is_root(node))
	4584	goto no_entry;
	4585
	4586	min = mas->max + 1;
	4587	if (min > max)
	4588	goto no_entry;
	4589
	4590	if (unlikely(mas_ascend(mas)))
	4591	return 1;
	4592
	4593	offset = mas->offset;
	4594	level++;
	4595	node = mas_mn(mas);
	4596	mt = mte_node_type(mas->node);
	4597	pivots = ma_pivots(node, mt);
	4598	} while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max)));
	4599
	4600	slots = ma_slots(node, mt);
	4601	pivot = mas_safe_pivot(mas, pivots, ++offset, mt);
	4602	while (unlikely(level > 1)) {
	4603	/* Descend, if necessary */
	4604	enode = mas_slot(mas, slots, offset);
	4605	if (unlikely(ma_dead_node(node)))
	4606	return 1;
	4607
	4608	mas->node = enode;
	4609	level--;
	4610	node = mas_mn(mas);
	4611	mt = mte_node_type(mas->node);
	4612	slots = ma_slots(node, mt);
	4613	pivots = ma_pivots(node, mt);
	4614	offset = 0;
	4615	pivot = pivots[0];
	4616	}
	4617
	4618	enode = mas_slot(mas, slots, offset);
	4619	if (unlikely(ma_dead_node(node)))
	4620	return 1;
	4621
	4622	mas->node = enode;
	4623	mas->min = min;
	4624	mas->max = pivot;
	4625	return 0;
	4626
	4627	no_entry:
	4628	if (unlikely(ma_dead_node(node)))
	4629	return 1;
	4630
	4631	mas->node = MAS_NONE;
	4632	return 0;
	4633	}
	4634
	4635	/*
	4636	* mas_next_nentry() - Get the next node entry
	4637	* @mas: The maple state
	4638	* @max: The maximum value to check
	4639	* @*range_start: Pointer to store the start of the range.
	4640	*
	4641	* Sets @mas->offset to the offset of the next node entry, @mas->last to the
	4642	* pivot of the entry.
	4643	*
	4644	* Return: The next entry, %NULL otherwise
	4645	*/
	4646	static inline void mas_next_nentry(struct ma_state mas,
	4647	struct maple_node *node, unsigned long max, enum maple_type type)
	4648	{
	4649	unsigned char count;
	4650	unsigned long pivot;
	4651	unsigned long *pivots;
	4652	void __rcu **slots;
	4653	void *entry;
	4654
	4655	if (mas->last == mas->max) {
	4656	mas->index = mas->max;
	4657	return NULL;
	4658	}
	4659
	4660	pivots = ma_pivots(node, type);
	4661	slots = ma_slots(node, type);
	4662	mas->index = mas_safe_min(mas, pivots, mas->offset);
	4663	if (ma_dead_node(node))
	4664	return NULL;
	4665
	4666	if (mas->index > max)
	4667	return NULL;
	4668
	4669	count = ma_data_end(node, type, pivots, mas->max);
	4670	if (mas->offset > count)
	4671	return NULL;
	4672
	4673	while (mas->offset < count) {
	4674	pivot = pivots[mas->offset];
	4675	entry = mas_slot(mas, slots, mas->offset);
	4676	if (ma_dead_node(node))
	4677	return NULL;
	4678
	4679	if (entry)
	4680	goto found;
	4681
	4682	if (pivot >= max)
	4683	return NULL;
	4684
	4685	mas->index = pivot + 1;
	4686	mas->offset++;
	4687	}
	4688
	4689	if (mas->index > mas->max) {
	4690	mas->index = mas->last;
	4691	return NULL;
	4692	}
	4693
	4694	pivot = mas_safe_pivot(mas, pivots, mas->offset, type);
	4695	entry = mas_slot(mas, slots, mas->offset);
	4696	if (ma_dead_node(node))
	4697	return NULL;
	4698
	4699	if (!pivot)
	4700	return NULL;
	4701
	4702	if (!entry)
	4703	return NULL;
	4704
	4705	found:
	4706	mas->last = pivot;
	4707	return entry;
	4708	}
	4709
	4710	static inline void mas_rewalk(struct ma_state *mas, unsigned long index)
	4711	{
	4712
	4713	retry:
	4714	mas_set(mas, index);
	4715	mas_state_walk(mas);
	4716	if (mas_is_start(mas))
	4717	goto retry;
	4718
	4719	return;
	4720
	4721	}
	4722
	4723	/*
	4724	* mas_next_entry() - Internal function to get the next entry.
	4725	* @mas: The maple state
	4726	* @limit: The maximum range start.
	4727	*
	4728	* Set the @mas->node to the next entry and the range_start to
	4729	* the beginning value for the entry. Does not check beyond @limit.
	4730	* Sets @mas->index and @mas->last to the limit if it is hit.
	4731	* Restarts on dead nodes.
	4732	*
	4733	* Return: the next entry or %NULL.
	4734	*/
	4735	static inline void mas_next_entry(struct ma_state mas, unsigned long limit)
	4736	{
	4737	void *entry = NULL;
	4738	struct maple_enode *prev_node;
	4739	struct maple_node *node;
	4740	unsigned char offset;
	4741	unsigned long last;
	4742	enum maple_type mt;
	4743
	4744	last = mas->last;
	4745	retry:
	4746	offset = mas->offset;
	4747	prev_node = mas->node;
	4748	node = mas_mn(mas);
	4749	mt = mte_node_type(mas->node);
	4750	mas->offset++;
	4751	if (unlikely(mas->offset >= mt_slots[mt])) {
	4752	mas->offset = mt_slots[mt] - 1;
	4753	goto next_node;
	4754	}
	4755
	4756	while (!mas_is_none(mas)) {
	4757	entry = mas_next_nentry(mas, node, limit, mt);
	4758	if (unlikely(ma_dead_node(node))) {
	4759	mas_rewalk(mas, last);
	4760	goto retry;
	4761	}
	4762
	4763	if (likely(entry))
	4764	return entry;
	4765
	4766	if (unlikely((mas->index > limit)))
	4767	break;
	4768
	4769	next_node:
	4770	prev_node = mas->node;
	4771	offset = mas->offset;
	4772	if (unlikely(mas_next_node(mas, node, limit))) {
	4773	mas_rewalk(mas, last);
	4774	goto retry;
	4775	}
	4776	mas->offset = 0;
	4777	node = mas_mn(mas);
	4778	mt = mte_node_type(mas->node);
	4779	}
	4780
	4781	mas->index = mas->last = limit;
	4782	mas->offset = offset;
	4783	mas->node = prev_node;
	4784	return NULL;
	4785	}
	4786
	4787	/*
	4788	* mas_prev_nentry() - Get the previous node entry.
	4789	* @mas: The maple state.
	4790	* @limit: The lower limit to check for a value.
	4791	*
	4792	* Return: the entry, %NULL otherwise.
	4793	*/
	4794	static inline void mas_prev_nentry(struct ma_state mas, unsigned long limit,
	4795	unsigned long index)
	4796	{
	4797	unsigned long pivot, min;
	4798	unsigned char offset;
	4799	struct maple_node *mn;
	4800	enum maple_type mt;
	4801	unsigned long *pivots;
	4802	void __rcu **slots;
	4803	void *entry;
	4804
	4805	retry:
	4806	if (!mas->offset)
	4807	return NULL;
	4808
	4809	mn = mas_mn(mas);
	4810	mt = mte_node_type(mas->node);
	4811	offset = mas->offset - 1;
	4812	if (offset >= mt_slots[mt])
	4813	offset = mt_slots[mt] - 1;
	4814
	4815	slots = ma_slots(mn, mt);
	4816	pivots = ma_pivots(mn, mt);
	4817	if (offset == mt_pivots[mt])
	4818	pivot = mas->max;
	4819	else
	4820	pivot = pivots[offset];
	4821
	4822	if (unlikely(ma_dead_node(mn))) {
	4823	mas_rewalk(mas, index);
	4824	goto retry;
	4825	}
	4826
	4827	while (offset && ((!mas_slot(mas, slots, offset) && pivot >= limit) \|\|
	4828	!pivot))
	4829	pivot = pivots[--offset];
	4830
	4831	min = mas_safe_min(mas, pivots, offset);
	4832	entry = mas_slot(mas, slots, offset);
	4833	if (unlikely(ma_dead_node(mn))) {
	4834	mas_rewalk(mas, index);
	4835	goto retry;
	4836	}
	4837
	4838	if (likely(entry)) {
	4839	mas->offset = offset;
	4840	mas->last = pivot;
	4841	mas->index = min;
	4842	}
	4843	return entry;
	4844	}
	4845
	4846	static inline void mas_prev_entry(struct ma_state mas, unsigned long min)
	4847	{
	4848	void *entry;
	4849
	4850	retry:
	4851	while (likely(!mas_is_none(mas))) {
	4852	entry = mas_prev_nentry(mas, min, mas->index);
	4853	if (unlikely(mas->last < min))
	4854	goto not_found;
	4855
	4856	if (likely(entry))
	4857	return entry;
	4858
	4859	if (unlikely(mas_prev_node(mas, min))) {
	4860	mas_rewalk(mas, mas->index);
	4861	goto retry;
	4862	}
	4863
	4864	mas->offset++;
	4865	}
	4866
	4867	mas->offset--;
	4868	not_found:
	4869	mas->index = mas->last = min;
	4870	return NULL;
	4871	}
	4872
	4873	/*
	4874	* mas_rev_awalk() - Internal function. Reverse allocation walk. Find the
	4875	* highest gap address of a given size in a given node and descend.
	4876	* @mas: The maple state
	4877	* @size: The needed size.
	4878	*
	4879	* Return: True if found in a leaf, false otherwise.
	4880	*
	4881	*/
	4882	static bool mas_rev_awalk(struct ma_state *mas, unsigned long size)
	4883	{
	4884	enum maple_type type = mte_node_type(mas->node);
	4885	struct maple_node *node = mas_mn(mas);
	4886	unsigned long pivots, gaps;
	4887	void __rcu **slots;
	4888	unsigned long gap = 0;
	4889	unsigned long max, min, index;
	4890	unsigned char offset;
	4891
	4892	if (unlikely(mas_is_err(mas)))
	4893	return true;
	4894
	4895	if (ma_is_dense(type)) {
	4896	/* dense nodes. */
	4897	mas->offset = (unsigned char)(mas->index - mas->min);
	4898	return true;
	4899	}
	4900
	4901	pivots = ma_pivots(node, type);
	4902	slots = ma_slots(node, type);
	4903	gaps = ma_gaps(node, type);
	4904	offset = mas->offset;
	4905	min = mas_safe_min(mas, pivots, offset);
	4906	/* Skip out of bounds. */
	4907	while (mas->last < min)
	4908	min = mas_safe_min(mas, pivots, --offset);
	4909
	4910	max = mas_safe_pivot(mas, pivots, offset, type);
	4911	index = mas->index;
	4912	while (index <= max) {
	4913	gap = 0;
	4914	if (gaps)
	4915	gap = gaps[offset];
	4916	else if (!mas_slot(mas, slots, offset))
	4917	gap = max - min + 1;
	4918
	4919	if (gap) {
	4920	if ((size <= gap) && (size <= mas->last - min + 1))
	4921	break;
	4922
	4923	if (!gaps) {
	4924	/* Skip the next slot, it cannot be a gap. */
	4925	if (offset < 2)
	4926	goto ascend;
	4927
	4928	offset -= 2;
	4929	max = pivots[offset];
	4930	min = mas_safe_min(mas, pivots, offset);
	4931	continue;
	4932	}
	4933	}
	4934
	4935	if (!offset)
	4936	goto ascend;
	4937
	4938	offset--;
	4939	max = min - 1;
	4940	min = mas_safe_min(mas, pivots, offset);
	4941	}
	4942
	4943	if (unlikely(index > max)) {
	4944	mas_set_err(mas, -EBUSY);
	4945	return false;
	4946	}
	4947
	4948	if (unlikely(ma_is_leaf(type))) {
	4949	mas->offset = offset;
	4950	mas->min = min;
	4951	mas->max = min + gap - 1;
	4952	return true;
	4953	}
	4954
	4955	/* descend, only happens under lock. */
	4956	mas->node = mas_slot(mas, slots, offset);
	4957	mas->min = min;
	4958	mas->max = max;
	4959	mas->offset = mas_data_end(mas);
	4960	return false;
	4961
	4962	ascend:
	4963	if (mte_is_root(mas->node))
	4964	mas_set_err(mas, -EBUSY);
	4965
	4966	return false;
	4967	}
	4968
	4969	static inline bool mas_anode_descend(struct ma_state *mas, unsigned long size)
	4970	{
	4971	enum maple_type type = mte_node_type(mas->node);
	4972	unsigned long pivot, min, gap = 0;
	4973	unsigned char count, offset;
	4974	unsigned long gaps = NULL, pivots = ma_pivots(mas_mn(mas), type);
	4975	void __rcu **slots = ma_slots(mas_mn(mas), type);
	4976	bool found = false;
	4977
	4978	if (ma_is_dense(type)) {
	4979	mas->offset = (unsigned char)(mas->index - mas->min);
	4980	return true;
	4981	}
	4982
	4983	gaps = ma_gaps(mte_to_node(mas->node), type);
	4984	offset = mas->offset;
	4985	count = mt_slots[type];
	4986	min = mas_safe_min(mas, pivots, offset);
	4987	for (; offset < count; offset++) {
	4988	pivot = mas_safe_pivot(mas, pivots, offset, type);
	4989	if (offset && !pivot)
	4990	break;
	4991
	4992	/* Not within lower bounds */
	4993	if (mas->index > pivot)
	4994	goto next_slot;
	4995
	4996	if (gaps)
	4997	gap = gaps[offset];
	4998	else if (!mas_slot(mas, slots, offset))
	4999	gap = min(pivot, mas->last) - max(mas->index, min) + 1;

2131 more lines aren’t shown here.

lib/show_mem.c+2−2

@@ -8,13 +8,13 @@ · 7 unmodified lines
8	#include <linux/mm.h>	8	#include <linux/mm.h>
9	#include <linux/cma.h>	9	#include <linux/cma.h>
10		10
11	void show_mem(unsigned int filter, nodemask_t *nodemask)	11	void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
12	{	12	{
13	pg_data_t *pgdat;	13	pg_data_t *pgdat;
14	unsigned long total = 0, reserved = 0, highmem = 0;	14	unsigned long total = 0, reserved = 0, highmem = 0;
15		15
16	printk("Mem-Info:\n");	16	printk("Mem-Info:\n");
17	show_free_areas(filter, nodemask);	17	__show_free_areas(filter, nodemask, max_zone_idx);
18		18
19	for_each_online_pgdat(pgdat) {	19	for_each_online_pgdat(pgdat) {
20	int zoneid;	20	int zoneid;

lib/stackdepot.c+24−5

 #define STACK_ALLOC_OFFSET_BITS (STACK_ALLOC_ORDER + PAGE_SHIFT - \
 					STACK_ALLOC_ALIGN)
 #define STACK_ALLOC_INDEX_BITS (DEPOT_STACK_BITS - \
-		STACK_ALLOC_NULL_PROTECTION_BITS - STACK_ALLOC_OFFSET_BITS)
+		STACK_ALLOC_NULL_PROTECTION_BITS - \
+		STACK_ALLOC_OFFSET_BITS - STACK_DEPOT_EXTRA_BITS)
 #define STACK_ALLOC_SLABS_CAP 8192
 #define STACK_ALLOC_MAX_SLABS \
 	(((1LL << (STACK_ALLOC_INDEX_BITS)) < STACK_ALLOC_SLABS_CAP) ? \
 		u32 slabindex : STACK_ALLOC_INDEX_BITS;
 		u32 offset : STACK_ALLOC_OFFSET_BITS;
 		u32 valid : STACK_ALLOC_NULL_PROTECTION_BITS;
+		u32 extra : STACK_DEPOT_EXTRA_BITS;
 	};
 };
 static size_t depot_offset;
 static DEFINE_RAW_SPINLOCK(depot_lock);
+unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle)
+{
+	union handle_parts parts = { .handle = handle };
+	return parts.extra;
+}
+EXPORT_SYMBOL(stack_depot_get_extra_bits);
 static bool init_stack_slab(void **prealloc)
 {
 	if (!*prealloc)
 	stack->handle.slabindex = depot_index;
 	stack->handle.offset = depot_offset >> STACK_ALLOC_ALIGN;
 	stack->handle.valid = 1;
+	stack->handle.extra = 0;
 	memcpy(stack->entries, entries, flex_array_size(stack, entries, size));
 	depot_offset += required_size;
  *
  * @entries:		Pointer to storage array
  * @nr_entries:		Size of the storage array
+ * @extra_bits:		Flags to store in unused bits of depot_stack_handle_t
  * @alloc_flags:	Allocation gfp flags
  * @can_alloc:		Allocate stack slabs (increased chance of failure if false)
  *
  * If the stack trace in @entries is from an interrupt, only the portion up to
  * interrupt entry is saved.
  *
+ * Additional opaque flags can be passed in @extra_bits, stored in the unused
+ * bits of the stack handle, and retrieved using stack_depot_get_extra_bits()
+ * without calling stack_depot_fetch().
+ *
  * Context: Any context, but setting @can_alloc to %false is required if
  *          alloc_pages() cannot be used from the current context. Currently
  *          this is the case from contexts where neither %GFP_ATOMIC nor
  */
 depot_stack_handle_t __stack_depot_save(unsigned long *entries,
 					unsigned int nr_entries,
+					unsigned int extra_bits,
 					gfp_t alloc_flags, bool can_alloc)
 {
 	struct stack_record *found = NULL, **bucket;
-	depot_stack_handle_t retval = 0;
+	union handle_parts retval = { .handle = 0 };
 	struct page *page = NULL;
 	void *prealloc = NULL;
 	unsigned long flags;
 		free_pages((unsigned long)prealloc, STACK_ALLOC_ORDER);
 	}
 	if (found)
-		retval = found->handle.handle;
+		retval.handle = found->handle.handle;
 fast_exit:
-	return retval;
+	retval.extra = extra_bits;
+	return retval.handle;
 }
 EXPORT_SYMBOL_GPL(__stack_depot_save);
 				      unsigned int nr_entries,
 				      gfp_t alloc_flags)
 {
-	return __stack_depot_save(entries, nr_entries, alloc_flags, true);
+	return __stack_depot_save(entries, nr_entries, 0, alloc_flags, true);
 }
 EXPORT_SYMBOL_GPL(stack_depot_save);

lib/string.c+8−0

@@ -197,6 +197,14 @@ ssize_t strscpy(char dest, const char src, size_t count) · 196 unmodified lines
197	max = 0;	197	max = 0;
198	#endif	198	#endif
199		199
		200	/*
		201	* read_word_at_a_time() below may read uninitialized bytes after the
		202	* trailing zero and use them in comparisons. Disable this optimization
		203	* under KMSAN to prevent false positive reports.
		204	*/
		205	if (IS_ENABLED(CONFIG_KMSAN))
		206	max = 0;
		207
200	while (max >= sizeof(unsigned long)) {	208	while (max >= sizeof(unsigned long)) {
201	unsigned long c, data;	209	unsigned long c, data;
202		210

lib/test_hmm.c+10−3

  */
 struct dmirror_device {
 	struct cdev		cdevice;
-	struct hmm_devmem	*devmem;
 	unsigned int            zone_device_type;
+	struct device		device;
 	unsigned int		devmem_capacity;
 	unsigned int		devmem_count;
 	cdev_init(&mdevice->cdevice, &dmirror_fops);
 	mdevice->cdevice.owner = THIS_MODULE;
-	ret = cdev_add(&mdevice->cdevice, dev, 1);
+	device_initialize(&mdevice->device);
+	mdevice->device.devt = dev;
+	ret = dev_set_name(&mdevice->device, "hmm_dmirror%u", id);
+	if (ret)
+		return ret;
+	ret = cdev_device_add(&mdevice->cdevice, &mdevice->device);
 	if (ret)
 		return ret;
 		kfree(mdevice->devmem_chunks);
 	}
-	cdev_del(&mdevice->cdevice);
+	cdev_device_del(&mdevice->cdevice, &mdevice->device);
 }
 static int __init hmm_dmirror_init(void)

lib/test_maple_tree.c+38307−0

@@ -0,0 +1,38307 @@
	1	// SPDX-License-Identifier: GPL-2.0+
	2	/*
	3	* test_maple_tree.c: Test the maple tree API
	4	* Copyright (c) 2018 Liam R. Howlett
	5	* Author: Liam R. Howlett <Liam.Howlett@Oracle.com>
	6	*/
	7
	8	#include <linux/maple_tree.h>
	9	#include <linux/module.h>
	10	#include <stdlib.h>
	11	#include <time.h>
	12
	13	#define MTREE_ALLOC_MAX 0x2000000000000Ul
	14	#define CONFIG_DEBUG_MAPLE_TREE
	15	#define CONFIG_MAPLE_SEARCH
	16	/* #define BENCH_SLOT_STORE */
	17	/* #define BENCH_NODE_STORE */
	18	/* #define BENCH_AWALK */
	19	/* #define BENCH_WALK */
	20	/* #define BENCH_MT_FOR_EACH */
	21	/* #define BENCH_FORK */
	22	static
	23	int mtree_insert_index(struct maple_tree *mt, unsigned long index, gfp_t gfp)
	24	{
	25	return mtree_insert(mt, index, xa_mk_value(index & LONG_MAX), gfp);
	26	}
	27
	28	static void mtree_erase_index(struct maple_tree *mt, unsigned long index)
	29	{
	30	MT_BUG_ON(mt, mtree_erase(mt, index) != xa_mk_value(index & LONG_MAX));
	31	MT_BUG_ON(mt, mtree_load(mt, index) != NULL);
	32	}
	33
	34	static int mtree_test_insert(struct maple_tree *mt, unsigned long index,
	35	void *ptr)
	36	{
	37	return mtree_insert(mt, index, ptr, GFP_KERNEL);
	38	}
	39
	40	static int mtree_test_store_range(struct maple_tree *mt, unsigned long start,
	41	unsigned long end, void *ptr)
	42	{
	43	return mtree_store_range(mt, start, end, ptr, GFP_KERNEL);
	44	}
	45
	46	static int mtree_test_store(struct maple_tree *mt, unsigned long start,
	47	void *ptr)
	48	{
	49	return mtree_test_store_range(mt, start, start, ptr);
	50	}
	51
	52	static int mtree_test_insert_range(struct maple_tree *mt, unsigned long start,
	53	unsigned long end, void *ptr)
	54	{
	55	return mtree_insert_range(mt, start, end, ptr, GFP_KERNEL);
	56	}
	57
	58	static void mtree_test_load(struct maple_tree mt, unsigned long index)
	59	{
	60	return mtree_load(mt, index);
	61	}
	62
	63	static void mtree_test_erase(struct maple_tree mt, unsigned long index)
	64	{
	65	return mtree_erase(mt, index);
	66	}
	67
	68	static noinline void check_mtree_alloc_range(struct maple_tree *mt,
	69	unsigned long start, unsigned long end, unsigned long size,
	70	unsigned long expected, int eret, void *ptr)
	71	{
	72
	73	unsigned long result = expected + 1;
	74	int ret;
	75
	76	ret = mtree_alloc_range(mt, &result, ptr, size, start, end,
	77	GFP_KERNEL);
	78	MT_BUG_ON(mt, ret != eret);
	79	if (ret)
	80	return;
	81
	82	MT_BUG_ON(mt, result != expected);
	83	}
	84
	85	static noinline void check_mtree_alloc_rrange(struct maple_tree *mt,
	86	unsigned long start, unsigned long end, unsigned long size,
	87	unsigned long expected, int eret, void *ptr)
	88	{
	89
	90	unsigned long result = expected + 1;
	91	int ret;
	92
	93	ret = mtree_alloc_rrange(mt, &result, ptr, size, start, end - 1,
	94	GFP_KERNEL);
	95	MT_BUG_ON(mt, ret != eret);
	96	if (ret)
	97	return;
	98
	99	MT_BUG_ON(mt, result != expected);
	100	}
	101
	102	static noinline void check_load(struct maple_tree *mt, unsigned long index,
	103	void *ptr)
	104	{
	105	void *ret = mtree_test_load(mt, index);
	106
	107	if (ret != ptr)
	108	pr_err("Load %lu returned %p expect %p\n", index, ret, ptr);
	109	MT_BUG_ON(mt, ret != ptr);
	110	}
	111
	112	static noinline void check_store_range(struct maple_tree *mt,
	113	unsigned long start, unsigned long end, void *ptr, int expected)
	114	{
	115	int ret = -EINVAL;
	116	unsigned long i;
	117
	118	ret = mtree_test_store_range(mt, start, end, ptr);
	119	MT_BUG_ON(mt, ret != expected);
	120
	121	if (ret)
	122	return;
	123
	124	for (i = start; i <= end; i++)
	125	check_load(mt, i, ptr);
	126	}
	127
	128	static noinline void check_insert_range(struct maple_tree *mt,
	129	unsigned long start, unsigned long end, void *ptr, int expected)
	130	{
	131	int ret = -EINVAL;
	132	unsigned long i;
	133
	134	ret = mtree_test_insert_range(mt, start, end, ptr);
	135	MT_BUG_ON(mt, ret != expected);
	136
	137	if (ret)
	138	return;
	139
	140	for (i = start; i <= end; i++)
	141	check_load(mt, i, ptr);
	142	}
	143
	144	static noinline void check_insert(struct maple_tree *mt, unsigned long index,
	145	void *ptr)
	146	{
	147	int ret = -EINVAL;
	148
	149	ret = mtree_test_insert(mt, index, ptr);
	150	MT_BUG_ON(mt, ret != 0);
	151	}
	152
	153	static noinline void check_erase(struct maple_tree *mt, unsigned long index,
	154	void *ptr)
	155	{
	156	MT_BUG_ON(mt, mtree_test_erase(mt, index) != ptr);
	157	}
	158
	159	static noinline void check_dup_insert(struct maple_tree *mt,
	160	unsigned long index, void *ptr)
	161	{
	162	int ret = -EINVAL;
	163
	164	ret = mtree_test_insert(mt, index, ptr);
	165	MT_BUG_ON(mt, ret != -EEXIST);
	166	}
	167
	168
	169	static noinline
	170	void check_index_load(struct maple_tree *mt, unsigned long index)
	171	{
	172	return check_load(mt, index, xa_mk_value(index & LONG_MAX));
	173	}
	174
	175	static noinline void check_nomem(struct maple_tree *mt)
	176	{
	177	MA_STATE(ms, mt, 1, 1);
	178
	179	MT_BUG_ON(mt, !mtree_empty(mt));
	180	/* Ensure no bypassing of allocation failures */
	181	mt_set_non_kernel(0);
	182
	183	/* Storing something at 1 requires memory allocation */
	184	MT_BUG_ON(mt, mtree_insert(mt, 1, &ms, GFP_ATOMIC) != -ENOMEM);
	185	/* Storing something at 0 does not */
	186	MT_BUG_ON(mt, mtree_insert(mt, 0, &ms, GFP_ATOMIC) != 0);
	187
	188	/*
	189	* Simulate two threads racing; the first one fails to allocate
	190	* memory to insert an entry at 1, then the second one succeeds
	191	* in allocating memory to insert an entry at 2. The first one
	192	* then needs to free the node it allocated. LeakSanitizer will
	193	* notice this, as will the 'nr_allocated' debugging aid in the
	194	* userspace test suite.
	195	*/
	196	mtree_lock(mt);
	197	mas_store(&ms, &ms); /* insert 1 -> &ms, fails. */
	198	MT_BUG_ON(mt, ms.node != MA_ERROR(-ENOMEM));
	199	mas_nomem(&ms, GFP_KERNEL); /* Node allocated in here. */
	200	MT_BUG_ON(mt, ms.node != MAS_START);
	201	mtree_unlock(mt);
	202	MT_BUG_ON(mt, mtree_insert(mt, 2, mt, GFP_KERNEL) != 0);
	203	mtree_lock(mt);
	204	mas_store(&ms, &ms); /* insert 1 -> &ms */
	205	mas_nomem(&ms, GFP_KERNEL); /* Node allocated in here. */
	206	mtree_unlock(mt);
	207	mtree_destroy(mt);
	208	}
	209
	210	static inline int not_empty(struct maple_node *node)
	211	{
	212	int i;
	213
	214	if (node->parent)
	215	return 1;
	216
	217	for (i = 0; i < ARRAY_SIZE(node->slot); i++)
	218	if (node->slot[i])
	219	return 1;
	220
	221	return 0;
	222	}
	223
	224	static noinline void check_new_node(struct maple_tree *mt)
	225	{
	226
	227	struct maple_node mn, mn2, *mn3;
	228	struct maple_alloc *smn;
	229	struct maple_node *nodes[100];
	230	int i, j, total;
	231
	232	MA_STATE(mas, mt, 0, 0);
	233
	234	/* Try allocating 3 nodes */
	235	mtree_lock(mt);
	236	/* request 3 nodes to be allocated. */
	237	mas_node_count(&mas, 3);
	238	/* Allocation request of 3. */
	239	MT_BUG_ON(mt, mas_alloc_req(&mas) != 3);
	240	/* Allocate failed. */
	241	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
	242	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	243
	244	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
	245	mn = mas_pop_node(&mas);
	246	MT_BUG_ON(mt, not_empty(mn));
	247	MT_BUG_ON(mt, mn == NULL);
	248	MT_BUG_ON(mt, mas.alloc == NULL);
	249	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
	250	mas_push_node(&mas, mn);
	251	mas_nomem(&mas, GFP_KERNEL); /* free */
	252	mtree_unlock(mt);
	253
	254
	255	/* Try allocating 1 node, then 2 more */
	256	mtree_lock(mt);
	257	/* Set allocation request to 1. */
	258	mas_set_alloc_req(&mas, 1);
	259	/* Check Allocation request of 1. */
	260	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
	261	mas_set_err(&mas, -ENOMEM);
	262	/* Validate allocation request. */
	263	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	264	/* Eat the requested node. */
	265	mn = mas_pop_node(&mas);
	266	MT_BUG_ON(mt, not_empty(mn));
	267	MT_BUG_ON(mt, mn == NULL);
	268	MT_BUG_ON(mt, mn->slot[0] != NULL);
	269	MT_BUG_ON(mt, mn->slot[1] != NULL);
	270	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	271
	272	ma_free_rcu(mn);
	273	mas.node = MAS_START;
	274	mas_nomem(&mas, GFP_KERNEL);
	275	/* Allocate 3 nodes, will fail. */
	276	mas_node_count(&mas, 3);
	277	/* Drop the lock and allocate 3 nodes. */
	278	mas_nomem(&mas, GFP_KERNEL);
	279	/* Ensure 3 are allocated. */
	280	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
	281	/* Allocation request of 0. */
	282	MT_BUG_ON(mt, mas_alloc_req(&mas) != 0);
	283
	284	MT_BUG_ON(mt, mas.alloc == NULL);
	285	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
	286	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
	287	/* Ensure we counted 3. */
	288	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
	289	/* Free. */
	290	mas_nomem(&mas, GFP_KERNEL);
	291
	292	/* Set allocation request to 1. */
	293	mas_set_alloc_req(&mas, 1);
	294	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
	295	mas_set_err(&mas, -ENOMEM);
	296	/* Validate allocation request. */
	297	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	298	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
	299	/* Check the node is only one node. */
	300	mn = mas_pop_node(&mas);
	301	MT_BUG_ON(mt, not_empty(mn));
	302	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	303	MT_BUG_ON(mt, mn == NULL);
	304	MT_BUG_ON(mt, mn->slot[0] != NULL);
	305	MT_BUG_ON(mt, mn->slot[1] != NULL);
	306	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	307	mas_push_node(&mas, mn);
	308	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
	309	MT_BUG_ON(mt, mas.alloc->node_count);
	310
	311	mas_set_alloc_req(&mas, 2); /* request 2 more. */
	312	MT_BUG_ON(mt, mas_alloc_req(&mas) != 2);
	313	mas_set_err(&mas, -ENOMEM);
	314	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	315	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
	316	MT_BUG_ON(mt, mas.alloc == NULL);
	317	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
	318	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
	319	for (i = 2; i >= 0; i--) {
	320	mn = mas_pop_node(&mas);
	321	MT_BUG_ON(mt, mas_allocated(&mas) != i);
	322	MT_BUG_ON(mt, !mn);
	323	MT_BUG_ON(mt, not_empty(mn));
	324	ma_free_rcu(mn);
	325	}
	326
	327	total = 64;
	328	mas_set_alloc_req(&mas, total); /* request 2 more. */
	329	MT_BUG_ON(mt, mas_alloc_req(&mas) != total);
	330	mas_set_err(&mas, -ENOMEM);
	331	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	332	for (i = total; i > 0; i--) {
	333	unsigned int e = 0; /* expected node_count */
	334
	335	if (i >= 35)
	336	e = i - 35;
	337	else if (i >= 5)
	338	e = i - 5;
	339	else if (i >= 2)
	340	e = i - 2;
	341	MT_BUG_ON(mt, mas.alloc->node_count != e);
	342	mn = mas_pop_node(&mas);
	343	MT_BUG_ON(mt, not_empty(mn));
	344	MT_BUG_ON(mt, mas_allocated(&mas) != i - 1);
	345	MT_BUG_ON(mt, !mn);
	346	ma_free_rcu(mn);
	347	}
	348
	349	total = 100;
	350	for (i = 1; i < total; i++) {
	351	mas_set_alloc_req(&mas, i);
	352	mas_set_err(&mas, -ENOMEM);
	353	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	354	for (j = i; j > 0; j--) {
	355	mn = mas_pop_node(&mas);
	356	MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
	357	MT_BUG_ON(mt, !mn);
	358	MT_BUG_ON(mt, not_empty(mn));
	359	mas_push_node(&mas, mn);
	360	MT_BUG_ON(mt, mas_allocated(&mas) != j);
	361	mn = mas_pop_node(&mas);
	362	MT_BUG_ON(mt, not_empty(mn));
	363	MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
	364	ma_free_rcu(mn);
	365	}
	366	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	367
	368	mas_set_alloc_req(&mas, i);
	369	mas_set_err(&mas, -ENOMEM);
	370	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	371	for (j = 0; j <= i/2; j++) {
	372	MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
	373	nodes[j] = mas_pop_node(&mas);
	374	MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
	375	}
	376
	377	while (j) {
	378	j--;
	379	mas_push_node(&mas, nodes[j]);
	380	MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
	381	}
	382	MT_BUG_ON(mt, mas_allocated(&mas) != i);
	383	for (j = 0; j <= i/2; j++) {
	384	MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
	385	mn = mas_pop_node(&mas);
	386	MT_BUG_ON(mt, not_empty(mn));
	387	ma_free_rcu(mn);
	388	MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
	389	}
	390	MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL));
	391
	392	}
	393
	394	/* Set allocation request. */
	395	total = 500;
	396	mas_node_count(&mas, total);
	397	/* Drop the lock and allocate the nodes. */
	398	mas_nomem(&mas, GFP_KERNEL);
	399	MT_BUG_ON(mt, !mas.alloc);
	400	i = 1;
	401	smn = mas.alloc;
	402	while (i < total) {
	403	for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) {
	404	i++;
	405	MT_BUG_ON(mt, !smn->slot[j]);
	406	if (i == total)
	407	break;
	408	}
	409	smn = smn->slot[0]; /* next. */
	410	}
	411	MT_BUG_ON(mt, mas_allocated(&mas) != total);
	412	mas_nomem(&mas, GFP_KERNEL); /* Free. */
	413
	414	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	415	for (i = 1; i < 128; i++) {
	416	mas_node_count(&mas, i); /* Request */
	417	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	418	MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
	419	for (j = i; j > 0; j--) { /Free the requests /
	420	mn = mas_pop_node(&mas); /* get the next node. */
	421	MT_BUG_ON(mt, mn == NULL);
	422	MT_BUG_ON(mt, not_empty(mn));
	423	ma_free_rcu(mn);
	424	}
	425	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	426	}
	427
	428	for (i = 1; i < MAPLE_NODE_MASK + 1; i++) {
	429	MA_STATE(mas2, mt, 0, 0);
	430	mas_node_count(&mas, i); /* Request */
	431	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	432	MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
	433	for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */
	434	mn = mas_pop_node(&mas); /* get the next node. */
	435	MT_BUG_ON(mt, mn == NULL);
	436	MT_BUG_ON(mt, not_empty(mn));
	437	mas_push_node(&mas2, mn);
	438	MT_BUG_ON(mt, mas_allocated(&mas2) != j);
	439	}
	440	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	441	MT_BUG_ON(mt, mas_allocated(&mas2) != i);
	442
	443	for (j = i; j > 0; j--) { /Free the requests /
	444	MT_BUG_ON(mt, mas_allocated(&mas2) != j);
	445	mn = mas_pop_node(&mas2); /* get the next node. */
	446	MT_BUG_ON(mt, mn == NULL);
	447	MT_BUG_ON(mt, not_empty(mn));
	448	ma_free_rcu(mn);
	449	}
	450	MT_BUG_ON(mt, mas_allocated(&mas2) != 0);
	451	}
	452
	453
	454	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	455	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
	456	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
	457	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	458	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
	459	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
	460
	461	mn = mas_pop_node(&mas); /* get the next node. */
	462	MT_BUG_ON(mt, mn == NULL);
	463	MT_BUG_ON(mt, not_empty(mn));
	464	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS);
	465	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 2);
	466
	467	mas_push_node(&mas, mn);
	468	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
	469	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
	470
	471	/* Check the limit of pop/push/pop */
	472	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */
	473	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
	474	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
	475	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
	476	MT_BUG_ON(mt, mas_alloc_req(&mas));
	477	MT_BUG_ON(mt, mas.alloc->node_count);
	478	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
	479	mn = mas_pop_node(&mas);
	480	MT_BUG_ON(mt, not_empty(mn));
	481	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
	482	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
	483	mas_push_node(&mas, mn);
	484	MT_BUG_ON(mt, mas.alloc->node_count);
	485	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
	486	mn = mas_pop_node(&mas);
	487	MT_BUG_ON(mt, not_empty(mn));
	488	ma_free_rcu(mn);
	489	for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) {
	490	mn = mas_pop_node(&mas);
	491	MT_BUG_ON(mt, not_empty(mn));
	492	ma_free_rcu(mn);
	493	}
	494	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
	495
	496
	497	for (i = 3; i < MAPLE_NODE_MASK * 3; i++) {
	498	mas.node = MA_ERROR(-ENOMEM);
	499	mas_node_count(&mas, i); /* Request */
	500	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	501	mn = mas_pop_node(&mas); /* get the next node. */
	502	mas_push_node(&mas, mn); /* put it back */
	503	mas_destroy(&mas);
	504
	505	mas.node = MA_ERROR(-ENOMEM);
	506	mas_node_count(&mas, i); /* Request */
	507	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	508	mn = mas_pop_node(&mas); /* get the next node. */
	509	mn2 = mas_pop_node(&mas); /* get the next node. */
	510	mas_push_node(&mas, mn); /* put them back */
	511	mas_push_node(&mas, mn2);
	512	mas_destroy(&mas);
	513
	514	mas.node = MA_ERROR(-ENOMEM);
	515	mas_node_count(&mas, i); /* Request */
	516	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	517	mn = mas_pop_node(&mas); /* get the next node. */
	518	mn2 = mas_pop_node(&mas); /* get the next node. */
	519	mn3 = mas_pop_node(&mas); /* get the next node. */
	520	mas_push_node(&mas, mn); /* put them back */
	521	mas_push_node(&mas, mn2);
	522	mas_push_node(&mas, mn3);
	523	mas_destroy(&mas);
	524
	525	mas.node = MA_ERROR(-ENOMEM);
	526	mas_node_count(&mas, i); /* Request */
	527	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	528	mn = mas_pop_node(&mas); /* get the next node. */
	529	ma_free_rcu(mn);
	530	mas_destroy(&mas);
	531
	532	mas.node = MA_ERROR(-ENOMEM);
	533	mas_node_count(&mas, i); /* Request */
	534	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	535	mn = mas_pop_node(&mas); /* get the next node. */
	536	ma_free_rcu(mn);
	537	mn = mas_pop_node(&mas); /* get the next node. */
	538	ma_free_rcu(mn);
	539	mn = mas_pop_node(&mas); /* get the next node. */
	540	ma_free_rcu(mn);
	541	mas_destroy(&mas);
	542	}
	543
	544	mas.node = MA_ERROR(-ENOMEM);
	545	mas_node_count(&mas, 5); /* Request */
	546	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	547	MT_BUG_ON(mt, mas_allocated(&mas) != 5);
	548	mas.node = MA_ERROR(-ENOMEM);
	549	mas_node_count(&mas, 10); /* Request */
	550	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	551	mas.node = MAS_START;
	552	MT_BUG_ON(mt, mas_allocated(&mas) != 10);
	553	mas_destroy(&mas);
	554
	555	mas.node = MA_ERROR(-ENOMEM);
	556	mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */
	557	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	558	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1);
	559	mas.node = MA_ERROR(-ENOMEM);
	560	mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */
	561	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
	562	mas.node = MAS_START;
	563	MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1);
	564	mas_destroy(&mas);
	565
	566	mtree_unlock(mt);
	567	}
	568
	569	static noinline void check_rev_seq(struct maple_tree *mt, unsigned long max,
	570	bool verbose)
	571	{
	572	unsigned long i = max, j;
	573
	574	MT_BUG_ON(mt, !mtree_empty(mt));
	575
	576	mt_zero_nr_tallocated();
	577	while (i) {
	578	MT_BUG_ON(mt, mtree_insert_index(mt, i, GFP_KERNEL));
	579	for (j = i; j <= max; j++)
	580	check_index_load(mt, j);
	581
	582	check_load(mt, i - 1, NULL);
	583	mt_set_in_rcu(mt);
	584	MT_BUG_ON(mt, !mt_height(mt));
	585	mt_clear_in_rcu(mt);
	586	MT_BUG_ON(mt, !mt_height(mt));
	587	i--;
	588	}
	589	check_load(mt, max + 1, NULL);
	590
	591	if (verbose) {
	592	rcu_barrier();
	593	mt_dump(mt);
	594	pr_info(" %s test of 0-%lu %luK in %d active (%d total)\n",
	595	__func__, max, mt_get_alloc_size()/1024, mt_nr_allocated(),
	596	mt_nr_tallocated());
	597	}
	598	}
	599
	600	static noinline void check_seq(struct maple_tree *mt, unsigned long max,
	601	bool verbose)
	602	{
	603	unsigned long i, j;
	604
	605	MT_BUG_ON(mt, !mtree_empty(mt));
	606
	607	mt_zero_nr_tallocated();
	608	for (i = 0; i <= max; i++) {
	609	MT_BUG_ON(mt, mtree_insert_index(mt, i, GFP_KERNEL));
	610	for (j = 0; j <= i; j++)
	611	check_index_load(mt, j);
	612
	613	if (i)
	614	MT_BUG_ON(mt, !mt_height(mt));
	615	check_load(mt, i + 1, NULL);
	616	}
	617	if (verbose) {
	618	rcu_barrier();
	619	mt_dump(mt);
	620	pr_info(" seq test of 0-%lu %luK in %d active (%d total)\n",
	621	max, mt_get_alloc_size()/1024, mt_nr_allocated(),
	622	mt_nr_tallocated());
	623	}
	624	}
	625
	626	static noinline void check_lb_not_empty(struct maple_tree *mt)
	627	{
	628	unsigned long i, j;
	629	unsigned long huge = 4000UL * 1000 * 1000;
	630
	631
	632	i = huge;
	633	while (i > 4096) {
	634	check_insert(mt, i, (void *) i);
	635	for (j = huge; j >= i; j /= 2) {
	636	check_load(mt, j-1, NULL);
	637	check_load(mt, j, (void *) j);
	638	check_load(mt, j+1, NULL);
	639	}
	640	i /= 2;
	641	}
	642	mtree_destroy(mt);
	643	}
	644
	645	static noinline void check_lower_bound_split(struct maple_tree *mt)
	646	{
	647	MT_BUG_ON(mt, !mtree_empty(mt));
	648	check_lb_not_empty(mt);
	649	}
	650
	651	static noinline void check_upper_bound_split(struct maple_tree *mt)
	652	{
	653	unsigned long i, j;
	654	unsigned long huge = 4000UL * 1000 * 1000;
	655
	656	MT_BUG_ON(mt, !mtree_empty(mt));
	657
	658	i = 4096;
	659	while (i < huge) {
	660	check_insert(mt, i, (void *) i);
	661	for (j = i; j >= huge; j *= 2) {
	662	check_load(mt, j-1, NULL);
	663	check_load(mt, j, (void *) j);
	664	check_load(mt, j+1, NULL);
	665	}
	666	i *= 2;
	667	}
	668	mtree_destroy(mt);
	669	}
	670
	671	static noinline void check_mid_split(struct maple_tree *mt)
	672	{
	673	unsigned long huge = 8000UL * 1000 * 1000;
	674
	675	check_insert(mt, huge, (void *) huge);
	676	check_insert(mt, 0, xa_mk_value(0));
	677	check_lb_not_empty(mt);
	678	}
	679
	680	static noinline void check_rev_find(struct maple_tree *mt)
	681	{
	682	int i, nr_entries = 200;
	683	void *val;
	684	MA_STATE(mas, mt, 0, 0);
	685
	686	for (i = 0; i <= nr_entries; i++)
	687	mtree_store_range(mt, i10, i10 + 5,
	688	xa_mk_value(i), GFP_KERNEL);
	689
	690	mas_set(&mas, 1000);
	691	val = mas_find_rev(&mas, 1000);
	692	MT_BUG_ON(mt, val != xa_mk_value(100));
	693	val = mas_find_rev(&mas, 1000);
	694	MT_BUG_ON(mt, val != NULL);
	695
	696	mas_set(&mas, 999);
	697	val = mas_find_rev(&mas, 997);
	698	MT_BUG_ON(mt, val != NULL);
	699
	700	mas_set(&mas, 1000);
	701	val = mas_find_rev(&mas, 900);
	702	MT_BUG_ON(mt, val != xa_mk_value(100));
	703	val = mas_find_rev(&mas, 900);
	704	MT_BUG_ON(mt, val != xa_mk_value(99));
	705
	706	mas_set(&mas, 20);
	707	val = mas_find_rev(&mas, 0);
	708	MT_BUG_ON(mt, val != xa_mk_value(2));
	709	val = mas_find_rev(&mas, 0);
	710	MT_BUG_ON(mt, val != xa_mk_value(1));
	711	val = mas_find_rev(&mas, 0);
	712	MT_BUG_ON(mt, val != xa_mk_value(0));
	713	val = mas_find_rev(&mas, 0);
	714	MT_BUG_ON(mt, val != NULL);
	715	}
	716
	717	static noinline void check_find(struct maple_tree *mt)
	718	{
	719	unsigned long val = 0;
	720	unsigned long count = 20;
	721	unsigned long max;
	722	unsigned long last = 0, index = 0;
	723	void entry, entry2;
	724
	725	MA_STATE(mas, mt, 0, 0);
	726
	727	/* Insert 0. */
	728	MT_BUG_ON(mt, mtree_insert_index(mt, val++, GFP_KERNEL));
	729
	730	for (int i = 0; i <= count; i++) {
	731	if (val != 64)
	732	MT_BUG_ON(mt, mtree_insert_index(mt, val, GFP_KERNEL));
	733	else
	734	MT_BUG_ON(mt, mtree_insert(mt, val,
	735	XA_ZERO_ENTRY, GFP_KERNEL));
	736
	737	val <<= 2;
	738	}
	739
	740	val = 0;
	741	mas_set(&mas, val);
	742	mas_lock(&mas);
	743	while ((entry = mas_find(&mas, 268435456)) != NULL) {
	744	if (val != 64)
	745	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	746	else
	747	MT_BUG_ON(mt, entry != XA_ZERO_ENTRY);
	748
	749	val <<= 2;
	750	/* For zero check. */
	751	if (!val)
	752	val = 1;
	753	}
	754	mas_unlock(&mas);
	755
	756	val = 0;
	757	mas_set(&mas, val);
	758	mas_lock(&mas);
	759	mas_for_each(&mas, entry, ULONG_MAX) {
	760	if (val != 64)
	761	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	762	else
	763	MT_BUG_ON(mt, entry != XA_ZERO_ENTRY);
	764	val <<= 2;
	765	/* For zero check. */
	766	if (!val)
	767	val = 1;
	768	}
	769	mas_unlock(&mas);
	770
	771	/* Test mas_pause */
	772	val = 0;
	773	mas_set(&mas, val);
	774	mas_lock(&mas);
	775	mas_for_each(&mas, entry, ULONG_MAX) {
	776	if (val != 64)
	777	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	778	else
	779	MT_BUG_ON(mt, entry != XA_ZERO_ENTRY);
	780	val <<= 2;
	781	/* For zero check. */
	782	if (!val)
	783	val = 1;
	784
	785	mas_pause(&mas);
	786	mas_unlock(&mas);
	787	mas_lock(&mas);
	788	}
	789	mas_unlock(&mas);
	790
	791	val = 0;
	792	max = 300; /* A value big enough to include XA_ZERO_ENTRY at 64. */
	793	mt_for_each(mt, entry, index, max) {
	794	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	795	val <<= 2;
	796	if (val == 64) /* Skip zero entry. */
	797	val <<= 2;
	798	/* For zero check. */
	799	if (!val)
	800	val = 1;
	801	}
	802
	803	val = 0;
	804	max = 0;
	805	index = 0;
	806	MT_BUG_ON(mt, mtree_insert_index(mt, ULONG_MAX, GFP_KERNEL));
	807	mt_for_each(mt, entry, index, ULONG_MAX) {
	808	if (val == 4398046511104)
	809	MT_BUG_ON(mt, entry !=
	810	xa_mk_value(ULONG_MAX & LONG_MAX));
	811	else
	812	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	813	val <<= 2;
	814	if (val == 64) /* Skip zero entry. */
	815	val <<= 2;
	816	/* For zero check. */
	817	if (!val)
	818	val = 1;
	819	max++;
	820	MT_BUG_ON(mt, max > 25);
	821	}
	822	mtree_erase_index(mt, ULONG_MAX);
	823
	824	mas_reset(&mas);
	825	index = 17;
	826	entry = mt_find(mt, &index, 512);
	827	MT_BUG_ON(mt, xa_mk_value(256) != entry);
	828
	829	mas_reset(&mas);
	830	index = 17;
	831	entry = mt_find(mt, &index, 20);
	832	MT_BUG_ON(mt, entry != NULL);
	833
	834
	835	/* Range check.. */
	836	/* Insert ULONG_MAX */
	837	MT_BUG_ON(mt, mtree_insert_index(mt, ULONG_MAX, GFP_KERNEL));
	838
	839	val = 0;
	840	mas_set(&mas, 0);
	841	mas_lock(&mas);
	842	mas_for_each(&mas, entry, ULONG_MAX) {
	843	if (val == 64)
	844	MT_BUG_ON(mt, entry != XA_ZERO_ENTRY);
	845	else if (val == 4398046511104)
	846	MT_BUG_ON(mt, entry != xa_mk_value(ULONG_MAX & LONG_MAX));
	847	else
	848	MT_BUG_ON(mt, xa_mk_value(val) != entry);
	849	val <<= 2;
	850
	851	/* For zero check. */
	852	if (!val)
	853	val = 1;
	854	mas_pause(&mas);
	855	mas_unlock(&mas);
	856	mas_lock(&mas);
	857	}
	858	mas_unlock(&mas);
	859
	860	mas_set(&mas, 1048576);
	861	mas_lock(&mas);
	862	entry = mas_find(&mas, 1048576);
	863	mas_unlock(&mas);
	864	MT_BUG_ON(mas.tree, entry == NULL);
	865
	866	/*
	867	* Find last value.
	868	* 1. get the expected value, leveraging the existence of an end entry
	869	* 2. delete end entry
	870	* 3. find the last value but searching for ULONG_MAX and then using
	871	* prev
	872	*/
	873	/* First, get the expected result. */
	874	mas_lock(&mas);
	875	mas_reset(&mas);
	876	mas.index = ULONG_MAX; /* start at max.. */
	877	entry = mas_find(&mas, ULONG_MAX);
	878	entry = mas_prev(&mas, 0);
	879	index = mas.index;
	880	last = mas.last;
	881
	882	/* Erase the last entry. */
	883	mas_reset(&mas);
	884	mas.index = ULONG_MAX;
	885	mas.last = ULONG_MAX;
	886	mas_erase(&mas);
	887
	888	/* Get the previous value from MAS_START */
	889	mas_reset(&mas);
	890	entry2 = mas_prev(&mas, 0);
	891
	892	/* Check results. */
	893	MT_BUG_ON(mt, entry != entry2);
	894	MT_BUG_ON(mt, index != mas.index);
	895	MT_BUG_ON(mt, last != mas.last);
	896
	897
	898	mas.node = MAS_NONE;
	899	mas.index = ULONG_MAX;
	900	mas.last = ULONG_MAX;
	901	entry2 = mas_prev(&mas, 0);
	902	MT_BUG_ON(mt, entry != entry2);
	903
	904	mas_set(&mas, 0);
	905	MT_BUG_ON(mt, mas_prev(&mas, 0) != NULL);
	906
	907	mas_unlock(&mas);
	908	mtree_destroy(mt);
	909	}
	910
	911	static noinline void check_find_2(struct maple_tree *mt)
	912	{
	913	unsigned long i, j;
	914	void *entry;
	915
	916	MA_STATE(mas, mt, 0, 0);
	917	rcu_read_lock();
	918	mas_for_each(&mas, entry, ULONG_MAX)
	919	MT_BUG_ON(mt, true);
	920	rcu_read_unlock();
	921
	922	for (i = 0; i < 256; i++) {
	923	mtree_insert_index(mt, i, GFP_KERNEL);
	924	j = 0;
	925	mas_set(&mas, 0);
	926	rcu_read_lock();
	927	mas_for_each(&mas, entry, ULONG_MAX) {
	928	MT_BUG_ON(mt, entry != xa_mk_value(j));
	929	j++;
	930	}
	931	rcu_read_unlock();
	932	MT_BUG_ON(mt, j != i + 1);
	933	}
	934
	935	for (i = 0; i < 256; i++) {
	936	mtree_erase_index(mt, i);
	937	j = i + 1;
	938	mas_set(&mas, 0);
	939	rcu_read_lock();
	940	mas_for_each(&mas, entry, ULONG_MAX) {
	941	if (xa_is_zero(entry))
	942	continue;
	943
	944	MT_BUG_ON(mt, entry != xa_mk_value(j));
	945	j++;
	946	}
	947	rcu_read_unlock();
	948	MT_BUG_ON(mt, j != 256);
	949	}
	950
	951	/MT_BUG_ON(mt, !mtree_empty(mt)); /
	952	}
	953
	954	#define erase_ptr(i) entry[i%2]
	955	#define erase_check_load(mt, i) check_load(mt, set[i], entry[i%2])
	956	#define erase_check_insert(mt, i) check_insert(mt, set[i], entry[i%2])
	957	#define erase_check_erase(mt, i) check_erase(mt, set[i], entry[i%2])
	958
	959	static noinline void check_erase_testset(struct maple_tree *mt)
	960	{
	961	unsigned long set[] = { 5015, 5014, 5017, 25, 1000,
	962	1001, 1002, 1003, 1005, 0,
	963	6003, 6002, 6008, 6012, 6015,
	964	7003, 7002, 7008, 7012, 7015,
	965	8003, 8002, 8008, 8012, 8015,
	966	9003, 9002, 9008, 9012, 9015,
	967	10003, 10002, 10008, 10012, 10015,
	968	11003, 11002, 11008, 11012, 11015,
	969	12003, 12002, 12008, 12012, 12015,
	970	13003, 13002, 13008, 13012, 13015,
	971	14003, 14002, 14008, 14012, 14015,
	972	15003, 15002, 15008, 15012, 15015,
	973	};
	974
	975
	976	void *ptr = &set;
	977	void *entry[2] = { ptr, mt };
	978	void *root_node;
	979
	980
	981	rcu_register_thread();
	982	mt_set_in_rcu(mt);
	983	for (int i = 0; i < 4; i++)
	984	erase_check_insert(mt, i);
	985	for (int i = 0; i < 4; i++)
	986	erase_check_load(mt, i);
	987
	988	mt_set_non_kernel(2);
	989	erase_check_erase(mt, 1);
	990	erase_check_load(mt, 0);
	991	check_load(mt, set[1], NULL);
	992	for (int i = 2; i < 4; i++)
	993	erase_check_load(mt, i);
	994
	995
	996	erase_check_erase(mt, 2);
	997	erase_check_load(mt, 0);
	998	check_load(mt, set[1], NULL);
	999	check_load(mt, set[2], NULL);
	1000
	1001	erase_check_insert(mt, 1);
	1002	erase_check_insert(mt, 2);
	1003
	1004	for (int i = 0; i < 4; i++)
	1005	erase_check_load(mt, i);
	1006
	1007	/* Check erase and load without an allocation. */
	1008	erase_check_load(mt, 3);
	1009	erase_check_erase(mt, 1);
	1010	erase_check_load(mt, 0);
	1011	check_load(mt, set[1], NULL);
	1012	for (int i = 2; i < 4; i++)
	1013	erase_check_load(mt, i);
	1014
	1015	/*
	1016	* Set the newly erased node. This will produce a different allocated
	1017	* node to avoid busy slots.
	1018	*/
	1019	root_node = mt->ma_root;
	1020	erase_check_insert(mt, 1);
	1021
	1022	erase_check_load(mt, 0);
	1023	check_load(mt, 5016, NULL);
	1024	erase_check_load(mt, 1);
	1025	check_load(mt, 5013, NULL);
	1026	erase_check_load(mt, 2);
	1027	check_load(mt, 5018, NULL);
	1028	erase_check_load(mt, 3);
	1029
	1030	erase_check_erase(mt, 2); /* erase 5017 to check append */
	1031	erase_check_load(mt, 0);
	1032	check_load(mt, 5016, NULL);
	1033	erase_check_load(mt, 1);
	1034	check_load(mt, 5013, NULL);
	1035	check_load(mt, set[2], NULL);
	1036	check_load(mt, 5018, NULL);
	1037
	1038	erase_check_load(mt, 3);
	1039
	1040	root_node = mt->ma_root;
	1041	erase_check_insert(mt, 2);
	1042
	1043	erase_check_load(mt, 0);
	1044	check_load(mt, 5016, NULL);
	1045	erase_check_load(mt, 1);
	1046	check_load(mt, 5013, NULL);
	1047	erase_check_load(mt, 2);
	1048	check_load(mt, 5018, NULL);
	1049	erase_check_load(mt, 3);
	1050
	1051	mt_set_non_kernel(1);
	1052	erase_check_erase(mt, 2); /* erase 5017 to check append */
	1053	erase_check_load(mt, 0);
	1054	check_load(mt, 5016, NULL);
	1055	check_load(mt, set[2], NULL);
	1056	erase_check_erase(mt, 0); /* erase 5015 to check append */
	1057	check_load(mt, set[0], NULL);
	1058	check_load(mt, 5016, NULL);
	1059	erase_check_insert(mt, 4); /* 1000 < Should not split. */
	1060	check_load(mt, set[0], NULL);
	1061	check_load(mt, 5016, NULL);
	1062	erase_check_load(mt, 1);
	1063	check_load(mt, 5013, NULL);
	1064	check_load(mt, set[2], NULL);
	1065	check_load(mt, 5018, NULL);
	1066	erase_check_load(mt, 4);
	1067	check_load(mt, 999, NULL);
	1068	check_load(mt, 1001, NULL);
	1069	erase_check_load(mt, 4);
	1070	if (mt_in_rcu(mt))
	1071	MT_BUG_ON(mt, root_node == mt->ma_root);
	1072	else
	1073	MT_BUG_ON(mt, root_node != mt->ma_root);
	1074
	1075	/* Should not have split. */
	1076	MT_BUG_ON(mt, !mte_is_leaf(mt->ma_root));
	1077
	1078
	1079	/* Coalesce testing */
	1080	erase_check_insert(mt, 0);
	1081	erase_check_insert(mt, 2);
	1082
	1083	for (int i = 5; i < 25; i++) {
	1084	erase_check_insert(mt, i);
	1085	for (int j = i; j >= 0; j--)
	1086	erase_check_load(mt, j);
	1087	}
	1088
	1089	erase_check_erase(mt, 14); /6015 /
	1090	for (int i = 0; i < 25; i++) {
	1091	if (i == 14)
	1092	check_load(mt, set[i], NULL);
	1093	else
	1094	erase_check_load(mt, i);
	1095	}
	1096	erase_check_erase(mt, 16); /7002 /
	1097	for (int i = 0; i < 25; i++) {
	1098	if (i == 16 \|\| i == 14)
	1099	check_load(mt, set[i], NULL);
	1100	else
	1101	erase_check_load(mt, i);
	1102	}
	1103
	1104
	1105	mt_set_non_kernel(1);
	1106	erase_check_erase(mt, 13); /6012 /
	1107	for (int i = 0; i < 25; i++) {
	1108	if (i == 16 \|\| i == 14 \|\| i == 13)
	1109	check_load(mt, set[i], NULL);
	1110	else
	1111	erase_check_load(mt, i);
	1112	}
	1113
	1114	erase_check_erase(mt, 15); /7003 /
	1115	for (int i = 0; i < 25; i++) {
	1116	if (i <= 16 && i >= 13)
	1117	check_load(mt, set[i], NULL);
	1118	else
	1119	erase_check_load(mt, i);
	1120	}
	1121
	1122	mt_set_non_kernel(2);
	1123	erase_check_erase(mt, 17); /7008 should* cause coalesce. */
	1124	for (int i = 0; i < 25; i++) {
	1125	if (i <= 17 && i >= 13)
	1126	check_load(mt, set[i], NULL);
	1127	else
	1128	erase_check_load(mt, i);
	1129	}
	1130
	1131	erase_check_erase(mt, 18); /7012 /
	1132	for (int i = 0; i < 25; i++) {
	1133	if (i <= 18 && i >= 13)
	1134	check_load(mt, set[i], NULL);
	1135	else
	1136	erase_check_load(mt, i);
	1137	}
	1138
	1139	mt_set_non_kernel(2);
	1140	erase_check_erase(mt, 19); /7015 /
	1141	for (int i = 0; i < 25; i++) {
	1142	if (i <= 19 && i >= 13)
	1143	check_load(mt, set[i], NULL);
	1144	else
	1145	erase_check_load(mt, i);
	1146	}
	1147
	1148	erase_check_erase(mt, 20); /8003 /
	1149	for (int i = 0; i < 25; i++) {
	1150	if (i <= 20 && i >= 13)
	1151	check_load(mt, set[i], NULL);
	1152	else
	1153	erase_check_load(mt, i);
	1154	}
	1155
	1156	erase_check_erase(mt, 21); /8002 /
	1157	for (int i = 0; i < 25; i++) {
	1158	if (i <= 21 && i >= 13)
	1159	check_load(mt, set[i], NULL);
	1160	else
	1161	erase_check_load(mt, i);
	1162	}
	1163
	1164	mt_set_non_kernel(2);
	1165	erase_check_erase(mt, 22); /8008 /
	1166	for (int i = 0; i < 25; i++) {
	1167	if (i <= 22 && i >= 13)
	1168	check_load(mt, set[i], NULL);
	1169	else
	1170	erase_check_load(mt, i);
	1171	}
	1172	for (int i = 23; i < 25; i++)
	1173	erase_check_erase(mt, i);
	1174
	1175	for (int i = 0; i < 25; i++) {
	1176	if (i <= 25 && i >= 13)
	1177	check_load(mt, set[i], NULL);
	1178	else
	1179	erase_check_load(mt, i);
	1180	}
	1181
	1182	/* Shrinking tree test. */
	1183
	1184	for (int i = 13; i < ARRAY_SIZE(set); i++)
	1185	erase_check_insert(mt, i);
	1186
	1187	mt_set_non_kernel(99);
	1188	for (int i = 18; i < ARRAY_SIZE(set); i++) {
	1189	erase_check_erase(mt, i);
	1190	for (int j = 0; j < ARRAY_SIZE(set); j++) {
	1191	if (j < 18 \|\| j > i)
	1192	erase_check_load(mt, j);
	1193	else
	1194	check_load(mt, set[j], NULL);
	1195	}
	1196	}
	1197	mt_set_non_kernel(35);
	1198	for (int i = 0; i < 18; i++) {
	1199	erase_check_erase(mt, i);
	1200	for (int j = 0; j < ARRAY_SIZE(set); j++) {
	1201	if (j < 18 && j > i)
	1202	erase_check_load(mt, j);
	1203	else
	1204	check_load(mt, set[j], NULL);
	1205	}
	1206	}
	1207	erase_check_insert(mt, 8);
	1208	erase_check_insert(mt, 9);
	1209	erase_check_erase(mt, 8);
	1210	rcu_unregister_thread();
	1211	}
	1212
	1213	#define erase_check_store_range(mt, a, i, ptr) mtree_test_store_range(mt, \
	1214	a[(i)], a[(i + 1)], ptr)
	1215	#define STORE 1
	1216	#define SNULL 2
	1217	#define ERASE 3
	1218	#define ec_type_str(x) \
	1219	(((x) == STORE) ? \
	1220	"STORE" : \
	1221	(((x) == SNULL) ? \
	1222	"SNULL" : "ERASE") \
	1223	)
	1224	#define check_erase2_debug 0
	1225	void mas_next(struct ma_state mas, unsigned long max);
	1226
	1227	/* Calculate the overwritten entries. */
	1228	int mas_ce2_over_count(struct ma_state mas_start, struct ma_state mas_end,
	1229	void *s_entry, unsigned long s_min,
	1230	void *e_entry, unsigned long e_max,
	1231	unsigned long *set, int i, bool null_entry)
	1232	{
	1233	int count = 0, span = 0;
	1234	unsigned long retry = 0;
	1235	void *entry;
	1236	struct ma_state tmp;
	1237
	1238
	1239	/* count slots */
	1240	memcpy(&tmp, mas_start, sizeof(tmp));
	1241	entry = mas_next(&tmp, mas_end->last);
	1242	while (entry) {
	1243	BUG_ON(retry > 50); /* stop infinite retry on testing. */
	1244	if (xa_is_zero(s_entry)) {
	1245	retry++;
	1246	continue;
	1247	}
	1248	count++;
	1249	span++;
	1250	entry = mas_next(&tmp, mas_end->last);
	1251	}
	1252
	1253	if (null_entry) {
	1254	/* Check splitting end. */
	1255	if (e_entry && (e_max > mas_end->last))
	1256	count--;
	1257
	1258	/* check overwrite of entire start */
	1259	if (s_entry && (s_min == mas_start->index))
	1260	count++;
	1261	} else { /* !null_entry (store) */
	1262	bool esplit = e_max > mas_end->last;
	1263	bool ssplit = s_min != mas_start->index;
	1264
	1265	if (s_entry && e_entry) {
	1266	if (esplit && ssplit)
	1267	count--;
	1268	else if (ssplit)
	1269	count--;
	1270	else if (esplit) {
	1271	if (span)
	1272	count--;
	1273	}
	1274	} else if (s_entry && !e_entry) {
	1275	if (ssplit)
	1276	count--;
	1277	} else if (!s_entry && e_entry) {
	1278	if (esplit)
	1279	count--;
	1280	count--;
	1281	} else {
	1282	count--;
	1283	}
	1284	}
	1285	return count;
	1286	}
	1287
	1288	/*
	1289	* mas_node_walk() - Walk a maple node to offset of the index.
	1290	* @mas: The maple state
	1291	* @type: The maple node type
	1292	* @*range_min: Pointer to store the minimum range of the offset
	1293	* @*range_max: Pointer to store the maximum range of the offset
	1294	*
	1295	* The offset will be stored in the maple state.
	1296	*
	1297	*/
	1298	static inline void mas_node_walk(struct ma_state mas, struct maple_node node,
	1299	enum maple_type type, unsigned long *range_min,
	1300	unsigned long *range_max)
	1301
	1302	{
	1303	unsigned long *pivots;
	1304	unsigned char count;
	1305	unsigned long prev, max;
	1306	unsigned char offset;
	1307	unsigned long index;
	1308
	1309	if (unlikely(ma_is_dense(type))) {
	1310	(range_max) = (range_min) = mas->index;
	1311	if (unlikely(ma_dead_node(node)))
	1312	return;
	1313
	1314	mas->offset = mas->index = mas->min;
	1315	return;
	1316	}
	1317
	1318	pivots = ma_pivots(node, type);
	1319	max = pivots[0];
	1320	if (unlikely(ma_dead_node(node)))
	1321	return;
	1322
	1323	offset = 0;
	1324	prev = mas->min;
	1325	index = mas->index;
	1326	if (unlikely(index <= max))
	1327	goto offset_zero;
	1328
	1329	count = mt_pivots[type];
	1330	while (++offset < count) {
	1331	prev = max;
	1332	max = pivots[offset];
	1333	if (unlikely(ma_dead_node(node)))
	1334	return;
	1335
	1336	if (index <= max)
	1337	goto offset_found;
	1338	else if (unlikely(!max))
	1339	goto mas_max;
	1340	}
	1341
	1342	prev = max;
	1343	mas_max:
	1344	max = mas->max;
	1345	offset_found:
	1346	prev++;
	1347	offset_zero:
	1348	mas->offset = offset;
	1349	if (ma_is_leaf(type)) {
	1350	*range_max = max;
	1351	*range_min = prev;
	1352	} else {
	1353	mas->max = max;
	1354	mas->min = prev;
	1355	}
	1356	}
	1357
	1358	/*
	1359	* mas_descend_walk(): Locates a value and sets the mas->node and slot
	1360	* accordingly. range_min and range_max are set to the range which the entry is
	1361	* valid.
	1362	* @mas: The maple state
	1363	* @*range_min: A pointer to store the minimum of the range
	1364	* @*range_max: A pointer to store the maximum of the range
	1365	*
	1366	* Check mas->node is still valid on return of any value.
	1367	*
	1368	* Return: true if pointing to a valid node and offset. False otherwise.
	1369	*/
	1370	static inline bool mas_descend_walk(struct ma_state *mas,
	1371	unsigned long range_min, unsigned long range_max)
	1372	{
	1373	struct maple_enode *next;
	1374	struct maple_node *node;
	1375	enum maple_type type;
	1376
	1377	next = mas->node;
	1378	while (true) {
	1379	node = mte_to_node(next);
	1380	type = mte_node_type(next);
	1381	mas_node_walk(mas, node, type, range_min, range_max);
	1382	next = mas_slot(mas, ma_slots(node, type), mas->offset);
	1383	if (unlikely(ma_dead_node(node)))
	1384	return false;
	1385
	1386	if (unlikely(ma_is_leaf(type)))
	1387	return true;
	1388
	1389	/* Descend. */
	1390	mas->node = next;
	1391	}
	1392	return false;
	1393	}
	1394
	1395	/*
	1396	* mas_tree_walk() - Walk to @mas->index and set the range values.
	1397	* @mas: The maple state.
	1398	* @*range_min: The minimum range to be set.
	1399	* @*range_max: The maximum range to be set.
	1400	*
	1401	* Ranges are only valid if there is a valid entry at @mas->index.
	1402	*
	1403	* Return: True if a value exists, false otherwise.
	1404	*/
	1405	static inline bool mas_tree_walk(struct ma_state mas, unsigned long range_min,
	1406	unsigned long *range_max)
	1407	{
	1408	bool ret;
	1409
	1410	retry:
	1411	ret = false;
	1412	mas_start(mas);
	1413	if (mas_is_none(mas))
	1414	goto not_found;
	1415
	1416	if (mas_is_ptr(mas)) {
	1417	range_min = range_max = 0;
	1418	if (!mas->index)
	1419	return true;
	1420
	1421	goto not_found;
	1422	}
	1423
	1424	ret = mas_descend_walk(mas, range_min, range_max);
	1425	if (unlikely(mte_dead_node(mas->node))) {
	1426	mas->node = MAS_START;
	1427	goto retry;
	1428	}
	1429
	1430	return ret;
	1431
	1432	not_found:
	1433	mas->offset = MAPLE_NODE_SLOTS;
	1434	return false;
	1435	}
	1436
	1437	static inline void mas_range_load(struct ma_state mas,
	1438	unsigned long range_min, unsigned long range_max)
	1439
	1440	{
	1441	void *entry = NULL;
	1442	unsigned long index = mas->index;
	1443
	1444	if (mas_is_none(mas) \|\| mas_is_paused(mas))
	1445	mas->node = MAS_START;
	1446	retry:
	1447	if (mas_tree_walk(mas, range_min, range_max))
	1448	if (unlikely(mas->node == MAS_ROOT))
	1449	return mas_root(mas);
	1450
	1451	if (likely(mas->offset != MAPLE_NODE_SLOTS))
	1452	entry = mas_get_slot(mas, mas->offset);
	1453
	1454	if (mas_dead_node(mas, index))
	1455	goto retry;
	1456
	1457	return entry;
	1458	}
	1459	static noinline void check_erase2_testset(struct maple_tree *mt,
	1460	unsigned long *set, unsigned long size)
	1461	{
	1462	int entry_count = 0;
	1463	int check = 0;
	1464	void *foo;
	1465	unsigned long addr = 0;
	1466	void s_entry = NULL, e_entry = NULL;
	1467
	1468	MA_STATE(mas, mt, 0, 0);
	1469
	1470	for (int i = 0; i < size; i += 3) {
	1471	unsigned long s_min, s_max;
	1472	unsigned long e_min, e_max;
	1473	void *value = NULL;
	1474
	1475	MA_STATE(mas_start, mt, set[i+1], set[i+1]);
	1476	MA_STATE(mas_end, mt, set[i+2], set[i+2]);
	1477	mt_set_non_kernel(127);
	1478	#if check_erase2_debug
	1479	pr_err("%s: %d %s %lu - %lu\n", __func__, i,
	1480	ec_type_str(set[i]),
	1481	set[i+1], set[i+2]);
	1482	#endif
	1483	s_entry = mas_range_load(&mas_start, &s_min, &s_max);
	1484	e_entry = mas_range_load(&mas_end, &e_min, &e_max);
	1485
	1486	switch (set[i]) {
	1487	case SNULL:
	1488	if ((s_min == set[i+1]) && (s_max == set[i+2])) {
	1489	if (s_entry)
	1490	entry_count--;
	1491	} else if ((s_min != set[i+1]) && (s_max != set[i+2])) {
	1492	entry_count++;
	1493	} else if ((mas_start.node != mas_end.node) \|\|
	1494	(mas_start.offset != mas_end.offset)) {
	1495	entry_count -=
	1496	mas_ce2_over_count(&mas_start, &mas_end,
	1497	s_entry, s_min,
	1498	e_entry, e_max, set, i,
	1499	true);
	1500	}
	1501
	1502
	1503	erase_check_store_range(mt, set, i + 1, value);
	1504	break;
	1505	case STORE:
	1506	value = xa_mk_value(set[i + 1]);
	1507	if (mas_start.offset > mt_slot_count(mas_start.node)) {
	1508	entry_count++; /* appending an entry. */
	1509	} else if ((s_min == e_min) && (s_max == e_max)) {
	1510	if (!entry_count)
	1511	entry_count++;
	1512
	1513	else if (s_entry) {
	1514	if (e_max > mas_end.last)
	1515	entry_count++;
	1516
	1517	if (s_min < mas_start.index)
	1518	entry_count++;
	1519
	1520	} else {
	1521	entry_count++;
	1522	}
	1523	} else {
	1524	entry_count -=
	1525	mas_ce2_over_count(&mas_start, &mas_end,
	1526	s_entry, s_min,
	1527	e_entry, e_max, set, i,
	1528	false);
	1529	}
	1530
	1531	erase_check_store_range(mt, set, i + 1, value);
	1532	break;
	1533	case ERASE:
	1534	if (!s_entry)
	1535	break;
	1536	check_erase(mt, set[i+1], xa_mk_value(set[i+1]));
	1537	entry_count--;
	1538	break;
	1539	}
	1540	mt_validate(mt);
	1541	if (entry_count)
	1542	MT_BUG_ON(mt, !mt_height(mt));
	1543	#if check_erase2_debug > 1
	1544	mt_dump(mt);
	1545	#endif
	1546	#if check_erase2_debug
	1547	pr_err("Done\n");
	1548	#endif
	1549
	1550	check = 0;
	1551	addr = 0;
	1552	mt_for_each(mt, foo, addr, ULONG_MAX) {
	1553	check++;
	1554	#if check_erase2_debug > 2
	1555	pr_err("mt: %lu -> %p (%d)\n", addr+1, foo, check);
	1556	#endif
	1557	if (check > entry_count)
	1558	break;
	1559	}
	1560
	1561	#if check_erase2_debug > 2
	1562	pr_err("mt_for_each %d and count %d\n", check, entry_count);
	1563	#endif
	1564
	1565	MT_BUG_ON(mt, check != entry_count);
	1566
	1567	check = 0;
	1568	addr = 0;
	1569	mas_reset(&mas);
	1570	mas.index = 0;
	1571	rcu_read_lock();
	1572	mas_for_each(&mas, foo, ULONG_MAX) {
	1573	if (xa_is_zero(foo)) {
	1574	if (addr == mas.index) {
	1575	mt_dump(mas.tree);
	1576	pr_err("retry failed %lu - %lu\n",
	1577	mas.index, mas.last);
	1578	MT_BUG_ON(mt, 1);
	1579	}
	1580	addr = mas.index;
	1581	continue;
	1582	}
	1583	#if check_erase2_debug > 2
	1584	pr_err("mas: %lu -> %p\n", mas.index, foo);
	1585	#endif
	1586	check++;
	1587	if (check > entry_count)
	1588	break;
	1589	}
	1590	rcu_read_unlock();
	1591	#if check_erase2_debug > 2
	1592	pr_err("mas_for_each %d and count %d\n", check, entry_count);
	1593	mt_validate(mt);
	1594	#endif
	1595
	1596	MT_BUG_ON(mt, check != entry_count);
	1597
	1598	MT_BUG_ON(mt, mtree_load(mas.tree, 0) != NULL);
	1599	}
	1600	}
	1601
	1602
	1603	/* These tests were pulled from kvm tests. */
	1604	static noinline void check_erase2_sets(struct maple_tree *mt)
	1605	{
	1606	void *entry;
	1607	unsigned long start = 0;
	1608	unsigned long set[] = {
	1609	STORE, 140737488347136, 140737488351231,
	1610	STORE, 140721266458624, 140737488351231,
	1611	ERASE, 140721266458624, 140737488351231,
	1612	STORE, 140721266458624, 140721266462719,
	1613	STORE, 94735788949504, 94735789121535,
	1614	ERASE, 94735788949504, 94735789121535,
	1615	STORE, 94735788949504, 94735788965887,
	1616	STORE, 94735788965888, 94735789121535,
	1617	ERASE, 94735788965888, 94735789121535,
	1618	STORE, 94735788965888, 94735789068287,
	1619	STORE, 94735789068288, 94735789109247,
	1620	STORE, 94735789109248, 94735789121535,
	1621	STORE, 140253902692352, 140253902864383,
	1622	ERASE, 140253902692352, 140253902864383,
	1623	STORE, 140253902692352, 140253902696447,
	1624	STORE, 140253902696448, 140253902864383,
	1625	};
	1626	unsigned long set2[] = {
	1627	STORE, 140737488347136, 140737488351231,
	1628	STORE, 140735933583360, 140737488351231,
	1629	ERASE, 140735933583360, 140737488351231,
	1630	STORE, 140735933583360, 140735933587455,
	1631	STORE, 94811003260928, 94811003432959,
	1632	ERASE, 94811003260928, 94811003432959,
	1633	STORE, 94811003260928, 94811003277311,
	1634	STORE, 94811003277312, 94811003432959,
	1635	ERASE, 94811003277312, 94811003432959,
	1636	STORE, 94811003277312, 94811003379711,
	1637	STORE, 94811003379712, 94811003420671,
	1638	STORE, 94811003420672, 94811003432959,
	1639	STORE, 140277094653952, 140277094825983,
	1640	ERASE, 140277094653952, 140277094825983,
	1641	STORE, 140277094653952, 140277094658047,
	1642	STORE, 140277094658048, 140277094825983,
	1643	ERASE, 140277094658048, 140277094825983,
	1644	STORE, 140277094658048, 140277094780927,
	1645	STORE, 140277094780928, 140277094813695,
	1646	STORE, 140277094813696, 140277094821887,
	1647	STORE, 140277094821888, 140277094825983,
	1648	STORE, 140735933906944, 140735933911039,
	1649	};
	1650	unsigned long set3[] = {
	1651	STORE, 140737488347136, 140737488351231,
	1652	STORE, 140735790264320, 140737488351231,
	1653	ERASE, 140735790264320, 140737488351231,
	1654	STORE, 140735790264320, 140735790268415,
	1655	STORE, 94016597282816, 94016597454847,
	1656	ERASE, 94016597282816, 94016597454847,
	1657	STORE, 94016597282816, 94016597299199,
	1658	STORE, 94016597299200, 94016597454847,
	1659	ERASE, 94016597299200, 94016597454847,
	1660	STORE, 94016597299200, 94016597401599,
	1661	STORE, 94016597401600, 94016597442559,
	1662	STORE, 94016597442560, 94016597454847,
	1663	STORE, 140496959283200, 140496959455231,
	1664	ERASE, 140496959283200, 140496959455231,
	1665	STORE, 140496959283200, 140496959287295,
	1666	STORE, 140496959287296, 140496959455231,
	1667	ERASE, 140496959287296, 140496959455231,
	1668	STORE, 140496959287296, 140496959410175,
	1669	STORE, 140496959410176, 140496959442943,
	1670	STORE, 140496959442944, 140496959451135,
	1671	STORE, 140496959451136, 140496959455231,
	1672	STORE, 140735791718400, 140735791722495,
	1673	STORE, 140735791706112, 140735791718399,
	1674	STORE, 47135835713536, 47135835721727,
	1675	STORE, 47135835721728, 47135835729919,
	1676	STORE, 47135835729920, 47135835893759,
	1677	ERASE, 47135835729920, 47135835893759,
	1678	STORE, 47135835729920, 47135835742207,
	1679	STORE, 47135835742208, 47135835893759,
	1680	STORE, 47135835840512, 47135835893759,
	1681	STORE, 47135835742208, 47135835840511,
	1682	ERASE, 47135835742208, 47135835840511,
	1683	STORE, 47135835742208, 47135835840511,
	1684	STORE, 47135835885568, 47135835893759,
	1685	STORE, 47135835840512, 47135835885567,
	1686	ERASE, 47135835840512, 47135835885567,
	1687	STORE, 47135835840512, 47135835893759,
	1688	ERASE, 47135835840512, 47135835893759,
	1689	STORE, 47135835840512, 47135835885567,
	1690	STORE, 47135835885568, 47135835893759,
	1691	};
	1692
	1693	unsigned long set4[] = {
	1694	STORE, 140737488347136, 140737488351231,
	1695	STORE, 140728251703296, 140737488351231,
	1696	ERASE, 140728251703296, 140737488351231,
	1697	STORE, 140728251703296, 140728251707391,
	1698	STORE, 94668429205504, 94668429377535,
	1699	ERASE, 94668429205504, 94668429377535,
	1700	STORE, 94668429205504, 94668429221887,
	1701	STORE, 94668429221888, 94668429377535,
	1702	ERASE, 94668429221888, 94668429377535,
	1703	STORE, 94668429221888, 94668429324287,
	1704	STORE, 94668429324288, 94668429365247,
	1705	STORE, 94668429365248, 94668429377535,
	1706	STORE, 47646523273216, 47646523445247,
	1707	ERASE, 47646523273216, 47646523445247,
	1708	STORE, 47646523273216, 47646523277311,
	1709	STORE, 47646523277312, 47646523445247,
	1710	ERASE, 47646523277312, 47646523445247,
	1711	STORE, 47646523277312, 47646523400191,
	1712	};
	1713
	1714	unsigned long set5[] = {
	1715	STORE, 140737488347136, 140737488351231,
	1716	STORE, 140726874062848, 140737488351231,
	1717	ERASE, 140726874062848, 140737488351231,
	1718	STORE, 140726874062848, 140726874066943,
	1719	STORE, 94248892870656, 94248893042687,
	1720	ERASE, 94248892870656, 94248893042687,
	1721	STORE, 94248892870656, 94248892887039,
	1722	STORE, 94248892887040, 94248893042687,
	1723	ERASE, 94248892887040, 94248893042687,
	1724	STORE, 94248892887040, 94248892989439,
	1725	STORE, 94248892989440, 94248893030399,
	1726	STORE, 94248893030400, 94248893042687,
	1727	STORE, 47884786266112, 47884786438143,
	1728	ERASE, 47884786266112, 47884786438143,
	1729	STORE, 47884786266112, 47884786270207,
	1730	STORE, 47884786270208, 47884786438143,
	1731	ERASE, 47884786270208, 47884786438143,
	1732	STORE, 47884786270208, 47884786393087,
	1733	STORE, 47884786393088, 47884786425855,
	1734	STORE, 47884786425856, 47884786434047,
	1735	STORE, 47884786434048, 47884786438143,
	1736	STORE, 140726874513408, 140726874517503,
	1737	STORE, 140726874501120, 140726874513407,
	1738	STORE, 47884786438144, 47884786446335,
	1739	STORE, 47884786446336, 47884786454527,
	1740	STORE, 47884786454528, 47884786618367,
	1741	ERASE, 47884786454528, 47884786618367,
	1742	STORE, 47884786454528, 47884786466815,
	1743	STORE, 47884786466816, 47884786618367,
	1744	STORE, 47884786565120, 47884786618367,
	1745	STORE, 47884786466816, 47884786565119,
	1746	ERASE, 47884786466816, 47884786565119,
	1747	STORE, 47884786466816, 47884786565119,
	1748	STORE, 47884786610176, 47884786618367,
	1749	STORE, 47884786565120, 47884786610175,
	1750	ERASE, 47884786565120, 47884786610175,
	1751	STORE, 47884786565120, 47884786618367,
	1752	ERASE, 47884786565120, 47884786618367,
	1753	STORE, 47884786565120, 47884786610175,
	1754	STORE, 47884786610176, 47884786618367,
	1755	ERASE, 47884786610176, 47884786618367,
	1756	STORE, 47884786610176, 47884786618367,
	1757	STORE, 47884786618368, 47884789669887,
	1758	STORE, 47884787163136, 47884789669887,
	1759	STORE, 47884786618368, 47884787163135,
	1760	ERASE, 47884787163136, 47884789669887,
	1761	STORE, 47884787163136, 47884789448703,
	1762	STORE, 47884789448704, 47884789669887,
	1763	STORE, 47884788858880, 47884789448703,
	1764	STORE, 47884787163136, 47884788858879,
	1765	ERASE, 47884787163136, 47884788858879,
	1766	STORE, 47884787163136, 47884788858879,
	1767	STORE, 47884789444608, 47884789448703,
	1768	STORE, 47884788858880, 47884789444607,
	1769	ERASE, 47884788858880, 47884789444607,
	1770	STORE, 47884788858880, 47884789444607,
	1771	STORE, 47884789653504, 47884789669887,
	1772	STORE, 47884789448704, 47884789653503,
	1773	ERASE, 47884789448704, 47884789653503,
	1774	STORE, 47884789448704, 47884789653503,
	1775	ERASE, 47884789653504, 47884789669887,
	1776	STORE, 47884789653504, 47884789669887,
	1777	STORE, 47884789669888, 47884791508991,
	1778	STORE, 47884789809152, 47884791508991,
	1779	STORE, 47884789669888, 47884789809151,
	1780	ERASE, 47884789809152, 47884791508991,
	1781	STORE, 47884789809152, 47884791468031,
	1782	STORE, 47884791468032, 47884791508991,
	1783	STORE, 47884791152640, 47884791468031,
	1784	STORE, 47884789809152, 47884791152639,
	1785	ERASE, 47884789809152, 47884791152639,
	1786	STORE, 47884789809152, 47884791152639,
	1787	STORE, 47884791463936, 47884791468031,
	1788	STORE, 47884791152640, 47884791463935,
	1789	ERASE, 47884791152640, 47884791463935,
	1790	STORE, 47884791152640, 47884791463935,
	1791	STORE, 47884791492608, 47884791508991,
	1792	STORE, 47884791468032, 47884791492607,
	1793	ERASE, 47884791468032, 47884791492607,
	1794	STORE, 47884791468032, 47884791492607,
	1795	ERASE, 47884791492608, 47884791508991,
	1796	STORE, 47884791492608, 47884791508991,
	1797	STORE, 47884791508992, 47884791644159,
	1798	ERASE, 47884791508992, 47884791644159,
	1799	STORE, 47884791508992, 47884791533567,
	1800	STORE, 47884791533568, 47884791644159,
	1801	STORE, 47884791595008, 47884791644159,
	1802	STORE, 47884791533568, 47884791595007,
	1803	ERASE, 47884791533568, 47884791595007,
	1804	STORE, 47884791533568, 47884791595007,
	1805	STORE, 47884791619584, 47884791644159,
	1806	STORE, 47884791595008, 47884791619583,
	1807	ERASE, 47884791595008, 47884791619583,
	1808	STORE, 47884791595008, 47884791644159,
	1809	ERASE, 47884791595008, 47884791644159,
	1810	STORE, 47884791595008, 47884791619583,
	1811	STORE, 47884791619584, 47884791644159,
	1812	STORE, 47884791627776, 47884791644159,
	1813	STORE, 47884791619584, 47884791627775,
	1814	ERASE, 47884791619584, 47884791627775,
	1815	STORE, 47884791619584, 47884791627775,
	1816	ERASE, 47884791627776, 47884791644159,
	1817	STORE, 47884791627776, 47884791644159,
	1818	STORE, 47884791644160, 47884791664639,
	1819	ERASE, 47884791644160, 47884791664639,
	1820	STORE, 47884791644160, 47884791648255,
	1821	STORE, 47884791648256, 47884791664639,
	1822	STORE, 47884791652352, 47884791664639,
	1823	STORE, 47884791648256, 47884791652351,
	1824	ERASE, 47884791648256, 47884791652351,
	1825	STORE, 47884791648256, 47884791652351,
	1826	STORE, 47884791656448, 47884791664639,
	1827	STORE, 47884791652352, 47884791656447,
	1828	ERASE, 47884791652352, 47884791656447,
	1829	STORE, 47884791652352, 47884791664639,
	1830	ERASE, 47884791652352, 47884791664639,
	1831	STORE, 47884791652352, 47884791656447,
	1832	STORE, 47884791656448, 47884791664639,
	1833	ERASE, 47884791656448, 47884791664639,
	1834	STORE, 47884791656448, 47884791664639,
	1835	STORE, 47884791664640, 47884791672831,
	1836	ERASE, 47884791468032, 47884791492607,
	1837	STORE, 47884791468032, 47884791484415,
	1838	STORE, 47884791484416, 47884791492607,
	1839	ERASE, 47884791656448, 47884791664639,
	1840	STORE, 47884791656448, 47884791660543,
	1841	STORE, 47884791660544, 47884791664639,
	1842	ERASE, 47884791619584, 47884791627775,
	1843	STORE, 47884791619584, 47884791623679,
	1844	STORE, 47884791623680, 47884791627775,
	1845	};
	1846
	1847	unsigned long set6[] = {
	1848	STORE, 140737488347136, 140737488351231,
	1849	STORE, 140722999021568, 140737488351231,
	1850	ERASE, 140722999021568, 140737488351231,
	1851	STORE, 140722999021568, 140722999025663,
	1852	STORE, 94901500268544, 94901500440575,
	1853	ERASE, 94901500268544, 94901500440575,
	1854	STORE, 94901500268544, 94901500284927,
	1855	STORE, 94901500284928, 94901500440575,
	1856	ERASE, 94901500284928, 94901500440575,
	1857	STORE, 94901500284928, 94901500387327,
	1858	STORE, 94901500387328, 94901500428287,
	1859	STORE, 94901500428288, 94901500440575,
	1860	STORE, 47430426660864, 47430426832895,
	1861	ERASE, 47430426660864, 47430426832895,
	1862	STORE, 47430426660864, 47430426664959,
	1863	STORE, 47430426664960, 47430426832895,
	1864	ERASE, 47430426664960, 47430426832895,
	1865	STORE, 47430426664960, 47430426787839,
	1866	STORE, 47430426787840, 47430426820607,
	1867	STORE, 47430426820608, 47430426828799,
	1868	STORE, 47430426828800, 47430426832895,
	1869	STORE, 140722999115776, 140722999119871,
	1870	STORE, 140722999103488, 140722999115775,
	1871	STORE, 47430426832896, 47430426841087,
	1872	STORE, 47430426841088, 47430426849279,
	1873	STORE, 47430426849280, 47430427013119,
	1874	ERASE, 47430426849280, 47430427013119,
	1875	STORE, 47430426849280, 47430426861567,
	1876	STORE, 47430426861568, 47430427013119,
	1877	STORE, 47430426959872, 47430427013119,
	1878	STORE, 47430426861568, 47430426959871,
	1879	ERASE, 47430426861568, 47430426959871,
	1880	STORE, 47430426861568, 47430426959871,
	1881	STORE, 47430427004928, 47430427013119,
	1882	STORE, 47430426959872, 47430427004927,
	1883	ERASE, 47430426959872, 47430427004927,
	1884	STORE, 47430426959872, 47430427013119,
	1885	ERASE, 47430426959872, 47430427013119,
	1886	STORE, 47430426959872, 47430427004927,
	1887	STORE, 47430427004928, 47430427013119,
	1888	ERASE, 47430427004928, 47430427013119,
	1889	STORE, 47430427004928, 47430427013119,
	1890	STORE, 47430427013120, 47430430064639,
	1891	STORE, 47430427557888, 47430430064639,
	1892	STORE, 47430427013120, 47430427557887,
	1893	ERASE, 47430427557888, 47430430064639,
	1894	STORE, 47430427557888, 47430429843455,
	1895	STORE, 47430429843456, 47430430064639,
	1896	STORE, 47430429253632, 47430429843455,
	1897	STORE, 47430427557888, 47430429253631,
	1898	ERASE, 47430427557888, 47430429253631,
	1899	STORE, 47430427557888, 47430429253631,
	1900	STORE, 47430429839360, 47430429843455,
	1901	STORE, 47430429253632, 47430429839359,
	1902	ERASE, 47430429253632, 47430429839359,
	1903	STORE, 47430429253632, 47430429839359,
	1904	STORE, 47430430048256, 47430430064639,
	1905	STORE, 47430429843456, 47430430048255,
	1906	ERASE, 47430429843456, 47430430048255,
	1907	STORE, 47430429843456, 47430430048255,
	1908	ERASE, 47430430048256, 47430430064639,
	1909	STORE, 47430430048256, 47430430064639,
	1910	STORE, 47430430064640, 47430431903743,
	1911	STORE, 47430430203904, 47430431903743,
	1912	STORE, 47430430064640, 47430430203903,
	1913	ERASE, 47430430203904, 47430431903743,
	1914	STORE, 47430430203904, 47430431862783,
	1915	STORE, 47430431862784, 47430431903743,
	1916	STORE, 47430431547392, 47430431862783,
	1917	STORE, 47430430203904, 47430431547391,
	1918	ERASE, 47430430203904, 47430431547391,
	1919	STORE, 47430430203904, 47430431547391,
	1920	STORE, 47430431858688, 47430431862783,
	1921	STORE, 47430431547392, 47430431858687,
	1922	ERASE, 47430431547392, 47430431858687,
	1923	STORE, 47430431547392, 47430431858687,
	1924	STORE, 47430431887360, 47430431903743,
	1925	STORE, 47430431862784, 47430431887359,
	1926	ERASE, 47430431862784, 47430431887359,
	1927	STORE, 47430431862784, 47430431887359,
	1928	ERASE, 47430431887360, 47430431903743,
	1929	STORE, 47430431887360, 47430431903743,
	1930	STORE, 47430431903744, 47430432038911,
	1931	ERASE, 47430431903744, 47430432038911,
	1932	STORE, 47430431903744, 47430431928319,
	1933	STORE, 47430431928320, 47430432038911,
	1934	STORE, 47430431989760, 47430432038911,
	1935	STORE, 47430431928320, 47430431989759,
	1936	ERASE, 47430431928320, 47430431989759,
	1937	STORE, 47430431928320, 47430431989759,
	1938	STORE, 47430432014336, 47430432038911,
	1939	STORE, 47430431989760, 47430432014335,
	1940	ERASE, 47430431989760, 47430432014335,
	1941	STORE, 47430431989760, 47430432038911,
	1942	ERASE, 47430431989760, 47430432038911,
	1943	STORE, 47430431989760, 47430432014335,
	1944	STORE, 47430432014336, 47430432038911,
	1945	STORE, 47430432022528, 47430432038911,
	1946	STORE, 47430432014336, 47430432022527,
	1947	ERASE, 47430432014336, 47430432022527,
	1948	STORE, 47430432014336, 47430432022527,
	1949	ERASE, 47430432022528, 47430432038911,
	1950	STORE, 47430432022528, 47430432038911,
	1951	STORE, 47430432038912, 47430432059391,
	1952	ERASE, 47430432038912, 47430432059391,
	1953	STORE, 47430432038912, 47430432043007,
	1954	STORE, 47430432043008, 47430432059391,
	1955	STORE, 47430432047104, 47430432059391,
	1956	STORE, 47430432043008, 47430432047103,
	1957	ERASE, 47430432043008, 47430432047103,
	1958	STORE, 47430432043008, 47430432047103,
	1959	STORE, 47430432051200, 47430432059391,
	1960	STORE, 47430432047104, 47430432051199,
	1961	ERASE, 47430432047104, 47430432051199,
	1962	STORE, 47430432047104, 47430432059391,
	1963	ERASE, 47430432047104, 47430432059391,
	1964	STORE, 47430432047104, 47430432051199,
	1965	STORE, 47430432051200, 47430432059391,
	1966	ERASE, 47430432051200, 47430432059391,
	1967	STORE, 47430432051200, 47430432059391,
	1968	STORE, 47430432059392, 47430432067583,
	1969	ERASE, 47430431862784, 47430431887359,
	1970	STORE, 47430431862784, 47430431879167,
	1971	STORE, 47430431879168, 47430431887359,
	1972	ERASE, 47430432051200, 47430432059391,
	1973	STORE, 47430432051200, 47430432055295,
	1974	STORE, 47430432055296, 47430432059391,
	1975	ERASE, 47430432014336, 47430432022527,
	1976	STORE, 47430432014336, 47430432018431,
	1977	STORE, 47430432018432, 47430432022527,
	1978	};
	1979	unsigned long set7[] = {
	1980	STORE, 140737488347136, 140737488351231,
	1981	STORE, 140729808330752, 140737488351231,
	1982	ERASE, 140729808330752, 140737488351231,
	1983	STORE, 140729808330752, 140729808334847,
	1984	STORE, 94629632020480, 94629632192511,
	1985	ERASE, 94629632020480, 94629632192511,
	1986	STORE, 94629632020480, 94629632036863,
	1987	STORE, 94629632036864, 94629632192511,
	1988	ERASE, 94629632036864, 94629632192511,
	1989	STORE, 94629632036864, 94629632139263,
	1990	STORE, 94629632139264, 94629632180223,
	1991	STORE, 94629632180224, 94629632192511,
	1992	STORE, 47439981776896, 47439981948927,
	1993	ERASE, 47439981776896, 47439981948927,
	1994	STORE, 47439981776896, 47439981780991,
	1995	STORE, 47439981780992, 47439981948927,
	1996	ERASE, 47439981780992, 47439981948927,
	1997	STORE, 47439981780992, 47439981903871,
	1998	STORE, 47439981903872, 47439981936639,
	1999	STORE, 47439981936640, 47439981944831,
	2000	STORE, 47439981944832, 47439981948927,
	2001	STORE, 140729808474112, 140729808478207,
	2002	STORE, 140729808461824, 140729808474111,
	2003	STORE, 47439981948928, 47439981957119,
	2004	STORE, 47439981957120, 47439981965311,
	2005	STORE, 47439981965312, 47439982129151,
	2006	ERASE, 47439981965312, 47439982129151,
	2007	STORE, 47439981965312, 47439981977599,
	2008	STORE, 47439981977600, 47439982129151,
	2009	STORE, 47439982075904, 47439982129151,
	2010	STORE, 47439981977600, 47439982075903,
	2011	ERASE, 47439981977600, 47439982075903,
	2012	STORE, 47439981977600, 47439982075903,
	2013	STORE, 47439982120960, 47439982129151,
	2014	STORE, 47439982075904, 47439982120959,
	2015	ERASE, 47439982075904, 47439982120959,
	2016	STORE, 47439982075904, 47439982129151,
	2017	ERASE, 47439982075904, 47439982129151,
	2018	STORE, 47439982075904, 47439982120959,
	2019	STORE, 47439982120960, 47439982129151,
	2020	ERASE, 47439982120960, 47439982129151,
	2021	STORE, 47439982120960, 47439982129151,
	2022	STORE, 47439982129152, 47439985180671,
	2023	STORE, 47439982673920, 47439985180671,
	2024	STORE, 47439982129152, 47439982673919,
	2025	ERASE, 47439982673920, 47439985180671,
	2026	STORE, 47439982673920, 47439984959487,
	2027	STORE, 47439984959488, 47439985180671,
	2028	STORE, 47439984369664, 47439984959487,
	2029	STORE, 47439982673920, 47439984369663,
	2030	ERASE, 47439982673920, 47439984369663,
	2031	STORE, 47439982673920, 47439984369663,
	2032	STORE, 47439984955392, 47439984959487,
	2033	STORE, 47439984369664, 47439984955391,
	2034	ERASE, 47439984369664, 47439984955391,
	2035	STORE, 47439984369664, 47439984955391,
	2036	STORE, 47439985164288, 47439985180671,
	2037	STORE, 47439984959488, 47439985164287,
	2038	ERASE, 47439984959488, 47439985164287,
	2039	STORE, 47439984959488, 47439985164287,
	2040	ERASE, 47439985164288, 47439985180671,
	2041	STORE, 47439985164288, 47439985180671,
	2042	STORE, 47439985180672, 47439987019775,
	2043	STORE, 47439985319936, 47439987019775,
	2044	STORE, 47439985180672, 47439985319935,
	2045	ERASE, 47439985319936, 47439987019775,
	2046	STORE, 47439985319936, 47439986978815,
	2047	STORE, 47439986978816, 47439987019775,
	2048	STORE, 47439986663424, 47439986978815,
	2049	STORE, 47439985319936, 47439986663423,
	2050	ERASE, 47439985319936, 47439986663423,
	2051	STORE, 47439985319936, 47439986663423,
	2052	STORE, 47439986974720, 47439986978815,
	2053	STORE, 47439986663424, 47439986974719,
	2054	ERASE, 47439986663424, 47439986974719,
	2055	STORE, 47439986663424, 47439986974719,
	2056	STORE, 47439987003392, 47439987019775,
	2057	STORE, 47439986978816, 47439987003391,
	2058	ERASE, 47439986978816, 47439987003391,
	2059	STORE, 47439986978816, 47439987003391,
	2060	ERASE, 47439987003392, 47439987019775,
	2061	STORE, 47439987003392, 47439987019775,
	2062	STORE, 47439987019776, 47439987154943,
	2063	ERASE, 47439987019776, 47439987154943,
	2064	STORE, 47439987019776, 47439987044351,
	2065	STORE, 47439987044352, 47439987154943,
	2066	STORE, 47439987105792, 47439987154943,
	2067	STORE, 47439987044352, 47439987105791,
	2068	ERASE, 47439987044352, 47439987105791,
	2069	STORE, 47439987044352, 47439987105791,
	2070	STORE, 47439987130368, 47439987154943,
	2071	STORE, 47439987105792, 47439987130367,
	2072	ERASE, 47439987105792, 47439987130367,
	2073	STORE, 47439987105792, 47439987154943,
	2074	ERASE, 47439987105792, 47439987154943,
	2075	STORE, 47439987105792, 47439987130367,
	2076	STORE, 47439987130368, 47439987154943,
	2077	STORE, 47439987138560, 47439987154943,
	2078	STORE, 47439987130368, 47439987138559,
	2079	ERASE, 47439987130368, 47439987138559,
	2080	STORE, 47439987130368, 47439987138559,
	2081	ERASE, 47439987138560, 47439987154943,
	2082	STORE, 47439987138560, 47439987154943,
	2083	STORE, 47439987154944, 47439987175423,
	2084	ERASE, 47439987154944, 47439987175423,
	2085	STORE, 47439987154944, 47439987159039,
	2086	STORE, 47439987159040, 47439987175423,
	2087	STORE, 47439987163136, 47439987175423,
	2088	STORE, 47439987159040, 47439987163135,
	2089	ERASE, 47439987159040, 47439987163135,
	2090	STORE, 47439987159040, 47439987163135,
	2091	STORE, 47439987167232, 47439987175423,
	2092	STORE, 47439987163136, 47439987167231,
	2093	ERASE, 47439987163136, 47439987167231,
	2094	STORE, 47439987163136, 47439987175423,
	2095	ERASE, 47439987163136, 47439987175423,
	2096	STORE, 47439987163136, 47439987167231,
	2097	STORE, 47439987167232, 47439987175423,
	2098	ERASE, 47439987167232, 47439987175423,
	2099	STORE, 47439987167232, 47439987175423,
	2100	STORE, 47439987175424, 47439987183615,
	2101	ERASE, 47439986978816, 47439987003391,
	2102	STORE, 47439986978816, 47439986995199,
	2103	STORE, 47439986995200, 47439987003391,
	2104	ERASE, 47439987167232, 47439987175423,
	2105	STORE, 47439987167232, 47439987171327,
	2106	STORE, 47439987171328, 47439987175423,
	2107	ERASE, 47439987130368, 47439987138559,
	2108	STORE, 47439987130368, 47439987134463,
	2109	STORE, 47439987134464, 47439987138559,
	2110	};
	2111	unsigned long set8[] = {
	2112	STORE, 140737488347136, 140737488351231,
	2113	STORE, 140722482974720, 140737488351231,
	2114	ERASE, 140722482974720, 140737488351231,
	2115	STORE, 140722482974720, 140722482978815,
	2116	STORE, 94121505034240, 94121505206271,
	2117	ERASE, 94121505034240, 94121505206271,
	2118	STORE, 94121505034240, 94121505050623,
	2119	STORE, 94121505050624, 94121505206271,
	2120	ERASE, 94121505050624, 94121505206271,
	2121	STORE, 94121505050624, 94121505153023,
	2122	STORE, 94121505153024, 94121505193983,
	2123	STORE, 94121505193984, 94121505206271,
	2124	STORE, 47708483284992, 47708483457023,
	2125	ERASE, 47708483284992, 47708483457023,
	2126	STORE, 47708483284992, 47708483289087,
	2127	STORE, 47708483289088, 47708483457023,
	2128	ERASE, 47708483289088, 47708483457023,
	2129	STORE, 47708483289088, 47708483411967,
	2130	STORE, 47708483411968, 47708483444735,
	2131	STORE, 47708483444736, 47708483452927,
	2132	STORE, 47708483452928, 47708483457023,
	2133	STORE, 140722483142656, 140722483146751,
	2134	STORE, 140722483130368, 140722483142655,
	2135	STORE, 47708483457024, 47708483465215,
	2136	STORE, 47708483465216, 47708483473407,
	2137	STORE, 47708483473408, 47708483637247,
	2138	ERASE, 47708483473408, 47708483637247,
	2139	STORE, 47708483473408, 47708483485695,
	2140	STORE, 47708483485696, 47708483637247,
	2141	STORE, 47708483584000, 47708483637247,
	2142	STORE, 47708483485696, 47708483583999,
	2143	ERASE, 47708483485696, 47708483583999,
	2144	STORE, 47708483485696, 47708483583999,
	2145	STORE, 47708483629056, 47708483637247,
	2146	STORE, 47708483584000, 47708483629055,
	2147	ERASE, 47708483584000, 47708483629055,
	2148	STORE, 47708483584000, 47708483637247,
	2149	ERASE, 47708483584000, 47708483637247,
	2150	STORE, 47708483584000, 47708483629055,
	2151	STORE, 47708483629056, 47708483637247,
	2152	ERASE, 47708483629056, 47708483637247,
	2153	STORE, 47708483629056, 47708483637247,
	2154	STORE, 47708483637248, 47708486688767,
	2155	STORE, 47708484182016, 47708486688767,
	2156	STORE, 47708483637248, 47708484182015,
	2157	ERASE, 47708484182016, 47708486688767,
	2158	STORE, 47708484182016, 47708486467583,
	2159	STORE, 47708486467584, 47708486688767,
	2160	STORE, 47708485877760, 47708486467583,
	2161	STORE, 47708484182016, 47708485877759,
	2162	ERASE, 47708484182016, 47708485877759,
	2163	STORE, 47708484182016, 47708485877759,
	2164	STORE, 47708486463488, 47708486467583,
	2165	STORE, 47708485877760, 47708486463487,
	2166	ERASE, 47708485877760, 47708486463487,
	2167	STORE, 47708485877760, 47708486463487,
	2168	STORE, 47708486672384, 47708486688767,
	2169	STORE, 47708486467584, 47708486672383,
	2170	ERASE, 47708486467584, 47708486672383,
	2171	STORE, 47708486467584, 47708486672383,
	2172	ERASE, 47708486672384, 47708486688767,
	2173	STORE, 47708486672384, 47708486688767,
	2174	STORE, 47708486688768, 47708488527871,
	2175	STORE, 47708486828032, 47708488527871,
	2176	STORE, 47708486688768, 47708486828031,
	2177	ERASE, 47708486828032, 47708488527871,
	2178	STORE, 47708486828032, 47708488486911,
	2179	STORE, 47708488486912, 47708488527871,
	2180	STORE, 47708488171520, 47708488486911,
	2181	STORE, 47708486828032, 47708488171519,
	2182	ERASE, 47708486828032, 47708488171519,
	2183	STORE, 47708486828032, 47708488171519,
	2184	STORE, 47708488482816, 47708488486911,
	2185	STORE, 47708488171520, 47708488482815,
	2186	ERASE, 47708488171520, 47708488482815,
	2187	STORE, 47708488171520, 47708488482815,
	2188	STORE, 47708488511488, 47708488527871,
	2189	STORE, 47708488486912, 47708488511487,
	2190	ERASE, 47708488486912, 47708488511487,
	2191	STORE, 47708488486912, 47708488511487,
	2192	ERASE, 47708488511488, 47708488527871,
	2193	STORE, 47708488511488, 47708488527871,

36114 more lines aren’t shown here.

lib/usercopy.c+2−1

This diff is large: its 11 lines aren’t shown here.

mm/Kconfig+27−7

This diff is large: its 55 lines aren’t shown here.

mm/Makefile+6−2

This diff is large: its 29 lines aren’t shown here.

mm/backing-dev.c+1−5

This diff is large: its 20 lines aren’t shown here.

mm/cma_debug.c+1−4

This diff is large: its 13 lines aren’t shown here.

mm/compaction.c+17−7

This diff is large: its 68 lines aren’t shown here.

mm/damon/Kconfig+3−0

This diff is large: its 10 lines aren’t shown here.

mm/damon/core-test.h+26−3

This diff is large: its 63 lines aren’t shown here.

mm/damon/core.c+184−97

This diff is large: its 520 lines aren’t shown here.

mm/damon/dbgfs.c+30−22

This diff is large: its 134 lines aren’t shown here.

mm/damon/lru_sort.c+87−293

This diff is large: its 447 lines aren’t shown here.

mm/damon/modules-common.h+46−0

This diff is large: its 47 lines aren’t shown here.

mm/damon/ops-common.c+8−42

This diff is large: its 76 lines aren’t shown here.

mm/damon/ops-common.h+1−1

This diff is large: its 9 lines aren’t shown here.

mm/damon/paddr.c+21−22

This diff is large: its 111 lines aren’t shown here.

mm/damon/reclaim.c+57−232

This diff is large: its 343 lines aren’t shown here.

mm/damon/sysfs.c+63−80

This diff is large: its 392 lines aren’t shown here.

mm/damon/vaddr-test.h+12−24

This diff is large: its 63 lines aren’t shown here.

mm/damon/vaddr.c+58−44

This diff is large: its 223 lines aren’t shown here.

mm/debug.c+6−8

This diff is large: its 44 lines aren’t shown here.

mm/filemap.c+66−58

This diff is large: its 247 lines aren’t shown here.

mm/folio-compat.c+6−0

This diff is large: its 13 lines aren’t shown here.

mm/gup.c+141−107

This diff is large: its 369 lines aren’t shown here.

mm/hmm.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/huge_memory.c+117−88

This diff is large: its 541 lines aren’t shown here.

mm/hugetlb.c+499−293

This diff is large: its 1626 lines aren’t shown here.

mm/hugetlb_cgroup.c+12−15

This diff is large: its 73 lines aren’t shown here.

mm/hugetlb_vmemmap.c+7−3

This diff is large: its 26 lines aren’t shown here.

mm/hwpoison-inject.c+2−2

This diff is large: its 16 lines aren’t shown here.

mm/init-mm.c+2−2

This diff is large: its 17 lines aren’t shown here.

mm/internal.h+11−37

This diff is large: its 100 lines aren’t shown here.

mm/kasan/Makefile+8−0

This diff is large: its 16 lines aren’t shown here.

mm/kasan/common.c+16−161

This diff is large: its 256 lines aren’t shown here.

mm/kasan/generic.c+143−11

This diff is large: its 172 lines aren’t shown here.

mm/kasan/hw_tags.c+1−38

This diff is large: its 70 lines aren’t shown here.

mm/kasan/kasan.h+117−54

This diff is large: its 292 lines aren’t shown here.

mm/kasan/kasan_test.crenamed from lib/test_kasan.c+25−1

This diff is large: its 47 lines aren’t shown here.

mm/kasan/kasan_test_module.crenamed from lib/test_kasan_module.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/kasan/report.c+69−69

This diff is large: its 239 lines aren’t shown here.

mm/kasan/report_generic.c+45−1

This diff is large: its 65 lines aren’t shown here.

mm/kasan/report_tags.c+94−29

This diff is large: its 137 lines aren’t shown here.

mm/kasan/sw_tags.c+4−1

This diff is large: its 12 lines aren’t shown here.

mm/kasan/tags.c+114−29

This diff is large: its 169 lines aren’t shown here.

mm/kfence/core.c+9−13

This diff is large: its 41 lines aren’t shown here.

mm/khugepaged.c+739−429

This diff is large: its 2018 lines aren’t shown here.

mm/kmemleak.c+9−12

This diff is large: its 57 lines aren’t shown here.

mm/kmsan/Makefile+28−0

This diff is large: its 29 lines aren’t shown here.

mm/kmsan/core.c+450−0

This diff is large: its 451 lines aren’t shown here.

mm/kmsan/hooks.c+384−0

This diff is large: its 385 lines aren’t shown here.

mm/kmsan/init.c+235−0

This diff is large: its 236 lines aren’t shown here.

mm/kmsan/instrumentation.c+307−0

This diff is large: its 308 lines aren’t shown here.

mm/kmsan/kmsan.h+209−0

This diff is large: its 210 lines aren’t shown here.

mm/kmsan/kmsan_test.c+581−0

This diff is large: its 582 lines aren’t shown here.

mm/kmsan/report.c+219−0

This diff is large: its 220 lines aren’t shown here.

mm/kmsan/shadow.c+298−0

This diff is large: its 299 lines aren’t shown here.

mm/ksm.c+190−185

This diff is large: its 1066 lines aren’t shown here.

mm/madvise.c+33−27

This diff is large: its 155 lines aren’t shown here.

mm/memblock.c+3−3

This diff is large: its 24 lines aren’t shown here.

mm/memcontrol.c+184−97

This diff is large: its 607 lines aren’t shown here.

mm/memory-failure.c+57−69

This diff is large: its 339 lines aren’t shown here.

mm/memory-tiers.c+732−0

This diff is large: its 733 lines aren’t shown here.

mm/memory.c+160−114

This diff is large: its 672 lines aren’t shown here.

mm/memory_hotplug.c+4−7

This diff is large: its 43 lines aren’t shown here.

mm/mempolicy.c+38−28

This diff is large: its 205 lines aren’t shown here.

mm/memremap.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/migrate.c+116−535

This diff is large: its 994 lines aren’t shown here.

mm/migrate_device.c+16−3

This diff is large: its 33 lines aren’t shown here.

mm/mlock.c+19−18

This diff is large: its 95 lines aren’t shown here.

mm/mm_init.c+4−2

This diff is large: its 19 lines aren’t shown here.

mm/mm_slot.h+55−0

This diff is large: its 56 lines aren’t shown here.

mm/mmap.c+1135−1124

This diff is large: its 3057 lines aren’t shown here.

mm/mmzone.c+2−0

This diff is large: its 9 lines aren’t shown here.

mm/mprotect.c+13−4

This diff is large: its 73 lines aren’t shown here.

mm/mremap.c+30−11

This diff is large: its 99 lines aren’t shown here.

mm/msync.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/nommu.c+141−119

This diff is large: its 462 lines aren’t shown here.

mm/oom_kill.c+5−6

This diff is large: its 43 lines aren’t shown here.

mm/page_alloc.c+105−91

This diff is large: its 613 lines aren’t shown here.

mm/page_counter.c+7−8

This diff is large: its 40 lines aren’t shown here.

mm/page_ext.c+108−9

This diff is large: its 206 lines aren’t shown here.

mm/page_io.c+24−19

This diff is large: its 107 lines aren’t shown here.

mm/page_isolation.c+8−9

This diff is large: its 52 lines aren’t shown here.

mm/page_owner.c+76−27

This diff is large: its 303 lines aren’t shown here.

mm/page_table_check.c+9−5

This diff is large: its 68 lines aren’t shown here.

mm/page_vma_mapped.c+3−3

This diff is large: its 27 lines aren’t shown here.

mm/pagewalk.c+10−2

This diff is large: its 26 lines aren’t shown here.

mm/rmap.c+106−78

This diff is large: its 326 lines aren’t shown here.

mm/rodata_test.c+3−5

This diff is large: its 31 lines aren’t shown here.

mm/secretmem.c+3−5

This diff is large: its 23 lines aren’t shown here.

mm/shmem.c+224−205

This diff is large: its 1082 lines aren’t shown here.

mm/shuffle.c+10−11

This diff is large: its 34 lines aren’t shown here.

mm/slab.h+1−0

This diff is large: its 8 lines aren’t shown here.

mm/slab_common.c+2−0

This diff is large: its 16 lines aren’t shown here.

mm/slub.c+41−0

This diff is large: its 90 lines aren’t shown here.

mm/swap.c+58−15

This diff is large: its 135 lines aren’t shown here.

mm/swap.h+7−11

This diff is large: its 52 lines aren’t shown here.

mm/swap_cgroup.c+6−0

This diff is large: its 20 lines aren’t shown here.

mm/swap_slots.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/swap_state.c+57−54

This diff is large: its 267 lines aren’t shown here.

mm/swapfile.c+90−88

This diff is large: its 371 lines aren’t shown here.

mm/truncate.c+1−1

This diff is large: its 9 lines aren’t shown here.

mm/userfaultfd.c+15−13

This diff is large: its 90 lines aren’t shown here.

mm/util.c+3−33

This diff is large: its 60 lines aren’t shown here.

mm/vmacache.c+0−117

This diff is large: its 118 lines aren’t shown here.

mm/vmalloc.c+26−9

This diff is large: its 109 lines aren’t shown here.

mm/vmscan.c+3184−372

This diff is large: its 4138 lines aren’t shown here.

mm/vmstat.c+1−8

This diff is large: its 51 lines aren’t shown here.

mm/workingset.c+108−2

This diff is large: its 146 lines aren’t shown here.

mm/zsmalloc.c+11−11

This diff is large: its 75 lines aren’t shown here.

mm/zswap.c+1−1

This diff is large: its 9 lines aren’t shown here.

scripts/Makefile.kmsan+8−0

This diff is large: its 9 lines aren’t shown here.

scripts/Makefile.lib+9−0

This diff is large: its 16 lines aren’t shown here.

security/Kconfig.hardening+4−0

This diff is large: its 32 lines aren’t shown here.

tools/include/asm-generic/hugetlb_encode.h+13−13

This diff is large: its 32 lines aren’t shown here.

tools/include/linux/slab.h+4−0

This diff is large: its 9 lines aren’t shown here.

tools/include/uapi/asm-generic/mman-common.h+2−0

This diff is large: its 9 lines aren’t shown here.

tools/objtool/check.c+20−0

This diff is large: its 27 lines aren’t shown here.

tools/testing/memblock/linux/mmzone.h+2−0

This diff is large: its 9 lines aren’t shown here.

tools/testing/radix-tree/.gitignore+2−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/radix-tree/Makefile+7−2

This diff is large: its 38 lines aren’t shown here.

tools/testing/radix-tree/generated/autoconf.h+1−0

This diff is large: its 3 lines aren’t shown here.

tools/testing/radix-tree/linux.c+156−4

This diff is large: its 210 lines aren’t shown here.

tools/testing/radix-tree/linux/kernel.h+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/radix-tree/linux/lockdep.h+2−0

This diff is large: its 7 lines aren’t shown here.

tools/testing/radix-tree/linux/maple_tree.h+7−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/radix-tree/maple.c+59−0

This diff is large: its 60 lines aren’t shown here.

tools/testing/radix-tree/trace/events/maple_tree.h+5−0

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/cgroup/config+0−1

This diff is large: its 6 lines aren’t shown here.

tools/testing/selftests/damon/Makefile+1−0

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/damon/debugfs_duplicate_context_creation.sh+27−0

This diff is large: its 28 lines aren’t shown here.

tools/testing/selftests/vm/.gitignore+0−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/vm/Makefile+3−20

This diff is large: its 47 lines aren’t shown here.

tools/testing/selftests/vm/check_config.sh+0−31

This diff is large: its 32 lines aren’t shown here.

tools/testing/selftests/vm/hmm-tests.c+73−35

This diff is large: its 185 lines aren’t shown here.

tools/testing/selftests/vm/khugepaged.c+861−334

This diff is large: its 1625 lines aren’t shown here.

tools/testing/selftests/vm/mremap_test.c+48−1

This diff is large: its 75 lines aren’t shown here.

tools/testing/selftests/vm/run_vmtests.sh+10−5

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/vm/soft-dirty.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/vm/split_huge_page_test.c+4−8

This diff is large: its 40 lines aren’t shown here.

tools/testing/selftests/vm/test_hmm.sh+0−10

This diff is large: its 22 lines aren’t shown here.

tools/testing/selftests/vm/userfaultfd.c+199−48

This diff is large: its 477 lines aren’t shown here.

tools/testing/selftests/vm/vm_util.c+27−9

This diff is large: its 68 lines aren’t shown here.

tools/testing/selftests/vm/vm_util.h+4−1

This diff is large: its 10 lines aren’t shown here.

tools/vm/page_owner_sort.c+6−1

This diff is large: its 14 lines aren’t shown here.

Merge tag 'mm-stable-2022-10-08' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code